Help - Running Redrock on HET VIRUS Spectra in a Time-Limited Jupyter Environment

Question

I am trying to classify 1D spectra from the HET VIRTUS instrument using Redrock based classifications and  compare the results to the official Diagnose classifications published in the HETVIPS survey paper. I use an older Redrock build (0.17) provided in the DESI 23.1 kernel. I essentially take the HET .fits files, wrap them into a Redrock "Spectrum" object and put that into a Redrock "Target" object. Then, Redrock requires a large “rebin” step before fitting: it loads each template file (star/galaxy/quasar), expands it across a broad redshift range, and interpolates onto our HET VIRUS wavelength grid. This is building a DistTemplate. I then call zfind(targets, templates) to get the desired redshifts, classifications, and chi squared value. Firstly, I am struggling with the rebinning step, as my jupyter server restarts before completion every time. Is there a way to handle these large template expansions in a stable environment that won't kill the workflow. And secondly, is this approach even the best option or possible? If my main goal is to extract redshift data from the HET spectra, is there a better approach? I feel deeply outside my depth with this project, any assistance would be immensely appreciated. 

Answer 1

Thank you for your interest in Data Lab and redrock. Could you provide any additional details about the Jupyter Server restarts?

Redrock is somewhat finely tuned for a high-performance computing environment, although a lack of adequate scratch space could also cause problems, so we need to know as much as we can.

For more detailed support for redrock performance, you should bring this to the attention of the redrock developers: https://github.com/desihub/redrock/issues.