Welcome to the October 2020 edition of the Astro Data Lab Newsletter! The purpose of this occasional newsletter is to communicate significant developments within the Astro Data Lab and to solicit feedback from you, our registered users.
Astro Data Lab is operated by the Community Science and Data Center (CSDC) at NSF’s National Optical-Infrared Astronomy Research Laboratory (NOIRLab).
Astro Data Lab Updates
Moving to JupyterLab
Jupyter notebooks at Astro Data Lab will soon run on a new server and the default interface will be JupyterLab. The URL remains the same: /devbooks/ We anticipate to complete the change on October 26, 2020.
After that date, when you log in to the notebook server, you should see the new JupyterLab interface, but the "classic" Jupyter Notebook interface is still available: simply select from the menu Help --> Launch Classic Notebook (it will open in a new browser tab).
JupyterLab offers several usability enhancements, e.g. splitting your view into separate sub-windows all connected to the same kernel, an integrated file browser, a built-in interactive Python console with command history, and more. A complete documentation of JupyterLab, with easy introduction chapters, can be found here: https://jupyterlab.readthedocs.io/
Our updated User Manual introduces JupyterLab, and retains the classic interface description as well. We are happy to assist users with the transition if needed. Please send an email to datalab@noirlab.edu.
End of Python 2 support at Data Lab
As announced in the January 2019 Newsletter, Python-2 support is ending at Astro Data Lab. Coinciding with rolling out the new notebook server, notebooks will no longer work correctly if they use Python-2 syntax. We are confident that the changes for Astro Data Lab users are minimal – for instance change a print "Hello World" statement to a print("Hello World") function. A section in the Data Lab User Manual describes the most common changes needed for older notebooks to work with a new Python-3 kernel.
New datasets
Collaboration with Gemini Large-And-Long Programs (LLP) – GOGREEN & GCLASS DR1
Astro Data Lab and Gemini Observatory have recently launched a cross-NOIRLab initiative for hosting high-level science products from the Gemini Large-and-Long Programs (LLP) within Astro Data Lab. In the pilot phase the GOGREEN Survey team agreed to work together with us, and we are happy to announce that Data Lab (in addition to CADC) now hosts the GOGREEN+GLCASS First Data Release. At Data Lab, we have exposed the clusters, redshift, and photometry catalogs through our Table Access Protocol (TAP) service, have stood up a Simple Image Access (SIA) image cutout service for the HST images collected by the survey for all 26 galaxy clusters, and exposed the entire survey file collection through a public VOSpace file service. Jupyter notebooks demonstrate data access and possible analyses of the GOGREEN data.
Work is underway to host data from other Gemini LLP surveys at Astro Data Lab in the future.
SDSS DR16 and other SDSS updates
- We have loaded the full spectroscopic catalog (“specObjAll”) from the Sloan Digital Sky Survey (SDSS DR16) in the sdss_dr16 database schema. This schema also includes the supplemental Firefly catalog (“sdssEbossFirefly”) of galaxy properties. Additional supplemental catalogs will be added in the near future.
- We have added the full SDSS DR12 spectroscopic catalog in the sdss_dr12 schema, as well as the (“emissionLinesPort” table). This table remains the most recent, comprehensive catalog of emission line measurements on SDSS spectra. Further catalogs of galaxy properties based on DR12 will be added in the near future.
- SDSS DR14 data are also available in the sdss_dr14 schema.
- We have added the capability to join across data releases using a new, indexed key called sdss_joinid. Tables with this column can be joined to tables with the same column in other data releases. For example, the DR16 spectroscopic catalog can be joined to the DR12 emission line table described above.
- Although some SDSS DR13 data are still available via the sdss_dr13 schema, that schema will be completely reloaded in the near future. Use this data set with caution.
- Finally, all file-based SDSS data from DR8 through DR16 (SDSS-III and SDSS-IV) are available via the Data Lab File Service.
Hipparcos-2 and Tycho2 catalogs
We are pleased to host catalogs from the Hipparcos satellite mission: the Hipparcos-2 catalog is a 2007 reduction with improved accuracies of the original 16 volume (printed) Hipparcos catalog (118,218 stars) from 1997, derived from the ESA Hipparcos Space Astrometry Mission (ESA, 1997, The Hipparcos and Tycho Catalogues, ESA SP-1200). In addition, we also host the Tycho-2 catalog (TYCHO2 - Tycho-2 Catalog of the 2.5 Million Brightest Stars) which is much bigger (~2.5 million stars) than the original Tycho catalog, also created during that mission. We have added the standard Data Lab columns and indexes to these catalogs as well, to aid in cross-matching and other science use cases.
Updated notebooks
Most notebooks curated by the Data Lab team have seen upgrades in the past few months, either for technical reasons or to improve user experience. The EPO (Education and Public Outreach) subdirectory has seen new additions in the TeenAstronomyCafe section, and we are happy to announce that Astro Data Lab now hosts various notebooks from the La Serena School for Data Science, which was unfortunately canceled in 2020 due to the COVID-19 pandemic. Several notebooks have been added in the ScienceExamples section, e.g. notebooks on the new GOGREEN+GCLASS DR1 dataset hosted at Data Lab. The current Astro Data Lab notebook landscape looks something like this:
Updated client package
While Data Lab services are typically meant to be run on the remote server, you can download and install the datalab client package to run the clients on your local hardware. The latest version is 2.18.12 and contains all recent features and bug fixes.
Astro Data Lab at JupyterCon
This week, the Data Lab is presenting at the JupyterCon 2020 virtual conference. If you are attending, please join us for an overview talk “Jupyter-enabled astrophysical analysis for researchers and students” and a poster presentation on “End-to-end notebook life cycle at Astro Data Lab”. If you are not attending, the presentations will also be available on the JupyterCon YouTube channel one or two weeks after the event.
Astro Data Lab & ANTARES are hiring Software Engineers
NOIRLab’s Community Science and Data Center is seeking to hire two Software Engineers to work on Astro Data Lab and the ANTARES alert broker system. Please check out the job ad, apply if you are interested, and make your friends and colleagues aware:
The positions will remain open until filled. Please send inquiries about the positions via email: datalab@noirlab.edu
Contact Data Lab
You can visit our website, use the Helpdesk, and follow us on Twitter! If you want to sign up or opt out from receiving these communications, send us an email with the subject “subscribe” or “unsubscribe”.
