Welcome to the January 2021 edition of the Astro Data Lab Newsletter! The purpose of this occasional newsletter is to communicate significant developments within the Astro Data Lab and to solicit feedback from you, our registered users.
Astro Data Lab is operated by the Community Science and Data Center (CSDC) at NSF’s National Optical-Infrared Astronomy Research Laboratory (NOIRLab).
Astro Data Lab at AAS237 Virtual Meeting
We have many activities planned at the AAS237 Virtual Meeting, and are looking forward to be engaging with our users. On Tuesday Jan 12 (4:10-5:40pm ET) we will hold a Special Session 224 “The Data Lab Science Platform and Open-Data Ecosystem at NSF’s NOIRLab”, with several presentations by Data Lab users and team members, and time for Q&A. Wednesday Jan 13 (4:10-5:40pm ET) will see the associated iPoster Session 334. On Thursday Jan 14 (4:10-5:40pm ET) we will then feature in the Splinter Session “NOIRLab’s Data Services: A Practical Demo Built on Science with DES DR2”, where in four demo presentations we will showcase how integrated data services facilitate scientific discovery from archival surveys, detect variability alerts and trigger follow-up observations. Members of the Astro Data Lab team will also be present at the NOIRLab booth in the virtual Exhibit Hall, and we are looking forward to talking with you, answering any question you might have about Data Lab, or giving you a quick demo of our data holdings and data services.
Astro Data Lab Updates
Spectroscopic Service
Astro Data Lab has so far served massive photometric catalogs and image data sets. We are now introducing a new capability: a performant and feature-rich service to access spectroscopic data. In this first release we introduce specClient which provides users with functions to search for the IDs of spectra based on spatial and other constraints, and then to retrieve spectra for a supplied list of IDs. We begin with SDSS spectra: DR16 is the flagship dataset, but DR8 through DR15 are also available. We will expand to other spectroscopic datasets in the coming months, and are ramping up for data from the Dark Energy Spectroscopic Instrument (currently undergoing survey validation at the Mayall 4-meter telescope).
We developed the service with ease-of-use and speed in mind. For instance, hundreds of SDSS spectra can be retrieved in mere seconds. To use the service, simply import specClient in your notebook or Python code. An intro notebook, a how-to notebook, and a science case notebook showcase how to use the new functionality efficiently. specClient is part of the latest version of the datalab package.
In these early stages of the spectral service launch we particularly welcome feedback and suggestions from our users (please email to datalab@noirlab.edu).
New Datasets
Dark Energy Survey (DES) DR2
On Friday Jan 15, DES DR2 will become available simultaneously at NCSA, Astro Data Lab, and LineA Science Server. The survey is based on years Y1 through Y6 of operations, features over 691 million individual sources, and median catalog depths in g, r, i, z, Y = 24.7, 24.4, 23.8, 23.1, 21.7 mag at S/N~10. At Astro Data Lab, we will provide web-based and programmatic query access to the main, mag, flux, and coverage tables via the TAP protocol, which include additional columns (e.g. Galactic coordinates, pre-computed colors), and also pre-computed cross-match tables against important reference datasets (Gaia DR2, NSC DR2, SDSS DR16, AllWISE, unWISE DR1) for very fast spatial queries.
We are also standing up Simple Image Access (SIA) services for queries on image metadata and cutout retrieval. Both the single-epoch images from all survey years Y1 through Y6 are available, as well as the DR2 co-added images. A File Service provides file-based access to the survey data products as a single collection (catalog tables as FITS bintables, image frames per-tile). New Jupyter notebooks showcasing DES DR2 will be added to the default Astro Data Lab notebook suite after the AAS meeting.
DECam Local Volume Exploration Survey (DELVE) DR1
The DELVE survey combines archival DECam data with 126 nights of new observations to study dwarf satellite galaxies over a broad range of luminosities and environments, and is divided into three survey components (WIDE, MC, and DEEP). Astro Data Lab hosts Data Release 1, consisting of catalog-level coadds in g, r, i, z bands from the DELVE-WIDE processing. DR1 covers >5,000 deg2 in each individual band and ~4,000 deg2 in all bands simultaneously. The DELVE DR1 catalog contains ~520 million unique objects. We offer TAP access to query the catalog, a SIA service for image cutouts, and an example notebook showcasing data access to DELVE DR1.
Gaia Early Data Release 3
Gaia Early Data Release 3 is available at Astro Data Lab. We begin with the main table ‘gaia_source’, and are in the process of adding other tables released by Gaia. The main table holds over 1.8 billion sources, with proper motion information for 1.47 billion sources.
Pre-computed cross-match tables
We have added hundreds of cross-match tables for many of our hosted datasets, against standard astrometric, photometric, and spectroscopic reference datasets (Gaia DR2, AllWISE, unWISE DR1, NSC DR1, and SDSS DR16). All cross-match tables were computed with a default search radius of 1.5 arcseconds, list the single closest match in the other table, do not include missed rows, and follow the same consistent structure. Pre-computed cross-match tables are much faster for quick spatial counterpart searches than dedicated spatial cross-match queries (using a simple JOIN statement on a common key). In our schema browser, look for tables named like schema1.xNpN__table1__schema2__table2, for instance ls_dr8.x1p5__tractor_n__gaia_dr2__gaia_source – this here is a cross-match table (indicated by the leading x), located in the ls_dr8 schema, and it matches the ls_dr8.tractor_n table with the gaia_dr2.gaia_source table (which lives in the gaia_dr2 schema) within a 1.5 arcseconds radius (1p5). A new notebook shows how to use the cross-match tables.
Coming Soon: Legacy Surveys DR9
The Legacy Imaging Surveys Data Release 9 (LS DR9) will include over 1.6 billion unique sources over a footprint surpassing 20,000 square degrees, and will be used for the official target selection of the Dark Energy Spectroscopic Instrument (DESI) five-year survey to yield 35 million spectra of galaxies and quasars, and 10 million spectra of stars. LS DR9 includes data from its three primary surveys (DECaLS, MzLS and BASS) as well as archival images from the NOIRLab Astro Data Archive from other projects, such as the Dark Energy Survey. LS DR9 also reports mid-infrared photometry at 3.4, 4.5, 12, and 22 microns for all optically detected sources, using forced photometry based on LS priors applied to the deepest WISE images available (unWISE coadds). The LS DR9 photometry reaches depths (5-sigma for galaxies) of g=24.7, r=23.9, z=23.0, W1=20.72, W2=19.97 AB magnitudes. In addition to the main Tractor photometric catalog, DR9 will also include single-epoch forced photometry catalogs, as well as photometric redshift catalogs. Our team is currently ingesting the full dataset and will be releasing the catalogs listed above, an image cutout tool, and example Jupyter Notebooks.
Updated client package
The newest version 2.19.0 of the datalab client package is now available through PyPI and on GitHub, if you would like to have the Data Lab functionality available on your local computer. It adds the new specClient, adds some functionality to queryClient (in your catalog queries you can for instance request a CSV-formatted result but without a header line), and contains of course the latest maintenance and bug fixes.
Opportunity to prepare for science with Rubin Observatory
On June 30, 2021 the Rubin Observatory will release Data Preview 0 (DP0), the first of three data previews, which will make simulated LSST-like data products available in the Rubin Science Platform (RSP) to Rubin Observatory staff and up to 300 individuals from the science community (“DP0 delegates”). One of Rubin Observatory's goals for DP0 is to have experienced users of science platforms engage with the RSP and provide feedback. Thus, current users of Astro Data Lab are encouraged to learn more about DP0 (links below) and consider participating. Applications to participate in DP0 will open in early March 2021. The only prerequisite for participation in DP0 is to have Rubin Observatory data rights. As described in the Data Policy astronomers working in the US and Chile have data rights (including students), as do named individuals on International Contributor teams.
Further information:
- Invitation to join: virtual information sessions about Data Preview 0
- Data Preview 0: An early opportunity to prepare for science with Rubin Observatory
- Data Preview 0: The Simulated Data Set from the DESC’s DC2
Contact Data Lab
You can visit our website, use the Helpdesk, and follow us on Twitter! If you want to sign up or opt out from receiving these communications, send us an email with the subject “subscribe” or “unsubscribe”.
