Astro Data Lab

We are happy to share with you the latest developments at Astro Data Lab in this January 2022 newsletter.

New datasets at Data Lab

Astro Data Lab has recently ingested several new survey datasets: the DESI Legacy Surveys (LS DR9), the Southern Photometric Local Universe Survey (S-PLUS DR2), the UKIRT InfraRed Deep Sky Surveys (UKIDSS), the VISTA Hemisphere Survey (VHS DR5), the SkyMapper Southern Sky Survey (SkyMapper DR2), and the Rubin/LSST TRILEGAL simulated survey (LSST-SIM DR2). The five observed surveys’ object tables yield over 4.3 billion rows (see table below), and together cover almost 35,700 square degrees, with some mutually overlapping areas. The LSST-SIM dataset simulates almost 27,000 square degrees, and covers practically the entire footprint of SkyMapper DR2, VHS DR5, S-PLUS DR2, two-thirds of the UKIDSS DR11plus footprint, and half of LS DR9’s footprint.

Object number density (per square degree) in Mollweide projection for the six new surveys at Astro Data Lab. LS DR9 and S-PLUS DR2 (left column) are shown in Equatorial coordinates, and UKIDSS DR11+, VHS DR5, SkyMapper DR2, and LSST-SIM DR2 are shown in Galactic coordinates. View a larger version.

The Data Lab team has already pre-crossmatched the five observational surveys with our reference datasets: Gaia DR2 or EDR3 (for astrometry), AllWISE, NSC DR2, unWISE DR1 (for photometry), and SDSS DR16 (for spectroscopy). We also routinely add other useful columns such as nest4096, ring256, and htm9 for Healpix-based and Hierarchical Triangular Mesh (HTM)-based sky tessellation use cases.
The Astro Data Lab team evaluates periodically which external survey datasets we should source, ingest, and serve. We appreciate requests and suggestions from our users. Please contact us at datalab@noirlab.edu to send your request and, if possible, mention an example scientific use case.

LS DR9

This is the ninth public data release of the DESI Legacy Imaging Surveys, and official target selection catalog for the DOE-funded Dark Energy Spectroscopic Survey (DESI) survey. Catalogs were produced from the Tractor inference model of the >20,000 square degrees of extragalactic sky visible from both hemispheres in three optical bands g,r,z and four infrared bands from deep unWISE images spanning the original WISE survey up to six years of NEOWISE. The LS sky coverage is approximately bounded by -18° < δ < +84° in celestial coordinates and |b| > 18° in Galactic coordinates. To achieve this goal, the Legacy Surveys have conducted three imaging projects on different telescopes: the Beijing-Arizona Sky Survey (BASS), the DECam Legacy Survey (DECaLS), The Mayall z-band Legacy Survey (MzLS). For details on the Legacy Surveys and this data release, please visit: https://www.legacysurvey.org/dr9/ For information on how to access the data products at the Astro Data Lab, please visit https://datalab.noirlab.edu/ls/dataAccess.php

S-PLUS DR2

The Southern Photometric Local Universe Survey (S-PLUS) is an international survey led by our Brazilian colleagues using an 0.83-m robotic telescope at CTIO. The ongoing survey photographs the sky through twelve Javalambre filters - 7 narrow-bands centered on [OII], Ca H+K, H𝛿, G-band, Mgb triplet, H𝛼, and Ca triplet and five Sloan-like filters. Data Release 2 comprises 950 square degrees observed between 2016 and 2020, to a S/N>3 depth of g=21.3 mag, and includes and updates the previous DR1 (which covers the Stripe 82 region). All data products are available from the survey team, while Astro Data Lab hosts the photometric object catalog, a photo-z table created using deep learning, and a star-galaxy-quasar classification table. Data Lab also hosts the DR1 catalog of S-PLUS.

UKIDSS DR11plus

The eleventh data release of the UKIRT InfraRed Deep Sky Surveys (UKIDSS) from the WFCAM Science Archive (WSA) generated by the Wide Field Camera (WFCAM) on the United Kingdom Infrared Telescope (UKIRT) comprises five surveys: Large Area Survey (LAS), Galactic Plane Survey (GPS), Galactic Clusters Survey (GCS), Deep Extragalactic Survey (DXS), and Ultra Deep Survey (UDS). Each of these near-infrared sub-surveys uses several of the Z, Y, J, H, K filters. Data Lab hosts the main object tables from these northern-hemisphere surveys: lassource (88M rows), gpssource (1B), gcssource (71M), dxssource (3.4M), udssource (300K). Their 5-sigma depths range from H=18.8 mag for LAS to H=23.3 mag for UDS.

VHS DR5

Courtesy of the ESO Science Archive, Astro Data Lab now hosts a copy of the latest release of the VISTA Hemisphere Survey, VHS DR5, generated by the dedicated 4.1-m VISTA telescope. VHS is one of five VISTA surveys, and DR5 delivers almost 1.4 billion individual objects detected in the NIR (bands Y, J, H, Ks) over a 16,500 square degrees area in the southern hemisphere visible from the Paranal Observatory in Chile. VHS DR5 reaches 5-sigma point source depth of J=20.8 and Ks=20.0 mag. For a subset of 4,500 square degrees (the Southern Galactic Cap) the limiting magnitudes are deeper, at J=21.4, H=20.7 and Ks=20.3 mag. Astro Data Lab hosts the multi-band photometric catalog, which also contains, e.g., star/galaxy classification columns. For column description please see our table browser, and also refer to the ESO data page.

SkyMapper DR2

SkyMapper is a 1.35-m wide-field survey telescope built and operated by our colleagues at the Australia National University. Located at the Siding Spring Observatory, it currently conducts the SkyMapper Southern Sky Survey. Its data release 2 delivers half a billion objects over 17,000 square degrees of the Southern sky, detected in six optical filters (u, v, g, r, i, z), with some portions of the sky reaching >21 mag in the g and r bands. Astro Data Lab hosts the main photometric object table, and more detailed per-image/CCD detection catalogs, including shape information. Data Lab also hosts DR1 of SkyMapper.

LSST-SIM DR2

Simulations are becoming ever more important for wide-field surveys, and the community needs to find ways to serve large-area simulations as if they were observed datasets, meaning that we employ the same data formats, access protocols, and tools as are used for any other observed survey. Data Lab paves the way by hosting the second release of the simulated Milky Way dataset over the entire LSST footprint (LSST-SIM DR2), generated by Leo Girardi and collaborators using their TRILEGAL stellar population code. The catalog provides positions, proper motions, and LSST photometry (u, g, r, i, z, y) for 10.5 billion simulated stars in the Milky Way, down to the LSST stacked depth limit of 27.5 mag in the r band. Because this is a simulated sky, everything is known about the objects in this catalog, including, e.g., their distance, age, metallicity, and extinction along our line of sight toward them. For a description of the TRILEGAL model please see Girardi et al. (2012): https://ui.adsabs.harvard.edu/abs/2012ASSP...26..165G/abstract

Note that this dataset was not pre-crossmatched at Data Lab against our reference dataset since the simulated RA and Dec positions are drawn randomly from appropriate distributions. We have, however, added additional coordinate columns, as well as a random_id column that helps in selecting random subsamples from this truly large dataset.

The usefulness of pre-crossmatched tables

We now serve over 350 pre-crossmatched tables for nearly all of our hosted datasets. Each catalog at Astro Data Lab has been cross-matched against standard astrometric, photometric, and spectroscopic reference datasets (Gaia DR2 or EDR3, AllWISE, unWISE DR1, NSC DR2, and SDSS DR16). All pre-crossmatched tables use a default search radius of 1.5 arcseconds, list for every object the single closest match in the other table, omit rows for missing matches, and follow the same consistent column structure ( ra1, dec1, id1, ra2, dec2, id2, distance). The pre-computed crossmatch tables allow users to find object counterparts much quicker via simple JOIN operations than by running dedicated spatial crossmatch queries. In our schema browser look for tables named like schema1.xNpN__table1__schema2__table2, for instance skymapper_dr2.x1p5__master__nsc_dr2__object, which is a cross-match table (indicated by the leading x) located in the skymapper_dr2 schema. It matches the skymapper_dr2.master table with the nsc_dr2.object table (which lives in the nsc_dr2 schema) within a 1.5 arcseconds radius (1p5). See this notebook that shows how to use the pre-crossmatched tables at Data Lab and its accompanying examples notebook.

Improved performance and reliability of Data Lab services

The Astro Data Lab team has improved the performance and reliability of several services. First up is VOSpace, our remote user file storage. It also powers the File Services, which hold heterogeneous survey file collections. We replaced some internal query logic, added more compact indexes on the metadata database, switched to a faster XML parser, and deployed bugfixes. Users can now expect a more reliable and performant subsystem. Uploads and downloads of many files (for instance using the Data Lab storeClient), by many concurrent users, are now much more reliable, the speed of several commands (such as 'ls') on large directories is now up to 40 times faster, and various other operations using the storeClient are also speedier.

We have improved the operational behavior of our Simple Image Access service (for querying Data Lab about available image datasets) to safekeep from becoming overwhelmed with large numbers of programmatic queries.  We further enhanced how SIA query results are filtered by the database, resulting in a more stable subsystem that is now both leaner and consumes less memory.

Astro Data Lab users will experience the benefits of this work when they access their VOSpace remote storage, any of the Data Lab survey file services, or the SIA service.

Migration to datalab.noirlab.edu domain complete

Now that the former noao.edu domain is fully deactivated, we have completed the final steps of transitioning all our services to the new ‘noirlab.edu’ domain. Please update any bookmarks that you might have to point to https://datalab.noirlab.edu. Note also that the structure of URLs remains identical, so in most cases you can simply substitute 'noirlab.edu' for 'noao.edu'. Please see also the last section in this Newsletter on how Astro Data Lab should be referenced in your scientific publications.

Updated Data Lab software

The 'datalab' command-line client and the 'dl' Python package have been updated recently. The recent version v2.20.0 works with the new noirlab.edu domain address for all services. The software also contains a number of bugfixes and performance improvements. Users who have previously installed 'datalab' on their local machines, or who want to do so now, can simply run

pip install --ignore-installed --no-cache-dir noaodatalab

or follow the manual installation instructions shown in our GitHub repository: https://github.com/noaodatalab/datalab. (Note that the package name 'noaodatalab' is historical.)

Users of Data Lab's Jupyter notebook server need not take any further action, as the new software is installed for all users globally.

Referencing Astro Data Lab

Many of you are using Astro Data Lab's resources and datasets for your research, and are referencing Astro Data Lab in your publications, which is terrific! As many of you know, NOAO was integrated as part of NOIRLab and the web domain was recently transitioned to noirlab.edu. We list here a few reminders of how Astro Data Lab should be referenced in your publications. Thank you!

• If your research makes use of Astro Data Lab resources (e.g. datasets, Jupyter notebook server, notebooks, data services, etc.), please acknowledge this in your Acknowledgments section, and if appropriate also in the main body of your manuscript. Standard phrasing is provided at: https://datalab.noirlab.edu/acknowledgments.php

• If the journal supports a \facilities{} keyword (such as the AAS Publishing Journals) please include \facilities{Astro Data Lab}.

• Please also consider linking to the Astro Data Lab website https://datalab.noirlab.edu or any relevant sub-page.

• Please no longer refer to the science platform as "NOAO Data Lab" or similar, but instead use "Astro Data Lab", or "NOIRLab's Astro Data Lab", or "the Astro Data Lab science platform operated at CSDC/NOIRLab", or any suitable combination of those. "Data Lab" (without "Astro") is also acceptable.

• Finally, it can be helpful to your readers to share, e.g., the specific SQL queries you ran at Data Lab, or a notebook demonstrating your analysis. Most journals have facilities for this, for instance in the Appendix or Supplements sections.

Community Science and Data Center is hiring full-stack software engineers

The CSDC is looking to hire three full-stack software engineers to work on exciting projects such as the Astro Data Lab science platform, the Astro Data Archive, the ANTARES alert broker system, and a DECam rapid reduction pipeline. Please consider applying.

Contact Us

You can visit our website, use the Helpdesk, reach us via email, and follow us on Twitter.

Currently registered and active users have been subscribed to this Newsletter mailing list. To unsubscribe, send an empty email to datalab-newsletter-unsubscribe@taurus.tuc.noirlab.edu and follow the instructions in the verification email to confirm.