Is the gaia_dr2.light_curves table missing some data or am I misunderstanding it?

Question

Currently, when I run the query SELECT count(DISTINCT source_id) FROM gaia_dr2.light_curves in DataLab (in a Jupyter notebook) it returns 233,000 (unique source IDs). On the GAIA DR2 page https://www.cosmos.esa.int/web/gaia/dr2, it says that there should be 550,737 "variable sources" -- should this number match the number of light curves?

Also, I noticed that http://cdn.gea.esac.esa.int/Gaia/gdr2/light_curves/csv/ lists some files like light_curves_6835506044324750976_6858007034391055616, and the gaia_dr2.light_curves table in DataLab does not contain source ID 6835506044324750976 (whereas it does contain earlier source IDs, like 1042504286338226688).

Am I misunderstanding something?

Thanks!!

Answer 1

I can't explain how/when it happened, but you are right that the gaia_dr2.light_curves table is short for some reason.  I've reloaded it from the origin FITS images downloaded shortly after the data release and the table now contains 548,320 unique source_id values which is still short of the number of variable sources you quote.  If I look at the CSV files of light curves downloaded at the same time, there are 549,371 uniq source_ids -- again short of the number you mention but also different than the number derived from the FITS files.

FITS files were not available for all tables just after the data release and so some tables were loaded from CSV files, my guess would be that the download files were still being generated/modified at the time we got them, however I cannot say whether files obtained today would be different yet again.  The specific source_id you mention *is* now in the table because of the reload.

Lastly, different data centers may have constructed/augmented the Gaia tables in different ways.  The Data Lab tables were built from what was available as a data download, not as an explicit mirror of ESA or other databases.  If there is something missing that you need we will try to provide it.  Please let us know if you find problems with any of the other tables.