mydb tables suddenly empty

Question

Most of my mydb tables (3° stripes of southern hemisphere in CatWISE2020 and VHS) are all of a sudden empty/hold no lines. According to my logs when creating the tables they should hold 10,000+ to millions of lines each but when I checked yesterday, all VHS tables and most of the CatWISE tables show a row count of zero as if they were truncated.

I haven't done anything like that myself. Do mydb tables get cleaned/removed after some time?

Answer 1

Hi Martin,

We discovered a problem in how the backup system works that could at times overlap and explain the truncated tables.  This is now being fixed and your tables should be restored.  I've taken the liberty of adding the indexes you might need as well.  Please let us know if you continue to have problems.

Cheers,

-Mike

Comments

Cool. You mean indexes for all the files? Wow, thanks a lot. :)
I guess this is what the new "tbl_stat" file is for?

btw. I'm glad you could fix a problem with the backup system.

---

The 'tbl_stat' table is something we'll explain in a HowTo notebook and future newsletter, but basically it is a utility table with some information (e.g. nrows, ncols, etc) about the tables in your mydb.  You can query it yourself  with something like queryClient.query('select * from mydb://tbl_stat') but we intend to use it on the backend to try to catch cases where tables suddenly start to appear truncated and to audit usage.

---

BTW, the indexes aren't related to tbl_stat, but will speed up your crossmatching.  By default, MyDB tables aren't indexed at all, but can be using the queryClient.mydb_index() function.

---

> … it is a utility table with some information (e.g. nrows, ncols, etc) about the tables in your mydb.

Nice. I sort of proposed something similar back in July in a forum discussion:
https://datalab.noirlab.edu/help/index.php?qa=1258

> MyDB tables aren't indexed at all, but can be using the queryClient.mydb_index() function.

Someone (user datalab) proposed it to me some days ago at https://datalab.noirlab.edu/help/index.php?qa=1295&show=1309#c1309

Adding indexes would've been my next step but you beat me to it. :)

Cheers
Martin