0 votes
249 views
Most of my mydb tables (3° stripes of southern hemisphere in CatWISE2020 and VHS) are all of a sudden empty/hold no lines. According to my logs when creating the tables they should hold 10,000+ to millions of lines each but when I checked yesterday, all VHS tables and most of the CatWISE tables show a row count of zero as if they were truncated.

I haven't done anything like that myself. Do mydb tables get cleaned/removed after some time?
by martinkb (720 points)
edited by 2231 | 249 views

1 Answer

0 votes
Best answer
Hi Martin,

We discovered a problem in how the backup system works that could at times overlap and explain the truncated tables.  This is now being fixed and your tables should be restored.  I've taken the liberty of adding the indexes you might need as well.  Please let us know if you continue to have problems.

Cheers,

-Mike
by datalab (20.7k points)
selected by 2231
Cool. You mean indexes for all the files? Wow, thanks a lot. :)
I guess this is what the new "tbl_stat" file is for?

btw. I'm glad you could fix a problem with the backup system.
The 'tbl_stat' table is something we'll explain in a HowTo notebook and future newsletter, but basically it is a utility table with some information (e.g. nrows, ncols, etc) about the tables in your mydb.  You can query it yourself  with something like queryClient.query('select * from mydb://tbl_stat') but we intend to use it on the backend to try to catch cases where tables suddenly start to appear truncated and to audit usage.
BTW, the indexes aren't related to tbl_stat, but will speed up your crossmatching.  By default, MyDB tables aren't indexed at all, but can be using the queryClient.mydb_index() function.
> … it is a utility table with some information (e.g. nrows, ncols, etc) about the tables in your mydb.

Nice. I sort of proposed something similar back in July in a forum discussion:
https://datalab.noirlab.edu/help/index.php?qa=1258

> MyDB tables aren't indexed at all, but can be using the queryClient.mydb_index() function.

Someone (user datalab) proposed it to me some days ago at https://datalab.noirlab.edu/help/index.php?qa=1295&show=1309#c1309

Adding indexes would've been my next step but you beat me to it. :)

Cheers
Martin

414 questions

430 answers

436 comments

635 users

Welcome to Data Lab Help Desk, where you can ask questions and receive answers from other members of the community.

Categories