0 votes
120 views
Dear Datalab Colleague,

I am still working on the cross match. But I found that the query before the cross match ran too long and can not be completed. I change the criteria of the query as following:

    job = qc.query('select * from ls_dr9.tractor where dered_mag_r between 16.0 and 23.0', out='mydb://ls_dered_mag_rdeep',async_=True)
status = qc.status(token, job)
    while (status != 'COMPLETED' and status != 'ERROR'):
         time.sleep(60)
        status = qc.status(token,job)
        #print (status)   
    

    qc.mydb_index(token,'mydb://ls_dered_mag_rdeep','ra')
    qc.mydb_index(token,'mydb://ls_dered_mag_rdeep','dec')
    qc.mydb_index(token,'mydb://ls_dered_mag_rdeep','', q3c='ra,dec', cluster=True)

Is it because of the internet speed limit or other issues?

Thanks and Happy New Year!
by huanian (250 points) | 120 views
I would need to change the cut to dered_mag_r between 16.0 and 24.0

1 Answer

0 votes
The magnitude constraint produces a result of ~673M rows (out of 1.96B in ls_dr9.tractor), and because you are selecting all columns the resulting table in ~800GB in size.  The time needed to read that much data out of the database exceeds the various timeouts between services currently configured in the system and so a table of that size is currently beyond our capability.  Depending on how you plan to use it, limiting the selection to only the object id and/or position columns might enable the query to finish but will still require an async query.
by datalab (20.7k points)

414 questions

430 answers

436 comments

635 users

Welcome to Data Lab Help Desk, where you can ask questions and receive answers from other members of the community.

Categories