timeout for long query

Question

Dear Datalab Colleague,

I am still working on the cross match. But I found that the query before the cross match ran too long and can not be completed. I change the criteria of the query as following:

    job = qc.query('select * from ls_dr9.tractor where dered_mag_r between 16.0 and 23.0', out='mydb://ls_dered_mag_rdeep',async_=True)
status = qc.status(token, job)
    while (status != 'COMPLETED' and status != 'ERROR'):
         time.sleep(60)
        status = qc.status(token,job)
        #print (status)   
    

    qc.mydb_index(token,'mydb://ls_dered_mag_rdeep','ra')
    qc.mydb_index(token,'mydb://ls_dered_mag_rdeep','dec')
    qc.mydb_index(token,'mydb://ls_dered_mag_rdeep','', q3c='ra,dec', cluster=True)

Is it because of the internet speed limit or other issues?

Thanks and Happy New Year!

Comments

I would need to change the cut to dered_mag_r between 16.0 and 24.0

Answer 1

The magnitude constraint produces a result of ~673M rows (out of 1.96B in ls_dr9.tractor), and because you are selecting all columns the resulting table in ~800GB in size.  The time needed to read that much data out of the database exceeds the various timeouts between services currently configured in the system and so a table of that size is currently beyond our capability.  Depending on how you plan to use it, limiting the selection to only the object id and/or position columns might enable the query to finish but will still require an async query.