0 votes
97 views
Dear Datalab Team,

I am currently trying to do an all-sky search for some interesting objects, therefore I need to download a quite large catalog. However, it will exceed the time limit even if I divide the entire SQL search into a few slices. I wonder whether there is any way for me to bulk download those large catalog?

Thank you very much,

Huanian
asked Mar 1 by Huanian | 97 views

1 Answer

0 votes
Hi Huanian, thanks for reaching out.
I suppose that you are not trying to query to download an entire large catalog; if yes, please add qualifying 'WHERE' conditions to your SQL to ask for only the sources that match your criteria.

Then, standard synchronous queries have a runtime limit of 300 seconds. You can increase that limit per-query to max. 600 seconds, like this:

result = qc.query(sql='SELECT ...',timeout=600)

For even longer-running queries, you will need to use the async mode, for instance like this:

jobid = qc.query(sql='SELECT ...',async_=True)
print(jobid)

`jobid` is a unique job ID. From time to time you can check the status of the job:

print(qc.status(jobid))

While the query is still running, this will print 'EXECUTING'. Once it says 'COMPLETED', you can then fetch the results of your query:

result = qc.results(jobid)

Final point: you can store the results of your queries directly to your Data Lab file storage (vospace) or as tables in your personal database (mydb). This will spare your having to download the files (i.e. you can continue to work with them on the Data Lab system). Run such queries e.g. like this:

qc.query(sql='SELECT ...',async_=True,timeout=3600,out='vos://myquery1.csv')

or

qc.query(sql='SELECT ...',async_=True,timeout=3600,out='mydb://myquery1')

Please let us know if this was useful, or if we can help otherwise.

Cheers,

Robert for the DL team
answered Mar 1 by datalab (12,020 points)
Dear Robert,

Thanks for the quick reply. I tried to increase the timeout limit, it basically worked. But I got an error which looks very strange "error message  Error: list index out of range" when I tried to save the query result. Do you have any ideas what might be causing this?
It looks like that the file can be saved in the VOS. So ignore it.
Glad you got it to work.
Robert

271 questions

273 answers

311 comments

1,841 users

Welcome to Data Lab Help Desk, where you can ask questions and receive answers from other members of the community.