What is the proper way to download large files from virtual storage? Getting consistent "network error" messages.

Question

I'm trying to download a lot of tables resulting from queries on the delve_dr1 database. I've broken things up in chunks of 1 GB and amy trying to download them from query VOS space.  Unfortunately, 80% oft the transfers always fail at ~700 -1000MB, regardless of browser and internet connection I use (university or home), and I'm finding I can't reliably transfer anything . This has been the case for several months. Thanks in advance for any help!

Paolo

Answer 1

Hi Paolo, thanks for reaching out. To diagnose, we will need a bit more information. You can send an email to datalab@noirlab.edu, and we'll take it from there.

Can you please tell us:

1) How are you running the queries? 'datalab' command line client? A notebook running on Data Lab's NB server? Or a NB running locally on your computer? Something else? (please post an example command that fails).

2) Are your queries maybe timing out? (if yes, try a higher timeout, up to timeout=600 (seconds) for sync queries; and there is also the async_=True mode, if you need longer-running queries).

3) Show us an example query that fails so we can try to reproduce it.

4) Are you running one query at a time, or maybe concurrently? And if concurrently, how?

Thanks,

Robert

Comments

Paolo, I initially missed that you wrote you are downloading files from your vospace. Still, please email to us:
a) A query that results in a file on vospace, which then subsequently fails to download.
b) Please describe how exactly you are attempting to download the file(s)
c) Maybe you can also point us to a file in your vospace?
We will try to reproduce your problem.
Thanks,
Robert