Failed Download from VOS Public Directory

Question

Hello Datalab Team,

Numerous times I have attempted to download a 2GB csv file that is a result of a bulk/general query of Gaia EDR3.  The file is specifically stored in the vos://public/ directory and is names "test.csv".  The stated error from the Chrome Browser is "Failed-Network error".  However, I have verified that there are no existing network issues and that my connection is robust.  As a check, after one of my failed download attempts from Datalab, I performed a query with ESA/cosmos and was able to retrieve a large file of similar size without any issue (this download occurred over a several minute period).  

I realize the file size on the public folder is large.  I will attempt the download again with the Datalab python package (and CLT interface).  Yet, I should be able to successfully download large files with the Datalab API/web-based platform.

Thank you in advance for any help that you provide,

Jen S.

Answer 1

Hi Jen, thanks for reaching out.

I just tried to download the file from your VOSpace, i.e. public/test.csv, and it worked well in several ways:

1. Logged in to the DL notebook server, I was able to copy it to my own VOSpace, like this (you of course don't have to do this step):

from dl import storeClient as sc
sc.cp(fr='YOURUSERNAME://public/test.csv',to='vos://test.csv',verbose=True)
'OK'

This took 18 seconds. Then, in the JupyterLab directory/file panel on the left hand side, I navigated to my 'vospace/' directory, right-clicked on the test.csv fil , and selected "Download". Then said where to download to on my local laptop, and after 2m15s I had the file (speed will of course depend on your network speed). My browser is Vivaldi, but it's based in Chrome, so should be very similar in behavior.

2. If you have the 'datalab' command-line package installed on your local computer, you can also download the file from a local terminal.
Install the package via:

$pip install --ignore-installed --no-cache-dir noaodatalab

Then verify that it installed correctly:

$datalab version
Task Version:  2.19.1

Finally, log in once:

$datalab login
(and provide your DL username and password)

Now you can download the file like this:

$datalab get fr="vos://public/test.csv" to=test.csv verbose=True
(1/1) [====================] [   2.5G] k://public/test.csv

Please let us know if this works for you. And if not, please provide as much diagnostic info as you can. You can for instance send screenshots etc. via email to datalab@noirlab.edu.

Best,
Robert

Comments

Hi.  Thank you for the rapid response.  As mentioned in my original comment, I understood that it was possible to download the file with Datalab command-line package.  So, my question was oriented more toward the web API and why it was not possible to use this interface and the associated functionality.  So, I would navigate to the virtual storage site (https://datalab.noirlab.edu/query.php) and select "Download" for the desired file (ostensibly, this functionality is provided as "Download" represent a hyperlink that initially seems to allow for file download; the problem was that the file download terminated early).