External file upload produces "Bad Gateway" error but only for very large files

Question

Hello,

I am trying to upload a CSV file with data on some ~2.6 million galaxies to MyDB to X-match it with DES DR2 to get some more information I need for a project of mine. However, whenever I attempt to upload this file to MyDB, I get this: 

"Upload started.
File uploaded. Waiting for response.
Failed!

Progress: 100%

The server saw it as:

Bad Gateway

The proxy server received an invalid response from an upstream server.

---

Apache/2.2.15 (CentOS) Server at datalab.noirlab.edu Port 443"

In essence, it seems to load the entire file but then it fails to push it through to MyDB. The part that confuses me is that I don't get this same error if I upload the first 10,000 or 100,000 rows; it seems to work just fine for smaller files.

Does anyone know what the issue may be? What would be the best way to go about doing this?

Thanks!

Answer 1

Hi,

sorry for the delay in replying to you. This is a problem we are trying to improve on, hopefully with a new release soon. Fortunately there are workarounds you can use to improve on the ingestion of large tables.

In the below link I'm providing two code examples: method 1 and method 2, that manage large inserts the fast way.

https://drive.google.com/drive/folders/1alf7jJ9Bmy2ANvbupqv7N5O60MINujTl?usp=sharing

• Method 1:

  • It uses a small sample of your data, it could be as small as one row to create the table via the import method.

  • Once that's done you can go and insert the rest.

  • Look at the import_a_big_table_method_1.py example script for more information.

• Method 2:

  • You know the types of your data, with that information create the table.

  • Once the table is created insert the data.

  • Look at the import_a_big_table_method_2.py example script for more information.

Along with the scripts there are mocked up sample data, of 10k rows, I created for the examples. Loading a file of 2M rows to mydb took me about 25 seconds.

Do let us know if you run into any problems,

Thanks,

Igor