0 votes
20 views
Hello! I am downloading many thousands of cutouts of galaxies from the DES DR2, and as it stands this is a process that takes multiple days. This is even with parallelization of the downloading procedure by spawning several python processes in a shell script loop. The download process is on the order of a couple of minutes for batches of 20 images, where each image is being downloaded by a separate python process. I have made sure that neither the CPU nor RAM are bottlenecks. Is there some way to optimize this procedure through datalab? I am aware of the existence of pre-created datasets such as through galaxy zoo, however I explicitly must work through FITS files, which is why I have avoided these sets.
by | 20 views

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
Anti-spam verification:
To avoid this verification in future, please log in or register.

452 questions

465 answers

474 comments

659 users

Welcome to Data Lab Help Desk, where you can ask questions and receive answers from other members of the community.

Categories