Section author: Alice Jacques <alice.jacques@noirlab.edu>, Adam Scott <adam.scott@noirlab.edu>

Version: 20210105

1.6.3. Job ManagerΒΆ

The Job Manager (JM) is a batch-queue style service with integreated resource management that ultimately controls who can do what, when, and where. In this sense, everything can be a job: e.g. table query, SIA query, image cutout, x-match, copying files, or running analysis notebooks. The JM draws inspiration from High Performance Computing (HPC) queue managers such as the Slurm Workload Manager and PBS, including:

  • resource scheduling

  • several work queues of varying priorities (e.g. long-can-wait vs. fast-urgent)

  • result storage and delivery

  • user notification

The JM will allow us to circumvent some bottlenecks such as long async queries or x-matches and large user tables exceeding service timeouts (currently capped at 300 seconds for users). Additionally, the JM would allow us to offer user-defined batch jobs.

NOTE: the Job Manager at Data Lab is currently in the works and has not yet been deployed for user operation. Stay tuned for its release.