Trying to find information on why db.OpsimDatabase in the LSST SIMS MAF kernel is now running so so slowly.

Question

The following call is hanging for >10 minutes when it should be fairly swift. Not sure if this is a kernel problem or an OpSim package problem or a DataLab problem....

The call is:

opsdb = db.OpsimsDatabase(/sims_maf/fbs_1.6/potential_schedulers/barebones_nexp2_v1.6_10yrs.db)

The package is from:

import lsst.sims.maf.db as db

Answer 1

This problem should now be resolved.  It appears to be and issue with using sqlite databases over NFS mounts, and the way file locking done by sqlite is broken in some NFS systems.  With the recent addition of new sims data, the /sims_maf files had to be moved to an NFS partition with more storage.  At the moment, the files are still on NFS partitions, but one that appears to work.  To fully resolve the issue we will need to install more local disk on the machine, but for now please let us know if the problem returns.

Comments

I can confirm that my Jupyter notebook is now executing db.OpsimDatabase() immediately, thank you!

Answer 2

Hi Melissa, some Data Lab systems may still be recovering from a data center-wide shutdown last weekend, which impacted the storage system in particular.  Thanks for the data point...

Comments

I should have mentioned -- this was also happening last Friday, right before the shutdown... but yes hopefully it is just related to systems coming back up. Thanks Knut!

---

Ah, ok.  Last Friday systems were being taken down in anticipation of shutdown, but we have also been having issues with NFS, which might have caused the slowness.  Work is underway to address both.

Answer 3

I have a similar question, therefore I won't open another thread: my notebook hangs too at

import lsst.sims.maf.db as db

dbFile = '/sims_maf/fbs_1.5/baseline/baseline_v1.5_10yrs.db'

opsdb = db.OpsimDatabase(dbFile)

but not for just 10 minutes, but indefinitely. I have tried different baselines, I also tried to call directly OpsimDatabaseV4 or OpsimDatabaseV3 but there is no change at all. I have tried to import a custom OpSimDatabase (identical to the original one but with some print commands to find the problem), e.g.

from OpsimDatabase_ours import OpsimDatabaseV4

and I found that the hanging happens when inheriting the class

        super(OpsimDatabaseV4, self).__init__(database=database, driver=driver, host=host, port=port,
                                              defaultTable=defaultTable, longstrings=longstrings,
                                              verbose=verbose)

I could not understand more than this.