+3 votes
396 views

The following call is hanging for >10 minutes when it should be fairly swift. Not sure if this is a kernel problem or an OpSim package problem or a DataLab problem....

The call is:

opsdb = db.OpsimsDatabase(/sims_maf/fbs_1.6/potential_schedulers/barebones_nexp2_v1.6_10yrs.db)

The package is from:

import lsst.sims.maf.db as db

by melissagraham (220 points)
edited by 661 | 396 views

3 Answers

0 votes
Best answer
This problem should now be resolved.  It appears to be and issue with using sqlite databases over NFS mounts, and the way file locking done by sqlite is broken in some NFS systems.  With the recent addition of new sims data, the /sims_maf files had to be moved to an NFS partition with more storage.  At the moment, the files are still on NFS partitions, but one that appears to work.  To fully resolve the issue we will need to install more local disk on the machine, but for now please let us know if the problem returns.
by datalab (20.7k points)
selected by 661
I can confirm that my Jupyter notebook is now executing db.OpsimDatabase() immediately, thank you!
0 votes
Hi Melissa, some Data Lab systems may still be recovering from a data center-wide shutdown last weekend, which impacted the storage system in particular.  Thanks for the data point...
by knutago (180 points)
I should have mentioned -- this was also happening last Friday, right before the shutdown... but yes hopefully it is just related to systems coming back up. Thanks Knut!
Ah, ok.  Last Friday systems were being taken down in anticipation of shutdown, but we have also been having issues with NFS, which might have caused the slowness.  Work is underway to address both.
0 votes

I have a similar question, therefore I won't open another thread: my notebook hangs too at

import lsst.sims.maf.db as db

dbFile = '/sims_maf/fbs_1.5/baseline/baseline_v1.5_10yrs.db'

opsdb = db.OpsimDatabase(dbFile)

but not for just 10 minutes, but indefinitely. I have tried different baselines, I also tried to call directly OpsimDatabaseV4 or OpsimDatabaseV3 but there is no change at all. I have tried to import a custom OpSimDatabase (identical to the original one but with some print commands to find the problem), e.g.

from OpsimDatabase_ours import OpsimDatabaseV4

and I found that the hanging happens when inheriting the class

        super(OpsimDatabaseV4, self).__init__(database=database, driver=driver, host=host, port=port,
                                              defaultTable=defaultTable, longstrings=longstrings,
                                              verbose=verbose)

I could not understand more than this.

by

414 questions

430 answers

436 comments

635 users

Welcome to Data Lab Help Desk, where you can ask questions and receive answers from other members of the community.

Categories