dl package

Submodules

dl.Util module

dl.Util.add_doc(value)[source]

Decorator to set the ‘Call docstring’ ipython field.

dl.Util.def_token(tok)[source]

Get a default token. If no token is provided, check for an existing $HOME/.datalab/id_token.<user> file and return that if it exists, otherwise default to the ANON_TOKEN.

If a token string is provided, return it directly. The value may also simply be a username, in which case the same check for a token ID file is done.

dl.Util.encode_multipart(fields, files, boundary=None)[source]

Encode dict of form fields and dict of files as multipart/form-data. Return tuple of (body_string, headers_dict). Each value in files is a dict with required keys ‘filename’ and ‘content’, and optional ‘mimetype’ (if not specified, tries to guess mime type or uses ‘application/octet-stream’).

..code-block:: python

>>> body, headers = encode_multipart({'FIELD': 'VALUE'},
...                                  {'FILE': {'filename': 'F.TXT', 'content': 'CONTENT'}},
...                                  boundary='BOUNDARY')
>>> print('\n'.join(repr(l) for l in body.split('\r\n')))
'--BOUNDARY'
'Content-Disposition: form-data; name="FIELD"'
''
'VALUE'
'--BOUNDARY'
'Content-Disposition: form-data; name="FILE"; filename="F.TXT"'
'Content-Type: text/plain'
''
'CONTENT'
'--BOUNDARY--'
''
>>> print(sorted(headers.items()))
[('Content-Length', '193'), ('Content-Type', 'multipart/form-data; boundary=BOUNDARY')]
>>> len(body)
193
dl.Util.multimethod(module, nargs, cm)[source]

Wrapper function to implement multimethod for functions. The identifying signature in this case is the number of required method parameters. When methods are called, all original arguments and keywords are passed.

dl.Util.readTokenFile(tok_file)[source]

dl.authClient module

dl.authClient.acToString(s)[source]

acToString – Force a return value to be type ‘string’ for all Python versions.

class dl.authClient.authClient[source]

Bases: object

AUTHCLIENT – Client-side methods to access the Data Lab

Authentication Service.

debug(debug_val)[source]

Toggle debug flag.

getConfig(section, param)[source]

Get a value from the configuration file.

getFromURL(svc_url, path, token)[source]

Get something from a URL. Return a ‘response’ object.

getHeaders(token)[source]

Get default tracking headers.

get_profile()[source]

Get the requested service profile.

Parameters

None

Returns

profile – The currently requested service profile.

Return type

str

Example

from dl import authClient
profile = authClient.client.get_profile()
get_svc_url()[source]

Return the currently-used Authentication Service URL.

Parameters

None

Returns

service_url – The currently-used Authentication Service URL.

Return type

str

Example

from dl import authClient
service_url = authClient.get_svc_url()
hasAccess(token, resource)[source]

See whether the given token has access to the named Resource.

Parameters
  • token (str) – User login token

  • resource (str) – Resource identifier to check

Returns

status – True if user owns or has access, False otherwise

Return type

bool

Example

from dl import authClient
status = authClient.hasAccess(token,'vos://test.dat')
isAlive(svc_url='https://datalab.noao.edu/auth')[source]
Check whether the AuthManager service at the given URL is

alive and responding. This is a simple call to the root service URL or ping() method.

Parameters

service_url (str) – The Query Service URL to ping.

Returns

result – True if service responds properly, False otherwise

Return type

bool

Example

from dl import authClient
if authClient.isAlive():
    print("Auth Manager is alive")
isTokenLoggedIn(token)[source]

See whether the user identified by the token is currently logged in.

Parameters

token (str) – User login token

Returns

status – True if token is marked as logged-in, False otherwise

Return type

bool

Example

isUserLoggedIn(user)[source]

See whether the user identified by the token is currently logged in.

Parameters

user (str) – User login name

Returns

status – True if user is currently logged-in, False otherwise

Return type

bool

Example

isValidPassword(user, password)[source]

See whether the password is valid for the user.

Parameters
  • user (str) – User login name

  • password (str) – Password for named user

Returns

status – True if password is valid for the user, False otherwise

Return type

bool

Example

isValidToken(token)[source]

See whether the current token is valid.

Parameters

token (str) – User login token

Returns

status – True if token is valid, False otherwise

Return type

bool

Example

isValidUser(user)[source]

See whether the specified user is valid.

Parameters

user (str) – User login name

Returns

status – True if ‘user’ is a valid user name, False otherwise

Return type

bool

Example

list_profiles(token, profile=None, format='text')[source]

List the service profiles which can be accessed by the user.

Parameters

token (str) – Valid auth service token.

Returns

profiles

Return type

JSON string

Example

from dl import authClient
profiles = authClient.client.list_profiles(token, profile, format)
loadConfig()[source]

Read the $HOME/.datalab/dl.conf file.

login(username, password, debug=False, verbose=False)[source]
Authenticate the user with the Authentication Service.

We first check for a valid login token in the user’s $HOME/.datalab/ directory and simply return that rather than make a service call to get a new token. If a token exists but is invalid, we remove it and get a new token. In either case, we save the token for later use.

Parameters
  • username (str) – User login name.

  • password (str) – User password. If not given, a valid ID token will be searched for in the $HOME/.datalab directory.

  • debug (bool) – Method debug flag.

  • verbose (bool) – Initialize session to print verbose output messages.

Returns

token – One-time security token for valid user(identified via ‘username’ and ‘password’).

Return type

str

Example

from dl import authClient
token = authClient.login('dldemo', 'dldemo')   # get security token
logout(token=None)[source]

Log the user out of the Data Lab.

Parameters

token (str) – User login token

Returns

Return type

‘OK’ string on success, or exception message

Example

from dl import authClient
status = authClient.logout(token)
passwordReset(token, username, password)[source]
Reset a user password reset. We require that the user provide

either a valid ‘root’ token or the token for the account being reset.

Parameters
  • token (str) – User login token

  • username (str) – User login name

  • password (str) – User password

Returns

Return type

‘OK’ string on success, or exception message

Example

from dl import authClient
status = authClient.logout(token)
retBoolValue(url)[source]

Utility method to call a boolean service at the given URL.

setConfig(section, param, value)[source]

Set a value and save the configuration file.

set_profile(profile)[source]

Set the requested service profile.

Parameters

profile (str) – Requested service profile string.

Returns

Return type

Nothing

Example

from dl import authClient
token = authClient.client.set_profile("dev")
set_svc_url(svc_url)[source]

Set the URL of the Authentication Service to be used.

Parameters

svc_url (str) – Authentication service base URL to call.

Returns

Return type

Nothing

Example

from dl import authClient
authClient.set_svc_url("http://localhost:7001/")
whoAmI()[source]

Return the currently logged-in user identity.

Parameters

None

Returns

name – Currently logged-in user name, or ‘anonymous’ if not logged-in

Return type

str

Example

from dl import authClient
name = authClient.whoAmI()
writeConfig()[source]

Write out the configuration file to disk.

exception dl.authClient.dlAuthError(message)[source]

Bases: Exception

A throwable error class.

dl.authClient.getClient()[source]

Get a new instance of the authClient client.

Parameters

None

Returns

client – An authClient object

Return type

authClient

Example

dl.authClient.get_profile()[source]

Get the requested service profile.

Parameters

None

Returns

profile – The currently requested service profile.

Return type

str

Example

from dl import authClient
profile = authClient.client.get_profile()
dl.authClient.get_svc_url()[source]

Return the currently-used Authentication Service URL.

Parameters

None

Returns

service_url – The currently-used Authentication Service URL.

Return type

str

Example

from dl import authClient
service_url = authClient.get_svc_url()
dl.authClient.hasAccess(user, resource)[source]

See whether the given token has access to the named Resource.

Parameters
  • token (str) – User login token

  • resource (str) – Resource identifier to check

Returns

status – True if user owns or has access, False otherwise

Return type

bool

Example

from dl import authClient
status = authClient.hasAccess(token,'vos://test.dat')
dl.authClient.isAlive(svc_url='https://datalab.noao.edu/auth')[source]
Check whether the AuthManager service at the given URL is

alive and responding. This is a simple call to the root service URL or ping() method.

Parameters

service_url (str) – The Query Service URL to ping.

Returns

result – True if service responds properly, False otherwise

Return type

bool

Example

from dl import authClient
if authClient.isAlive():
    print("Auth Manager is alive")
dl.authClient.isTokenLoggedIn(token)[source]

See whether the user identified by the token is currently logged in.

Parameters

token (str) – User login token

Returns

status – True if token is marked as logged-in, False otherwise

Return type

bool

Example

dl.authClient.isUserLoggedIn(user)[source]

See whether the user identified by the token is currently logged in.

Parameters

user (str) – User login name

Returns

status – True if user is currently logged-in, False otherwise

Return type

bool

Example

dl.authClient.isValidPassword(user, password)[source]

See whether the password is valid for the user.

Parameters
  • user (str) – User login name

  • password (str) – Password for named user

Returns

status – True if password is valid for the user, False otherwise

Return type

bool

Example

dl.authClient.isValidToken(token)[source]

See whether the current token is valid.

Parameters

token (str) – User login token

Returns

status – True if token is valid, False otherwise

Return type

bool

Example

dl.authClient.isValidUser(user)[source]

See whether the specified user is valid.

Parameters

user (str) – User login name

Returns

status – True if ‘user’ is a valid user name, False otherwise

Return type

bool

Example

dl.authClient.list_profiles(token, profile=None, format='text')[source]
dl.authClient.login(user, password=None, debug=False, verbose=False)[source]
Authenticate the user with the Authentication Service.

We first check for a valid login token in the user’s $HOME/.datalab/ directory and simply return that rather than make a service call to get a new token. If a token exists but is invalid, we remove it and get a new token. In either case, we save the token for later use.

Parameters
  • username (str) – User login name.

  • password (str) – User password. If not given, a valid ID token will be searched for in the $HOME/.datalab directory.

  • debug (bool) – Method debug flag.

  • verbose (bool) – Initialize session to print verbose output messages.

Returns

token – One-time security token for valid user(identified via ‘username’ and ‘password’).

Return type

str

Example

from dl import authClient
token = authClient.login('dldemo', 'dldemo')   # get security token
dl.authClient.logout(token=None)[source]

Log the user out of the Data Lab.

Parameters

token (str) – User login token

Returns

Return type

‘OK’ string on success, or exception message

Example

from dl import authClient
status = authClient.logout(token)
dl.authClient.passwordReset(token, username, password)[source]
Reset a user password reset. We require that the user provide

either a valid ‘root’ token or the token for the account being reset.

Parameters
  • token (str) – User login token

  • username (str) – User login name

  • password (str) – User password

Returns

Return type

‘OK’ string on success, or exception message

Example

from dl import authClient
status = authClient.logout(token)
dl.authClient.set_profile(profile)[source]

Set the requested service profile.

Parameters

profile (str) – Requested service profile string.

Returns

Return type

Nothing

Example

from dl import authClient
token = authClient.client.set_profile("dev")
dl.authClient.set_svc_url(svc_url)[source]

Set the URL of the Authentication Service to be used.

Parameters

svc_url (str) – Authentication service base URL to call.

Returns

Return type

Nothing

Example

from dl import authClient
authClient.set_svc_url("http://localhost:7001/")
dl.authClient.whoAmI()[source]

Return the currently logged-in user identity.

Parameters

None

Returns

name – Currently logged-in user name, or ‘anonymous’ if not logged-in

Return type

str

Example

from dl import authClient
name = authClient.whoAmI()

dl.dlinterface module

class dl.dlinterface.DLInteract[source]

Bases: object

Main class for Data Lab interactions

get(section, param)[source]

Get a value from the configuration file.

save(section, param, value)[source]

Save the configuration file.

class dl.dlinterface.Dlinterface(verbose=True)[source]

Bases: object

Data Lab python interface super-class with methods for each command.

copyurl(url=None, name=None)[source]

Copy a file to VOSpace using a URL.

Parameters
  • url (str) – The URL location of the file.

  • name (str) – The name of the file in VOSpace. The vos:// prefix is not necessary.

Example

Copy the file http://www.mywebsite.com/file1.fits to output1.fits in VOSpace.

dl.copyurl('http://www.mywebsite.com/file1.fits','output1.fits')
cp(source=None, destination=None, verbose=True)[source]

Copy a file in Data Lab VOSpace.

Parameters
  • source (str) – The name of the file in VOSpace to copy, e.g. file1.txt.

  • destination (str) – The new name of the file in VOSpace, e.g. newfile1.txt.

Example

Copy the file file.txt to newfile.txt.

dl.ls()
file1.txt

dl.cp('file1.txt','newfile.txt')

dl.ls()
file1.txt, newfile.txt
droptable(table=None)[source]

Drop a user’s MyDB table.

Parameters

table (str) – The name of a specific table in mydb to drop.

Returns

list – The list of properties of table or all tables in mydb.

Return type

str

Example

Drop the MyDB table called table.

print dl.listdb()
table
table2

dl.droptable('table')
table
table2

print dl.listdb()
table2
exporttable(table=None, name=None, fmt=None)[source]

Copy a user’s MyDB table to a file in VOSpace.

Parameters
  • table (str) – The name of a specific table in mydb to drop.

  • name (str) – The file name to save the table to.

  • fmt (str) – The output file format. The available formats are ‘csv’, ‘fits’ and ‘hdf5’. If this is not specified then the file extension is used to identify the format type.

Example

Export the MyDB table called table to file test.csv.

dl.exporttable('table','test.csv')
get(source=None, destination=None, verbose=True)[source]

Get one or more files from Data Lab.

Parameters
  • source (str) – The name of the source file on VOSpace, e.g. file2.txt.

  • destination (str) – The name of the local destination file, e.g. file1.txt.

Example

Get a query output table called table1_output.txt from VOSpace.

dl.get('table1_output.txt','table1_output.txt')
(1/1) [====================] [   9.1K] table1_output.txt
help(command=None)[source]

Print out useful help information on the Data Lab python interface and it’s commands.

listdb(table='')[source]

List the user’s MyDB tables.

Parameters

table (str) – The name of a specific table in mydb. If this is blank then all tables will be listed.

Returns

list – The list of properties of table or all tables in mydb.

Return type

str

Example

List the MyDB tables.

print dl.listmydb()
table
table2
ln(target=None, link=None)[source]

Link a file in Data Lab VOSpace.

Parameters
  • target (str) – The name of the file in VOSpace to link to, e.g. file1.txt.

  • link (str) – The name of the link, e.g. file1link.

Example

Create a link called iamlink to the file file1.txt.

dl.ls()
file1.txt

dl.ln('file1.txt','iamlink')

dl.ls()
file1.txt, iamlink
load(name=None, inpfmt=None, fmt='pandas', ext=None)[source]

Save the string representation of a data object to a file in VOSpace.

Parameters
  • name (str) – The name of the file in load into memory. The vos:// prefix is necessary otherwise it is assumed a local file that should be read. Currently only FITS binary tables and CSV file formats are supported.

  • inpfmt (str) – Tne input format type. The currently supported types are FITS binary tables, HDF5, CSV, and ASCII files (‘string’). If this is not specified then the file extension of name is used to attempt to figure out the format.

  • fmt (str) –

    The data type format to output. Permitted values are:
    • ’csv’ the returned result is a comma-separated string that looks like a

      csv file (newlines at the end of every row)

    • ’ascii’ same as csv but tab-delimited

    • ’string’ just a straight read of ASCII into a string

    • ’array’ Numpy array

    • ’structarray’ Numpy structured / record array

    • ’pandas’ a Pandas data frame

    • ’table’ in Astropy Table format

    • ’votable’ result is a string XML-formatted as a VO table

    The output type for a FITS image is a numpy array. For other data ‘pandas’ is the default format.

  • ext (int) – The FITS extension to load for images. The default is 1 for a FITS binary table and 0 for a FITS image.

Example

Load the file “output.fits” into a pandas data frame.

df = dl.load('output.fits',fmt='pandas')

Load a FITS image “im1.fits”.

im,head = dl.load('im1.fits')
login(user=None)[source]

Login to Data Lab using username.

Parameters

user (str) – The Data lab username. If this is not given, then the user will be prompted for the information.

Example


Login and give the username,

dl.login(‘myusername’) Enter password: *** Welcome to the Data Lab, myusername

or,

dl.login() Enter user: myusername Enter password: ** Welcome to the Data Lab, myusername

logout(unmount=None, verbose=True)[source]

Logout out of the Data Lab.

Example

Logout of Data Lab.

dl.logout()
'myusername' is now logged out of the Data Lab
ls(name='vos://', format='csv', verbose=False)[source]

List files in VOSpace.

Parameters
  • name (str) – The name of a specific file to list. If name is blank then all files will be listed.

  • format (str) – The format to use.

  • verbose (bool) – Give more verbose output, or just a list of files. The default is verbose=True.

Returns

results – The list of files in VOSpace.

Return type

str

Example

List the files.

dl.ls()
test2  test1

Verbose listing of the files in the public/ directory.

dl.ls('public',verbose=True)
lrw-rw----  demo15      0B  17 May 2017 14:04:25  thisisalsoalink -> /public/smash2
lrw-rw----  demo15      0B  17 May 2017 13:58:04  thisisalink -> /smash1
-rw-rw-r--  demo15    3.4K  17 May 2017 09:40:13  smash2
-rw-rw-r--  demo15    3.4K  17 May 2017 07:34:54  smash1
drw-rw----  demo15      0B  17 May 2017 14:05:02  data/  tableingester,downloader,runner
mkdir(name=None)[source]

Create a directory in Data Lab VOSpace.

Parameters

name (str) – The name of the directory in VOSpace to create, e.g. results.

Example

Create the directory data1/.

dl.mkdir('data1')
mv(source=None, destination=None, verbose=True)[source]

Move a file in Data Lab VOSpace.

Parameters
  • source (str) – The name the file in VOSpace to move/rename, e.g. file1.txt.

  • destination (str) – The new name of the file in VOSpace (e.g. newfile1.txt) or the directory to move it to.

Example

Rename the file file.txt to newfile.txt.

dl.ls()
file.txt

dl.mv('file.txt','newfile.txt')

dl.ls()
newfile.txt

Move the file output.fits to the results/ directory.

dl.ls()
output.txt, results

dl.mv('output.fits','results/output.fits')

dl.ls()
results/output.txt
put(source=None, destination=None, verbose=True)[source]

Put files into Data Lab VOSpace.

Parameters
  • source (str) – The name of a local file to upload to VOSpace, e.g. file1.txt.

  • destination (str) – The name of the destination file with, e.g. file2.txt. The destination file can have the vos:// prefix but it is not required.

Example

Put a catalog called cat.fits into VOSpace.

dl.put('cat.fits','cat.fits')
(1 / 1) cat.fits -> vos://cat.fits
query(query=None, qtype='sql', fmt='csv', out=None, async_=False, drop=False, profile='default', verbose=True, **kw)[source]

Send a query to a remote query service.

Parameters
  • query (str) –

    The query string that will be passed to the queryClient and then to the DB query manager. This can either be in the SQL or ADQL format (specified by the “type” parameter). For example,

    'select ra,dec from gaia_dr1.gaia_source limit 3'
    

  • qtype (str) – The query format, SQL or ADQL. SQL is used by default.

  • fmt (str) –

    Format of the result to be returned by the query. Permitted values are.

    For file output and direct output to python: * ‘csv’ the returned result is a comma-separated string that looks like a csv file (newlines at the end of every row) * ‘ascii’ same as csv but tab-delimited * ‘votable’ result is a string XML-formatted as a VO table Only for direct output to python: * ‘array’ Numpy array * ‘structarray’ Numpy structured / record array * ‘pandas’ a Pandas data frame * ‘table’ in Astropy Table format Only for file output: * ‘fits’ FITS binary table. Only if the results are saved to a file with out=. * ‘hdf5’ HDF5 file. Only if the results are saved to a file with out=.

  • out (str or None) – The output name if the results are to be saved to mydb (mydb://tablename), to VOSpace (vos://filename), or the local file system (file:// and other names with no prefix). The files are in csv format.

  • async_ (bool) –

    If True, the query is asynchronous, i.e. a job is submitted to the DB, and a jobID is returned. The jobID must be then used to check the query’s status and to retrieve the result (when status is COMPLETE). Default is False, i.e. synchroneous query.

    async_ replaces the previous async parameter, because async was promoted to a keyword in Python 3.7. Users of Python versions prior to 3.7 can continue to use the async keyword.

  • drop (bool) – If True, then if the query is saving to mydb where the same table name already exists, it will overwrite the old mydb table.

Returns

result – If async_=False and out is not used, then the return value is the result of the query in the requested format (see fmt). If out is given then the query result is saved to a file or mydb. If async_=True the jobID is returned with which later the asynchronous query’s status can be checked (dl.querystatus()), and the result retrieved (see dl.queryresults().

Return type

str

Example

A simple query returned as a pandas data frame.

data = dl.query('SELECT * from smash_dr1.source LIMIT 100',fmt='pandas')
Returning Pandas dataframe

type(data)
pandas.core.frame.DataFrame

print data['ra'][0:3]
0    103.068355
1    103.071774
2    103.071598

Perform a query and save the results to a table called “table1.txt” in mydb.

res = dl.query('SELECT * from smash_dr1.source LIMIT 100',out='mydb://table1.txt')

 dl.listmydb()

Perform the same query and save it to a local file.

res = dl.query('SELECT * from smash_dr1.source LIMIT 100',out='table1.txt')

ls
table1.txt
queryhistory(async_=None, **kw)[source]

Report the history of queries made so far.

Parameters
  • async (bool) –

    A boolean (True/False) of whether to only show the ASYNC queries. By default all queries are shown.

    async_ replaces the previous async parameter, because async was promoted to a keyword in Python 3.7. Users of Python versions prior to 3.7 can continue to use the async keyword.

  • Results

  • -------

  • information on part queries is output to the screen with the following (The) –

  • columns (query ID, submission time, query type (sql/adql), sync or async query, jobid (for async queries),) – output format, status of query (or number of returned rows if sync query), query string

Examples

Perform some queries and then list the history.

data1 = dl.query('select ra,dec from smash_dr1.source limit 100',fmt='csv')
Returning CSV formatted table as a string

data2 = dl.query('select ra,dec from smash_dr1.source limit 500',fmt='pandas')
Returning Pandas dataframe

data3 = dl.query('select ra,dec from smash_dr1.source limit 1000',fmt='structarray')
Returning Numpy structured / record array

dl.queryhistory()
1  2017-05-16 13:27:34  sql  SYNC  pandas  100  --  'select ra,dec,gmag from smash_dr1.object limit 100'
2  2017-05-16 13:27:40  sql  SYNC  csv  500  --  'select ra,dec,gmag from smash_dr1.object limit 500'
3  2017-05-16 13:27:46  sql  SYNC  structarray  1000  --  'select ra,dec,gmag from smash_dr1.object limit 1000'
queryprofiles(profile=None)[source]

List the available Query Manager profiles to use with a dl.query().

Parameters

profile (str) – The name of a specific Query Manager profile to check. If this is blank then all of the available profile names will be listed.

Returns

results – The list of properties of profile profile, or a list of all available profiles.

Return type

str

Example

List of available profiles.

dl.queryprofiles()
default,IRSA,HEASARC,Vizier,GAVO,SIMBAD,zeus1,SDSS-DR9,STScI-RegTAP,GALEX-DR6,dldb1

Get profile information on profile dldb1.

dl.queryprofiles(‘dldb1’) {u’accessURL’: u’http://dldb1.sdm.noao.edu:8080/ivoa-dal/tap’, u’dbport’: 5432, u’password’: u’datalab’, u’description’: u’Development NOAO Data Lab TAP Service / Database on dldb1’, u’database’: u’tapdb’, u’host’: u’dldb1.sdm.noao.edu’, u’vosRoot’: u’vos://datalab.noao!vospace’, u’vosEndpoint’: u’http://dldb1.sdm.noao.edu:8080/vospace-2.0/vospace’, u’user’: u’dlquery’, u’vosRootDir’: u’/data/vospace/users’, u’type’: u’datalab’}

queryresults(jobid=None)[source]

Get the async query results.

Parameters

jobid (str) – This can be either (1) the Query ID (QID) returned by the dl.queryhistory() command, or (2) the unique job identifier for the asynchronous query which was returned by ql.query() when the query job was submitted.

Returns

result – The result of the query in the requested format (see fmt in dl.query().

Return type

str

Example

Submit an asynchronous query and then check the status.

jobid = dl.query('SELECT ra,dec from smash_dr1.source LIMIT 3',async_=True)
Asynchronous query JobID = uqrcs8a5n8s6d0je

dl.querystatus(jobid)
COMPLETED

results = dl.queryresults(jobid)
print results
ra,dec
103.068354922718,-37.973538878907299
103.071774116284,-37.973599429479599
103.071597827998,-37.972329108796401
querystatus(jobid=None)[source]

Get the async query job status.

Parameters

jobid (str) – This can be either (1) the Query ID (QID) returned by the dl.queryhistory() command, or (2) the unique job identifier for the asynchronous query which was returned by ql.query() when the query job was submitted.

Returns

status – The status of the query, which can be one of the following: QUEUED the query job is in queue and waiting to be executed. EXECUTING the query job is currently running. COMPLETED the query is done and the results are ready to be retrieved with dl.queryresults(). ERROR there was a problem with the query.

Return type

str

Example

Submit an asynchronous query and then check the status.

jobid = dl.query('SELECT ra,dec from smash_dr1.source LIMIT 100',async_=True)
Asynchronous query JobID = uqrcs8a5n8s6d0je

dl.querystatus(jobid)
COMPLETED
rm(name=None, verbose=True)[source]

Delete files in Data Lab VOSpace.

Parameters

name (str) – The name of the file in VOSpace to delete, e.g. file1.txt.

Example

Delete the file file1.txt.

dl.ls()
file1.txt, file2.txt

dl.rm('file1.txt')

dl.ls()
file2.txt
rmdir(name=None)[source]

Delete a directory in Data Lab VOSpace.

Parameters

name (str) – The name of the directory in VOSpace to delete, e.g. results.

Example

Delete the directory data1/.

dl.rmdir('data1')
save(data=None, name=None, fmt=None, clobber=False)[source]

Save the string representation of a data object to a file in VOSpace.

Parameters
  • data (str) – The data object such as a pandas data frame or numpy structured array.

  • name (str) – The name of the file in VOSpace to create. The vos:// prefix is not necessary.

  • fmt (str) –

    The format to use for the output file. If this is not specified then the file extension of name is used to attempt to figure out the format. The currently supported input and output formats are:

    Data type Format csv csv ascii ascii array csv/fits structarray csv/fits pandas csv/fits/hdf5 table csv/fits/hdf5 votable csv/fits/hdf5

  • clobber (bool) – Whether to overwrite an existing file. The default is False.

Example

Save the pandas data frame called “df” to a file called “data1.csv” in VOSpace.

dl.save(df,'data1.csv')
schema(val='', fmt='text', profile='default')[source]

Print information about data service schema.

Parameters
  • val (str) – Value to list ([[<schema>][.<table>][.<col>]]).

  • fmt (str) – Output format (csv|text|json).

  • profile (str) – Service profile.

Returns

results – The schema information is printed to the screen.

Return type

str

Example

Print out all the DL tables.

 datalab schema

 Schema Name   Description
-----------   -----------
   gaia_dr1   GAIA Data Release 1
       ivoa   IVOA ObsCore tables
   des_sva1   DES SVA1 Data Products
 tap_schema   TAP Schema Tables
       usno   USNO Astrometry Catalogs
  sdss_dr13
    neo_dr1   NEO Survey Data Release 1
     ls_dr3   The DECam Legacy Survey Data Release 3
  smash_dr1   SMASH Data Release 1

List all tables in a schema/catalog.

datalab schema val=smash_dr1

Schema: smash_dr1

Table Name   Description
----------   -----------
      chip   Info on each chip in the frame
  exposure   Info on each exposure
     field   Info on each target field (position, Num exposures, etc)
    object   Average photometry of each unique object
    source   All of the individual source measurements
     stars   View of object table to select for stars
  galaxies   View of object table to select for galaxies
    xmatch   Crossmatch of object against GAIA DR1 and WISE
servicestatus()[source]

This checks on the status of the DL services.

siaquery(ra=None, dec=None, dist=None, verbose=False)[source]

Perform a SIA query with a set of coordinates or from an uploaded file.

Parameters
  • ra (float) – The right ascension (in degrees) of the point to use for the search.

  • dec (float) – The declination (in degrees) of the point to use for the search.

  • dist (float) – The search distance (radius) in degrees. The default is 0.0085 deg.

  • verbose (bool) – Use verbose output. The default is False.

Returns

images – The list of images in Astropy table format.

Return type

votable

Example

Perform a simple SIA search.

itab = dl.siaquery(0.5,10.0,0.1)
The image list contains 6 entries

Download the first image using copyurl().

dl.copyurl(itab['access_url'][0],'im1.fits')
status()[source]

Print the status of the Data Lab connection.

Example

The “myusername” is logged in.

dl.status()
User myusername is logged into the Data Lab

No user is currently logged in.

dl.status()
No user is currently logged into the Data Lab
whoami()[source]

Print the current active user.

Example

dl.whoami()
myusername
dl.dlinterface.addFormatMapping(self)[source]

Add the format mapping information to the DL object

dl.dlinterface.areLoginsWorking()[source]

This checks if the Authentication Manager is returning proper tokens.

dl.dlinterface.areSyncQueriesWorking()[source]

This checks if the Query Manager is returning proper Sync queries.

dl.dlinterface.checkLogin(self)[source]

Check if the user is already logged in. If not, give a warning message

dl.dlinterface.convertTableToFormat(t, format)[source]

Convert Astropy table to a different format using StringIO and write method.

dl.dlinterface.convert_vospace_time_to_seconds(str_date)[source]

A convenience method that takes a string from a vospace time field and converts it to seconds since epoch.

Parameters

str_date (str) – string to parse into a VOSpace time

Returns

A datetime object for the provided string date

Return type

datetime

dl.dlinterface.getNodeInfo(self, xnode, lenpathbase, verbose=True)[source]

Get information on a node. The input is a “node” element of a XML ElementTree.

dl.dlinterface.getUserName(self)[source]

Get the currently logged-in user token. If we haven’t logged in return the anonymous username.

dl.dlinterface.getUserToken(self)[source]

Get the currently logged-in user token. If we haven’t logged in return the anonymous token.

dl.dlinterface.isListWorking()[source]

This checks if the Storage Manager is returning proper list queries:

dl.dlinterface.isTapWorking()[source]

This checks if the TAP service and Tomcat are running.

dl.dlinterface.readAscii(filename)[source]

Read in an ASCII file and return the data

dl.dlinterface.reformatQueryOutput(self, res=None, fmt='csv', verbose=True)[source]

Reformat the output of a query based on a format.

dl.dlinterface.writeAscii(filename, txt)[source]

Write data to ASCII file

dl.dltasks module

class dl.dltasks.AddCapability(datalab)[source]

Bases: dl.dltasks.Task

Add a capability to a VOSpace container

run()[source]
class dl.dltasks.Broadcast(datalab)[source]

Bases: dl.dltasks.Task

Broadcast a SAMP message

run()[source]
class dl.dltasks.Copy(datalab)[source]

Bases: dl.dltasks.Task

Copy a file in Data Lab

run()[source]
class dl.dltasks.DataLab[source]

Bases: object

Main class for Data Lab interactions

get(section, param)[source]

Get a value from the configuration file.

save(section, param, value)[source]

Save the configuration file.

class dl.dltasks.Delete(datalab)[source]

Bases: dl.dltasks.Task

Delete files in Data Lab

run()[source]
class dl.dltasks.DropMyDB(datalab)[source]

Bases: dl.dltasks.Task

Drop a user’s MyDB table. [DEPRECATED]

run()[source]
class dl.dltasks.Get(datalab)[source]

Bases: dl.dltasks.Task

Get one or more files from Data Lab.

run()[source]
class dl.dltasks.Init(datalab)[source]

Bases: dl.dltasks.Task

Print the current service URLS in use.

run()[source]

Initialize the Data Lab configuration.

class dl.dltasks.Launch(datalab)[source]

Bases: dl.dltasks.Task

Launch a plugin in Data Lab

run()[source]
class dl.dltasks.LaunchJob(datalab)[source]

Bases: dl.dltasks.Task

Execute a remote processing job in the Data Lab

getJob(job)[source]
run()[source]
validJob(job)[source]

Bases: dl.dltasks.Task

Link a file in Data Lab

run()[source]
class dl.dltasks.List(datalab)[source]

Bases: dl.dltasks.Task

List files in Data Lab

run()[source]
class dl.dltasks.ListCapability(datalab)[source]

Bases: dl.dltasks.Task

Add a capability to a VOSpace container

run()[source]
class dl.dltasks.ListMyDB(datalab)[source]

Bases: dl.dltasks.Task

List the user’s MyDB tables. [DEPRECATED]

run()[source]
class dl.dltasks.Login(datalab)[source]

Bases: dl.dltasks.Task

Log into the Data Lab

do_login()[source]
login()[source]

Login to the Data Lab.

run()[source]

Execute the Login Task.

class dl.dltasks.Logout(datalab)[source]

Bases: dl.dltasks.Task

Logout out of the Data Lab

run()[source]
class dl.dltasks.MkDir(datalab)[source]

Bases: dl.dltasks.Task

Create a directory in Data Lab

run()[source]
class dl.dltasks.Mountvofs(datalab)[source]

Bases: dl.dltasks.Task

Mount a VOSpace via FUSE

run()[source]
class dl.dltasks.Move(datalab)[source]

Bases: dl.dltasks.Task

Move files in Data Lab

run()[source]
class dl.dltasks.MyDB_Copy(datalab)[source]

Bases: dl.dltasks.Task

Copy a user MyDB table.

run()[source]
class dl.dltasks.MyDB_Create(datalab)[source]

Bases: dl.dltasks.Task

Create a user MyDB table.

run()[source]
class dl.dltasks.MyDB_Drop(datalab)[source]

Bases: dl.dltasks.Task

Drop a user’s MyDB table.

run()[source]
class dl.dltasks.MyDB_Import(datalab)[source]

Bases: dl.dltasks.Task

Import data into a user MyDB table.

run()[source]
class dl.dltasks.MyDB_Index(datalab)[source]

Bases: dl.dltasks.Task

Index data in a user’s MyDB table.

run()[source]
class dl.dltasks.MyDB_Insert(datalab)[source]

Bases: dl.dltasks.Task

Insert data into a user MyDB table.

run()[source]
class dl.dltasks.MyDB_List(datalab)[source]

Bases: dl.dltasks.Task

List the user’s MyDB tables.

run()[source]
class dl.dltasks.MyDB_Rename(datalab)[source]

Bases: dl.dltasks.Task

Rename a user MyDB table.

run()[source]
class dl.dltasks.MyDB_Truncate(datalab)[source]

Bases: dl.dltasks.Task

Truncate a user MyDB table.

run()[source]
class dl.dltasks.Option(name, value, description, display=None, default=None, required=False)[source]

Bases: object

Represents an option

class dl.dltasks.Put(datalab)[source]

Bases: dl.dltasks.Task

Put files into Data Lab.

run()[source]
class dl.dltasks.Query(datalab)[source]

Bases: dl.dltasks.Task

Send a query to a remote query service (OLD VERSION - NOT USED))

dbquery(h, url, token)[source]
httpquery(h, url, token)[source]
ivoquery(h, uri, token, out)[source]
run()[source]
class dl.dltasks.Query2(datalab)[source]

Bases: dl.dltasks.Task

Send a query to a remote query service. [Note: placeholder name until we figure out what to do with the old Query() functionality.]

run()[source]
class dl.dltasks.QueryProfiles(datalab)[source]

Bases: dl.dltasks.Task

List the available Query Manager profiles.

run()[source]
class dl.dltasks.QueryResults(datalab)[source]

Bases: dl.dltasks.Task

Get the async query results.

run()[source]
class dl.dltasks.QueryStatus(datalab)[source]

Bases: dl.dltasks.Task

Get the async query job status.

run()[source]
class dl.dltasks.Receiver(client)[source]

Bases: object

SAMP listener

point_select(private_key, send_id, mtype, params, extra)[source]
receive_notifications(private_key, sender_id, mtype, params, extra)[source]
receiver_call(private_key, sender_id, msg_id, mtype, params, extra)[source]
class dl.dltasks.Resolve(datalab)[source]

Bases: dl.dltasks.Task

Resolve a vos short form identifier – FIXME

run()[source]
class dl.dltasks.RmDir(datalab)[source]

Bases: dl.dltasks.Task

Delete a directory in Data Lab

run()[source]
class dl.dltasks.Schema(datalab)[source]

Bases: dl.dltasks.Task

Print information about data servicce schema

run()[source]
class dl.dltasks.Services(datalab)[source]

Bases: dl.dltasks.Task

Print the available data services.

run()[source]
class dl.dltasks.SiaQuery(datalab)[source]

Bases: dl.dltasks.Task

SIA query with an uploaded file

getUID()[source]

Get the UID for this user

run()[source]
class dl.dltasks.Status(datalab)[source]

Bases: dl.dltasks.Task

Status of the Data Lab connection

run()[source]
class dl.dltasks.StorageProfiles(datalab)[source]

Bases: dl.dltasks.Task

List the available Storage Manager profiles.

run()[source]
class dl.dltasks.SvcURLs(datalab)[source]

Bases: dl.dltasks.Task

Print the current service URLS in use.

run()[source]
class dl.dltasks.Tag(datalab)[source]

Bases: dl.dltasks.Task

Tag a file in Data Lab

run()[source]
class dl.dltasks.Task(datalab, name, description)[source]

Bases: object

Superclass to represent a Task

addLogger(logLevel, logFile)[source]

Add a Logger to the Task.

addOption(name, option)[source]

Add an option to the Task.

addStdOptions()[source]
broadcast(client, messageType, params, app=None)[source]

Broadcast a SAMP message.

getSampConnect()[source]

Get a SAMP listening client connection.

listen(client)[source]

Setup a listener for a specific SAMP message.

report(resp)[source]

Handle call response

run()[source]
setLogger(logLevel=None)[source]

Set the logger to be used.

setOption(name, value)[source]

Set a Task option.

class dl.dltasks.Version(datalab)[source]

Bases: dl.dltasks.Task

Print the task version.

run()[source]
class dl.dltasks.WhoAmI(datalab)[source]

Bases: dl.dltasks.Task

Print the current active user.

run()[source]
dl.dltasks.getUserName(self)[source]

Get the currently logged-in user token. If we haven’t logged in return the anonymous token.

dl.dltasks.getUserToken(self)[source]

Get the currently logged-in user token. If we haven’t logged in return the anonymous token.

dl.dltasks.parseSelf(obj)[source]

dl.queryClient module

dl.queryClient.abort[source]

Abort the specified asynchronous job.

Usage:

abort (token=None, jobId=None)

queryClient.abort (token, jobId) queryClient.abort (jobId) queryClient.abort (token, jobId=<id>) queryClient.abort (jobId=<str>)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • jobId (str) – The jobId to abort.

Returns

results

Return type

str

Example

# issue an async query (here a tiny one just for this example)
query = 'select ra,dec from gaia_dr1.gaia_source limit 3'
jobId = queryClient.query(token, adql=query, fmt='csv', async=True)

# ensure job completes...then check status and retrieve results
time.sleep(4)
if queryClient.status(token, jobId) == 'COMPLETED':
    results = queryClient.results(token,jobId)
    print type(results)
    print results

This prints

<type 'str'>
ra,dec
301.37502633933002,44.4946851014515588
301.371102372343785,44.4953207577355698
301.385106974224186,44.4963443903961604
dl.queryClient.drop[source]

Drop the specified table from the user’s MyDB

Usage:

drop (table=None, token=None)

queryClient.drop (token, table) queryClient.drop (table) queryClient.drop (token, table=<id>) queryClient.drop (table=<str>)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • table (str) – The specific table to drop

Example

# List the tables
queryClient.drop('foo1')
dl.queryClient.error[source]

Retrieve the error of an asynchronous query, once completed.

Usage:

error (token=None, jobId=None)

queryClient.error (jobId) queryClient.error (token, jobId) queryClient.error (token, jobId=<id>) queryClient.error (jobId=<str>)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • jobId (str) – The jobId returned when issuing an asynchronous query via queryClient.query() with async=True.

Returns

error

Return type

str

Example

# issue an async query (here a tiny one just for this example)
query = 'select ra,dec from gaia_dr1.gaia_source limit 3'
jobId = queryClient.query(token, adql=query, fmt='csv', async=True)

# ensure job completes...then check status and retrieve error
time.sleep(4)
if queryClient.status(token, jobId) == 'ERROR':
    error = queryClient.error(token,jobId)
    print type(error)
    print error

This prints

<type 'str'>
ra,dec
301.37502633933002,44.4946851014515588
301.371102372343785,44.4953207577355698
301.385106974224186,44.4963443903961604
dl.queryClient.getClient(profile='default', svc_url='https://datalab.noao.edu/query')[source]

Create a new queryClient object and set a default profile.

dl.queryClient.get_profile()[source]

Get the current query profile.

Parameters

None

Returns

profile – The name of the current profile used with the Query Manager service

Return type

str

Example

print ("Query Service profile = " + queryClient.get_profile())
dl.queryClient.get_svc_url()[source]

Get the currently-used Query Manager serice URL.

Parameters

None

Returns

service_url – The currently-used Query Service URL.

Return type

str

Example

print (queryClient.get_svc_url())
dl.queryClient.get_timeout_request()[source]

Get the current Sync query timeout value.

Parameters

None

Returns

result – Current sync query timeout value.

Return type

int

Example

# get the current timeout value
print (queryClient.get_timeout_request())
dl.queryClient.isAlive(svc_url='https://datalab.noao.edu/query', timeout=5)[source]
Check whether the QueryManager service at the given URL is

alive and responding. This is a simple call to the root service URL or ping() method.

Parameters
  • svc_url (str) – The Query Service URL to ping.

  • timeout (int) – Call will assume to have failed if ‘timeout’ seconds pass.

Returns

result – True if service responds properly, False otherwise

Return type

bool

Example

if queryClient.isAlive():
    print ("Query Manager is alive")
dl.queryClient.jobs[source]

Get a list of the user’s Async jobs.

Usage:

jobs (token=None, jobId=None, format=’text’, status=’all’)

queryClient.jobs (jobId) queryClient.jobs (token, jobId) queryClient.jobs (token, jobId=<id>) queryClient.jobs (jobId=<str>)

Use the authentication token and the jobId of a previously issued asynchronous query to check the query’s current status.

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • jobId (str) – The jobId returned when issuing an asynchronous query via queryClient.query() with async=True.

  • format (str) – Format of the result. Support values include ‘text’ for a simple formatted table suitable for printing, or ‘json’ for a JSON string of the full matching record(s).

  • status (str) –

    If status=’all’ then all async jobs are returned, otherwise this value may be used to return only those jobs with the specified status. Allowed values are:

    all Return all jobs EXECUTING Job is still running COMPLETED Job completed successfully ERROR Job exited with an error ABORTED Job was aborted by the user

  • option (str) – If ‘list’ then the matching records are returned, if ‘delete’ then the records are removed from the database (e.g. to clear up long job lists of completed jobs).

Returns

joblist – Returns a list of async query jobs submitted by the user in the last 30 days, possibly filtered by the ‘status’ parameter. The ‘json’ format option allows the caller to format the full contents of the job record beyond the supplied simple ‘text’ option.

Return type

str

Example

print (queryClient.jobs(token, jobId))

This prints

JobID              Start              End                 Status
tfu8zpn2tkrlfyr9e  07-22-20T13:10:22  07-22-20T13:34:12   COMPLETED
k8uznptrkkl29ryef  07-22-20T14:09:45                      EXECUTING
      :                   :                 :                :
dl.queryClient.list[source]

List the tables or table schema in the user’s MyDB.

Usage:

list (table=None, token=None)

queryClient.list (token, table) queryClient.list (table) queryClient.list (token, table=<id>) queryClient.list ()

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • table (str) – The specific table to list (returns the table schema), or an empty string to return a list of the names of all tables.

Returns

listing – The list of tables in the user’s MyDB or the schema of the named table

Return type

str

Example

# List the tables
queryClient.list()
dl.queryClient.list_profiles[source]

Retrieve the profiles supported by the query manager service.

Usage:

list_profiles (token=None, profile=None, format=’text’)

queryClient.list_profiles (token) queryClient.list_profiles ()

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • profile (str) – A specific profile configuration to list. If None, a list of profiles available to the given auth token is returned.

  • format (str) – Result format: One of ‘text’ or ‘json’

Returns

profiles – A list of the names of the supported profiles or a dictionary of the specific profile

Return type

list/dict

Example

profiles = queryClient.list_profiles()
profiles = queryClient.list_profiles(token)
dl.queryClient.mydb_copy[source]

Copy a table in the user’s MyDB to a new name

Usage:

mydb_copy (source, target, token=None)

queryClient.mydb_copy (token, source, target) queryClient.mydb_copy (source, target)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • source (str) – The old table name, i.e. the table to be copied

  • target (str) – The new table name, i.e. the table to be created

Example

# Copy table 'foo' to a new table, 'bar'
queryClient.mydb_copy ('foo', 'bar')
dl.queryClient.mydb_create[source]

Create a table in the user’s MyDB

Usage:

mydb_create (table, schema, token=None, **kw)

queryClient.mydb_create (token, table, <schema_dict>) queryClient.mydb_create (table, <schema_dict>)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • table (str) – The name of the table to create

  • schema (str or dict) – The schema is CSV text containing the name of the column and it’s PostgreSQL data type. If set as a ‘str’ type it is either a CSV string, or the name of a file containing the CSV. If passed as a ‘dict’ type, it is a dictionary object where keys are the column names and values are the data types.

  • drop (bool (optional)) – Drop any existing table of the same name before creating new one.

Example

# Create table in MyDB named 'foo' with columns defined by
# the file 'schema.txt'.  The schema file contains:
#
#     id,text
#     ra,double precision
#     dec,double precision
#
queryClient.mydb_create ('foo', 'schema.txt')
dl.queryClient.mydb_drop[source]

Drop the specified table from the user’s MyDB

Usage:

mydb_drop (table, token=None)

queryClient.mydb_drop (token, table) queryClient.mydb_drop (table) queryClient.mydb_drop (token, table=<id>)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • table (str) – The specific table to drop

Example

# Drop the 'foo1' table
queryClient.drop('foo1')
dl.queryClient.mydb_flush(token=None)[source]
dl.queryClient.mydb_import[source]

Import data into a table in the user’s MyDB

Usage:

mydb_import (table, data, token=None, **kw)

queryClient.mydb_import (token, table, data) queryClient.mydb_import (table, data)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • table (str) – The name of the table to be loaded

  • data (str or data object) –

    The data file or python object to be loaded. The ‘data’ value may be one of the following types:

    filename A CSV file of data string A string containing CSV data Pandas DataFrame A Pandas DataFrame object

    : : : :

    Additional object types can be added provided the data can be converted to a CSV format.

  • schema (str [OPTIONAL]) – If set, this is a filename or string containing a schema for the data table to be created. A schema contains a comma-delimited row for each column containing the column name and it’s Postgres data type. If not set, the schema is determined automatically from the data.

  • drop (bool [Optional]) – Drop any existing table of the same name before creating new one.

  • verbose (bool [Optional]) – Be verbose about operations.

Returns

  • schema A string containing the table schema

  • data_obj The CSV data to be imported (possibly converted)

Example

# Import data into a MyDB table named 'foo' from file 'data.csv'.
schema, data = queryClient.mydb_import ('foo', 'data.csv')
dl.queryClient.mydb_index[source]

Index the specified column in a table in the user’s MyDB

queryClient.mydb_index (table, colunm) queryClient.mydb_index (token, table, column) queryClient.mydb_index (table, column, token=None)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • table (str) – The table to be indexed

  • column (str) – The column

  • q3c (str) – A comma-delimited list of two column names giving the RA and Dec positions (decimal degrees) to be used to Q3C index the table. If None, no Q3C index will be computed.

  • cluster (bool) – If enabled, data table will be rewritten to cluster on the Q3C index for efficiency. Only used when ‘q3c’ columns are specified.

  • async_ (bool) – If enabled, index commands will be submitted asynchronously.

Returns

Return type

command status

Example

# Index the table's "id" column
queryClient.index('foo1', 'id')

# Index and cluster the table by position
queryClient.index('foo1', q3c='ra,dec', cluster=True)
dl.queryClient.mydb_insert[source]

Insert data into a table in the user’s MyDB

Usage:

mydb_insert (table, data, token=None, **kw)

queryClient.mydb_insert (token, table, <filename>) queryClient.mydb_insert (token, table, <data_object>) queryClient.mydb_insert (table, <filename>) queryClient.mydb_insert (table, <data_object>)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • table (str) – The name of the table to append

  • data (str or data object) – The schema is CSV text containing the name of the column and it’s PostgreSQL data type. If set as a ‘str’ type it is either a CSV string, or the name of a file containing the CSV data. If passed as a tabular data object, it is converted to CSV and sent to the service.

  • csv_header (bool [OPTIONAL]) – If True, then the CSV data object contains a CSV header line, i.e. the first line is a row of column names. Otherwise, no column names are assumed and the column order must match the table schema.

Example

# Insert data into a MyDB table named 'foo'.
queryClient.mydb_insert ('foo', 'data.csv')
dl.queryClient.mydb_list[source]

List the tables or table schema in the user’s MyDB.

Usage:

mydb_list (table=None, token=None)

queryClient.mydb_list (table) queryClient.mydb_list (token, table=<str>) queryClient.mydb_list (table=<str>)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • table (str) – The specific table to list (returns the table schema), or an empty string to return a list of the names of all tables.

Returns

listing – The list of tables in the user’s MyDB or the schema of the named table

Return type

str

Example

# List the tables
queryClient.mydb_list()
dl.queryClient.mydb_rename[source]

Rename a table in the user’s MyDB to a new name

Usage:

mydb_rename (source, target, token=None)

queryClient.mydb_rename (token, source, target) queryClient.mydb_rename (source, target)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • source (str) – The old table name

  • target (str) – The new table name

Example

# Copy table 'foo' to a new table, 'bar'
queryClient.mydb_rename ('foo', 'bar')
dl.queryClient.mydb_truncate[source]

Truncate the specified table in the user’s MyDB

Usage:

mydb_truncate (table, token=None)

queryClient.mydb_truncate (token, table) queryClient.mydb_truncate (table) queryClient.mydb_truncate (token, table=<id>)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • table (str) – The specific table to drop

Example

# Truncate the table 'foo'
queryClient.truncate('foo')
dl.queryClient.qcToString(s)[source]

qcToString – Force a return value to be type ‘string’ for all Python versions.

dl.queryClient.query[source]

Send an SQL or ADQL query to the database or TAP service.

Usage:
query (token=None, adql=None, sql=None, fmt=’csv’, out=None,

async_=False, drop=False, profile=’default’, **kw):

queryClient.query (token, query, <args>) queryClient.query (token | query, <args>)

Parameters
  • token (str) – Secure token obtained via authClient.login()

  • adql (str or None) –

    ADQL query string that will be passed to the DB query manager, e.g.

    adql='select top 3 ra,dec from gaia_dr1.gaia_source'
    

  • sql (str or None) –

    SQL query string that will be passed to the DB query manager, e.g.

    sql='select ra,dec from gaia_dr1.gaia_source limit 3'
    

  • fmt (str) –

    Format of result to be returned by the query. Permitted values are:

    • ’csv’ The returned result is a comma-separated string

      that looks like a csv file (newlines at the end of every row) [DEFAULT]

    • ’csv-noheader’ A csv result with no column headers (data only)

    • ’ascii’ Same, but the column separator is a tab

    • ’array’ Returns a NumPy array

    • ’pandas’ Returns a Pandas DataFrame

    • ’structarray’ Numpy structured array (aka ‘record array’)

    • ’table’ Returns an Astropy Table object

      The following formats may be used when saving a file to virtual storage on the server:

    • ’fits’ FITS binary

    • ’hdf5’ HDF5 file (NOT YET IMPLEMENTED)

    • ’votable’ An XML-formatted VOTable

  • out (str or None) – The output filename to create on the local machine, the URI of a VOSpace or MyDB resource to create, or None if the result is to be returned directly to the caller.

  • async_ (bool) –

    If True, the query is Asynchronous, i.e. a job is submitted to the DB, and a jobID token is returned the caller. The jobID must be then used to check the query’s status and to retrieve the result (when the job status is COMPLETED) or the error message (when the job status is ERROR). Default is False, i.e. the task runs a Synchroneous query.

    async_ replaces the previous async parameter, because async was promoted to a keyword in Python 3.7. Users of Python versions prior to 3.7 can continue to use the async keyword.

  • drop (bool) – If True, then if the query is saving to mydb where the same table name already exists, it will overwrite the old mydb table.

  • profile (str or None) – The Query Manager profile to use for this call. If None then the default profile is used. Available profiles may be listed using the queryClient.list_profiles()

  • **kw (dict) –

    Optional keyword arguments. Supported keywords currently include:

    wait = False

    Wait for asynchronous queries to complete? If enabled, the query() method will submit the job in async mode and then poll for results internally before returning. the default is to return the job ID immediately and let the client poll for job status and return results.

    timeout = 300

    Requested timeout (in seconds) for a query. For a Sync query, this value sets a session timeout request in the database that will abort the query at the specified time. A maximum value of 600 seconds is permitted. If the wait option is enabled for an ASync query, this is the maximum time the query will be allowed to run before an abort() is issued on the job. The maximum timeout for an ASync job is 24-hrs (86400 sec).

    poll = 1

    ASync job polling time in seconds.

    verbose = False

    Print verbose messages during ASync job.

Returns

result – If async=False, the return value is the result of the query as a formatted string (see fmt). Otherwise the result string is a job token, with which later the Asynchroneaous query’s status can be checked (queryClient.status()), and the result retrieved (see queryClient.result().

Return type

str

Example

Get security token first, see authClient.login(). Then:

query = 'select ra,dec from gaia_dr1.gaia_source limit 3'
response = queryClient.query(token, adql=query, fmt='csv')
print response

This prints

ra,dec
315.002571989537842,35.2662974820284489
315.00408275885701,35.2665448169895797
314.996334457679438,35.2673478725552698
class dl.queryClient.queryClient(profile='default', svc_url='https://datalab.noao.edu/query')[source]

Bases: object

QUERYCLIENT – Client-side methods to access the Data Lab

Query Manager Service.

abort(**kw)[source]

Call the appropriate instance of the function.

chunked_upload(token, local_file, remote_file)[source]

A streaming file uploader.

conequery(token, input=None, out=None, schema=None, table=None, ra=None, dec=None, search=0.5)[source]

Send a cone search query to the consearch service

dataType(val, current_type)[source]

Lexically scan a value to determine the datatype.

drop(**kw)[source]

Call the appropriate instance of the function.

error(**kw)[source]

Call the appropriate instance of the function.

getFromURL(svc_url, path, token)[source]

Get something from a URL. Return a ‘response’ object

getHeaders(token)[source]

Get default tracking headers,

getSchema(data, **kw)[source]

Generate a schema for mydb_create() from a CSV file or data object.

getStreamURL(url, headers, fname=None, chunk_size=1048576)[source]

Get the specified URL in a streaming fashion. This allows for large downloads without hitting timeout limits.

get_profile()[source]

Get the current query profile.

Parameters

None

Returns

profile – The name of the current profile used with the Query Manager service

Return type

str

Example

print ("Query Service profile = " + queryClient.get_profile())
get_svc_url()[source]

Get the currently-used Query Manager serice URL.

Parameters

None

Returns

service_url – The currently-used Query Service URL.

Return type

str

Example

print (queryClient.get_svc_url())
get_timeout_request()[source]

Get the current Sync query timeout value.

Parameters

None

Returns

result – Current sync query timeout value.

Return type

int

Example

# get the current timeout value
print (queryClient.get_timeout_request())
isAlive(svc_url=None, timeout=5)[source]
Check whether the QueryManager service at the given URL is

alive and responding. This is a simple call to the root service URL or ping() method.

Parameters
  • svc_url (str) – The Query Service URL to ping.

  • timeout (int) – Call will assume to have failed if ‘timeout’ seconds pass.

Returns

result – True if service responds properly, False otherwise

Return type

bool

Example

if queryClient.isAlive():
    print ("Query Manager is alive")
jobs(**kw)[source]

Call the appropriate instance of the function.

list(**kw)[source]

Call the appropriate instance of the function.

list_profiles(**kw)[source]

Call the appropriate instance of the function.

mydb_copy(**kw)[source]

Call the appropriate instance of the function.

mydb_create(**kw)[source]

Call the appropriate instance of the function.

mydb_drop(**kw)[source]

Call the appropriate instance of the function.

mydb_import(**kw)[source]

Call the appropriate instance of the function.

mydb_index(**kw)[source]

Call the appropriate instance of the function.

mydb_insert(**kw)[source]

Call the appropriate instance of the function.

mydb_list(**kw)[source]

Call the appropriate instance of the function.

mydb_rename(**kw)[source]

Call the appropriate instance of the function.

mydb_truncate(**kw)[source]

Call the appropriate instance of the function.

static pretty_print_POST(req)[source]

At this point it is completely built and ready to be fired; it is “prepared”.

However pay attention at the formatting used in this function because it is programmed to be pretty printed and may differ from the actual request.

query(**kw)[source]

Call the appropriate instance of the function.

results(**kw)[source]

Call the appropriate instance of the function.

schema(**kw)[source]

Call the appropriate instance of the function.

services(name=None, svc_type=None, mode='list', profile='default')[source]

Usage: queryClient.services ()

set_profile(profile)[source]

Set the service profile to be used.

Parameters

profile (str) – The name of the profile to use. The list of available profiles can be retrieved from the service (see function func:queryClient.list_profiles())

Returns

Return type

Nothing

Example

queryClient.set_profile('test')
set_svc_url(svc_url)[source]

Set the Query Manager service URL.

Parameters

svc_url (str) – The service URL of the Query Manager to call.

Returns

Return type

Nothing

Example

queryClient.set_svc_url ("http://localhost:7002")
set_timeout_request(nsec)[source]

Set the requested Sync query timeout value (in seconds).

Parameters

nsec (int) – The number of seconds requested before a sync query timeout occurs. The service may cap this as a server defined maximum.

Returns

Return type

Nothing

Example

# set the sync query timeout request to 30 seconds
queryClient.set_timeout_request(30)
siaquery(token, input=None, out=None, search=0.5)[source]

Send a SIA (Simple Image Access) query to the query manager service

status(**kw)[source]

Call the appropriate instance of the function.

wait(**kw)[source]

Call the appropriate instance of the function.

exception dl.queryClient.queryClientError(message)[source]

Bases: Exception

dl.queryClient.results[source]

Retrieve the results of an asynchronous query, once completed.

Usage:

results (token=None, jobId=None, delete=True)

queryClient.results (jobId) queryClient.results (token, jobId) queryClient.results (token, jobId=<id>) queryClient.results (jobId=<str>)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • jobId (str) – The jobId returned when issuing an asynchronous query via queryClient.query() with async=True.

Returns

results

Return type

str

Example

# issue an async query (here a tiny one just for this example)
query = 'select ra,dec from gaia_dr1.gaia_source limit 3'
jobId = queryClient.query(token, adql=query, fmt='csv', async=True)

# ensure job completes...then check status and retrieve results
time.sleep(4)
if queryClient.status(token, jobId) == 'COMPLETED':
    results = queryClient.results(token,jobId)
    print type(results)
    print results

This prints

<type 'str'>
ra,dec
301.37502633933002,44.4946851014515588
301.371102372343785,44.4953207577355698
301.385106974224186,44.4963443903961604
dl.queryClient.schema[source]

Return information about a data service schema.

Usage:

schema (value=’’, format=’text’, profile=None)

Parameters
  • value (str) – Schema object to return: Of the form <schema>[.<table>[.<column]]

  • profile (str) – The name of the service profile to use. The list of available profiles can be retrieved from the service (see function queryClient.list_profiles())

  • format (str) – Result format: One of ‘text’ or ‘json’ (NOT CURRENTLY USED)

Example

# List the available schema
queryClient.schema("", "text", "default")

# List the tables in the USNO schema
queryClient.schema("usno", "text", "default")

# List the columns of the USNO-A2 table
queryClient.schema("usno.a2", "text", "default")

# List the attributes of the USNO-A2 'raj2000' column
queryClient.schema("usno.a2.raj2000", "text", "default")
dl.queryClient.services(name=None, svc_type=None, mode='list', profile='default')[source]

Usage: queryClient.services ()

dl.queryClient.set_profile(profile)[source]

Set the service profile to be used.

Parameters

profile (str) – The name of the profile to use. The list of available profiles can be retrieved from the service (see function func:queryClient.list_profiles())

Returns

Return type

Nothing

Example

queryClient.set_profile('test')
dl.queryClient.set_svc_url(svc_url)[source]

Set the Query Manager service URL.

Parameters

svc_url (str) – The service URL of the Query Manager to call.

Returns

Return type

Nothing

Example

queryClient.set_svc_url ("http://localhost:7002")
dl.queryClient.set_timeout_request(nsec)[source]

Set the requested Sync query timeout value (in seconds).

Parameters

nsec (int) – The number of seconds requested before a sync query timeout occurs. The service may cap this as a server defined maximum.

Returns

Return type

Nothing

Example

# set the sync query timeout request to 30 seconds
queryClient.set_timeout_request(30)
dl.queryClient.status[source]

Get the status of an asynchronous query.

Usage:

status (token=None, jobId=None)

queryClient.status (jobId) queryClient.status (token, jobId) queryClient.status (token, jobId=<id>) queryClient.status (jobId=<str>)

Use the authentication token and the jobId of a previously issued asynchronous query to check the query’s current status.

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • jobId (str) – The jobId returned when issuing an asynchronous query via queryClient.query() with async=True.

Returns

status – Either ‘QUEUED’ or ‘EXECUTING’ or ‘COMPLETED’. If the token & jobId combination does not correspond to an actual job, then a HTML-formatted error message is returned. If there is a problem with the backend, the returned value can be ‘ERROR’.

When status is ‘COMPLETED’, you can retrieve the results of the query via queryClient.results()

Return type

str

Example

import time
query = 'select ra,dec from gaia_dr1.gaia_source limit 200000'
jobId = queryClient.query(token, adql=query, fmt='csv', async=True)
while True:
    status = queryClient.status(token, jobId)
    print "time index =", time.localtime()[5], "   status =", status
    if status == 'COMPLETED':
        break
    time.sleep(1)

This prints

time index = 16    status = EXECUTING
time index = 17    status = EXECUTING
time index = 18    status = COMPLETED
dl.queryClient.wait[source]

queryClient.wait (jobID=<str>)

Type

Usage

dl.resClient module

Client methods for the Data Lab Resource Management Service.

createUser (username, password, email, name, institute)

getUser (token, username, keyword) setUser (token, username, keyword, value)

deleteUser (token, username)

passwordReset (token, user, password)

sendPasswordLink (token, user)

listFields ()

createGroup (token, group)

getGroup (token, group, keyword) setGroup (token, group, keyword, value)

deleteGroup (token, group)

createResource (token, resource)

getResource (token, resource, keyword) setResource (token, resource, keyword, value)

deleteResource (token, resource)

createJob (token, jobid, job_type, query=None, task=None)

getJob (token, jobid, keyword) setJob (token, jobid, keyword, value)

deleteJob (token, jobid)
findJobs (token, jobid, format=’text’, status=’all’,

option=’list’)

set_svc_url (svc_url) get_svc_url () set_profile (profile) get_profile ()

list_profiles (token, profile=None, format=’text’)

Import via

from dl import resClient
dl.resClient.createGroup(token, group, profile='default')[source]

Create a new Group in the system.

Parameters
  • token (str) – User identity token

  • group (str) – Group name to create

Returns

resp – Service response

Return type

str

dl.resClient.createJob(token, jobid, job_type, query=None, task=None, profile='default')[source]
dl.resClient.createResource(token, resource, profile='default')[source]

Create a new Resource in the system.

Parameters
  • token (str) – User identity token

  • resource (str) – Resource URI to create

Returns

resp – Service response

Return type

str

dl.resClient.createUser(username, password, email, name, institute, profile='default')[source]

Create a new user in the system.

Parameters
  • username (str) – Account username

  • password (str) – Account password

  • email (str) – User’s contact email address

  • name (str) – User’s full name

  • institute (str) – User’s home institution

Returns

Return type

Service response

dl.resClient.deleteGroup(token, group, profile='default')[source]

Delete a Group in the system.

Parameters
  • token (str) – User identity token.

  • group (str) – Name of Group to be deleted. The token must identify the owner of the Group to be deleted.

dl.resClient.deleteJob(token, jobid, profile='default')[source]
dl.resClient.deleteResource(token, resource, profile='default')[source]

Delete a Resource in the system.

Parameters
  • token (str) – User identity token. The token must identify the owner of the Resource to be deleted.

  • resource (str) – Name of Resource to be deleted. The token must identify the owner of the Group to be deleted.

dl.resClient.deleteUser(token, username, profile='default')[source]

Delete a user in the system.

Parameters
  • token (str) – User identity token.

  • username (str) – Name of user to be deleted.

Returns

Return type

Service response

exception dl.resClient.dlResError(message)[source]

Bases: Exception

A throwable error class.

dl.resClient.findJobs(token, jobid, format='text', status='all', option='list')[source]
dl.resClient.getClient()[source]
dl.resClient.getGroup(token, group, keyword, profile='default')[source]

Read info about a Group in the system.

Parameters
  • token (str) – User identity token

  • group (str) – Group name to create

  • keyword (str) – Group record field to retrieve

Returns

value – Value of record field

Return type

str

dl.resClient.getJob(token, jobid, keyword, profile='default')[source]
dl.resClient.getResource(token, resource, keyword, profile='default')[source]

Read info about a Resource in the system.

Parameters
  • token (str) – User identity token

  • resource (str) – Resource URI to create

  • keyword (str) – Resource record to retrieve

Returns

value – Resource record value

Return type

str

dl.resClient.getUser(token, username, keyword, profile='default')[source]

Read info about a user in the system.

Parameters
  • username (str) – User name

  • keyword (str) – User record field to be set

Returns

Return type

Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")
dl.resClient.get_profile()[source]

Get the requested service profile.

Parameters

None

Returns

profile – The currently requested service profile.

Return type

str

Example

from dl import resClient
profile = resClient.client.get_profile()
dl.resClient.get_svc_url()[source]

Return the currently-used Resource Management Service URL.

Parameters

None

Returns

service_url – The currently-used Resource Management Service URL.

Return type

str

Example

from dl import resClient
service_url = resClient.client.get_svc_url()
dl.resClient.listFields(profile='default')[source]

List available user fields.

Parameters

None

Returns

Return type

Service response

dl.resClient.list_profiles(token, profile=None, format='text')[source]

List the service profiles which can be accessed by the user.

Returns

profiles

Return type

JSON string

Example

from dl import resClient
profiles = resClient.client.list_profiles(token, profile, format)
dl.resClient.passwordReset(token, user, password, profile='default')[source]

Change a user’s password.

Parameters
  • token (str) – User identity token

  • user (str) – User account name. Token must match the user or root token.

  • password (str) – New password

Returns

Return type

Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")
class dl.resClient.resClient[source]

Bases: object

RESCLIENT – Client-side methods to access the Data Lab

Resource Management Service.

approveUser(token, user, profile='default')[source]

Approve a pending user request.

Parameters
  • token (str) – User identity token

  • user (str) – User account to approve. Token must have authority to manage users.

Returns

Return type

Service response

clientDelete(token, what, query_args)[source]

Generic method to call a /delete service.

clientRead(token, what, key, keyword, profile='default')[source]

Generic method to call a /get service.

clientUpdate(token, what, key, keyword, value, profile='default')[source]

Generic method to call a /set service.

createGroup(token, group, profile='default')[source]

Create a new Group in the system.

Parameters
  • token (str) – User identity token

  • group (str) – Group name to create

Returns

resp – Service response

Return type

str

createJob(token, jobid, job_type, query=None, task=None, profile='default')[source]

Create a new Job record in the system.

Parameters
  • token (str) – User identity token

  • jobid (str) – Job ID to create

  • type (str) – Type of job: currently only ‘query’ or ‘compute’

  • query (str) – If ‘type’ is ‘query’, the SQL/ADQL query string

  • task (str) – If ‘type’ is ‘compute’, the name of the task being run

Returns

resp – Service response

Return type

str

createResource(token, resource, profile='default')[source]

Create a new Resource in the system.

Parameters
  • token (str) – User identity token

  • resource (str) – Resource URI to create

Returns

resp – Service response

Return type

str

createUser(username, password, email, name, institute, profile='default')[source]

Create a new user in the system.

Parameters
  • username (str) – Account username

  • password (str) – Account password

  • email (str) – User’s contact email address

  • name (str) – User’s full name

  • institute (str) – User’s home institution

Returns

Return type

Service response

debug(debug_val)[source]

Set the debug flag.

deleteGroup(token, group, profile='default')[source]

Delete a Group in the system.

Parameters
  • token (str) – User identity token.

  • group (str) – Name of Group to be deleted. The token must identify the owner of the Group to be deleted.

deleteJob(token, jobid, profile='default')[source]

Delete a Job in the system.

Parameters
  • token (str) – User identity token. The token must identify the owner of the Job to be deleted.

  • jobid (str) – ID of Job to be deleted. The token must identify the owner of the Job to be deleted.

deleteResource(token, resource, profile='default')[source]

Delete a Resource in the system.

Parameters
  • token (str) – User identity token. The token must identify the owner of the Resource to be deleted.

  • resource (str) – Name of Resource to be deleted. The token must identify the owner of the Group to be deleted.

deleteUser(token, username, profile='default')[source]

Delete a user in the system.

Parameters
  • token (str) – User identity token.

  • username (str) – Name of user to be deleted.

Returns

Return type

Service response

disapproveUser(token, user, profile='default')[source]

Disapprove a pending user request.

Parameters
  • token (str) – User identity token

  • user (str) – User account to decline. Token must have authority to manage users.

Returns

Return type

Service response

findJobs(token, jobid, format='text', status='all', option='list')[source]
Find job records. If jobid is None or ‘*’, all records for the user

identified by the token are returned, otherwise the specific job record is returned.

Parameters
  • token (str) – User identity token. The token must identify the owner of the Job to be deleted.

  • jobid (str) – Job ID to match.

  • format (str) – Output format: ‘text’ or ‘json’

  • status (str) – Job phase status to match. Default to ‘all’ but may be one of EXECUTING, COMPLETED, ERROR, ABORT or WAITING.

  • option (str) – Processing option: ‘list’ will return a listing of the matching records in the format specified by ‘format’; ‘delete’ will delete all matching records from the server except for EXECUTING jobs.

Returns

Return type

A JSON string of Job records matching the user or jobid.

getGroup(token, group, keyword, profile='default')[source]

Read info about a Group in the system.

Parameters
  • token (str) – User identity token

  • group (str) – Group name to create

  • keyword (str) – Group record field to retrieve

Returns

value – Value of record field

Return type

str

getJob(token, jobid, keyword, profile='default')[source]

Read info about a Job in the system.

Parameters
  • token (str) – User identity token

  • jobid (str) – Job ID to create

  • keyword (str) – Job record to retrieve

Returns

value – Job record value

Return type

str

getResource(token, resource, keyword, profile='default')[source]

Read info about a Resource in the system.

Parameters
  • token (str) – User identity token

  • resource (str) – Resource URI to create

  • keyword (str) – Resource record to retrieve

Returns

value – Resource record value

Return type

str

getUser(username, keyword, profile='default')[source]

Read info about a user in the system.

Parameters
  • username (str) – User name

  • keyword (str) – User record field to be set

Returns

Return type

Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")
get_profile()[source]

Get the requested service profile.

Parameters

None

Returns

profile – The currently requested service profile.

Return type

str

Example

from dl import resClient
profile = resClient.client.get_profile()
get_svc_url()[source]

Return the currently-used Resource Management Service URL.

Parameters

None

Returns

service_url – The currently-used Resource Management Service URL.

Return type

str

Example

from dl import resClient
service_url = resClient.client.get_svc_url()
isAlive(svc_url)[source]
Check whether the ResManager service at the given URL is

alive and responding. This is a simple call to the root service URL or ping() method.

Parameters

svc_url (str) – Resource Management service base URL to call.

Returns

Return type

Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")
listFields(profile='default')[source]

List available user fields.

Parameters

None

Returns

Return type

Service response

listPending(token, verbose=False, profile='default')[source]

List all pending user accounts.

Parameters
  • token (str) – User identity token

  • verbose (bool) – Return verbose listing?

Returns

Return type

List of use accounts pending approval

list_profiles(token, profile=None, format='text')[source]

List the service profiles which can be accessed by the user.

Returns

profiles

Return type

JSON string

Example

from dl import resClient
profiles = resClient.client.list_profiles(token, profile, format)
passwordReset(token, user, password, profile='default')[source]

Change a user’s password.

Parameters
  • token (str) – User identity token

  • user (str) – User account name. Token must match the user or root token.

  • password (str) – New password

Returns

Return type

Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")
retBoolValue(url)[source]

Utility method to call a boolean service at the given URL.

Send a password-reset link to the user.

Parameters
  • token (str) – User identity token

  • user (str) – User account name.

Returns

Return type

Service response

setField(token, user, field, value, profile='default')[source]

Set a specific user record field

Parameters
  • token (str) – User identity token

  • user (str) – User name to modify. If None then identity is take from token, a root token is required to modify other users.

  • field (str) – Record field to be set

  • value (str) – Field value

Returns

Return type

‘OK’ is field was set, else a service error message.

setGroup(token, group, keyword, value, profile='default')[source]

Update info about a Group in the system.

Parameters
  • token (str) – User identity token

  • group (str) – Group name to create

  • keyword (str) – Group record field to be set

  • value (str) – Value of field to set

setJob(token, jobid, keyword, value, profile='default')[source]

Update info about a Job in the system.

Parameters
  • token (str) – User identity token

  • jobid (str) – Job ID to create

  • keyword (str) – Job record field to be set

  • value (str) – Value of field to set

Returns

status – ‘OK’ if record was set

Return type

str

setResource(token, resource, keyword, value, profile='default')[source]

Update info about a Resource in the system.

Parameters
  • token (str) – User identity token

  • resource (str) – Resource URI to create

  • keyword (str) – Resource record field to be set

  • value (str) – Value of field to set

Returns

status – ‘OK’ if record was set

Return type

str

setUser(username, keyword, value, profile='default')[source]

Update info about a user in the system.

Parameters
  • username (str) – User name

  • keyword (str) – User record field to be set

  • value (str) – Value of field to set

Returns

Return type

Service response

set_profile(profile)[source]

Set the requested service profile.

Parameters

profile (str) – Requested service profile string.

Returns

Return type

Nothing

Example

from dl import resClient
token = resClient.client.set_profile("dev")
set_svc_url(svc_url)[source]

Set the URL of the Resource Management Service to be used.

Parameters

svc_url (str) – Resource Management service base URL to call.

Returns

Return type

Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")
svcGet(token, url)[source]

Utility method to call a Resource Manager service.

Parameters
  • token (str) – User identity token

  • url (str) – URL to call with HTTP/GET

Returns

Return type

Service response

userRecord(token, user, value, format, profile='default')[source]

Get a value from the User record.

Parameters
  • token (str) – User identity token

  • user (str) – User account name to retrieve. Token must match the use name or be a root token to access other records

  • value (str) – Value to retrieve. The special ‘all’ value will return all fields accessible to the token.

  • format (str) – ‘text’ for a single value, or ‘json’ for a complete record

Returns

Return type

User record

Send a password-reset link to the user.

Parameters
  • token (str) – User identity token

  • user (str) – User account name.

Returns

Return type

Service response

dl.resClient.setGroup(token, group, keyword, value, profile='default')[source]

Update info about a Group in the system.

Parameters
  • token (str) – User identity token

  • group (str) – Group name to create

  • keyword (str) – Group record field to be set

  • value (str) – Value of field to set

dl.resClient.setJob(token, jobid, keyword, value, profile='default')[source]
dl.resClient.setResource(token, resource, keyword, value, profile='default')[source]

Update info about a Resource in the system.

Parameters
  • token (str) – User identity token

  • resource (str) – Resource URI to create

  • keyword (str) – Resource record field to be set

  • value (str) – Value of field to set

Returns

status – ‘OK’ if record was set

Return type

str

dl.resClient.setUser(token, username, keyword, value, profile='default')[source]

Update info about a user in the system.

Parameters
  • username (str) – User name

  • keyword (str) – User record field to be set

  • value (str) – Value of field to set

Returns

Return type

Service response

dl.resClient.set_profile(profile)[source]

Set the requested service profile.

Parameters

profile (str) – Requested service profile string.

Returns

Return type

Nothing

Example

from dl import resClient
token = resClient.client.set_profile("dev")
dl.resClient.set_svc_url(svc_url)[source]

Set the URL of the Resource Management Service to be used.

Parameters

svc_url (str) – Resource Management service base URL to call.

Returns

Return type

Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")

dl.specClient module

dl.specClient.airtovac(l)[source]

Convert air wavelengths (greater than 2000A) to vacuum wavelengths.

dl.specClient.catalogs(context='default', profile='default', fmt='text')[source]

List available catalogs for a given dataset context

exception dl.specClient.dlSpecError(message)[source]

Bases: Exception

A throwable error class.

dl.specClient.getClient(context='default', profile='default')[source]

Get a new instance of the specClient client.

Parameters
  • context (str) – Dataset context

  • profile (str) – Service profile

Returns

client – An specClient object

Return type

specClient

Example

dl.specClient.getSpec(id_list, fmt='numpy', out=None, align=False, cutout=None, context=None, profile=None, **kw)[source]

Get spectra for a list of object IDs.

Usage:
getSpec(id_list, fmt=’numpy’, out=None, align=False, cutout=None,

context=None, profile=None, **kw)

Parameters
  • id_list (list object) – List of object identifiers.

  • fmt (str) – Return format of spectra

  • out – Output file or return to caller if None

  • align – Align spectra to common wavelength grid with zero-padding

  • cutout – Wavelength cutout range (as “<start>-<end>”)

  • context (str) – Dataset context.

  • profile (str) – Data service profile.

  • **kw (dict) –

    Optional keyword arguments. Supported keywords currently include:

    values = None

    Spectrum vectors to return.

    token = None

    Data Lab auth token.

    id_col = None

    Name of ID column in input table.

    verbose = False

    Print verbose messages during retrieval

    debug = False

    Print debug messages during retrieval

Returns

result

Return type

object or array of objects or ‘OK’ string

Example

  1. Retrieve spectra individually:

  1. Retrieve spectra in bulk:

dl.specClient.get_context()[source]

Get the requested dataset context.

Parameters

None

Returns

context – The currently requested dataset context.

Return type

str

Example

from dl import specClient
context = specClient.client.get_context()
dl.specClient.get_profile()[source]

Get the requested service profile.

Parameters

None

Returns

profile – The currently requested service profile.

Return type

str

Example

from dl import specClient
profile = specClient.client.get_profile()
dl.specClient.get_svc_url()[source]

Return the currently-used Spectroscopic Data Service URL.

Parameters

None

Returns

service_url – The currently-used Spectroscopic Data Service URL.

Return type

str

Example

from dl import specClient
service_url = specClient.get_svc_url()
dl.specClient.isAlive(svc_url='https://datalab.noao.edu/spec', timeout=5)[source]
dl.specClient.list_contexts[source]

Retrieve the contexts supported by the spectro data service.

Usage:

list_contexts ([context], fmt=’text’)

specClient.list_contexts (context) specClient.list_contexts ()

Parameters
  • contexts (str) – A specific contexts configuration to list. If None, a list of available contexts is returned.

  • format (str) – Result format: One of ‘text’ or ‘json’

Returns

contexts – A list of the names of the supported contexts or a dictionary of the specific contexts

Return type

list/dict

Example

contexts = specClient.list_contexts(context)
contexts = specClient.list_contexts()
dl.specClient.list_profiles[source]

Retrieve the profiles supported by the spectro data service.

Usage:

list_profiles ([profile], fmt=’text’)

specClient.list_profiles (profile) specClient.list_profiles ()

Parameters
  • profile (str) – A specific profile configuration to list. If None, a list of available profiles is returned.

  • format (str) – Result format: One of ‘text’ or ‘json’

Returns

profiles – A list of the names of the supported profiles or a dictionary of the specific profile

Return type

list/dict

Example

profiles = specClient.list_profiles(profile)
profiles = specClient.list_profiles()
dl.specClient.plot(spec, context=None, profile=None, out=None, **kw)[source]

Utility to batch plot a single spectrum.

Usage:

spec.plot(id, context=None, profile=None, **kw)

Parameters
  • spec (object ID or data array) – Spectrum to be plotted. If ‘spec’ is a numpy array or a Spectrum1D object the data are plotted directly, otherwise the value is assumed to be an object ID that will be retrieved from the service.

  • out (str) – Output filename. If specified, plot saved as PNG.

  • context (str) – Dataset context.

  • profile (str) – Data service profile.

  • **kw (dict) –

    Optional keyword arguments. Supported keywords currently include:

    rest_frame - Whether or not to plot the spectra in the

    rest-frame (def: True)

    z - Redshift value

    xlim - Set the xrange of the plot ylim - Set the yrange of the plot

    values - A comma-delimited string of which values to plot,

    a combination of ‘flux,model,sky,ivar’

    mark_lines - Which lines to mark. No lines marked if None or

    an empty string, otherwise one of ‘em|abs|all|both’

    grid - Plot grid lines (def: True) dark - Dark-mode plot colors (def: True)

    em_lines - List of emission lines to plot. If not given,

    all the lines in the default list will be plotted.

    abs_lines - Lines of absorption lines to plot. If not given,

    all the lines in the default list will be plotted.

    spec_args - Plotting kwargs for the spectrum

    model_args - Plotting kwargs for the model
    ivar_args - Plotting kwargs for the ivar

    sky_args - Plotting kwargs for the sky

Returns

Return type

Nothing

Example

  1. Plot a single spectrum, save to a virtual storage file

dl.specClient.plotGrid(id_list, nx, ny, page=0, context=None, profile=None, **kw)[source]

Get a grid of preview plots of a spectrum list.

Usage:
image = plotGrid(id_list, nx, ny, page=0,

context=None, profile=None, **kw):

Parameters
  • id_list (list object) – List of object identifiers.

  • nx (int) – Number of plots in the X dimension

  • ny (int) – Number of plots in the Y dimension

  • page (int) – Dataset context.

  • context (str) – Dataset context.

  • profile (str) – Data service profile.

  • **kw (dict) –

    Optional keyword arguments. Supported keywords currently include:

    verbose = False

    Print verbose messages during retrieval

    debug = False

    Print debug messages during retrieval

Returns

image

Return type

A PNG image object

Example

  1. Display a 5x5 grid of preview plots for a list:

dl.specClient.preview(spec, context=None, profile=None, **kw)[source]

Get a preview plot of a spectrum

Usage:

spec.preview(spec, context=None, profile=None, **kw):

Parameters
  • id_list (list object) – List of object identifiers.

  • context (str) – Dataset context.

  • profile (str) – Data service profile.

  • **kw (dict) –

    Optional keyword arguments. Supported keywords currently include:

    N/A

Returns

image

Return type

A PNG image object

Example

  1. Display a preview plot a given spectrum:

dl.specClient.prospect(spec, context=None, profile=None, **kw)[source]

Utility wrapper to launch the interactive PROSPECT tool.

Usage:

stat = prospect(spec, context=None, profile=None, **kw)

Parameters
  • spec (object ID or data array) – Spectrum to be plotted. If ‘spec’ is a numpy array or a Spectrum1D object the data are plotted directly, otherwise the value is assumed to be an object ID that will be retrieved from the service.

  • context (str) – Dataset context.

  • profile (str) – Data service profile.

  • **kw (dict) –

    Optional keyword arguments. Supported keywords currently include:

    TBD

Returns

result – Status ‘OK’ string or error message.

Return type

str

Example

  1. Plot ….

dl.specClient.query[source]
Query for a list of spectrum IDs that can then be retrieved from

the service.

Usage:
id_list = query(ra, dec, size, constraint=None, out=None,

context=None, profile=None, **kw)

id_list = query(pos, size, constraint=None, out=None,

context=None, profile=None, **kw)

id_list = query(region, constraint=None, out=None,

context=None, profile=None, **kw)

id_list = query(constraint=None, out=None,

context=None, profile=None, **kw)

Parameters
  • ra (float) – RA search center specified in degrees.

  • dec (float) – Dec of search center specified in degrees.

  • size (float) – Size of search center specified in degrees.

  • pos (Astropy Coord object) – Coordinate of search center specified as an Astropy Coord object.

  • region (float) – Array of polygon vertices (in degrees) defining a search region.

  • constraint (str) – A valid SQL syntax that can be used as a WHERE constraint in the search query.

  • out (str) – Output filename to create. If None or an empty string the query results are returned directly to the client. Otherwise, results are writeen to the named file with one identifier per line. A Data Lab ‘vos://’ prefix will save results to the named virtual storage file.

  • context (str) – Dataset context.

  • profile (str) – Data service profile.

  • **kw (dict) –

    Optional keyword arguments. Supported keywords currently include:

    For context=’sdss_dr16’ | ‘default’:
    fields:

    specobjid # or ‘bestobjid’, etc tuple # a plate/mjd/fiber tuple

    Service will always return array of ‘specobjid’ value, the p/m/f tuple is extracted from the bitmask value by the client.

    catalog:
    <schema>.<table> # alternative catalog to query e.g. a

    # VAC from earlier DR (must support an # ra/dec search and return a specobjid- # like value)

    For all contexts:
    verbose = False

    Print verbose messages during retrieval

    debug = False

    Print debug messages during retrieval

Returns

result – An array of spectrum IDs appropriate for the dataset context.

Return type

array

Example

  1. Query by position:

dl.specClient.set_context(context)[source]

Set the requested dataset context.

Parameters

context (str) – Requested dataset context string.

Returns

Return type

Nothing

Example

from dl import specClient
context = specClient.client.set_context("dev")
dl.specClient.set_profile(profile)[source]

Set the requested service profile.

Parameters

profile (str) – Requested service profile string.

Returns

Return type

Nothing

Example

from dl import specClient
profile = specClient.client.set_profile("dev")
dl.specClient.set_svc_url(svc_url)[source]

Set the URL of the Spectroscopic Data Service to be used.

Parameters

svc_url (str) – Spectroscopic Data service base URL to call.

Returns

Return type

Nothing

Example

from dl import specClient
specClient.set_svc_url("http://localhost:7001/")
dl.specClient.spcToString(s)[source]

spcToString – Force a return value to be type ‘string’ for all Python versions.

class dl.specClient.specClient(context='default', profile='default')[source]

Bases: object

SPECCLIENT – Client-side methods to access the Data Lab

Spectroscopic Data Service.

catalogs(context='default', profile='default', fmt='text')[source]

Usage: specClient.client.catalogs (…)

curl_get(url)[source]

Utility routine to use cURL to return a URL

debug(debug_val)[source]

Toggle debug flag.

extractIDList(id_list, id_col=None)[source]

Extract a 1-D array or single identifier from an input ID type.

getFromURL(svc_url, path, token)[source]

Get something from a URL. Return a ‘response’ object.

getHeaders(token)[source]

Get default tracking headers.

getSpec(id_list, fmt='numpy', out=None, align=False, cutout=None, context=None, profile=None, **kw)[source]

Get spectra for a list of object IDs.

Usage:
getSpec(id_list, fmt=’numpy’, out=None, align=False, cutout=None,

context=None, profile=None, **kw)

Parameters
  • id_list (list object) – List of object identifiers.

  • format – Return format of spectra

  • out – Output filename or return to caller if None

  • align – Align spectra to common wavelength grid with zero-padding

  • cutout – Wavelength cutout range (as “<start>-<end>”)

  • context (str) – Dataset context.

  • profile (str) – Data service profile.

  • **kw (dict) –

    Optional keyword arguments. Supported keywords currently include:

    values = None

    Spectrum vectors to return.

    token = None

    Data Lab auth token.

    id_col = None

    Name of ID column in input table.

    verbose = False

    Print verbose messages during retrieval

    debug = False

    Print debug messages during retrieval

Returns

result

Return type

object or array of objects or ‘OK’ string

Example

  1. Retrieve spectra individually:

  1. Retrieve spectra in bulk:

get_context()[source]

Get the requested dataset context.

Parameters

None

Returns

context – The currently requested dataset context.

Return type

str

Example

from dl import specClient
context = specClient.client.get_context()
get_profile()[source]

Get the requested service profile.

Parameters

None

Returns

profile – The currently requested service profile.

Return type

str

Example

from dl import specClient
profile = specClient.client.get_profile()
get_svc_url()[source]

Return the currently-used Spectroscopic Data Service URL.

Parameters

None

Returns

service_url – The currently-used Spectroscopic Data Service URL.

Return type

str

Example

from dl import specClient
service_url = specClient.get_svc_url()
isAlive(svc_url='https://datalab.noao.edu/spec')[source]
Check whether the service at the given URL is alive and responding.

This is a simple call to the root service URL or ping() method.

Parameters

service_url (str) – The Query Service URL to ping.

Returns

result – True if service responds properly, False otherwise

Return type

bool

Example

from dl import specClient
if specClient.isAlive():
    print("Spec Server is alive")
list_contexts(**kw)[source]

Call the appropriate instance of the function.

list_profiles(**kw)[source]

Call the appropriate instance of the function.

plot(spec, context=None, profile=None, out=None, **kw)[source]

Utility to batch plot a single spectrum.

Usage:

spec.plot(id, context=None, profile=None, **kw)

Parameters
  • spec (object ID or data array) – Spectrum to be plotted. If ‘spec’ is a numpy array or a Spectrum1D object the data are plotted directly, otherwise the value is assumed to be an object ID that will be retrieved from the service.

  • out (str) – Output filename. If specified, plot saved as PNG.

  • context (str) – Dataset context.

  • profile (str) – Data service profile.

  • **kw (dict) –

    Optional keyword arguments. Supported keywords currently include:

    rest_frame - Whether or not to plot the spectra in the

    rest-frame. If True, the wavelengths are assumed to have been already shifted and no ‘z’ value is required. If False, wavelengths are assumed to be in observed frame and a ‘z’ value is required to overplot absorption/emission lines. (def: True)

    z - Redshift value (def: None)

    xlim - Set the xrange of the plot ylim - Set the yrange of the plot

    title - Plot title (def: object ID)

    xlabel - Plot x-axis label (def: wavelength) ylabel - Plot y-axis label (def: flux units)

    out - Saved output filename.

    values - A comma-delimited string of which values to plot,

    a combination of ‘flux,model,sky,ivar’

    mark_lines - Which lines to mark. No lines marked if None or

    an empty string, otherwise one of ‘em|abs|all|both’

    grid - Plot grid lines (def: True) dark - Dark-mode plot colors (def: True)

    em_lines - List of emission lines to plot. If not given,

    all the lines in the default list will be plotted.

    abs_lines - Lines of absorption lines to plot. If not given,

    all the lines in the default list will be plotted.

    spec_args - Plotting kwargs for the spectrum

    model_args - Plotting kwargs for the model
    ivar_args - Plotting kwargs for the ivar

    sky_args - Plotting kwargs for the sky

Returns

Return type

Nothing

Example

  1. Plot a single spectrum, save to a virtual storage file

plotGrid(id_list, nx, ny, page=0, context=None, profile=None, **kw)[source]

Get a grid of preview plots of a spectrum list.

Usage:
image = plotGrid(id_list, nx, ny, page=0,

context=None, profile=None, **kw):

Parameters
  • id_list (list object) – List of object identifiers.

  • nx (int) – Number of plots in the X dimension

  • ny (int) – Number of plots in the Y dimension

  • page (int) – Dataset context.

  • context (str) – Dataset context.

  • profile (str) – Data service profile.

  • **kw (dict) –

    Optional keyword arguments. Supported keywords currently include:

    verbose = False

    Print verbose messages during retrieval

    debug = False

    Print debug messages during retrieval

Returns

image

Return type

A PNG image object

Example

  1. Display a 5x5 grid of preview plots for a list:

preview(spec, context=None, profile=None, **kw)[source]

Get a preview plot of a spectrum

Usage:

spec.preview(spec, context=None, profile=None, **kw):

Parameters
  • spec (objectID) – Object identifiers.

  • context (str) – Dataset context.

  • profile (str) – Data service profile.

  • **kw (dict) –

    Optional keyword arguments. Supported keywords currently include:

    N/A

Returns

image

Return type

A PNG image object

Example

  1. Display a preview plot a given spectrum:

prospect(spec, context=None, profile=None, **kw)[source]

Utility wrapper to launch the interactive PROSPECT tool.

Usage:

stat = prospect(spec, context=None, profile=None, **kw)

Parameters
  • spec (object ID or data array) – Spectrum to be plotted. If ‘spec’ is a numpy array or a Spectrum1D object the data are plotted directly, otherwise the value is assumed to be an object ID that will be retrieved from the service.

  • context (str) – Dataset context.

  • profile (str) – Data service profile.

  • **kw (dict) –

    Optional keyword arguments. Supported keywords currently include:

    TBD

Returns

result – Status ‘OK’ string or error message.

Return type

str

Example

  1. Plot ….

query(**kw)[source]

Call the appropriate instance of the function.

retBoolValue(url)[source]

Utility method to call a boolean service at the given URL.

set_context(context)[source]

Set the requested dataset context.

Parameters

context (str) – Requested dataset context string.

Returns

Return type

Nothing

Example

from dl import specClient
context = specClient.client.set_context("dev")
set_profile(profile)[source]

Set the requested service profile.

Parameters

profile (str) – Requested service profile string.

Returns

Return type

Nothing

Example

from dl import specClient
profile = specClient.client.set_profile("dev")
set_svc_url(svc_url)[source]

Set the URL of the Spectroscopic Data Service to be used.

Parameters

svc_url (str) – Spectroscopic Data service base URL to call.

Returns

Return type

Nothing

Example

from dl import specClient
specClient.set_svc_url("http://localhost:7001/")
stackedImage(id_list, align=False, yflip=False, context=None, profile=None, **kw)[source]

Get …

Usage:

Parameters
  • id_list (list object) – List of spectrum identifiers.

  • context (str) – Dataset context.

  • profile (str) – Data service profile.

  • **kw (dict) –

    Optional keyword arguments. Supported keywords currently include:

    verbose = False

    Print verbose messages during retrieval

    debug = False

    Print debug messages during retrieval

Returns

result

Return type

Example

  1. Query ….

to_Spectrum1D(npy_data)[source]

Convert a Numpy spectrum array to a Spectrum1D object.

to_Table(npy_data)[source]

Utility method to convert a Numpy array to an Astropy Table object.

to_pandas(npy_data)[source]

Utility method to convert a Numpy array to a Pandas DataFrame

dl.specClient.stackedImage(id_list, align=False, yflip=False, context=None, profile=None, **kw)[source]

Get …

Usage:

Parameters
  • id_list (list object) – List of spectrum identifiers.

  • context (str) – Dataset context.

  • profile (str) – Data service profile.

  • **kw (dict) –

    Optional keyword arguments. Supported keywords currently include:

    verbose = False

    Print verbose messages during retrieval

    debug = False

    Print debug messages during retrieval

Returns

result

Return type

Example

  1. Query ….

dl.specClient.to_Spectrum1D(npy_data)[source]

Utility method to convert a Numpy array to Spectrum1D

dl.specClient.to_Table(npy_data)[source]

Utility method to convert a Numpy array to an Astropy Table object.

dl.specClient.to_pandas(npy_data)[source]

Utility method to convert a Numpy array to a Pandas DataFrame

dl.storeClient module

dl.storeClient.access[source]

Determine whether the file can be accessed with the given node.

Usage:

access(path, mode=None, token=None, verbose=True)

storeClient.access(token, path, mode) storeClient.access(path, mode) storeClient.access(path)

Parameters
  • path (str) – A name or file template of the file status to retrieve.

  • mode (str) – Requested access mode. Modes are ‘r’ (read access), ‘w’ (write access), or ‘rw’ to test for both read/write access. If mode is None a simple existence check is made.

  • token (str) – Authentication token (see function authClient.login())

  • verbose (bool) – Verbose output flag.

Returns

result – True if the node can be access with the requested mode.

Return type

bool

Example

if storeClient.access('/mydata.csv')
    print('File exists')
elif storeClient.access('/mydata.csv','rw')
    print('File is both readable and writable')
dl.storeClient.chunked_upload(token, local_file, remote_file)[source]

A streaming file uploader.

dl.storeClient.cp[source]

Copy a file/directory within the store manager service

Usage:

cp(token=None, fr=’’, to=’’, verbose=False)

storeClient.cp(token, fr, to) storeClient.cp(fr, to) storeClient.cp(fr) storeClient.cp(fr=’’,to=’’)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • fr (str) – Name of the file to be copied (may not be a directory).

  • to (str) – Name of the file to be created

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Copy a file in vospace
storeClient.cp('foo', 'bar')
storeClient.cp('vos://foo', 'vos:///new/bar')
dl.storeClient.expandFileList(svc_url, token, pattern, format, full=False)[source]

Expand a filename pattern in a VOSpace URI to a list of files. We do this by getting a listing of the parent container contents from the service and then match the pattern on the client side.

dl.storeClient.get[source]

Retrieve a file from the store manager service

Usage:
get(token=None, fr=’’, to=’’, mode=’text’, verbose=True, debug=False,

timeout=30)

storeClient.get(token, fr, to) storeClient.get(fr, to) storeClient.get(fr) storeClient.get(token, fr, to)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • fr (str) – A name or file template of the file(s) to retrieve.

  • to (str) – Name of the file(s) to locally. If not specified, the contents of the file are returned to the caller.

  • mode ([binary | text | fileobj]) – Return data type if note saving to file. If set to ‘text’ the file contents are converted to string – this is appropriate when dealing with unicode but may fail with general binary data. If set to ‘binary’ the raw content of the HTTP response is returned – for Python 2 this will be a ‘string’, for Python 3 it will be a ‘bytes’ data type (the caller is responsible for conversion).

  • verbose (bool) – Print verbose output, e.g. progress indicators.

  • debug (bool) – Print debug output.

  • timeout (integer) – Retry timeout value. When processing long lists, download will pause every timeout files to lessen server load. For individual files, transfer will retry for timeout seconds before aborting. Failed transfers are automatically appended to the file list so they may be transferred again later.

Returns

result – A list of the names of the files retrieved, or the contents of a single file.

Return type

str

Example

# get a single file to a local file of a different name
data = storeClient.get('vos://mydata.csv', 'data.csv')

# get the contents of a single file to a local variable
data = storeClient.get('vos://mydata.csv')

# get a list of remote files to a local directory
flist = storeClient.get('vos://*.fits', './data/')
flist = storeClient.get('*.fits', './data/')
dl.storeClient.getClient(profile='default', svc_url='https://datalab.noao.edu/storage')[source]

Create a new storeClient object and set a default profile.

dl.storeClient.get_profile()[source]

Get the profile

Parameters

None

Returns

profile – The name of the current profile used with the Storage Manager

Return type

str

Example

print('Store Service profile = ' + storeClient.get_profile())
dl.storeClient.get_svc_url()[source]

Return the currently-used Storage Manager service URL.

Parameters

None

Returns

Return type

Current Storage Manager service URL

Example

print(storeClient.get_scv_url())
dl.storeClient.hasmeta(s)[source]

Determine whether a string contains filename meta-characters.

dl.storeClient.isAlive(svc_url='https://datalab.noao.edu/storage')[source]
Check whether the StorageManager service at the given URL is

alive and responding. This is a simple call to the root service URL or ping() method.

Parameters

svc_url (str) – The service URL of the storage manager to use

Example

# Check if service is responding
if storeClient.isAlive():
   .....stuff
dl.storeClient.is_vosDir(svc_url, token, path)[source]

Determine whether ‘path’ is a ContainerNode in the VOSpace.

dl.storeClient.list_profiles[source]

Retrieve the profiles supported by the storage manager service

Usage:

list_profiles(token=None, profile=None, format=’text’)

storeClient.list_profiles(token) # list default profile storeClient.list_profiles(profile) # list named profile storeClient.list_profiles() # list default profile

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • profile (str) – A specific profile to list

Returns

profiles – A list of the names of the supported profiles or a dictionary of the specific profile

Return type

list/dict

Example

# get the list of profiles
profiles = storeClient.list_profiles(token)
dl.storeClient.ln[source]

Create a link to a file/directory in the store manager service

Usage:

ln(token, fr=’’, target=’’, verbose=False)

storeClient.ln(token, fr, target) storeClient.ln(fr, target) storeClient.ln(fr, target)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • fr (str) – Name of the file to be linked (may not be a directory).

  • to (str) – Name of the link to be created

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Link a file in vospace
storeClient.ln('foo', 'bar')
storeClient.ln('vos://foo', 'vos:///new/bar')
dl.storeClient.load[source]

Load a file from a remote endpoint to the Store Manager service

Usage:

load(name, endpoint, token=None, is_vospace=False)

storeClient.load(token, name, endpoint) storeClient.load(name, endpoint)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • name (str) – Name or template of local files to upload.

  • endpoint (str) – Name of the file for destination directory on remote VOSpace.

Returns

result – An ‘OK’ message or error string

Return type

str

Example

# Load a file from a remote URL
storeClient.load('mydata.vot', 'http://example.com/data.vot')
dl.storeClient.ls[source]

Get a file/directory listing from the store manager service

Usage:

ls(name=’vos://’, token=None, format=’csv’, verbose=False)

storeClient.ls(token, name) storeClient.ls(name) storeClient.ls()

Parameters
  • token (str) – Secure token obtained via authClient.login()

  • name (str) – Valid name of file or directory, e.g. vos://somedir .. todo:: [20161110] currently doesn’t seem to work.

  • format (str) – Default csv. The long option produces an output similar to ‘ls -l’.

Example

listing = storeClient.ls(token, name='vos://somedir')
listing = storeClient.ls(token, 'vos://somedir')
listing = storeClient.ls('vos://somedir')
print(listing)

This prints for instance:

bar2.fits,foo1.csv,fancyfile.dat
dl.storeClient.mkdir[source]

Make a directory in the storage manager service

Usage:

mkdir(optval, name=’’, token=None)

storeClient.mkdir(token, name) storeClient.mkdir(name)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • name (str) – Name of the container (directory) to create.

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Create a directory in vospace
storeClient.mkdir('foo')
dl.storeClient.mv[source]

Move/rename a file/directory within the store manager service

Usage:

mv(token=None, fr=’’, to=’’, verbose=False)

storeClient.mv(token, fr, to) storeClient.mv(fr, to) storeClient.mv(token) storeClient.mv(fr=’’,to=’’)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • fr (str) – Name of the file to be moved

  • to (str) – Name of the file or directory to move the ‘fr’ file. If given as a directory the original filename is preserved.

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Move a file in vospace
storeClient.mv('foo', 'bar')             # rename file
storeClient.mv('foo', 'vos://newdir/')   # move to new directory
storeClient.mv('foo', 'newdir')
dl.storeClient.pull[source]

Load a file from a remote endpoint to the Store Manager service

Usage:

pull(name, endpoint, token=None, is_vospace=False)

storeClient.pull(token, name, endpoint) storeClient.pull(name, endpoint)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • name (str) – Name or template of local files to upload.

  • endpoint (str) – Name of the file for destination directory on remote VOSpace.

Returns

result – An ‘OK’ message or error string

Return type

str

Example

# Load a file from a remote URL
storeClient.load('mydata.vot', 'http://example.com/data.vot')
dl.storeClient.put[source]

Upload a file to the store manager service

Usage:

put(fr=’’, to=’vos://’, token=None, verbose=True, debug=False)

storeClient.put(token, fr, to) storeClient.put(fr, to) storeClient.put(fr) storeClient.put(fr=’’,to=’’)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • fr (str) – Name or template of local files to upload.

  • to (str) – Name of the file for destination directory on remote VOSpace.

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# get the contents of a single file
dl.storeClient.rm[source]

Delete a file from the store manager service

Usage:

rm(name=’’, token=None, verbose=False)

storeClient.rm(token, name) storeClient.rm(name)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • name (str) – Name of the file to delete

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Remove a file from vospace
storeClient.rm('foo.csv')
storeClient.rm('vos://foo.csv')
dl.storeClient.rmdir[source]

Delete a directory from the store manager service

Usage:

rmdir(name=’’, token=None, verbose=False)

storeClient.rmdir(token, name) storeClient.rmdir(name)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • name (str) – Name of the container (directory) to delete.

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Remove an empty directory from VOSpace.
storeClient.rmdir('datadir')
storeClient.rmdir('vos://datadir')
dl.storeClient.saveAs[source]

Save the string representation of a data object as a file.

Usage:

saveAs(data, name, token=None)

storeClient.saveAs(token, data, name) storeClient.saveAs(data, name)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • data (str) – Data object to be saved.

  • name (str) – Name of the file to create containing the string representation of the data.

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Save a data object to VOSpace
storeClient.saveAs(pandas_data, 'pandas.example')
storeClient.saveAs(json_data, 'json.example')
storeClient.saveAs(table_data, 'table.example')
dl.storeClient.scToString(s)[source]

scToString – Force a return value to be type ‘string’ for all Python versions. If there is an error, return the original.

dl.storeClient.services(name=None, svc_type='vos', format=None, profile='default')[source]
dl.storeClient.set_profile(profile='default')[source]

Set the service profile to be used.

Parameters

profile (str) – The name of the profile to use. The list of available ones can be retrieved from the service (see function storeClient.list_profiles())

Returns

Return type

Nothing

Example

storeClient.set_profile('test')
dl.storeClient.set_svc_url(svc_url='https://datalab.noao.edu/storage')[source]

Set the Storage Manager service URL.

Parameters

svc_url (str) – The service URL of the Storage Manager to use

Returns

Return type

Nothing

Example

storeClient.set_scv_url("http://demo.datalab.noao.edu:7003")
dl.storeClient.stat[source]

Get file status information, similar to stat().

Usage:

stat(path, token=None, verbose=True)

storeClient.stat(token, path) storeClient.stat(path)

Parameters
  • path (str) – A name or file template of the file status to retrieve.

  • token (str) – Authentication token (see function authClient.login())

  • verbose (bool) – Verbose output flag.

Returns

stat

A dictionary of node status values. Returned fields include:

name Name of node groupread List of group/owner names w/ read access groupwrite List of group/owner names w/ write access publicread Publicly readable (0=False, 1=True) owner Owner name perms Formatted unix-like permission string target Node target if LinkNode size Size of file node (bytes) type Node type (container|data|link)

Return type

dictionary

Example

# get status information for a specific node
stat = storeClient.stat('vos://mydata.csv')

if stat['type'] == 'container':
    print('This is a directory')
else:
    print('File size is: ' + stat['size'])
class dl.storeClient.storeClient(profile='default', svc_url='https://datalab.noao.edu/storage')[source]

Bases: object

STORECLIENT – Client-side methods to access the Data Lab

Storage Manager Service.

access(**kw)[source]

Call the appropriate instance of the function.

cp(**kw)[source]

Call the appropriate instance of the function.

get(**kw)[source]

Call the appropriate instance of the function.

getFromURL(svc_url, path, token)[source]

Get something from a URL. Return a ‘response’ object

get_profile()[source]

Get the profile

Parameters

None

Returns

profile – The name of the current profile used with the Storage Manager

Return type

str

Example

print('Store Service profile = ' + storeClient.get_profile())
get_svc_url()[source]

Return the currently-used Storage Manager service URL.

Parameters

None

Returns

Return type

Current Storage Manager service URL

Example

print(storeClient.get_scv_url())
isAlive(svc_url=None, timeout=2)[source]
Check whether the StorageManager service at the given URL is

alive and responding. This is a simple call to the root service URL or ping() method.

Parameters

svc_url (str) – The service URL of the storage manager to use

Example

# Check if service is responding
if storeClient.isAlive():
   .....stuff
list_profiles(**kw)[source]

Call the appropriate instance of the function.

ln(**kw)[source]

Call the appropriate instance of the function.

load(**kw)[source]

Call the appropriate instance of the function.

ls(**kw)[source]

Call the appropriate instance of the function.

mkdir(**kw)[source]

Call the appropriate instance of the function.

mv(**kw)[source]

Call the appropriate instance of the function.

pull(**kw)[source]

Call the appropriate instance of the function.

put(**kw)[source]

Call the appropriate instance of the function.

rm(**kw)[source]

Call the appropriate instance of the function.

rmdir(**kw)[source]

Call the appropriate instance of the function.

saveAs(**kw)[source]

Call the appropriate instance of the function.

services(name=None, svc_type='vos', format=None, profile='default')[source]
set_profile(profile)[source]

Set the service profile to be used.

Parameters

profile (str) – The name of the profile to use. The list of available ones can be retrieved from the service (see function storeClient.list_profiles())

Returns

Return type

Nothing

Example

storeClient.set_profile('test')
set_svc_url(svc_url)[source]

Set the Storage Manager service URL.

Parameters

svc_url (str) – The service URL of the Storage Manager to use

Returns

Return type

Nothing

Example

storeClient.set_scv_url("http://demo.datalab.noao.edu:7003")
stat(**kw)[source]

Call the appropriate instance of the function.

tag(**kw)[source]

Call the appropriate instance of the function.

exception dl.storeClient.storeClientError(message)[source]

Bases: Exception

dl.storeClient.tag[source]

Annotate a file/directory in the store manager service

Usage:

tag(token, name=’’, tag=’’)

storeClient.tag(token, name, tag) storeClient.tag(name, tag) storeClient.tag(token, name=’foo’, tag=’bar’)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • name (str) – Name of the file to tag

  • tag (str) – Annotation string for file

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Annotate a file in vospace
storeClient.tag('foo.csv', 'This is a test')