dl.storeClient module

dl.storeClient.access()

Determine whether the file can be accessed with the given node.

Usage:

access(path, mode=None, token=None, verbose=True)

MultiMethod Usage:

storeClient.access(token, path, mode)
storeClient.access(path, mode)
storeClient.access(path)

Parameters:

• path (str) – A name or file template of the file status to retrieve.

• mode (str) – Requested access mode. Modes are ‘r’ (read access), ‘w’ (write access), or ‘rw’ to test for both read/write access. If mode is None a simple existence check is made.

• token (str [Optional]) – Authentication token (see function authClient.login()).

• verbose (bool) – Verbose output flag.

Returns:

result – True if the node can be access with the requested mode.

Return type:

bool

Example

if storeClient.access('/mydata.csv'):
    print('File exists')
elif storeClient.access('/mydata.csv','rw'):
    print('File is both readable and writable')

dl.storeClient.chunked_upload(token, local_file, remote_file)

A streaming file uploader.

dl.storeClient.cp()

Copy a file/directory within the Storage Manager service.

Usage:

cp(token=None, fr='', to='', verbose=False)

MultiMethod Usage:

storeClient.cp(token, fr, to)
storeClient.cp(fr, to)
storeClient.cp(fr)
storeClient.cp(fr='',to='')

Parameters:

• token (str [Optional]) – Authentication token (see function authClient.login()).

• fr (str) – Name of the file to be copied (may not be a directory).

• to (str) – Name of the file to be created.

Returns:

result – An 'OK' message or error string for each copied file.

Return type:

str

Example

# Copy a file in VOSpace
storeClient.cp('foo', 'bar')
storeClient.cp('vos://foo', 'vos:///new/bar')

dl.storeClient.expandFileList(svc_url, token, pattern, format, full=False)

Expand a filename pattern in a VOSpace URI to a list of files. We do this by getting a listing of the parent container contents from the service and then match the pattern on the client side.

dl.storeClient.get()

Retrieve a file from the Storage Manager service.

Usage::

get(token=None, fr=’’, to=’’, mode=’text’, verbose=True, debug=False,

timeout=30)

MultiMethod Usage:

storeClient.get(token, fr, to)
storeClient.get(fr, to)
storeClient.get(fr)
storeClient.get(token, fr, to)

Parameters:

• token (str [Optional]) – Authentication token (see function authClient.login()).

• fr (str) – A name or file template of the file(s) to retrieve.

• to (str) – Name of the file(s) to locally save to. If not specified, the contents of the file are returned to the caller.

• mode ([binary | text | fileobj]) – Return data type if not saving to file. If set to ‘text’ the file contents are converted to string – this is appropriate when dealing with unicode but may fail with general binary data. If set to ‘binary’ the raw content of the HTTP response is returned – for Python 2 this will be a ‘string’, for Python 3 it will be a ‘bytes’ data type (the caller is responsible for conversion).

• verbose (bool) – Print verbose output, e.g. progress indicators.

• debug (bool) – Print debug output.

• timeout (integer) – Retry timeout value. When processing long lists, download will pause every timeout files to lessen server load. For individual files, transfer will retry for timeout seconds before aborting. Failed transfers are automatically appended to the file list so they may be transferred again later.

Returns:

result – A list of the names of the files retrieved, or the contents of a single file.

Return type:

str

Example

# get a single file to a local file of a different name
data = storeClient.get('vos://mydata.csv', 'data.csv')

# get the contents of a single file to a local variable
data = storeClient.get('vos://mydata.csv')

# get a list of remote files to a local directory
flist = storeClient.get('vos://*.fits', './data/')
flist = storeClient.get('*.fits', './data/')

dl.storeClient.getClient(profile='default', svc_url='https://datalab.noirlab.edu/storage')

Create a new storeClient object and set a default profile.

dl.storeClient.get_profile()

Get the Storage Manager service profile being used.

Parameters:

None

Returns:

profile – The name of the current profile used with the Storage Manager.

Return type:

str

Example

print('Store Service profile = ' + storeClient.get_profile())

dl.storeClient.get_svc_url()

Return the currently-used Storage Manager service URL.

Parameters:

None

Returns:

result – Current Storage Manager service URL.

Return type:

str

Example

print(storeClient.get_svc_url())

dl.storeClient.hasmeta(s)

Determine whether a string contains filename meta-characters.

dl.storeClient.isAlive(svc_url='https://datalab.noirlab.edu/storage')

Check whether the Storage Manager service at the given URL is

alive and responding. This is a simple call to the root service URL or ping() method.

Parameters:

svc_url (str [Optional]) – The service URL of the Storage Manager to use.

Example

# Check if service is responding
storeClient.isAlive()

This prints

True

dl.storeClient.is_vosDir(svc_url, token, path)

Determine whether ‘path’ is a ContainerNode in the VOSpace.

dl.storeClient.list_profiles()

Retrieve the profiles supported by the Storage Manager service.

Usage:

list_profiles(token=None, profile=None, format='text')

MultiMethod Usage:

storeClient.list_profiles(token)        # list default profile
storeClient.list_profiles(profile)      # list named profile
storeClient.list_profiles()             # list default profile

Parameters:

• token (str [Optional]) – Authentication token (see function authClient.login()).

• profile (str) – A specific profile to list.

Returns:

profiles – A list of the names of the supported profiles or a dictionary of the specific profile.

Return type:

list/dict

Example

# get the list of available profiles
profiles = storeClient.list_profiles()

dl.storeClient.ln()

Create a link to a file/directory in the Storage Manager service.

Usage:

ln(token, fr='', target='', verbose=False)

MultiMethod Usage:

storeClient.ln(token, fr, target)
storeClient.ln(fr, target)
storeClient.ln(fr, target)

Parameters:

• token (str [Optional]) – Authentication token (see function authClient.login()).

• fr (str) – Name of the file to be linked (may not be a directory).

• to (str) – Name of the link to be created.

Returns:

result – An 'OK' message or error string for each linked file.

Return type:

str

Example

# Link a file in VOSpace
storeClient.ln('foo', 'bar')
storeClient.ln('vos://foo', 'vos:///new/bar')

dl.storeClient.load()

Load a file from a remote endpoint to the Storage Manager service.

Usage:

load(name, endpoint, token=None, is_vospace=False)

MultiMethod Usage:

storeClient.load(token, name, endpoint)
storeClient.load(name, endpoint)

Parameters:

• token (str [Optional]) – Authentication token (see function authClient.login()).

• name (str) – Name or template of local files to upload.

• endpoint (str) – Name of the file for destination directory on remote VOSpace.

Returns:

result – An 'OK' message or error string for each loaded file.

Return type:

str

Example

# Load a file to VOSpace called ``mydata.fits`` from a remote URL
storeClient.load('vos://mydata.fits', 'http://example.com/data.fits')

dl.storeClient.ls()

Get a file/directory listing from the Storage Manager service.

Usage:

ls(name='vos://', token=None, format='csv', verbose=False)

MultiMethod Usage:

storeClient.ls(token, name)
storeClient.ls(name)
storeClient.ls()

Parameters:

• token (str [Optional]) – Authentication token (see function authClient.login()).

• name (str [Optional]) – Valid name of file or directory, e.g. vos://somedir. If no name is provided, it will return the top directory listing.

• format (str) – Default csv. The long option produces an output similar to ls -l.

Example

# Get the file/directory listing of the top directory
listing = storeClient.ls()
# Get the file/directory listing of the ``somedir`` directory
listing = storeClient.ls('vos://somedir')
print(listing)

This prints for instance:

bar2.fits,foo1.csv,fancyfile.dat

dl.storeClient.mkdir()

Make a directory in the Storage Manager service.

Usage:

mkdir(optval, name='', token=None)

MultiMethod Usage:

storeClient.mkdir(token, name)
storeClient.mkdir(name)

Parameters:

• token (str [Optional]) – Authentication token (see function authClient.login()).

• name (str) – Name of the container (directory) to create.

Returns:

result – An 'OK' message or error string.

Return type:

str

Example

# Create a directory in VOSpace called ``foo``
storeClient.mkdir('foo')

dl.storeClient.mv()

Move/rename a file/directory within the Storage Manager service.

Usage:

mv(token=None, fr='', to='', verbose=False)

MultiMethod Usage:

storeClient.mv(token, fr, to)
storeClient.mv(fr, to)
storeClient.mv(token)
storeClient.mv(fr='',to='')

Parameters:

• token (str [Optional]) – Authentication token (see function authClient.login()).

• fr (str) – Name of the file or directory to be moved or renamed.

• to (str) – Name of the file or directory to move or rename the 'fr' file or directory. If given as a directory the original filename is preserved.

Returns:

result – An 'OK' message or error string for each moved or renamed file/directory.

Return type:

str

Example

# Move a file in VOSpace
storeClient.mv('foo', 'bar')             # rename file
storeClient.mv('foo', 'vos://newdir/')   # move to new directory
storeClient.mv('foo', 'newdir')

dl.storeClient.pull()

Pull a file from a remote endpoint to the Storage Manager service.

Same as storeClient.load() function.

Usage:

pull(name, endpoint, token=None, is_vospace=False)

MultiMethod Usage:

storeClient.pull(token, name, endpoint)
storeClient.pull(name, endpoint)

Parameters:

• token (str [Optional]) – Authentication token (see function authClient.login()).

• name (str) – Name or template of local files to pull.

• endpoint (str) – Name of the file for destination directory on remote VOSpace.

Returns:

result – An 'OK' message or error string for each pulled file.

Return type:

str

Example

# Pull a file to VOSpace called ``mydata.fits`` from a remote URL
storeClient.pull('vos://mydata.fits', 'http://example.com/data.fits')

dl.storeClient.put()

Upload a file to the Storage Manager service.

Usage:

put(fr='', to='vos://', token=None, verbose=True, debug=False)

MultiMethod Usage:

storeClient.put(token, fr, to)
storeClient.put(fr, to)
storeClient.put(fr)
storeClient.put(fr='',to='')

Parameters:

• token (str [Optional]) – Authentication token (see function authClient.login()).

• fr (str) – Name or template of local file to upload.

• to (str) – Name of the file for destination directory on remote VOSpace.

Returns:

result – An 'OK' message or error string for each uploaded file.

Return type:

str

Example

# Put the contents of local file 'data.csv' into VOSpace file named 'data_uploaded.csv'.
storeClient.put('data.csv', 'vos://data_uploaded.csv')

dl.storeClient.rm()

Delete a file from the Storage Manager service.

Usage:

rm(name='', token=None, verbose=False)

MultiMethod Usage:

storeClient.rm(token, name)
storeClient.rm(name)

Parameters:

• token (str [Optional]) – Authentication token (see function authClient.login()).

• name (str) – Name of the file to delete.

Returns:

result – An 'OK' message or error string for each file deleted.

Return type:

str

Example

# Remove a file from VOSpace
storeClient.rm('foo.csv')
storeClient.rm('vos://foo.csv')

dl.storeClient.rmdir()

Delete a directory from the Storage Manager service.

Usage:

rmdir(name='', token=None, verbose=False)

MultiMethod Usage:

storeClient.rmdir(token, name)
storeClient.rmdir(name)

Parameters:

• token (str [Optional]) – Authentication token (see function authClient.login()).

• name (str) – Name of the container (directory) to delete.

Returns:

result – An 'OK' message or error string for each directory deleted.

Return type:

str

Example

# Remove an empty directory from VOSpace
storeClient.rmdir('datadir')
storeClient.rmdir('vos://datadir')

dl.storeClient.saveAs()

Save the string representation of a data object as a file.

Usage:

saveAs(data, name, token=None)

MultiMethod Usage:

storeClient.saveAs(token, data, name)
storeClient.saveAs(data, name)

Parameters:

• token (str [Optional]) – Authentication token (see function authClient.login()).

• data (str) – Data object to be saved.

• name (str) – Name of the file to create containing the string representation of the data.

Returns:

result – An 'OK' message or error string for each file saved.

Return type:

str

Example

# Save a data object called ``table_data`` to VOSpace as ``table.example``
storeClient.saveAs(table_data, 'table.example')

dl.storeClient.scToString(s)

scToString – Force a return value to be type ‘string’ for all Python versions. If there is an error, return the original.

dl.storeClient.services(name=None, svc_type='vos', format=None, profile='default')

dl.storeClient.set_profile(profile='default')

Set the service profile to be used.

Parameters:

profile (str) – The name of the profile to use. The list of available ones can be retrieved from the service (see function storeClient.list_profiles()).

Return type:

Nothing

Example

storeClient.set_profile('test')

dl.storeClient.set_svc_url(svc_url='https://datalab.noirlab.edu/storage')

Set the Storage Manager service URL.

Parameters:

svc_url (str) – The service URL of the Storage Manager to use.

Return type:

Nothing

Example

storeClient.set_svc_url("http://demo.datalab.noirlab.edu:7003")

dl.storeClient.stat()

Get file status information, similar to stat().

Usage::

stat(path, token=None, verbose=True)

MultiMethod Usage:

storeClient.stat(token, path)
storeClient.stat(path)

Parameters:

• path (str) – A name or file template of the file status to retrieve.

• token (str [Optional]) – Authentication token (see function authClient.login()).

• verbose (bool) – Verbose output flag.

Returns:

stat –

A dictionary of node status values. Returned fields include:

• name Name of node.

• groupread List of group/owner names with read access.

• groupwrite List of group/owner names with write access.

• publicread Publicly readable (0=False, 1=True).

• owner Owner name.

• perms Formatted unix-like permission string.

• target Node target if LinkNode.

• size Size of file node (bytes).

• type Node type (container|data|link).

Return type:

dictionary

Example

# get status information for a specific node
stat = storeClient.stat('vos://mydata.csv')

if stat['type'] == 'container':
    print('This is a directory')
else:
    print('File size is: ' + str(stat['size']))

class dl.storeClient.storeClient(profile='default', svc_url='https://datalab.noirlab.edu/storage')

Bases: object

STORECLIENT – Client-side methods to access the Data Lab

Storage Manager service.

access(**kw)

Call the appropriate instance of the function.

cp(token=None, fr='', to='', verbose=False)

Usage:: storeClient.cp(fr, to)

get(fr='', to='', token=None, mode='text', verbose=True, debug=False, timeout=30)

Usage:: storeClient.get(token, fr, to)

getFromURL(svc_url, path, token)

Get something from a URL. Return a ‘response’ object.

get_profile()

Get the Storage Manager service profile being used.

Parameters:

None

Returns:

profile – The name of the current profile used with the Storage Manager.

Return type:

str

Example

print('Store Service profile = ' + storeClient.get_profile())

get_svc_url()

Return the currently-used Storage Manager service URL.

Parameters:

None

Returns:

result – Current Storage Manager service URL.

Return type:

str

Example

print(storeClient.get_svc_url())

isAlive(svc_url=None, timeout=2)

Check whether the Storage Manager service at the given URL is

alive and responding. This is a simple call to the root service URL or ping() method.

Parameters:

svc_url (str [Optional]) – The service URL of the Storage Manager to use.

Example

# Check if service is responding
storeClient.isAlive()

This prints

True

list_profiles(token=None, profile=None, format='text')

Usage:: storeClient.list_profiles(….)

ln(**kw)

Call the appropriate instance of the function.

load(**kw)

Call the appropriate instance of the function.

ls(name='vos://', token=None, format='csv', verbose=False)

Usage:: storeClient.ls()

mkdir(**kw)

Call the appropriate instance of the function.

mv(token=None, fr='', to='', verbose=False)

Usage:: storeClient.mv(fr, to)

pull(**kw)

Call the appropriate instance of the function.

put(fr='', to='vos://', token=None, verbose=True, debug=False)

Usage:: storeClient.put(fr=’’,to=’’)

rm(name='', token=None, verbose=False)

Usage:: storeClient.rm(name)

rmdir(name='', token=None, verbose=False)

Usage:: storeClient.rmdir(name)

saveAs(**kw)

Call the appropriate instance of the function.

services(name=None, svc_type='vos', format=None, profile='default')

set_profile(profile)

Set the service profile to be used.

Parameters:

profile (str) – The name of the profile to use. The list of available ones can be retrieved from the service (see function storeClient.list_profiles()).

Return type:

Nothing

Example

storeClient.set_profile('test')

set_svc_url(svc_url)

Set the Storage Manager service URL.

Parameters:

svc_url (str) – The service URL of the Storage Manager to use.

Return type:

Nothing

Example

storeClient.set_svc_url("http://demo.datalab.noirlab.edu:7003")

stat(**kw)

Call the appropriate instance of the function.

tag(**kw)

Call the appropriate instance of the function.

exception dl.storeClient.storeClientError(message)

Bases: Exception

dl.storeClient.tag()

Annotate a file/directory in the Storage Manager service.

Usage:

tag(token, name='', tag='')

MultiMethod Usage:

storeClient.tag(token, name, tag)
storeClient.tag(name, tag)
storeClient.tag(token, name='foo', tag='bar')

Parameters:

• token (str [Optional]) – Authentication token (see function authClient.login()).

• name (str) – Name of the file to tag.

• tag (str) – Annotation string for file.

Returns:

result – An 'OK' message or error string for each file annotated.

Return type:

str

Example

# Annotate a file in VOSpace
storeClient.tag('foo.csv', 'This is a test')