aiida.transport.plugins package

Submodules

class aiida.transport.plugins.local.LocalTransport(**kwargs)

Bases: aiida.transport.Transport

Support copy and command execution on the same host on which AiiDA is running via direct file copy and execution commands.

__init__(**kwargs)

__init__ method of the Transport base class.

__module__ = 'aiida.transport.plugins.local'

__str__()

Return a description as a string.

_exec_command_internal(command)

Executes the specified command, first changing directory to the current working directory as returned by self.getcwd(). Does not wait for the calculation to finish.

For a higher-level exec_command that automatically waits for the job to finish, use exec_command_wait. Otherwise, to end the process, use the proc.wait() method.

Parameters:command – the command to execute Returns:a tuple with (stdin, stdout, stderr, proc), where stdin, stdout and stderr behave as file-like objects, proc is the process object as returned by the subprocess.Popen() class.

_local_listdir(path, pattern=None)

_os_path_split_asunder(path)

Used by makedirs. Takes path (a str) and returns a list deconcatenating the path

_valid_auth_params = []

chdir(path)

Changes directory to path, emulated internally. :param path: path to cd into :raise OSError: if the directory does not have read attributes.

chmod(path, mode)

Changes permission bits of object at path :param path: path to modify :param mode: permission bits

Raises:IOError – if path does not exist.

close()

Closes the local transport channel

Raises:InvalidOperation – if the channel is already open

copy(source, destination, dereference=False)

Copies a file or a folder from ‘remote’ source to ‘remote’ destination. Automatically redirects to copyfile or copytree.

Parameters:

• source – path to local file
• destination – path to remote file
• dereference – follow symbolic links. Default = False

Raises:

• ValueError – if ‘remote’ source or destination is not valid
• OSError – if source does not exist

copyfile(source, destination)

Copies a file from ‘remote’ source to ‘remote’ destination.

Parameters:

• source – path to local file
• destination – path to remote file

Raises:

• ValueError – if ‘remote’ source or destination is not valid
• OSError – if source does not exist

copytree(source, destination, dereference=False)

Copies a folder from ‘remote’ source to ‘remote’ destination.

Parameters:

• source – path to local file
• destination – path to remote file
• dereference – follow symbolic links. Default = False

Raises:

• ValueError – if ‘remote’ source or destination is not valid
• OSError – if source does not exist

curdir

Returns the _internal_dir, if the channel is open. If possible, use getcwd() instead!

exec_command_wait(command, stdin=None)

Executes the specified command and waits for it to finish.

Parameters:command – the command to execute Returns:a tuple with (return_value, stdout, stderr) where stdout and stderr are strings.

get(source, destination, dereference=True, overwrite=True, ignore_nonexisting=False)

Copies a folder or a file recursively from ‘remote’ source to ‘local’ destination. Automatically redirects to getfile or gettree.

Parameters:

• source – path to local file
• destination – absolute path to remote file
• dereference – follow symbolic links default = True
• overwrite – if True overwrites destination default = False

Raises:

• IOError – if ‘remote’ source is not valid
• ValueError – if ‘local’ destination is not valid

get_attribute(path)

Returns an object FileAttribute, as specified in aiida.transport. :param path: the path of the given file.

getcwd()

Returns the current working directory, emulated by the transport

getfile(source, destination, overwrite=True)

Copies a file recursively from ‘remote’ source to ‘local’ destination.

Parameters:

• source – path to local file
• destination – absolute path to remote file
• overwrite – if True overwrites destination. Default = False

:raise IOError if ‘remote’ source is not valid or not found :raise ValueError: if ‘local’ destination is not valid :raise OSError: if unintentionally overwriting

gettree(source, destination, dereference=True, overwrite=True)

Copies a folder recursively from ‘remote’ source to ‘local’ destination.

Parameters:

• source – path to local file
• destination – absolute path to remote file
• dereference – follow symbolic links. Default = True
• overwrite – if True overwrites destination. Default = False

Raises:

• IOError – if ‘remote’ source is not valid
• ValueError – if ‘local’ destination is not valid
• OSError – if unintentionally overwriting

gotocomputer_command(remotedir)

Return a string to be run using os.system in order to connect via the transport to the remote directory.

Expected behaviors:

• A new bash session is opened
• A reasonable error message is produced if the folder does not exist

Parameters:remotedir (str) – the full path of the remote directory

isdir(path)

Checks if ‘path’ is a directory. :return: a boolean

isfile(path)

Checks if object at path is a file. Returns a boolean.

listdir(path='.', pattern=None)

Returns:

a list containing the names of the entries in the directory.

Parameters:

• path – default =’.’
• pattern – if set, returns the list of files matching pattern. Unix only. (Use to emulate ls * for example)

makedirs(path, ignore_existing=False)

Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except that any intermediate path segment (not just the rightmost) will be created if it does not exist.

Parameters:

• path – directory to create
• ignore_existing – if set to true, it doesn’t give any error if the leaf directory does already exist

Raises:

OSError – If the directory already exists and is not ignore_existing

mkdir(path, ignore_existing=False)

Create a folder (directory) named path.

Parameters:

• path – name of the folder to create
• ignore_existing – if True, does not give any error if the directory already exists

Raises:

OSError – If the directory already exists.

normalize(path)

Normalizes path, eliminating double slashes, etc.. :param path: path to normalize

open()

Opens a local transport channel

Raises:InvalidOperation – if the channel is already open

path_exists(path)

Check if path exists

put(source, destination, dereference=True, overwrite=True, ignore_nonexisting=False)

Copies a file or a folder from source to destination. Automatically redirects to putfile or puttree.

Parameters:

• source – absolute path to local file
• destination – path to remote file
• dereference – if True follows symbolic links. Default = True
• overwrite – if True overwrites destination. Default = False

Raises:

• IOError – if destination is not valid
• ValueError – if source is not valid

putfile(source, destination, overwrite=True)

Copies a file from source to destination. Automatically redirects to putfile or puttree.

Parameters:

• source – absolute path to local file
• destination – path to remote file
• overwrite – if True overwrites destination Default = False

Raises:

• IOError – if destination is not valid
• ValueError – if source is not valid
• OSError – if source does not exist

puttree(source, destination, dereference=True, overwrite=True)

Copies a folder recursively from source to destination. Automatically redirects to putfile or puttree.

Parameters:

• source – absolute path to local file
• destination – path to remote file
• dereference – follow symbolic links. Default = True
• overwrite – if True overwrites destination. Default = False

Raises:

• IOError – if destination is not valid
• ValueError – if source is not valid
• OSError – if source does not exist

remove(path)

Removes a file at position path.

rename(src, dst)

Rename a file or folder from oldpath to newpath.

Parameters:

• oldpath (str) – existing name of the file or folder
• newpath (str) – new name for the file or folder

Raises:

• IOError – if src/dst is not found
• ValueError – if src/dst is not a valid string

rmdir(path)

Removes a folder at location path. :param path: path to remove

rmtree(path)

Remove tree as rm -r would do

Parameters:path – a string to path

symlink(remotesource, remotedestination)

Create a symbolic link between the remote source and the remote destination

Parameters:

• remotesource – remote source. Can contain a pattern.
• remotedestination – remote destination

class aiida.transport.plugins.ssh.SshTransport(machine, **kwargs)

Bases: aiida.transport.Transport

Support connection, command execution and data transfer to remote computers via SSH+SFTP.

__init__(machine, **kwargs)

Initialize the SshTransport class.

Parameters:

• machine – the machine to connect to
• load_system_host_keys – (optional, default False) if False, do not load the system host keys
• key_policy – (optional, default = paramiko.RejectPolicy()) the policy to use for unknown keys

Other parameters valid for the ssh connect function (see the self._valid_connect_params list) are passed to the connect function (as port, username, password, …); taken from the accepted paramiko.SSHClient.connect() params.

__module__ = 'aiida.transport.plugins.ssh'

__str__()

Return a useful string.

classmethod _convert_allow_agent_fromstring(string)

Convert the port from string.

classmethod _convert_compress_fromstring(string)

Convert the port from string.

classmethod _convert_gss_auth_fromstring(string)

Convert the gss auth. command from string.

classmethod _convert_gss_deleg_creds_fromstring(string)

Convert the gss auth. command from string.

classmethod _convert_gss_host_fromstring(string)

Convert the gss auth. command from string.

classmethod _convert_gss_kex_fromstring(string)

Convert the gss key exchange command from string.

classmethod _convert_key_filename_fromstring(string)

Convert the port from string.

classmethod _convert_key_policy_fromstring(string)

Convert the port from string.

classmethod _convert_load_system_host_keys_fromstring(string)

Convert the port from string.

classmethod _convert_look_for_keys_fromstring(string)

Convert the port from string.

classmethod _convert_port_fromstring(string)

Convert the port from string.

classmethod _convert_proxy_command_fromstring(string)

Convert the proxy command from string.

classmethod _convert_timeout_fromstring(string)

Convert the port from string.

classmethod _convert_username_fromstring(string)

Convert the username from string.

_exec_command_internal(command, combine_stderr=False, bufsize=-1)

Executes the specified command, first changing directory to the current working directory are returned by self.getcwd(). Does not wait for the calculation to finish.

For a higher-level _exec_command_internal that automatically waits for the job to finish, use exec_command_wait.

Parameters:

• command – the command to execute
• combine_stderr – (default False) if True, combine stdout and stderr on the same buffer (i.e., stdout). Note: If combine_stderr is True, stderr will always be empty.
• bufsize – same meaning of the one used by paramiko.

Returns:

a tuple with (stdin, stdout, stderr, channel), where stdin, stdout and stderr behave as file-like objects, plus the methods provided by paramiko, and channel is a paramiko.Channel object.

_exec_cp(cp_exe, cp_flags, src, dst)

classmethod _get_allow_agent_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_compress_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_gss_auth_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_gss_deleg_creds_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_gss_host_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_gss_kex_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_key_filename_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_key_policy_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_load_system_host_keys_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_look_for_keys_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_port_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_proxy_command_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_timeout_suggestion_string(computer)

Return a suggestion for the specific field.

Provide 60s as a default timeout for connections.

classmethod _get_username_suggestion_string(computer)

Return a suggestion for the specific field.

_local_listdir(path, pattern=None)

Acts on the local folder, for the rest, same as listdir

_os_path_split_asunder(path)

Used by makedirs. Takes path (a str) and returns a list deconcatenating the path

_valid_auth_params = ['username', 'port', 'look_for_keys', 'key_filename', 'timeout', 'allow_agent', 'proxy_command', 'compress', 'gss_auth', 'gss_kex', 'gss_deleg_creds', 'gss_host', 'load_system_host_keys', 'key_policy']

_valid_connect_params = ['username', 'port', 'look_for_keys', 'key_filename', 'timeout', 'allow_agent', 'proxy_command', 'compress', 'gss_auth', 'gss_kex', 'gss_deleg_creds', 'gss_host']

chdir(path)

Change directory of the SFTP session. Emulated internally by paramiko.

Differently from paramiko, if you pass None to chdir, nothing happens and the cwd is unchanged.

chmod(path, mode)

Change permissions to path

Parameters:

• path – path to file
• mode – new permission bits (integer)

close()

Close the SFTP channel, and the SSHClient.

Todo:correctly manage exceptions Raises:InvalidOperation – if the channel is already open

copy(remotesource, remotedestination, dereference=False)

Copy a file or a directory from remote source to remote destination. Flags used: -r: recursive copy; -f: force, makes the command non interactive; -L follows symbolic links

Parameters:

• remotesource – file to copy from
• remotedestination – file to copy to
• dereference – if True, copy content instead of copying the symlinks only Default = False.

Raises:

IOError – if the cp execution failed.

Note

setting dereference equal to True could cause infinite loops.

copyfile(remotesource, remotedestination, dereference=False, pattern=None)

Copy a file from remote source to remote destination Redirects to copy().

Parameters:

• remotesource –
• remotedestination –
• dereference –
• pattern –

copytree(remotesource, remotedestination, dereference=False, pattern=None)

copy a folder recursively from remote source to remote destination Redirects to copy()

Parameters:

• remotesource –
• remotedestination –
• dereference –
• pattern –

exec_command_wait(command, stdin=None, combine_stderr=False, bufsize=-1)

Executes the specified command and waits for it to finish.

Parameters:

• command – the command to execute
• stdin – (optional,default=None) can be a string or a file-like object.
• combine_stderr – (optional, default=False) see docstring of self._exec_command_internal()
• bufsize – same meaning of paramiko.

Returns:

a tuple with (return_value, stdout, stderr) where stdout and stderr are strings.

get(remotepath, localpath, callback=None, dereference=True, overwrite=True, ignore_nonexisting=False)

Get a file or folder from remote to local. Redirects to getfile or gettree.

Parameters:

• remotepath – a remote path
• localpath – an (absolute) local path
• dereference – follow symbolic links. Default = True (default behaviour in paramiko). False is not implemented.
• overwrite – if True overwrites files and folders. Default = False

Raises:

• ValueError – if local path is invalid
• IOError – if the remotepath is not found

get_attribute(path)

Returns the object Fileattribute, specified in aiida.transport Receives in input the path of a given file.

getcwd()

Return the current working directory for this SFTP session, as emulated by paramiko. If no directory has been set with chdir, this method will return None. But in __enter__ this is set explicitly, so this should never happen within this class.

getfile(remotepath, localpath, callback=None, dereference=True, overwrite=True)

Get a file from remote to local.

Parameters:

• remotepath – a remote path
• localpath – an (absolute) local path
• overwrite – if True overwrites files and folders. Default = False

Raises:

• ValueError – if local path is invalid
• OSError – if unintentionally overwriting

gettree(remotepath, localpath, callback=None, dereference=True, overwrite=True)

Get a folder recursively from remote to local.

Parameters:

• remotepath – a remote path
• localpath – an (absolute) local path
• dereference – follow symbolic links. Default = True (default behaviour in paramiko). False is not implemented.
• overwrite – if True overwrites files and folders. Default = False

Raises:

• ValueError – if local path is invalid
• IOError – if the remotepath is not found
• OSError – if unintentionally overwriting

gotocomputer_command(remotedir)

Specific gotocomputer string to connect to a given remote computer via ssh and directly go to the calculation folder.

isdir(path)

Return True if the given path is a directory, False otherwise. Return False also if the path does not exist.

isfile(path)

Return True if the given path is a file, False otherwise. Return False also if the path does not exist.

listdir(path='.', pattern=None)

Get the list of files at path.

Parameters:

• path – default = ‘.’
• pattern – returns the list of files matching pattern. Unix only. (Use to emulate ls * for example)

makedirs(path, ignore_existing=False)

Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except that any intermediate path segment (not just the rightmost) will be created if it does not exist.

NOTE: since os.path.split uses the separators as the host system (that could be windows), I assume the remote computer is Linux-based and use ‘/’ as separators!

Parameters:

• path – directory to create (string)
• ignore_existing – if set to true, it doesn’t give any error if the leaf directory does already exist (bool)

Raises:

OSError – If the directory already exists.

mkdir(path, ignore_existing=False)

Create a folder (directory) named path.

Parameters:

• path – name of the folder to create
• ignore_existing – if True, does not give any error if the directory already exists

Raises:

OSError – If the directory already exists.

normalize(path)

Returns the normalized path (removing double slashes, etc…)

open()

Open a SSHClient to the machine possibly using the parameters given in the __init__.

Also opens a sftp channel, ready to be used. The current working directory is set explicitly, so it is not None.

Raises:InvalidOperation – if the channel is already open

path_exists(path)

Check if path exists

put(localpath, remotepath, callback=None, dereference=True, overwrite=True, ignore_nonexisting=False)

Put a file or a folder from local to remote. Redirects to putfile or puttree.

Parameters:

• localpath – an (absolute) local path
• remotepath – a remote path
• dereference – follow symbolic links (boolean). Default = True (default behaviour in paramiko). False is not implemented.
• overwrite – if True overwrites files and folders (boolean). Default = False.

Raises:

• ValueError – if local path is invalid
• OSError – if the localpath does not exist

putfile(localpath, remotepath, callback=None, dereference=True, overwrite=True)

Put a file from local to remote.

Parameters:

• localpath – an (absolute) local path
• remotepath – a remote path
• overwrite – if True overwrites files and folders (boolean). Default = True.

Raises:

• ValueError – if local path is invalid
• OSError – if the localpath does not exist, or unintentionally overwriting

puttree(localpath, remotepath, callback=None, dereference=True, overwrite=True)

Put a folder recursively from local to remote.

Parameters:

• localpath – an (absolute) local path
• remotepath – a remote path
• dereference – follow symbolic links (boolean) Default = True (default behaviour in paramiko). False is not implemented.
• overwrite – if True overwrites files and folders (boolean). Default = True

Raises:

• ValueError – if local path is invalid
• OSError – if the localpath does not exist, or trying to overwrite
• IOError – if remotepath is invalid

Note

setting dereference equal to True could cause infinite loops. see os.walk() documentation

remove(path)

Remove a single file at ‘path’

rename(src, dst)

Rename a file or folder from src to dst.

Parameters:

• oldpath (str) – existing name of the file or folder
• newpath (str) – new name for the file or folder

Raises:

• IOError – if src/dst is not found
• ValueError – if src/dst is not a valid string

rmdir(path)

Remove the folder named ‘path’ if empty.

rmtree(path)

Remove a file or a directory at path, recursively Flags used: -r: recursive copy; -f: force, makes the command non interactive;

Parameters:path – remote path to delete Raises:IOError – if the rm execution failed.

sftp

sshclient

symlink(remotesource, remotedestination)

Create a symbolic link between the remote source and the remote destination.

Parameters:

• remotesource – remote source. Can contain a pattern.
• remotedestination – remote destination

aiida.transport.plugins.ssh.convert_to_bool(string)

aiida.transport.plugins.ssh.parse_sshconfig(computername)

This module contains a set of unittest test classes that can be loaded from the plugin. Every transport plugin should be able to pass all of these common tests. Plugin specific tests will be written in the plugin itself.

class aiida.transport.plugins.test_all_plugins.TestDirectoryManipulation(methodName='runTest')

Bases: unittest.case.TestCase

Tests to check, create and delete folders.

__module__ = 'aiida.transport.plugins.test_all_plugins'

test_chdir_to_empty_string()

The wrapper function that calls the subfunction for each transport.

test_dir_copy()

The wrapper function that calls the subfunction for each transport.

test_dir_creation_deletion()

The wrapper function that calls the subfunction for each transport.

test_dir_permissions_creation_modification()

The wrapper function that calls the subfunction for each transport.

test_dir_reading_permissions()

The wrapper function that calls the subfunction for each transport.

test_isfile_isdir_to_empty_string()

The wrapper function that calls the subfunction for each transport.

test_isfile_isdir_to_non_existing_string()

The wrapper function that calls the subfunction for each transport.

test_listdir()

The wrapper function that calls the subfunction for each transport.

test_listdir_withattributes()

The wrapper function that calls the subfunction for each transport.

test_makedirs()

The wrapper function that calls the subfunction for each transport.

test_rmtree()

The wrapper function that calls the subfunction for each transport.

class aiida.transport.plugins.test_all_plugins.TestExecuteCommandWait(methodName='runTest')

Bases: unittest.case.TestCase

Test some simple command executions and stdin/stdout management.

It also checks for escaping of the folder names.

__module__ = 'aiida.transport.plugins.test_all_plugins'

test_exec_pwd()

The wrapper function that calls the subfunction for each transport.

test_exec_with_stdin_filelike()

The wrapper function that calls the subfunction for each transport.

test_exec_with_stdin_string()

The wrapper function that calls the subfunction for each transport.

test_exec_with_stdin_unicode()

The wrapper function that calls the subfunction for each transport.

test_exec_with_wrong_stdin()

The wrapper function that calls the subfunction for each transport.

class aiida.transport.plugins.test_all_plugins.TestPutGetFile(methodName='runTest')

Bases: unittest.case.TestCase

Test to verify whether the put and get functions behave correctly on files. 1) they work 2) they need abs paths where necessary, i.e. for local paths 3) they reject empty strings

__module__ = 'aiida.transport.plugins.test_all_plugins'

test_put_and_get()

The wrapper function that calls the subfunction for each transport.

test_put_get_abs_path()

The wrapper function that calls the subfunction for each transport.

test_put_get_empty_string()

The wrapper function that calls the subfunction for each transport.

class aiida.transport.plugins.test_all_plugins.TestPutGetTree(methodName='runTest')

Bases: unittest.case.TestCase

Test to verify whether the put and get functions behave correctly on folders. 1) they work 2) they need abs paths where necessary, i.e. for local paths 3) they reject empty strings

__module__ = 'aiida.transport.plugins.test_all_plugins'

test_copy()

The wrapper function that calls the subfunction for each transport.

test_get()

The wrapper function that calls the subfunction for each transport.

test_put()

The wrapper function that calls the subfunction for each transport.

test_put_and_get()

The wrapper function that calls the subfunction for each transport.

test_put_and_get_overwrite()

The wrapper function that calls the subfunction for each transport.

test_put_get_abs_path()

The wrapper function that calls the subfunction for each transport.

test_put_get_empty_string()

The wrapper function that calls the subfunction for each transport.

aiida.transport.plugins.test_all_plugins.get_all_custom_transports()

Autodiscover all custom transports defined in the variable plugin_transpors inside each test_* file in this folder.

Therefore, do not move this function out of this file.

Returns:a dictionary of objects as defined in the various plugin_transport variables of the different files (the key is the module in which it was found)

aiida.transport.plugins.test_all_plugins.run_for_all_plugins(actual_test_method)

Decorator method that actually run the methods with an additional parameter (custom_transport), once for every custom_transport defined in the test_* files [except this one].

class aiida.transport.plugins.test_local.TestBasicConnection(methodName='runTest')

Bases: unittest.case.TestCase

Test basic connections.

__module__ = 'aiida.transport.plugins.test_local'

test_basic()

test_closed_connection()

test_invalid_param()

class aiida.transport.plugins.test_local.TestGeneric(methodName='runTest')

Bases: unittest.case.TestCase

Test whoami on localhost.

__module__ = 'aiida.transport.plugins.test_local'

test_whoami()

Test ssh plugin on localhost

class aiida.transport.plugins.test_ssh.TestBasicConnection(methodName='runTest')

Bases: unittest.case.TestCase

Test basic connections.

__module__ = 'aiida.transport.plugins.test_ssh'

test_auto_add_policy()

test_closed_connection_sftp()

test_closed_connection_ssh()

test_invalid_param()

test_no_host_key()