aiida.transports.plugins package

Submodules

Plugin for transport over SSH (and SFTP for file transfer).

aiida.transports.plugins.ssh.parse_sshconfig(computername)

Return the ssh configuration for a given computer name.

This parses the .ssh/config file in the home directory and returns the part of configuration of the given computer name.

Parameters

computername – the computer name for which we want the configuration.

aiida.transports.plugins.ssh.convert_to_bool(string)

Convert a string passed in the CLI to a valid bool.

Returns

the parsed bool value.

Raises

ValueError – If the value is not parsable as a bool

class aiida.transports.plugins.ssh.SshTransport(*args, **kwargs)

Bases: aiida.transports.transport.Transport

Support connection, command execution and data transfer to remote computers via SSH+SFTP.

__abstractmethods__ = frozenset({})

__init__(*args, **kwargs)

Initialize the SshTransport class.

Parameters

• machine – the machine to connect to

• load_system_host_keys – (optional, default False) if False, do not load the system host keys

• key_policy – (optional, default = paramiko.RejectPolicy()) the policy to use for unknown keys

Other parameters valid for the ssh connect function (see the self._valid_connect_params list) are passed to the connect function (as port, username, password, …); taken from the accepted paramiko.SSHClient.connect() params.

__module__ = 'aiida.transports.plugins.ssh'

__str__()

Return a useful string.

_abc_impl = <_abc_data object>

_exec_command_internal(command, combine_stderr=False, bufsize=-1)

Executes the specified command in bash login shell.

Before the command is executed, changes directory to the current working directory as returned by self.getcwd().

For executing commands and waiting for them to finish, use exec_command_wait.

Parameters

• command – the command to execute. The command is assumed to be already escaped using aiida.common.escaping.escape_for_bash().

• combine_stderr – (default False) if True, combine stdout and stderr on the same buffer (i.e., stdout). Note: If combine_stderr is True, stderr will always be empty.

• bufsize – same meaning of the one used by paramiko.

Returns

a tuple with (stdin, stdout, stderr, channel), where stdin, stdout and stderr behave as file-like objects, plus the methods provided by paramiko, and channel is a paramiko.Channel object.

_exec_cp(cp_exe, cp_flags, src, dst)

Execute the cp command on the remote machine.

classmethod _get_allow_agent_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_compress_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_gss_auth_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_gss_deleg_creds_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_gss_host_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_gss_kex_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_key_filename_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_key_policy_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_load_system_host_keys_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_look_for_keys_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_port_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_proxy_command_suggestion_string(computer)

Return a suggestion for the specific field.

classmethod _get_timeout_suggestion_string(computer)

Return a suggestion for the specific field.

Provide 60s as a default timeout for connections.

classmethod _get_username_suggestion_string(computer)

Return a suggestion for the specific field.

static _local_listdir(path, pattern=None)

Acts on the local folder, for the rest, same as listdir

static _os_path_split_asunder(path)

Used by makedirs. Takes path (a str) and returns a list deconcatenating the path

_valid_auth_options = [('username', {'prompt': 'User name', 'help': 'user name for the computer', 'non_interactive_default': True}), ('port', {'option': <aiida.cmdline.params.options.overridable.OverridableOption object>, 'prompt': 'port Nr', 'non_interactive_default': True}), ('look_for_keys', {'switch': True, 'prompt': 'Look for keys', 'help': 'switch automatic key file discovery on / off', 'non_interactive_default': True}), ('key_filename', {'type': ABSOLUTEPATHEMPTY, 'prompt': 'SSH key file', 'help': 'Manually pass a key file if default path is not set in ssh config', 'non_interactive_default': True}), ('timeout', {'type': <class 'int'>, 'prompt': 'Connection timeout in s', 'help': 'time in seconds to wait for connection before giving up', 'non_interactive_default': True}), ('allow_agent', {'switch': True, 'prompt': 'Allow ssh agent', 'help': 'switch to allow or disallow ssh agent', 'non_interactive_default': True}), ('proxy_command', {'prompt': 'SSH proxy command', 'help': 'SSH proxy command', 'non_interactive_default': True}), ('compress', {'switch': True, 'prompt': 'Compress file transfers', 'help': 'switch file transfer compression on / off', 'non_interactive_default': True}), ('gss_auth', {'type': <class 'bool'>, 'prompt': 'GSS auth', 'help': 'GSS auth for kerberos', 'non_interactive_default': True}), ('gss_kex', {'type': <class 'bool'>, 'prompt': 'GSS kex', 'help': 'GSS kex for kerberos', 'non_interactive_default': True}), ('gss_deleg_creds', {'type': <class 'bool'>, 'prompt': 'GSS deleg_creds', 'help': 'GSS deleg_creds for kerberos', 'non_interactive_default': True}), ('gss_host', {'prompt': 'GSS host', 'help': 'GSS host for kerberos', 'non_interactive_default': True}), ('load_system_host_keys', {'switch': True, 'prompt': 'Load system host keys', 'help': 'switch loading system host keys on / off', 'non_interactive_default': True}), ('key_policy', {'type': Choice(['RejectPolicy', 'WarningPolicy', 'AutoAddPolicy']), 'prompt': 'Key policy', 'help': 'SSH key policy', 'non_interactive_default': True})]

_valid_connect_options = [('username', {'prompt': 'User name', 'help': 'user name for the computer', 'non_interactive_default': True}), ('port', {'option': <aiida.cmdline.params.options.overridable.OverridableOption object>, 'prompt': 'port Nr', 'non_interactive_default': True}), ('look_for_keys', {'switch': True, 'prompt': 'Look for keys', 'help': 'switch automatic key file discovery on / off', 'non_interactive_default': True}), ('key_filename', {'type': ABSOLUTEPATHEMPTY, 'prompt': 'SSH key file', 'help': 'Manually pass a key file if default path is not set in ssh config', 'non_interactive_default': True}), ('timeout', {'type': <class 'int'>, 'prompt': 'Connection timeout in s', 'help': 'time in seconds to wait for connection before giving up', 'non_interactive_default': True}), ('allow_agent', {'switch': True, 'prompt': 'Allow ssh agent', 'help': 'switch to allow or disallow ssh agent', 'non_interactive_default': True}), ('proxy_command', {'prompt': 'SSH proxy command', 'help': 'SSH proxy command', 'non_interactive_default': True}), ('compress', {'switch': True, 'prompt': 'Compress file transfers', 'help': 'switch file transfer compression on / off', 'non_interactive_default': True}), ('gss_auth', {'type': <class 'bool'>, 'prompt': 'GSS auth', 'help': 'GSS auth for kerberos', 'non_interactive_default': True}), ('gss_kex', {'type': <class 'bool'>, 'prompt': 'GSS kex', 'help': 'GSS kex for kerberos', 'non_interactive_default': True}), ('gss_deleg_creds', {'type': <class 'bool'>, 'prompt': 'GSS deleg_creds', 'help': 'GSS deleg_creds for kerberos', 'non_interactive_default': True}), ('gss_host', {'prompt': 'GSS host', 'help': 'GSS host for kerberos', 'non_interactive_default': True})]

_valid_connect_params = ['username', 'port', 'look_for_keys', 'key_filename', 'timeout', 'allow_agent', 'proxy_command', 'compress', 'gss_auth', 'gss_kex', 'gss_deleg_creds', 'gss_host']

chdir(path)

Change directory of the SFTP session. Emulated internally by paramiko.

Differently from paramiko, if you pass None to chdir, nothing happens and the cwd is unchanged.

chmod(path, mode)

Change permissions to path

Parameters

• path – path to file

• mode – new permission bits (integer)

chown(path, uid, gid)

Change owner permissions of a file.

For now, this is not implemented for the SSH transport.

close()

Close the SFTP channel, and the SSHClient.

Todo

correctly manage exceptions

Raises

aiida.common.InvalidOperation – if the channel is already open

copy(remotesource, remotedestination, dereference=False, recursive=True)

Copy a file or a directory from remote source to remote destination. Flags used: -r: recursive copy; -f: force, makes the command non interactive; -L follows symbolic links

Parameters

• remotesource – file to copy from

• remotedestination – file to copy to

• dereference – if True, copy content instead of copying the symlinks only Default = False.

• recursive (bool) – if True copy directories recursively, otherwise only copy the specified file(s)

Raises

IOError – if the cp execution failed.

Note

setting dereference equal to True could cause infinite loops.

copyfile(remotesource, remotedestination, dereference=False)

Copy a file from remote source to remote destination (On the same remote machine)

Parameters

• remotesource (str) – path of the remote source directory / file

• remotedestination (str) – path of the remote destination directory / file

• dereference (bool) – if True copy the contents of any symlinks found, otherwise copy the symlinks themselves

Raises

IOError – if one of src or dst does not exist

copytree(remotesource, remotedestination, dereference=False)

Copy a folder from remote source to remote destination (On the same remote machine)

Parameters

• remotesource (str) – path of the remote source directory / file

• remotedestination (str) – path of the remote destination directory / file

• dereference (bool) – if True copy the contents of any symlinks found, otherwise copy the symlinks themselves

Raises

IOError – if one of src or dst does not exist

exec_command_wait(command, stdin=None, combine_stderr=False, bufsize=-1)

Executes the specified command and waits for it to finish.

Parameters

• command – the command to execute

• stdin – (optional,default=None) can be a string or a file-like object.

• combine_stderr – (optional, default=False) see docstring of self._exec_command_internal()

• bufsize – same meaning of paramiko.

Returns

a tuple with (return_value, stdout, stderr) where stdout and stderr are strings.

get(remotepath, localpath, callback=None, dereference=True, overwrite=True, ignore_nonexisting=False)

Get a file or folder from remote to local. Redirects to getfile or gettree.

Parameters

• remotepath – a remote path

• localpath – an (absolute) local path

• dereference – follow symbolic links. Default = True (default behaviour in paramiko). False is not implemented.

• overwrite – if True overwrites files and folders. Default = False

Raises

• ValueError – if local path is invalid

• IOError – if the remotepath is not found

get_attribute(path)

Returns the object Fileattribute, specified in aiida.transports Receives in input the path of a given file.

getcwd()

Return the current working directory for this SFTP session, as emulated by paramiko. If no directory has been set with chdir, this method will return None. But in __enter__ this is set explicitly, so this should never happen within this class.

getfile(remotepath, localpath, callback=None, dereference=True, overwrite=True)

Get a file from remote to local.

Parameters

• remotepath – a remote path

• localpath – an (absolute) local path

• overwrite – if True overwrites files and folders. Default = False

Raises

• ValueError – if local path is invalid

• OSError – if unintentionally overwriting

gettree(remotepath, localpath, callback=None, dereference=True, overwrite=True)

Get a folder recursively from remote to local.

Parameters

• remotepath – a remote path

• localpath – an (absolute) local path

• dereference – follow symbolic links. Default = True (default behaviour in paramiko). False is not implemented.

• overwrite – if True overwrites files and folders. Default = False

Raises

• ValueError – if local path is invalid

• IOError – if the remotepath is not found

• OSError – if unintentionally overwriting

gotocomputer_command(remotedir)

Specific gotocomputer string to connect to a given remote computer via ssh and directly go to the calculation folder.

isdir(path)

Return True if the given path is a directory, False otherwise. Return False also if the path does not exist.

isfile(path)

Return True if the given path is a file, False otherwise. Return False also if the path does not exist.

listdir(path='.', pattern=None)

Get the list of files at path.

Parameters

• path – default = ‘.’

• pattern – returns the list of files matching pattern. Unix only. (Use to emulate ls * for example)

makedirs(path, ignore_existing=False)

Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except that any intermediate path segment (not just the rightmost) will be created if it does not exist.

NOTE: since os.path.split uses the separators as the host system (that could be windows), I assume the remote computer is Linux-based and use ‘/’ as separators!

Parameters

• path – directory to create (string)

• ignore_existing – if set to true, it doesn’t give any error if the leaf directory does already exist (bool)

Raises

OSError – If the directory already exists.

mkdir(path, ignore_existing=False)

Create a folder (directory) named path.

Parameters

• path – name of the folder to create

• ignore_existing – if True, does not give any error if the directory already exists

Raises

OSError – If the directory already exists.

normalize(path='.')

Returns the normalized path (removing double slashes, etc…)

open()

Open a SSHClient to the machine possibly using the parameters given in the __init__.

Also opens a sftp channel, ready to be used. The current working directory is set explicitly, so it is not None.

Raises

aiida.common.InvalidOperation – if the channel is already open

path_exists(path)

Check if path exists

put(localpath, remotepath, callback=None, dereference=True, overwrite=True, ignore_nonexisting=False)

Put a file or a folder from local to remote. Redirects to putfile or puttree.

Parameters

• localpath – an (absolute) local path

• remotepath – a remote path

• dereference – follow symbolic links (boolean). Default = True (default behaviour in paramiko). False is not implemented.

• overwrite – if True overwrites files and folders (boolean). Default = False.

Raises

• ValueError – if local path is invalid

• OSError – if the localpath does not exist

putfile(localpath, remotepath, callback=None, dereference=True, overwrite=True)

Put a file from local to remote.

Parameters

• localpath – an (absolute) local path

• remotepath – a remote path

• overwrite – if True overwrites files and folders (boolean). Default = True.

Raises

• ValueError – if local path is invalid

• OSError – if the localpath does not exist, or unintentionally overwriting

puttree(localpath, remotepath, callback=None, dereference=True, overwrite=True)

Put a folder recursively from local to remote.

By default, overwrite.

Parameters

• localpath – an (absolute) local path

• remotepath – a remote path

• dereference – follow symbolic links (boolean) Default = True (default behaviour in paramiko). False is not implemented.

• overwrite – if True overwrites files and folders (boolean). Default = True

Raises

• ValueError – if local path is invalid

• OSError – if the localpath does not exist, or trying to overwrite

• IOError – if remotepath is invalid

Note

setting dereference equal to True could cause infinite loops. see os.walk() documentation

remove(path)

Remove a single file at ‘path’

rename(oldpath, newpath)

Rename a file or folder from oldpath to newpath.

Parameters

• oldpath (str) – existing name of the file or folder

• newpath (str) – new name for the file or folder

Raises

• IOError – if oldpath/newpath is not found

• ValueError – if sroldpathc/newpath is not a valid string

rmdir(path)

Remove the folder named ‘path’ if empty.

rmtree(path)

Remove a file or a directory at path, recursively Flags used: -r: recursive copy; -f: force, makes the command non interactive;

Parameters

path – remote path to delete

Raises

IOError – if the rm execution failed.

property sftp

property sshclient

symlink(remotesource, remotedestination)

Create a symbolic link between the remote source and the remote destination.

Parameters

• remotesource – remote source. Can contain a pattern.

• remotedestination – remote destination