Local transport
aiida.transports.plugins.local.
LocalTransport
Bases: aiida.transports.transport.Transport
aiida.transports.transport.Transport
Support copy and command execution on the same host on which AiiDA is running via direct file copy and execution commands.
Note that the environment variables are copied from the submitting process, so you might need to clean it with a prepend_text. For example, the AiiDA daemon sets a PYTHONPATH, so you might want to add unset PYTHONPATH if you plan on running calculations that use Python.
prepend_text
PYTHONPATH
unset PYTHONPATH
_DEFAULT_SAFE_OPEN_INTERVAL
__abstractmethods__
__init__
__init__ method of the Transport base class.
safe_interval – (optional, default self._DEFAULT_SAFE_OPEN_INTERVAL) Minimum time interval in seconds between opening new connections.
use_login_shell – (optional, default True) if False, do not use a login shell when executing command
__module__
__str__
Return a description as a string.
_abc_impl
_exec_command_internal
Executes the specified command in bash login shell.
Before the command is executed, changes directory to the current working directory as returned by self.getcwd().
For executing commands and waiting for them to finish, use exec_command_wait. Otherwise, to end the process, use the proc.wait() method.
The subprocess is set to have a different process group than the main process, so that it is shielded from signals sent to the parent.
command – the command to execute. The command is assumed to be already escaped using aiida.common.escaping.escape_for_bash().
aiida.common.escaping.escape_for_bash()
a tuple with (stdin, stdout, stderr, proc), where stdin, stdout and stderr behave as file-like objects, proc is the process object as returned by the subprocess.Popen() class.
_local_listdir
Act on the local folder, for the rest, same as listdir.
_os_path_split_asunder
Used by makedirs, Takes path (a str) and returns a list deconcatenating the path.
_valid_auth_options
chdir
Changes directory to path, emulated internally. :param path: path to cd into :raise OSError: if the directory does not have read attributes.
chmod
Changes permission bits of object at path :param path: path to modify :param mode: permission bits
IOError – if path does not exist.
chown
Change the owner (uid) and group (gid) of a file. As with python’s os.chown function, you must pass both arguments, so if you only want to change one, use stat first to retrieve the current owner and group.
path (str) – path to the file to change the owner and group of
uid (int) – new owner’s uid
gid (int) – new group id
close
Closes the local transport channel
aiida.common.InvalidOperation – if the channel is already open
copy
Copies a file or a folder from ‘remote’ remotesource to ‘remote’ remotedestination. Automatically redirects to copyfile or copytree.
remotesource – path to local file
remotedestination – path to remote file
dereference – follow symbolic links. Default = False
recursive (bool) – if True copy directories recursively, otherwise only copy the specified file(s)
ValueError – if ‘remote’ remotesource or remotedestinationis not valid
OSError – if remotesource does not exist
copyfile
Copies a file from ‘remote’ remotesource to ‘remote’ remotedestination.
dereference (bool) – if True copy the contents of any symlinks found, otherwise copy the symlinks themselves
ValueError – if ‘remote’ remotesource or remotedestination is not valid
copytree
Copies a folder from ‘remote’ remotesource to ‘remote’ remotedestination.
curdir
Returns the _internal_dir, if the channel is open. If possible, use getcwd() instead!
exec_command_wait
Executes the specified command and waits for it to finish.
command – the command to execute
a tuple with (return_value, stdout, stderr) where stdout and stderr are strings.
get
Copies a folder or a file recursively from ‘remote’ remotepath to ‘local’ localpath. Automatically redirects to getfile or gettree.
remotepath – path to local file
localpath – absolute path to remote file
dereference – follow symbolic links default = True
overwrite – if True overwrites localpath default = False
IOError – if ‘remote’ remotepath is not valid
ValueError – if ‘local’ localpath is not valid
get_attribute
Returns an object FileAttribute, as specified in aiida.transports. :param path: the path of the given file.
getcwd
Returns the current working directory, emulated by the transport
getfile
Copies a file recursively from ‘remote’ remotepath to ‘local’ localpath.
overwrite – if True overwrites localpath. Default = False
:raise IOError if ‘remote’ remotepath is not valid or not found :raise ValueError: if ‘local’ localpath is not valid :raise OSError: if unintentionally overwriting
gettree
Copies a folder recursively from ‘remote’ remotepath to ‘local’ localpath.
dereference – follow symbolic links. Default = True
OSError – if unintentionally overwriting
gotocomputer_command
Return a string to be run using os.system in order to connect via the transport to the remote directory.
Expected behaviors:
A new bash session is opened
A reasonable error message is produced if the folder does not exist
remotedir (str) – the full path of the remote directory
isdir
Checks if ‘path’ is a directory. :return: a boolean
isfile
Checks if object at path is a file. Returns a boolean.
listdir
a list containing the names of the entries in the directory.
path – default =’.’
pattern – if set, returns the list of files matching pattern. Unix only. (Use to emulate ls * for example)
makedirs
Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except that any intermediate path segment (not just the rightmost) will be created if it does not exist.
path – directory to create
ignore_existing – if set to true, it doesn’t give any error if the leaf directory does already exist
OSError – If the directory already exists and is not ignore_existing
mkdir
Create a folder (directory) named path.
path – name of the folder to create
ignore_existing – if True, does not give any error if the directory already exists
OSError – If the directory already exists.
normalize
Normalizes path, eliminating double slashes, etc.. :param path: path to normalize
open
Opens a local transport channel
path_exists
Check if path exists
put
Copies a file or a folder from localpath to remotepath. Automatically redirects to putfile or puttree.
localpath – absolute path to local file
remotepath – path to remote file
dereference – if True follows symbolic links. Default = True
overwrite – if True overwrites remotepath. Default = False
IOError – if remotepath is not valid
ValueError – if localpath is not valid
putfile
Copies a file from localpath to remotepath. Automatically redirects to putfile or puttree.
overwrite – if True overwrites remotepath Default = False
OSError – if localpath does not exist
puttree
Copies a folder recursively from localpath to remotepath. Automatically redirects to putfile or puttree.
remove
Removes a file at position path.
rename
Rename a file or folder from oldpath to newpath.
oldpath (str) – existing name of the file or folder
newpath (str) – new name for the file or folder
IOError – if src/dst is not found
ValueError – if src/dst is not a valid string
rmdir
Removes a folder at location path. :param path: path to remove
rmtree
Remove tree as rm -r would do
path – a string to path
symlink
Create a symbolic link between the remote source and the remote remotedestination
remotesource – remote source. Can contain a pattern.
remotedestination – remote destination
Plugin for transport over SSH (and SFTP for file transfer).
aiida.transports.plugins.ssh.
SshTransport
Support connection, command execution and data transfer to remote computers via SSH+SFTP.
_MAX_EXEC_COMMAND_LOG_SIZE
Initialize the SshTransport class.
machine – the machine to connect to
load_system_host_keys – (optional, default False) if False, do not load the system host keys
key_policy – (optional, default = paramiko.RejectPolicy()) the policy to use for unknown keys
Other parameters valid for the ssh connect function (see the self._valid_connect_params list) are passed to the connect function (as port, username, password, …); taken from the accepted paramiko.SSHClient.connect() params.
Return a useful string.
_close_proxies
Close all proxy connections (proxy_jump and proxy_command)
For executing commands and waiting for them to finish, use exec_command_wait.
combine_stderr – (default False) if True, combine stdout and stderr on the same buffer (i.e., stdout). Note: If combine_stderr is True, stderr will always be empty.
bufsize – same meaning of the one used by paramiko.
a tuple with (stdin, stdout, stderr, channel), where stdin, stdout and stderr behave as file-like objects, plus the methods provided by paramiko, and channel is a paramiko.Channel object.
_exec_cp
Execute the cp command on the remote machine.
cp
_get_allow_agent_suggestion_string
Return a suggestion for the specific field.
_get_compress_suggestion_string
_get_gss_auth_suggestion_string
_get_gss_deleg_creds_suggestion_string
_get_gss_host_suggestion_string
_get_gss_kex_suggestion_string
_get_key_filename_suggestion_string
_get_key_policy_suggestion_string
_get_load_system_host_keys_suggestion_string
_get_look_for_keys_suggestion_string
_get_port_suggestion_string
_get_proxy_command_suggestion_string
_get_proxy_jump_suggestion_string
Return an empty suggestion since Paramiko does not parse ProxyJump from the SSH config.
_get_timeout_suggestion_string
Provide 60s as a default timeout for connections.
_get_username_suggestion_string
Acts on the local folder, for the rest, same as listdir
Used by makedirs. Takes path (a str) and returns a list deconcatenating the path
_symlink
Wrap SFTP symlink call without breaking API
source – source of link
dest – link to create
_valid_connect_options
_valid_connect_params
Change directory of the SFTP session. Emulated internally by paramiko.
Differently from paramiko, if you pass None to chdir, nothing happens and the cwd is unchanged.
Change permissions to path
path – path to file
mode – new permission bits (integer)
Change owner permissions of a file.
For now, this is not implemented for the SSH transport.
Close the SFTP channel, and the SSHClient.
correctly manage exceptions
Copy a file or a directory from remote source to remote destination. Flags used: -r: recursive copy; -f: force, makes the command non interactive; -L follows symbolic links
-r
-f
-L
remotesource – file to copy from
remotedestination – file to copy to
dereference – if True, copy content instead of copying the symlinks only Default = False.
IOError – if the cp execution failed.
Note
setting dereference equal to True could cause infinite loops.
Copy a file from remote source to remote destination (On the same remote machine)
remotesource (str) – path of the remote source directory / file
remotedestination (str) – path of the remote destination directory / file
IOError – if one of src or dst does not exist
Copy a folder from remote source to remote destination (On the same remote machine)
stdin – (optional,default=None) can be a string or a file-like object.
combine_stderr – (optional, default=False) see docstring of self._exec_command_internal()
bufsize – same meaning of paramiko.
Get a file or folder from remote to local. Redirects to getfile or gettree.
remotepath – a remote path
localpath – an (absolute) local path
dereference – follow symbolic links. Default = True (default behaviour in paramiko). False is not implemented.
overwrite – if True overwrites files and folders. Default = False
ValueError – if local path is invalid
IOError – if the remotepath is not found
Returns the object Fileattribute, specified in aiida.transports Receives in input the path of a given file.
Return the current working directory for this SFTP session, as emulated by paramiko. If no directory has been set with chdir, this method will return None. But in __enter__ this is set explicitly, so this should never happen within this class.
Get a file from remote to local.
Get a folder recursively from remote to local.
Specific gotocomputer string to connect to a given remote computer via ssh and directly go to the calculation folder.
Return True if the given path is a directory, False otherwise. Return False also if the path does not exist.
Return True if the given path is a file, False otherwise. Return False also if the path does not exist.
Get the list of files at path.
path – default = ‘.’
pattern – returns the list of files matching pattern. Unix only. (Use to emulate ls * for example)
ls *
lstat
Retrieve information about a file on the remote system, without following symbolic links (shortcuts). This otherwise behaves exactly the same as stat.
path (str) – the filename to stat
a paramiko.sftp_attr.SFTPAttributes object containing attributes about the given file.
NOTE: since os.path.split uses the separators as the host system (that could be windows), I assume the remote computer is Linux-based and use ‘/’ as separators!
path – directory to create (string)
ignore_existing – if set to true, it doesn’t give any error if the leaf directory does already exist (bool)
Returns the normalized path (removing double slashes, etc…)
Open a SSHClient to the machine possibly using the parameters given in the __init__.
Also opens a sftp channel, ready to be used. The current working directory is set explicitly, so it is not None.
Put a file or a folder from local to remote. Redirects to putfile or puttree.
dereference – follow symbolic links (boolean). Default = True (default behaviour in paramiko). False is not implemented.
overwrite – if True overwrites files and folders (boolean). Default = False.
OSError – if the localpath does not exist
Put a file from local to remote.
overwrite – if True overwrites files and folders (boolean). Default = True.
OSError – if the localpath does not exist, or unintentionally overwriting
Put a folder recursively from local to remote.
By default, overwrite.
dereference – follow symbolic links (boolean) Default = True (default behaviour in paramiko). False is not implemented.
overwrite – if True overwrites files and folders (boolean). Default = True
OSError – if the localpath does not exist, or trying to overwrite
IOError – if remotepath is invalid
setting dereference equal to True could cause infinite loops. see os.walk() documentation
Remove a single file at ‘path’
IOError – if oldpath/newpath is not found
ValueError – if sroldpathc/newpath is not a valid string
Remove the folder named ‘path’ if empty.
Remove a file or a directory at path, recursively Flags used: -r: recursive copy; -f: force, makes the command non interactive;
path – remote path to delete
IOError – if the rm execution failed.
sftp
sshclient
stat
Retrieve information about a file on the remote system. The return value is an object whose attributes correspond to the attributes of Python’s stat structure as returned by os.stat, except that it contains fewer fields. The fields supported are: st_mode, st_size, st_uid, st_gid, st_atime, and st_mtime.
os.stat
st_mode
st_size
st_uid
st_gid
st_atime
st_mtime
Create a symbolic link between the remote source and the remote destination.
convert_to_bool
Convert a string passed in the CLI to a valid bool.
the parsed bool value.
ValueError – If the value is not parsable as a bool
parse_sshconfig
Return the ssh configuration for a given computer name.
This parses the .ssh/config file in the home directory and returns the part of configuration of the given computer name.
.ssh/config
computername – the computer name for which we want the configuration.