AiiDA communicates with remote computers via the SSH protocol. There are two ways of setting up an SSH connection for AiiDA:
Using a passwordless SSH key (easier, less safe) Using a password-protected SSH key through ssh-agent (one more step, safer)
Using a passwordless SSH key (easier, less safe)
Using a password-protected SSH key through ssh-agent (one more step, safer)
ssh-agent
There are numerous tutorials on the web, see e.g. here. Very briefly, first create a new private/public keypair (aiida/aiida.pub), leaving passphrase emtpy:
aiida
aiida.pub
$ ssh-keygen -t rsa -b 4096 -f ~/.ssh/aiida
Copy the public key to the remote machine, normally this will add the public key to the remote machine’s ~/.ssh/authorized_keys:
~/.ssh/authorized_keys
$ ssh-copy-id -i ~/.ssh/aiida YOURUSERNAME@YOURCLUSTERADDRESS
Add the following lines to your ~/.ssh/config file (or create it, if it does not exist):
~/.ssh/config
Host YOURCLUSTERADDRESS User YOURUSERNAME IdentityFile ~/.ssh/aiida
Note
If your cluster needs you to connect to another computer PROXY first, you can use the ProxyJump or ProxyCommand feature of SSH, see Connecting to a remote computer via a proxy server.
ProxyJump
ProxyCommand
You should now be able to access the remote computer (without the need to type a password) via:
$ ssh YOURCLUSTERADDRESS # this connection is used to copy files $ sftp YOURCLUSTERADDRESS
Connection closed failures
If the ssh command works, but the sftp command prints Connection closed, there may be a line in the ~/.bashrc file on the cluster that either produces text output or an error. Remove/comment lines from this file until no output or error is produced: this should make sftp work again.
ssh
sftp
Connection closed
~/.bashrc
Finally, if you are planning to use a batch scheduler on the remote computer, try also:
$ ssh YOURCLUSTERADDRESS QUEUE_VISUALIZATION_COMMAND
replacing QUEUE_VISUALIZATION_COMMAND by squeue (SLURM), qstat (PBSpro) or the equivalent command of your scheduler and check that it prints a list of the job queue without errors.
QUEUE_VISUALIZATION_COMMAND
squeue
qstat
Scheduler errors?
If the previous command errors with command not found, while the same QUEUE_VISUALIZATION_COMMAND works fine after you’ve logged in via SSH, it may be that a guard in the .bashrc file on the cluster prevents necessary modules from being loaded.
command not found
.bashrc
Look for lines like:
[ -z "$PS1" ] && return
or:
case $- in *i*) ;; *) return;; esac
which will prevent any instructions that follow from being executed.
You can either move relevant instructions before these lines or delete the guards entirely. If you are wondering whether the PATH environment variable is set correctly, you can check its value using:
PATH
$ ssh YOURCLUSTERADDRESS 'echo $PATH'
Tools like ssh-agent (available on most Linux distros and MacOS) allow you to enter the passphrase of a protected key once and provide access to the decrypted key for as long as the agent is running. This allows you to use a passphrase-protected key (required by some HPC centres), while making the decrypted key available to AiiDA for automatic SSH operations.
Start by following the instructions above for Using a passwordless SSH key, the only difference being that you enter a passphrase when creating the key (and when logging in to the remote computer).
Now provide the passphrase for your private key to the agent:
ssh-add ~/.ssh/aiida
The private key and the relative passphrase are now recorded in an instance of the agent.
The passphase is stored in the agent only until the next reboot. If you shut down or restart the AiiDA machine, before starting the AiiDA deamon remember to run the ssh-add command again.
ssh-add
On most modern Linux installations, the ssh-agent starts automatically at login (e.g. Ubuntu 16.04 and later or MacOS 10.5 and later). If you received an error Could not open a connection to your authentication agent, you will need to start the agent manually instead.
Could not open a connection to your authentication agent
Check whether you can start an ssh-agent in your current shell:
eval `ssh-agent`
In order to reuse the same agent instance everywhere (including the AiiDA daemon), the environment variables of ssh-agent need to be reused by all shells. Download the script load-singlesshagent.sh and place it e.g. in ~/bin. Then add the following lines to your ~/.bashrc file:
load-singlesshagent.sh
~/bin
if [ -f ~/bin/load-singlesshagent.sh ]; then . ~/bin/load-singlesshagent.sh fi
To check that it works:
Open a new shell (~/.bashrc file is sourced).
Run ssh-add.
Close the shell.
Open a new shell and try logging in to the remote computer.
Try logging in to the remote computer; it should no longer require a passphrase.
The key and its corresponding passphrase are now stored by the agent until it is stopped. After a reboot, remember to run ssh-add ~/.ssh/aiida again before starting the AiiDA daemon.
On OSX Sierra and later, the native ssh-add client allows passphrases to be stored persistently in the OSX keychain. Store the passphrase in the keychain using the OSX-specific -k argument:
-k
ssh-add -k ~/.ssh/aiida
To instruct ssh to look in the OSX keychain for key passphrases, add the following lines to ~/.ssh/config:
Host * UseKeychain yes
When configuring the computer in AiiDA, simply make sure that Allow ssh agent is set to true (default).
Allow ssh agent
true
Some compute clusters require you to connect to an intermediate server PROXY, from which you can then connect to the cluster TARGET on which you run your calculations. This section explains how to use the ProxyJump or ProxyCommand feature of ssh in order to make this jump automatically.
Tip
This method can also be used to avoid having to start a virtual private network (VPN) client if you have an SSH account on a proxy/jumphost server which is accessible from your current network and from which you can access the TARGET machine directly.
To decide whether to use the ProxyJump (recommended) or the ProxyCommand directive, please check the version of your SSH client first with ssh -V. The ProxyJump directive has been added in version 7.3 of OpenSSH, hence if you are using an older version of SSH (on your machine or the PROXY) you have to use the older ProxyCommand.
ssh -V
To setup the proxy configuration with ProxyJump, edit the ~/.ssh/config file on the computer on which you installed AiiDA (or create it if missing) and add the following lines:
Host SHORTNAME_TARGET Hostname FULLHOSTNAME_TARGET User USER_TARGET IdentityFile ~/.ssh/aiida ProxyJump USER_PROXY@FULLHOSTNAME_PROXY Host FULLHOSTNAME_PROXY IdentityFile ~/.ssh/aiida
Replace the ..._TARGET and ..._PROXY variables with the host/user names of the respective servers.
..._TARGET
..._PROXY
To setup the proxy configuration with ProxyCommand instead, edit the ~/.ssh/config file on the computer on which you installed AiiDA (or create it if missing) and add the following lines: Host SHORTNAME_TARGET Hostname FULLHOSTNAME_TARGET User USER_TARGET IdentityFile ~/.ssh/aiida ProxyCommand ssh -W %h:%p USER_PROXY@FULLHOSTNAME_PROXY Host FULLHOSTNAME_PROXY IdentityFile ~/.ssh/aiida
To setup the proxy configuration with ProxyCommand instead, edit the ~/.ssh/config file on the computer on which you installed AiiDA (or create it if missing) and add the following lines:
Host SHORTNAME_TARGET Hostname FULLHOSTNAME_TARGET User USER_TARGET IdentityFile ~/.ssh/aiida ProxyCommand ssh -W %h:%p USER_PROXY@FULLHOSTNAME_PROXY Host FULLHOSTNAME_PROXY IdentityFile ~/.ssh/aiida
In both cases, this should allow you to directly connect to the TARGET server using
$ ssh SHORTNAME_TARGET
If the user directory is not shared between the PROXY and the TARGET (in most supercomputing facilities your user directory is shared between the machines), you need to follow the instructions for a passwordless connection twice: once for the connection from your computer to the PROXY server, and once for the connection from the PROXY server to the TARGET server (e.g. the public key must be listed in the ~/.ssh/authorized_keys file of both the PROXY and the TARGET server).
When configuring the computer in AiiDA, AiiDA will automatically parse most of required information from your ~/.ssh/config file. A notable exception to this is the proxy_jump directive, which must be specified manually.
proxy_jump
Simply copy & paste the same instructions as you have used for ProxyJump in your ~/.ssh/config to the input for proxy_jump:
$ verdi computer configure ssh SHORTNAME_TARGET ... Allow ssh agent [True]: SSH proxy jump []: USER_PROXY@FULLHOSTNAME_PROXY
A chain of proxies can be specified as a comma-separated list. If you need to specify a different username, you can so with USER_PROXY@.... If no username is specified for the proxy the same username as for the TARGET is used.
USER_PROXY@...
Important
Specifying the proxy_command manually
proxy_command
When specifying or updating the proxy_command option via verdi computer configure ssh, please do not use placeholders %h and %p but provide the actual hostname and port. AiiDA replaces them only when parsing from the ~/.ssh/config file.
verdi computer configure ssh
%h
%p
If the remote machine requires authentication through a Kerberos token (that you need to obtain before using ssh), you typically need to
install libffi (sudo apt-get install libffi-dev under Ubuntu) install the ssh_kerberos extra during the installation of aiida-core (see Setup).
install libffi (sudo apt-get install libffi-dev under Ubuntu)
libffi
sudo apt-get install libffi-dev
install the ssh_kerberos extra during the installation of aiida-core (see Setup).
ssh_kerberos
If you provide all necessary GSSAPI options in your ~/.ssh/config file, verdi computer configure should already pick up the appropriate values for all the gss-related options.
GSSAPI
verdi computer configure