Supported databases¶

For those who are interested in the details, there are three supported database backends:

• SQLite The SQLite backend is the fastest to configure: in fact, it does not really use a “real” database, but stores everything in a file. This is great if you never configured a database before and you just want to give AiiDA a try. However, keep in mind that it has many big shortcomings for a real AiiDA usage!

  In fact, since everything is stored on a single file, each access (especially when writing or doing a transaction) to the database locks it: this means that a second thread wanting to access the database has to wait that the lock is released. We set up a timeout of about 60 seconds for each thread to retry to connect to the database, but after that time you will get an exception, with the risk of storing corrupted data in the AiiDA repository.

  Therefore, it is OK to use SQLite for testing, but as soon as you want to use AiiDA in production, with more than one calculation submitted at each given time, please switch to a real database backend, like PostgreSQL.

  Note

  note, however, that typically SQLite is pretty fast for queries, once the database is loaded into memory, so it could be an interesting solution if you do not want to launch new calculations, but only to import the results and then query them (in a single-user approach).

• PostgreSQL This is the database backend that the we, the AiiDA developers, suggest to use, because it is the one with most features.

• MySQL This is another possible backend that you could use. However, we suggest that you use PostgreSQL instead of MySQL, due to some MySQL limitations (unless you have very strong reasons to prefer MySQL over PostgreSQL). In particular, some of the limitations of MySQL are:

  • Only a precision of 1 second is possible for time objects, while PostgreSQL supports microsecond precision. This can be relevant for a proper sorting of calculations launched almost simultaneously.
  • Indexed text columns can have an hardcoded maximum length. This can give issues with attributes, if you have very long key names or nested dictionaries/lists. These cannot be natively stored and therefore you either end up storing a JSON (therefore partially losing query capability) or you can even incur in problems.

Setup instructions¶

For any database, you may need to install a specific python package using pip; this typically also requires to have the development libraries installed (the .h C header files). Refer to the installation documentation for more details.

sudo apt-get install postgresql-9.1
sudo apt-get install postgresql-client-9.1

su - postgres

psql template1

CREATE USER aiida WITH PASSWORD 'the_aiida_password';

ALTER USER aiida PASSWORD 'new_aiida_password';

CREATE DATABASE aiidadb OWNER aiida;

GRANT ALL PRIVILEGES ON DATABASE aiidadb to aiida;

psql -h localhost -d aiidadb -U aiida -W

Database engine: postgresql
PostgreSQL host: localhost
PostgreSQL port: 5432
AiiDA Database name: aiidadb
AiiDA Database user: aiida
AiiDA Database password: the_aiida_password

mysql -h localhost mysql -u root -p

CREATE USER 'aiida'@'localhost' IDENTIFIED BY 'the_aiida_password';

CREATE DATABASE aiidadb;

GRANT ALL PRIVILEGES on aiidadb.* to aiida@localhost;

CREATE DATABASE test_aiidadb;
GRANT ALL PRIVILEGES on test_aiidadb.* to aiida@localhost;

Database engine: mysql
mySQL host: localhost
mySQL port: 3306
AiiDA Database name: aiidadb
AiiDA Database user: aiida
AiiDA Database password: the_aiida_passwd

How to backup the databases¶

It is strongly advised to backup the content of your database daily. Below are instructions to set this up for the SQLite, PostgreSQL and MySQL databases, under Ubuntu (tested with version 12.04).

#!/bin/bash
AIIDAUSER=aiida
AIIDADB=aiidadb
AIIDAPORT=5432
## STORE THE PASSWORD, IN THE PROPER FORMAT, IN THE ~/.pgpass file
## see http://www.postgresql.org/docs/current/static/libpq-pgpass.html
AIIDALOCALTMPDUMPFILE=~/.aiida/${AIIDADB}-backup.psql.gz


if [ -e ${AIIDALOCALTMPDUMPFILE} ]
then
    mv ${AIIDALOCALTMPDUMPFILE} ${AIIDALOCALTMPDUMPFILE}~
fi

# NOTE: password stored in ~/.pgpass, where pg_dump will read it automatically
pg_dump -h localhost -p $AIIDAPORT -U $AIIDAUSER $AIIDADB | gzip > $AIIDALOCALTMPDUMPFILE || rm $AIIDALOCALTMPDUMPFILE

localhost:5432:aiidadb:aiida:YOUR_DATABASE_PASSWORD

#!/bin/bash
su USERNAME -c "/home/USERNAME/.aiida/backup_postgresql.sh"

sudo chmod +x /etc/cron.daily/backup-aiidadb-USERNAME

psql -h localhost -U aiida -d aiidadb -f aiidadb-backup.psql

Supported architecture¶

AiiDA has a few strict requirements, in its current version: first, it will run only on Unix-like systems - it is tested (and developed) in Mac OS X and Linux (Ubuntu), but other Unix flavours should work as well.

Moreover, on the clusters (computational resources) side, it expects to find a Unix system, and the default shell is required to be bash.

Installing python¶

AiiDA requires python 2.7.x (only CPython has been tested). It is probable that you already have a version of python installed on your computer. To check, open a terminal and type:

python -V

that will print something like this:

Python 2.7.3

If you don’t have python installed, or your version is outdated, please install a suitable version of python (either refer to the manual of your Linux distribution, or for instance you can download the ActiveState Python from ActiveState. Choose the appropriate distribution corresponding to your architecture, and with version 2.7.x.x).

Installation of the core dependencies¶

sudo apt-get install git
sudo apt-get install python-pip
sudo apt-get install ipython
sudo apt-get install python2.7-dev
sudo apt-get install libsqlite3-dev
sudo apt-get install postgresql-server-dev-9.1

Downloading the code¶

Download the code using git in a directory of your choice (~/git/aiida in this tutorial), using the following command:

git clone https://USERNAME@bitbucket.org/aiida_team/aiida.git ~/git/aiida

(or use git@bitbucket.org:aiida_team/aiida.git if you are downloading through SSH; note that this requires your ssh key to be added on the Bitbucket account.)

Python dependencies¶

Python dependencies are managed using pip, that you have installed in the previous steps.

As a first step, check that pip is at its most recent version.

One possible way of doing this is to update pip with itself, with a command similar to the following:

sudo pip install -U pip

Then, install the python dependencies is as simple as this:

cd ~/git/aiida # or the folder where you downloaded AiiDA
pip install --user -U -r requirements.txt

(this will download and install requirements that are listed in the requirements.txt file; the --user option allows to install the packages as a normal user, without the need of using sudo or becoming root). Check that every package is installed correctly.

OSError: [Errno 13] Permission denied: '/usr/local/bin/easy_install'

sudo pip install -U -r requirements.txt

If everything went smoothly, congratulations! Now the code is installed! However, we need still a few steps to properly configure AiiDA for your user.

Cannot fetch index base URL https://pypi.python.org/simple/

sudo easy_install pip==1.2.1

apt-get install libqp-dev

export PATH=/the/path/to/the/pg_config/file:${PATH}

export DYLD_FALLBACK_LIBRARY_PATH=/Library/PostgreSQL/9.3/lib:$DYLD_FALLBACK_LIBRARY_PATH

AiiDA configuration¶

export PYTHONPATH=~/git/aiida:${PYTHONPATH}
export PATH=~/git/aiida/bin:${PATH}

export PATH=~/git/aiida/bin:~/.local/bin:${PATH}

export PATH=~/git/aiida/bin:~/Library/Python/2.7/bin:${PATH}

eval "$(verdi completioncommand)"

verdi completioncommand

if [ -f ~/.bashrc ]
then
    . ~/.bashrc
fi

verdi install

import pytz
print pytz.all_timezones

Insert your timezone: Europe/Zurich
Default user email: richard.wagner@leipzig.de
Database engine: sqlite3
AiiDA Database location: /home/wagner/.aiida/aiida.db
AiiDA repository directory: /home/wagner/.aiida/repository/
[...]
Configuring a new user with email 'richard.wagner@leipzig.de'
First name: Richard
Last name: Wagner
Institution: BRUHL, LEIPZIG
The user has no password, do you want to set one? [y/N] y
Insert the new password:
Insert the new password (again):

verdi daemon configureuser

verdi daemon start

verdi daemon status

* aiida-daemon[0]        RUNNING    pid 12076, uptime 0:39:05
* aiida-daemon-beat[0]   RUNNING    pid 12075, uptime 0:39:05

verdi daemon stop

Optional dependencies¶

sudo pip install pycifrw==3.6.2.1
sudo apt-get install jmol

curl https://wiki.fysik.dtu.dk/ase-files/python-ase-3.8.1.3440.tar.gz > python-ase-3.8.1.3440.tar.gz
gunzip python-ase-3.8.1.3440.tar.gz
tar -xvf python-ase-3.8.1.3440.tar
cd python-ase-3.8.1.3440
setup.py build
setup.py install
export PYTHONPATH=$(pwd):$PYTHONPATH

Further comments and troubleshooting¶

• For some reasons, on some machines (notably often on Mac OS X) there is no default locale defined, and when you run verdi install for the first time it fails (see also this issue of django). To solve the problem, first remove the sqlite database that was created.

  Then, run in your terminal (or maybe even better, add to your .bashrc, but then remember to open a new shell window!):

  export LANG="en_US.UTF-8"
  export LC_ALL="en_US.UTF-8"
  

  and then run verdi install again.

• [Only for developers] The developer tests of the SSH transport plugin are performed connecting to localhost. The tests will fail if a passwordless ssh connection is not set up. Therefore, if you want to run the tests:

  • make sure to have a ssh server. On Ubuntu, for instance, you can install it using:

    sudo apt-get install openssh-server
    

  • Configure a ssh key for your user on your machine, and then add your public key to the authorized keys of localhsot. The easiest way to achieve this is to run:

    ssh-copy-id localhost
    

    (it will ask your password, because it is connecting via ssh to localhost to install your public key inside ~/.ssh/authorized_keys).

Remote computer requirements¶

A computer in AiiDA denotes any computational resource (with a batch job scheduler) on which you will run your calculations. Computers typically are clusters or supercomputers.

Requirements for a computer are:

• It must run a Unix-like operating system
• The default shell must be bash
• It should have a batch scheduler installed (see here for a list of supported batch schedulers)
• It must be accessible from the machine that runs AiiDA using one of the available transports (see below).

The first step is to choose the transport to connect to the computer. Typically, you will want to use the SSH transport, apart from a few special cases where SSH connection is not possible (e.g., because you cannot setup a password-less connection to the computer). In this case, you can install AiiDA directly on the remote cluster, and use the local transport (in this way, commands to submit the jobs are simply executed on the AiiDA machine, and files are simply copied on the disk instead of opening an SFTP connection).

If you plan to use the local transport, you can skip to the next section.

If you plan to use the SSH transport, you have to configure a password-less login from your user to the cluster. To do so type first (only if you do not already have some keys in your local ~/.ssh directory - i.e. files like id_rsa.pub):

ssh-keygen -t rsa

Then copy your keys to the remote computer (in ~/.ssh/authorized_keys) with:

ssh-copy-id YOURUSERNAME@YOURCLUSTERADDRESS

replacing YOURUSERNAME and YOURCLUSTERADDRESS by respectively your username and cluster address. Finally add the following lines to ~/.ssh/config (leaving an empty line before and after):

Host YOURCLUSTERADDRESS
  User YOURUSERNAME
  HostKeyAlgorithms ssh-rsa
  IdentityFile YOURRSAKEY

replacing YOURRSAKEY by the path to the rsa private key you want to use (it should look like ~/.ssh/id_rsa).

Before proceeding to setup the computer, be sure that you are able to connect to your cluster using:

ssh YOURCLUSTERADDRESS

without the need to type a password. Moreover, make also sure you can connect via sftp (needed to copy files). The following command:

sftp YOURCLUSTERADDRESS

should show you a prompt without errors (possibly with a message saying Connected to YOURCLUSTERADDRESS).

The authenticity of host 'YOURCLUSTERADDRESS (IP)' can't be established.
RSA key fingerprint is xx:xx:xx:xx:xx.
Are you sure you want to continue connecting (yes/no)?

Finally, try also:

ssh YOURCLUSTERADDRESS QUEUE_VISUALIZATION_COMMAND

replacing QUEUE_VISUALIZATION_COMMAND by the scheduler command that prints on screen the status of the queue on the cluster (i.e. qstat for PBSpro scheduler, squeue for SLURM, etc.). It should print a snapshot of the queue status, without any errors.

export PATH=$PATH:/opt/pbs/default/bin

Computer setup and configuration¶

The configuration of computers happens in two steps.

1. Setup of the computer, using the:

   verdi computer setup
   

   command. This command allows to create a new computer instance in the DB.

   Tip

   The code will ask you a few pieces of information. At every prompt, you can type the ? character and press <enter> to get a more detailed explanation of what is being asked.

   Tip

   You can press <CTRL>+C at any moment to abort the setup process. Nothing will be stored in the DB.

   Note

   For multiline inputs (like the prepend text and the append text, see below) you have to press <CTRL>+D to complete the input, even if you do not want any text.

   Here is a list of what is asked, together with an explanation.

   • Computer name: the (user-friendly) name of the new computer instance which is about to be created in the DB (the name is used for instance when you have to pick up a computer to launch a calculation on it). Names must be unique. This command should be thought as a AiiDA-wise configuration of computer, independent of the AiiDA user that will actually use it.

   • Fully-qualified hostname: the fully-qualified hostname of the computer to which you want to connect (i.e., with all the dots: bellatrix.epfl.ch, and not just bellatrix). Type localhost for the local transport.

   • Description: A human-readable description of this computer; this is useful if you have a lot of computers and you want to add some text to distinguish them (e.g.: “cluster of computers at EPFL, installed in 2012, 2 GB of RAM per CPU”)

   • Enabled: either True or False; if False, the computer is disabled and calculations associated with it will not be submitted. This allows to disable temporarily a computer if it is giving problems or it is down for maintenance, without the need to delete it from the DB.

   • Transport type: The name of the transport to be used. A list of valid transport types can be obtained typing ?

   • Scheduler type: The name of the plugin to be used to manage the job scheduler on the computer. A list of valid scheduler plugins can be obtained typing ?. See here for a documentation of scheduler plugins in AiiDA.

   • AiiDA work directory: The absolute path of the directory on the remote computer where AiiDA will run the calculations (often, it is the scratch of the computer). You can (should) use the {username} replacement, that will be replaced by your username on the remote computer automatically: this allows the same computer to be used by different users, without the need to setup a different computer for each one. Example:

     /scratch/{username}/aiida_work/
     

   • mpirun command: The mpirun command needed on the cluster to run parallel MPI programs. You can (should) use the {tot_num_mpiprocs} replacement, that will be replaced by the total number of cpus, or the other scheduler-dependent fields (see the scheduler docs for more information). Some examples:

     mpirun -np {tot_num_mpiprocs}
     aprun -n {tot_num_mpiprocs}
     poe
     

   • Text to prepend to each command execution: This is a multiline string, whose content will be prepended inside the submission script before the real execution of the job. It is your responsibility to write proper bash code! This is intended for computer-dependent code, like for instance loading a module that should always be loaded on that specific computer. Remember to end the input by pressing <CTRL>+D. A practical example:

     export NEWVAR=1
     source some/file
     

     A not-to-do example:

     #PBS -l nodes=4:ppn=12
     

     (it’s the plugin that will do this!)

   • Text to append to each command execution: This is a multiline string, whose content will be appended inside the submission script after the real execution of the job. It is your responsibility to write proper bash code! This is intended for computer-dependent code. Remember to end the input by pressing <CTRL>+D.

At the end, you will get a confirmation command, and also the ID in the database (pk, i.e. the principal key, and uuid).

2. Configuration of the computer, using the:

   verdi computer configure COMPUTERNAME
   

   command. This will allow to access more detailed configurations, that are often user-dependent and also depend on the specific transport (for instance, if the transport is SSH, it will ask for username, port, ...).

   The command will try to provide automatically default answers, mainly reading the existing ssh configuration in ~/.ssh/config, and in most cases one simply need to press enter a few times.

   Note

   At the moment, the in-line help (i.e., just typing ? to get some help) is not yet supported in verdi configure, but only in verdi setup.

   For local transport, you need to run the command, even if nothing will be asked to you. For ssh transport, the following will be asked:

   • username: your username on the remote machine
   • port: the port to connect to (the default SSH port is 22)
   • look_for_keys: automatically look for the private key in ~/.ssh. Default: True.
   • key_filename: the absolute path to your private SSH key. You can leave it empty to use the default SSH key, if you set look_for_keys to True.
   • timeout: A timeout in seconds if there is no response (e.g., the machine is down. You can leave it empty to use the default value.
   • allow_agent: If True, it will try to use an SSH agent.
   • proxy_command: Leave empty if you do not need a proxy command (i.e., if you can directly connect to the machine). If you instead need to connect to an intermediate computer first, you need to provide here the command for the proxy: see documentation here for how to use this option, and in particular the notes here for the format of this field.
   • compress: True to compress the traffic (recommended)
   • load_system_host_keys: True to load the known hosts keys from the default SSH location (recommended)
   • key_policy: What is the policy in case the host is not known. It is a string among the following:
     • RejectPolicy (default, recommended): reject the connection if the host is not known.
     • WarningPolicy (not recommended): issue a warning if the host is not known.
     • AutoAddPolicy (not recommended): automatically add the host key at the first connection to the host.

After these two steps have been completed, your computer is ready to go!

verdi computer test COMPUTERNAME

verdi computer list

verdi computer show COMPUTERNAME

verdi computer rename OLDCOMPUTERNAME NEWCOMPUTERNAME

verdi computer delete COMPUTERNAME

verdi computer enable COMPUTERNAME
verdi computer disable COMPUTERNAME

verdi computer disable COMPUTERNAME --only-for-user USER_EMAIL

Code setup and configuration¶

Once you have at least one computer configured, you can configure the codes.

In AiiDA, for full reproducibility of each calculation, we store each code in the database, and attach to each calculation a given code. This has the further advantage to make very easy to query for all calculations that were run with a given code (for instance because I am looking for phonon calculations, or because I discovered that a specific version had a bug and I want to rerun the calculations).

In AiiDA, we distinguish two types of codes: remote codes and local codes, where the distinction between the two is described here below.

verdi code

verdi code setup

verdi code relabel "ID"

verdi code list

verdi code show "ID"

verdi code delete "ID"

Available plugins¶

cif_filter --help
cif_filter --usage

verdi shell¶

By running verdi shell on the terminal, a new interactive IPython shell will be opened (this requires that IPython is installed on your computer).

Note that simply opening IPython and loading the AiiDA modules will not work (unless you perform the operations described in the following section) because the database settings are not loaded by default and AiiDA does not know how to access the database.

Moreover, by calling verdi shell, you have the additional advantage that some classes and modules are automatically loaded, in particular at start-up the following modules are loaded, as described here.

A further advantage is that bash completion is enabled, allowing to press the TAB key to see available submethods of a given object (see for instance the documentation of the ResultManager).

Writing python scripts for AiiDA¶

Alternatively, if you do not need an interactive shell but you prefer to write a script and then launch it from the command line, you can just write a standard python .py file. The only modification that you need to do is to add, at the beginning of the file and before loading any other AiiDA module, the following two lines:

from aiida import load_dbenv
load_dbenv()

that will load the database settings and allow AiiDA to reach your database. Then, you can load as usual python and AiiDA modules and classes, and use them. If you want to have the same environment of the verdi shell interactive shell, you can also add (below the load_dbenv call) the following lines:

from aiida.orm import Calculation, Code, Computer, Data, Node
from aiida.orm import CalculationFactory, DataFactory
from aiida.djsite.db import models

or simply import the only modules that you will need in the script.

While this method will work, we strongly suggest to use instead the verdi run command, described here below.

verdi group list -t autogroup.run

#!/usr/bin/env runaiida
import sys

pk = int(sys.argv[1])
node = load_node(pk)
print "Node {} is: {}".format(pk, repr(node))

import aiida
print "AiiDA version is: {}".format(aiida.get_version())

General comments¶

This section contains an example of how you can use the StructureData object to create complex crystals.

With the StructureData class we did not try to have a full set of features to manipulate crystal structures. Indeed, other libraries such as ASE exist, and we simply provide easy ways to convert between the ASE and the AiiDA formats. On the other hand, we tried to define a “standard” format for structures in AiiDA, that can be used across different codes.

Tutorial¶

Take a look at the following example:

alat = 4. # angstrom
cell = [[alat, 0., 0.,],
        [0., alat, 0.,],
        [0., 0., alat,],
       ]
s = StructureData(cell=cell)
s.append_atom(position=(0.,0.,0.), symbols='Fe')
s.append_atom(position=(alat/2.,alat/2.,alat/2.), symbols='O')

With the commands above, we have created a crystal structure s with a cubic unit cell and lattice parameter of 4 angstrom, and two atoms in the cell: one iron (Fe) atom in the origin, and one oxygen (O) at the center of the cube (this cell has been just chosen as an example and most probably does not exist).

In this way, any periodic structure can be defined. If you want to import from ASE in order to specify the coordinates, e.g., in terms of the crystal lattice vectors, see the guide on the conversion to/from ASE below.

When using the append_atom() method, further parameters can be passed. In particular, one can specify the mass of the atom, particularly important if you want e.g. to run a phonon calculation. If no mass is specified, the mass provided by NIST (retrieved in October 2014) is going to be used. The list of masses is stored in the module aiida.common.constants, in the elements dictionary.

Moreover, in the StructureData class of AiiDA we also support the storage of crystal structures with alloys, vacancies or partial occupancies. In this case, the argument of the parameter symbols should be a list of symbols, if you want to consider an alloy; moreover, you must pass a weights list, with the same length as symbols, and with values between 0. (no occupancy) and 1. (full occupancy), to specify the fractional occupancy of that site for each of the symbols specified in the symbols list. The sum of all occupancies must be lower or equal to one; if the sum is lower than one, it means that there is a given probability of having a vacancy at that specific site position.

As an example, you could use:

s.append_atom(position=(0.,0.,0.),symbols=['Ba','Ca'],weights=[0.9,0.1])

to add a site at the origin of a structure s consisting of an alloy of 90% of Barium and 10% of Calcium (again, just an example).

The following line instead:

s.append_atom(position=(0.,0.,0.),symbols='Ca',weights=0.9)

would create a site with 90% probability of being occupied by Calcium, and 10% of being a vacancy.

Utility methods s.is_alloy() and s.has_vacancies() can be used to verify, respectively, if more than one element if given in the symbols list, and if the sum of all weights is smaller than one.

>>> s = StructureData(cell=[[4,0,0],[0,4,0],[0,0,4]])
>>> s.append_atom(position=(0.,0.,0.), symbols=['Fe', 'O'], weights=[1.,0.])
>>> s.is_alloy()
True

Internals: Kinds and Sites¶

Internally, the append_atom() method works by manipulating the kinds and sites of the current structure. Kinds are instances of the Kind class and represent a chemical species, with given properties (composing element or elements, occupancies, mass, ...) and identified by a label (normally, simply the element chemical symbol).

Sites are instances of the Site class and represent instead each single site. Each site refers to a Kind to identify its properties (which element it is, the mass, ...) and to its three spatial coordinates.

The append_atom() works in the following way:

• It creates a new Kind class with the properties passed as parameters (i.e., all parameters except position).

• It tries to identify if an identical Kind already exists in the list of kinds of the structure (e.g., in the same atom with the same mass was already previously added). Comparison of kinds is performed using aiida.orm.data.structure.Kind.compare_with(), and in particular it returns True if the mass and the list of symbols and of weights are identical (within a threshold). If an identical kind k is found, it simply adds a new site referencing to kind k and with the provided position. Otherwise, it appends k to the list of kinds of the current structure and then creates the site referencing to k. The name of the kind is chosen, by default, equal to the name of the chemical symbol (e.g., “Fe” for iron).

• If you pass more than one species for the same chemical symbol, but e.g. with different masses, a new kind is created and the name is obtained postponing an integer to the chemical symbol name. For instance, the following lines:

  s.append_atom(position = [0,0,0], symbols='Fe', mass = 55.8)
  s.append_atom(position = [1,1,1], symbols='Fe', mass = 57)
  s.append_atom(position = [1,1,1], symbols='Fe', mass = 59)
  

  will automatically create three kinds, all for iron, with names Fe, Fe1 and Fe2, and masses 55.8, 57. and 59. respecively.

• In case of alloys, the kind name is obtained concatenating all chemical symbols names (and a X is the sum of weights is less than one). The same rules as above are used to append a digit to the kind name, if needed.

• Finally, you can simply specify the kind_name to automatically generate a new kind with a specific name. This is the case if you want a name different from the automatically generated one, or for instance if you want to create two different species with the same properties (same mass, symbols, ...). This is for instance the case in Quantum ESPRESSO in order to describe an antiferromagnetic cyrstal, with different magnetizations on the different atoms in the unit cell.

  In this case, you can for instance use:

  s.append_atom(position = [0,0,0], symbols='Fe', mass = 55.845, name='Fe1')
  s.append_atom(position = [2,2,2], symbols='Fe', mass = 55.845, name='Fe2')
  

  To create two species Fe1 and Fe2 for iron, with the same mass.

  Note

  You do not need to specify explicitly the mass if the default one is ok for you. However, when you pass explicitly a name and it coincides with the name of an existing species, all properties that you specify must be identical to the ones of the existing species, or the method will raise an exception.

  Note

  If you prefer to work with the internal Kind and Site classes, you can obtain the same result of the two lines above with:

  from aiida.orm.data.structure import Kind, Site
  s.append_kind(Kind(symbols='Fe', mass=55.845, name='Fe1'))
  s.append_kind(Kind(symbols='Fe', mass=55.845, name='Fe1'))
  s.append_site(Site(kind_name='Fe1', position=[0.,0.,0.]))
  s.append_site(Site(kind_name='Fe2', position=[2.,2.,2.]))
  

Conversion to/from ASE¶

If you have an AiiDA structure, you can get an ase.Atom object by just calling the get_ase method:

ase_atoms = aiida_structure.get_ase()

If instead you have as ASE Atoms object and you want to load the structure from it, just pass it when initializing the class:

StructureData = DataFactory('structure')
# or:
# from aiida.orm.data.structure import StructureData
aiida_structure = StructureData(ase = ase_atoms)

>>> import ase
>>> StructureData = DataFactory("structure")
>>> asecell = ase.Atoms('Fe2')
>>> asecell[0].mass = 55.
>>> asecell[1].mass = 56.
>>> s = StructureData(ase=asecell)
>>> for kind in s.kinds:
>>>     print kind.name, kind.mass
Fe 55.0
Fe1 56.0

>>> import ase
>>> StructureData = DataFactory("structure")
>>> asecell = ase.Atoms('Fe2')
>>> asecell[0].tag = 1
>>> asecell[1].tag = 2
>>> s = StructureData(ase=asecell)
>>> for kind in s.kinds:
>>>     print kind.name
Fe1
Fe2

Your classic pw.x input file¶

This is the input file of Quantum Espresso that we will try to execute. It consists in the total energy calculation of a 5 atom cubic cell of BaTiO3. Note also that AiiDA is a tool to use other codes: if the following input is not clear to you, please refer to the Quantum Espresso Documentation.

&CONTROL
  calculation = 'scf'
  outdir = './out/'
  prefix = 'aiida'
  pseudo_dir = './pseudo/'
  restart_mode = 'from_scratch'
  verbosity = 'high'
  wf_collect = .true.
/
&SYSTEM
  ecutrho =   2.4000000000d+02
  ecutwfc =   3.0000000000d+01
  ibrav = 0
  nat = 5
  ntyp = 3
/
&ELECTRONS
  conv_thr =   1.0000000000d-06
/
ATOMIC_SPECIES
Ba     137.33    Ba.pbesol-spn-rrkjus_psl.0.2.3-tot-pslib030.UPF
Ti     47.88     Ti.pbesol-spn-rrkjus_psl.0.2.3-tot-pslib030.UPF
O      15.9994   O.pbesol-n-rrkjus_psl.0.1-tested-pslib030.UPF
ATOMIC_POSITIONS angstrom
Ba           0.0000000000       0.0000000000       0.0000000000
Ti           2.0000000000       2.0000000000       2.0000000000
O            2.0000000000       2.0000000000       0.0000000000
O            2.0000000000       0.0000000000       2.0000000000
O            0.0000000000       2.0000000000       2.0000000000
K_POINTS automatic
4 4 4 0 0 0
CELL_PARAMETERS angstrom
      4.0000000000       0.0000000000       0.0000000000
      0.0000000000       4.0000000000       0.0000000000
      0.0000000000       0.0000000000       4.0000000000

In the old way, not only you had to prepare ‘manually’ this file, but also prepare the scheduler submission script, send everything on the cluster, etc. We are going instead to prepare everything in a more programmatic way.

Quantum Espresso Pw Walkthrough¶

We’ve got to prepare a script to submit a job to your local installation of AiiDA. This example will be a rather long script: in fact there is still nothing in your database, so that we will have to load everything, like the pseudopotential files and the structure. In a more practical situation, you might load data from the database and perform a small modification to re-use it.

Let’s say that through the verdi command you have already installed a cluster, say TheHive, and that you also compiled Quantum Espresso on the cluster, and installed the code pw.x with verdi with label pw-5.1 for instance, so that in the rest of this tutorial we will reference to the code as pw-5.1@TheHive.

Let’s start writing the python script. First of all, we need to load the configuration concerning your particular installation, in particular, the details of your database installation:

#!/usr/bin/env python
from aiida import load_dbenv
load_dbenv()

Code¶

Now we have to select the code. Note that in AiiDA the object ‘code’ in the database is meant to represent a specific executable, i.e. a given compiled version of a code. This means that if you install Quantum Espresso (QE) on two computers A and B, you will need to have two different ‘codes’ in the database (although the source of the code is the same, the binary file is different).

If you setup the code pw-5.1 on machine TheHive correctly, then it is sufficient to write:

codename = 'pw-5.1@TheHive'
from aiida.orm import Code
code = Code.get_from_string(codename)

Where in the last line we just load the database object representing the code.

code = Code.get(label='pw-5.1', machinename='TheHive',
                useremail='user@domain.com')

code = Code.get_subclass_from_pk(PK)

Structure¶

We now proceed in setting up the structure.

There are two ways to do that in AiiDA, a first one is to use the AiiDA Structure, which we will explain in the following; the second choice is the Atomic Simulation Environment (ASE) which provides excellent tools to manipulate structures (the ASE Atoms object needs to be converted into an AiiDA Structure, see the note at the end of the section).

We first have to load the abstract object class that describes a structure. We do it in the following way: we load the DataFactory, which is a tool to load the classes by their name, and then call StructureData the abstract class that we loaded. (NB: it’s not yet a class instance!) (If you are not familiar with the terminology of object programming, we could take Wikipedia and see their short explanation: in common speech that one refers to a file as a class, while the file is the object or the class instance. In other words, the class is our definition of the object Structure, while its instance is what will be saved as an object in the database):

from aiida.orm import DataFactory
StructureData = DataFactory('structure')

We define the cell with a 3x3 matrix (we choose the convention where each ROW represents a lattice vector), which in this case is just a cube of size 4 Angstroms:

alat = 4. # angstrom
cell = [[alat, 0., 0.,],
        [0., alat, 0.,],
        [0., 0., alat,],
       ]

Now, we create the StructureData instance, assigning immediately the cell. Then, we append to the empty crystal cell the atoms, specifying their element name and their positions:

# BaTiO3 cubic structure
s = StructureData(cell=cell)
s.append_atom(position=(0.,0.,0.),symbols='Ba')
s.append_atom(position=(alat/2.,alat/2.,alat/2.),symbols='Ti')
s.append_atom(position=(alat/2.,alat/2.,0.),symbols='O')
s.append_atom(position=(alat/2.,0.,alat/2.),symbols='O')
s.append_atom(position=(0.,alat/2.,alat/2.),symbols='O')

To see more methods associated to the class StructureData, look at the Structure documentation.

s.store()

For an extended tutorial about the creation of Structure objects, check this tutorial.

s = StructureData(ase=ase_s)

Parameters¶

Now we need to provide also the parameters of a Quantum Espresso calculation, like the cutoff for the wavefunctions, some convergence threshold, etc... The Quantum ESPRESSO pw.x plugin requires to pass this information within a ParameterData object, that is a specific AiiDA data node that can store a dictionary (even nested) of basic data types: integers, floats, strings, lists, dates, ... We first load the class through the DataFactory, just like we did for the Structure. Then we create the instance of the object parameter. To represent closely the structure of the QE input file, ParameterData is a nested dictionary, at the first level the namelists (capitalized), and then the variables with their values (in lower case).

Note also that numbers and booleans are written in Python, i.e. False and not the Fortran string .false.!

ParameterData = DataFactory('parameter')

parameters = ParameterData(dict={
          'CONTROL': {
              'calculation': 'scf',
              'restart_mode': 'from_scratch',
              'wf_collect': True,
              },
          'SYSTEM': {
              'ecutwfc': 30.,
              'ecutrho': 240.,
              },
          'ELECTRONS': {
              'conv_thr': 1.e-6,
              }})

parameters = ParameterData(dict={...}).store()

The experienced QE user will have noticed also that a couple of variables are missing: the prefix, the pseudo directory and the scratch directory are reserved to the plugin which will use default values, and there are specific AiiDA methods to restart from a previous calculation.

The k-points have to be saved in another kind of data, namely KpointsData:

KpointsData = DataFactory('array.kpoints')
kpoints = KpointsData()
kpoints.set_kpoints_mesh([4,4,4])

In this case it generates a 4*4*4 mesh without offset. To add an offset one can replace the last line by:

kpoints.set_kpoints_mesh([4,4,4],offset=(0.5,0.5,0.5))

You can also specify kpoints manually, by inputing a list of points in crystal coordinates (here they all have equal weights):

import numpy
kpoints.set_kpoints([[i,i,0] for i in numpy.linspace(0,1,10)],
    weights = [1. for i in range(10)])

settings = ParameterData(dict={'gamma_only':True})

kpoints.set_kpoints_mesh([1,1,1])

calc.use_settings(settings)

As a further comment, this is specific to the way the plugin for Quantum Espresso works. Other codes may need more than two ParameterData, or even none of them. And also how this parameters have to be written depends on the plugin: what is discussed here is just the format that we decided for the Quantum Espresso plugins.

Calculation¶

Now we proceed to set up the calculation. Since during the setup of the code we already set the code to be a quantumespresso.pw code, there is a simple method to create a new calculation:

calc = code.new_calc()

We have to specify the details required by the scheduler. For example, on a SLURM or PBS scheduler, we have to specify the number of nodes (num_machines), possibly the number of MPI processes per node (num_mpiprocs_per_machine) if we want to run with a different number of MPI processes with respect to the default value configured when setting up the computer in AiiDA, the job walltime, the queue name (if desired), ...:

calc.set_max_wallclock_seconds(30*60) # 30 min
calc.set_resources({"num_machines": 1})
## OPTIONAL, use only if you need to explicitly specify a queue name
# calc.set_queue_name("the_queue_name")

(For the complete scheduler documentation, see Supported schedulers)

calc = code.new_calc()
calc.set_max_wallclock_seconds(3600)
calc.set_resources({"num_machines": 1})

calc = code.new_calc(max_wallclock_seconds=3600,
    resources={"num_machines": 1})

At this point, we just created a “lone” calculation, that still does not know anything about the inputs that we created before. We need therefore to tell the calculation to use the parameters that we prepared before, by properly linking them using the use_ methods:

calc.use_structure(s)
calc.use_code(code)
calc.use_parameters(parameters)
calc.use_kpoints(kpoints)

In practice, when you say calc.use_structure(s), you are setting a link between the two nodes (s and calc), that means that s is the input structure for calculation calc. Also these links are cached and do not require to store anything in the database yet.

In the case of the gamma-only computation (see above), you also need to add:

calc.use_settings(settings)

Pseudopotentials¶

There is still one missing piece of information, that is the pseudopotential files, one for each element of the structure.

In AiiDA, it is possible to specify manually which pseudopotential files to use for each atomic species. However, for any practical use, it is convenient to use the pseudopotential families. Its use is documented in Pseudopotential families tutorial. If you got one installed, you can simply tell the calculation to use the pseudopotential family with a given name, and AiiDA will take care of linking the proper pseudopotentials to the calculation, one for each atomic species present in the input structure. This can be done using:

calc.use_pseudos_from_family('my_pseudo_family')

Labels and comments¶

Sometimes it is useful to attach some notes to the calculation, that may help you later understand why you did such a calculation, or note down what you understood out of it. Comments are a special set of properties of the calculation, in the sense that it is one of the few properties that can be changed, even after the calculation has run.

Comments come in various flavours. The most basic one is the label property, a string of max 255 characters, which is meant to be the title of the calculation. To create it, simply write:

calc.label = "A generic title"

The label can be later accessed as a class property, i.e. the command:

calc.label

will return the string you previously set (empty by default). Another important property to set is the description, which instead does not have a limitation on the maximum number of characters:

calc.description = "A much longer description"

And finally, there is the possibility to add comments to any calculation (actually, to any node). The peculiarity of comments is that they are user dependent (like the comments that you can post on facebook pages), so it is best suited to calculation exposed on a website, where you want to remember the comments of each user. To set a comment, you need first to import the django user, and then write it with a dedicated method:

from aiida.djsite.utils import get_automatic_user
calc.add_comment("Some comment", user=get_automatic_user())

The comments can be accessed with this function:

calc.get_comments_tuple()

Execute¶

If we are satisfied with what you created, it is time to store everything in the database. Note that after storing it, it will not be possible to modify it (nor you should: you risk of compromising the integrity of the database)!

Unless you already stored all the inputs beforehand, you will need to store the inputs before being able to store the calculation itself. Since this is a very common operation, there is an utility method that will automatically store both all the input nodes of calc and then calc itself:

calc.store_all()

Once we store the calculation, it is useful to print its PK (principal key, that is its identifier) that is useful in the following to interact with it:

print "created calculation; with uuid='{}' and PK={}".format(calc.uuid,calc.pk)

Summarizing, we created all the inputs needed by a PW calculation, that are: parameters, kpoints, pseudopotential files and the structure. We then created the calculation, where we specified that it is a PW calculation and we specified the details of the remote cluster. We set the links between the inputs and the calculation (calc.use_***) and finally we stored all this objects in the database (.store_all()).

That’s all that the calculation needs. Now we just need to submit it:

calc.submit()

Everything else will be managed by AiiDA: the inputs will be checked to verify that it is consistent with a PW input. If the input is complete, the pw input file will be prepared in a folder together with all the other files required for the execution (pseudopotentials, etc.). It will be then sent on cluster, submitted, and after execution automatically retrieved and parsed.

To know how to monitor and check the state of submitted calculations, go to Calculations.

To continue the tutorial with the ph.x phonon code of Quantum ESPRESSO, continue here: Quantum Espresso Phonon user-tutorial.

Script: source code¶

In this section you’ll find two scripts that do what explained in the tutorial. The compact is a script with a minimal configuration required. You can copy and paste it (or download it), modify the two strings codename and pseudo_family with the correct values, and execute it with:

python pw_short_example.py

(It requires to have one family of pseudopotentials configured).

You will also find a longer version, with more exception checks, error management and user interaction. Note that the configuration of the computer resources (like number of nodes and machines) is hardware and scheduler dependent. The configuration used below should work for a pbspro or slurm cluster, asking to run on 1 node only.

Compact script¶

Download: this example script

#!/usr/bin/env python
from aiida import load_dbenv
load_dbenv()

from aiida.orm import Code, DataFactory
StructureData = DataFactory('structure')
ParameterData = DataFactory('parameter')
KpointsData = DataFactory('array.kpoints')

###############################
# Set your values here
codename = 'pw-5.1@TheHive'
pseudo_family = 'lda_pslibrary'
###############################

code = Code.get_from_string(codename)

# BaTiO3 cubic structure
alat = 4. # angstrom
cell = [[alat, 0., 0.,],
        [0., alat, 0.,],
        [0., 0., alat,],
       ]
s = StructureData(cell=cell)
s.append_atom(position=(0.,0.,0.),symbols='Ba')
s.append_atom(position=(alat/2.,alat/2.,alat/2.),symbols='Ti')
s.append_atom(position=(alat/2.,alat/2.,0.),symbols='O')
s.append_atom(position=(alat/2.,0.,alat/2.),symbols='O')
s.append_atom(position=(0.,alat/2.,alat/2.),symbols='O')

parameters = ParameterData(dict={
          'CONTROL': {
              'calculation': 'scf',
              'restart_mode': 'from_scratch',
              'wf_collect': True,
              },
          'SYSTEM': {
              'ecutwfc': 30.,
              'ecutrho': 240.,
              },
          'ELECTRONS': {
              'conv_thr': 1.e-6,
              }})

kpoints = KpointsData()
kpoints.set_kpoints_mesh([4,4,4])

calc = code.new_calc(max_wallclock_seconds=3600,
    resources={"num_machines": 1})
calc.label = "A generic title"
calc.description = "A much longer description"

calc.use_structure(s)
calc.use_code(code)
calc.use_parameters(parameters)
calc.use_kpoints(kpoints)
calc.use_pseudos_from_family(pseudo_family)

calc.store_all()
print "created calculation with PK={}".format(calc.pk)
calc.submit()

Exception tolerant code¶

You can find a more sophisticated example, that checks the possible exceptions and prints nice error messages inside your AiiDA folder, under examples/submission/test_pw.py.

Imports and database environement¶

As usual, we load the database environment and load the PwimmigrantCalculation class using the CalculationFactory.

from aiida import load_dbenv
from aiida.orm.code import Code
from aiida.orm import CalculationFactory

# Load the database environment.
load_dbenv()

# Load the PwimmigrantCalculation class.
PwimmigrantCalculation = CalculationFactory('quantumespresso.pwimmigrant')

Code, computer, and resources¶

Next, we load the code and computer that have already been configured to be representative of those used to perform the calculation. We also define the resources representive of those that were used to run the calculation.

# Load the Code node representative of the one used to perform the calculations.
code = Code.get('pw_on_TheHive')

# Get the Computer node representative of the one the calculations were run on.
computer = code.get_remote_computer()

# Define the computation resources used for the calculations.
resources = {'num_machines': 1, 'num_mpiprocs_per_machine': 1}

Initialization of the calculation¶

Now, we are ready to initialize the immigrated calculation objects from the PwimmigrantCalculation class. We will pass the necessary parameters as keywords during the initialization calls. Then, we link the code from above as an input node.

# Initialize the pw_job1 calculation node.
calc1 = PwimmigrantCalculation(computer=computer,
                               resources=resources,
                               remote_workdir='/scratch/',
                               input_file_name='pw_job1.in',
                               output_file_name='pw_job1.out')

# Initialize the pw_job2 calculation node.
calc2 = PwimmigrantCalculation(computer=computer,
                               resources=resources,
                               remote_workdir='/scratch/',
                               input_file_name='pw_job2.in',
                               output_file_name='pw_job2.out')

# Link the code that was used to run the calculations.
calc1.use_code(code)
calc2.use_code(code)

The user may have noticed the additional initialization keywords/parameters–remote_wordir, input_file_name, and output_file_name–passed here. These are necessary in order to tell AiiDA which files to use to automatically generate the calculation`s input nodes in the next step.

The immigration¶

Now that AiiDA knows where to look for the input files of the calculations we are immigrating, all we need to do in order to generate all the input nodes is call the create_input_nodes method. This method is the most helpful method of the PwimmigrantCalculation class. It parses the job’s input file and creates and links the follow types of input nodes:

• ParameterData – based on the namelists and their variable-value pairs
• KpointsData – based on the K_POINTS card
• SturctureData – based on the ATOMIC_POSITIONS and CELL_PARAMETERS cards (and the a or celldm(1) of the &SYSTEM namelist, if alat is specified through these variables)
• UpfData – one for each of the atomic species, based on the pseudopotential files specified in the ATOMIC_SPECIES card
• settings ParameterData – if there are any fixed coordinates, or if the gamma kpoint is used

All units conversion and/or coordinate transformations are handled automatically, and the input nodes are generated in the correct units and coordinates required by AiiDA.

Finally, the last step of the immigration is to call the prepare_for_retrieval_and_parsing method. This method stores the calculation and it’s input nodes in the database, copies the original input file to the calculation’s repository folder, and then tells the daemon to retrieve and parse the calculation’s output files.

Note

If the daemon is not currently running, the retrieval and parsing process will not begin until it is started.

Because the input and pseudopotential files need to be retrieved from the computer, the computer’s transport plugin needs to be open. Rather than opening and closing the transport for each calculation, we instead require the user to pass an open transport instance as a parameter to the create_input_nodes and prepare_for_retrieval_and_parsing methods. This minimizes the number of transport opening and closings, which is highly beneficial when immigrating a large number of calculations.

Calling these methods with an open transport is performed as follows:

# Get the computer's transport and create an instance.
Transport = computer.get_transport_class()
transport = Transport()

# Open the transport for the duration of the immigrations, so it's not
# reopened for each one. This is best performed using the transport's
# context guard through the ``with`` statement.
with transport as open_transport:

    # Parse the calculations' input files to automatically generate and link the
    # calculations' input nodes.
    calc1.create_input_nodes(open_transport)
    calc2.create_input_nodes(open_transport)

    # Store the calculations and their input nodes and tell the daeomon the output
    # is ready to be retrieved and parsed.
    calc1.prepare_for_retrieval_and_parsing(open_transport)
    calc2.prepare_for_retrieval_and_parsing(open_transport)

The process above is easily expanded to large-scale immigrations of multiple jobs.

Compact script¶

Download: this example script

#!/usr/bin/env python
from aiida import load_dbenv
from aiida.orm.code import Code
from aiida.orm import CalculationFactory

# Load the database environment.
load_dbenv()

# Load the PwimmigrantCalculation class.
PwimmigrantCalculation = CalculationFactory('quantumespresso.pwimmigrant')

# Load the Code node representative of the one used to perform the calculations.
code = Code.get('pw_on_TheHive')

# Get the Computer node representative of the one the calculations were run on.
computer = code.get_remote_computer()

# Define the computation resources used for the calculations.
resources = {'num_machines': 1, 'num_mpiprocs_per_machine': 1}

# Initialize the pw_job1 calculation node.
calc1 = PwimmigrantCalculation(computer=computer,
                               resources=resources,
                               remote_workdir='/scratch/',
                               input_file_name='pw_job1.in',
                               output_file_name='pw_job1.out')

# Initialize the pw_job2 calculation node.
calc2 = PwimmigrantCalculation(computer=computer,
                               resources=resources,
                               remote_workdir='/scratch/',
                               input_file_name='pw_job2.in',
                               output_file_name='pw_job2.out')

# Link the code that was used to run the calculations.
calc1.use_code(code)
calc2.use_code(code)

# Get the computer's transport and create an instance.
Transport = computer.get_transport_class()
transport = Transport()

# Open the transport for the duration of the immigrations, so it's not
# reopened for each one. This is best performed using the transport's
# context guard through the ``with`` statement.
with transport as open_transport:

    # Parse the calculations' input files to automatically generate and link the
    # calculations' input nodes.
    calc1.create_input_nodes(open_transport)
    calc2.create_input_nodes(open_transport)

    # Store the calculations and their input nodes and tell the daeomon the output
    # is ready to be retrieved and parsed.
    calc1.prepare_for_retrieval_and_parsing(open_transport)
    calc2.prepare_for_retrieval_and_parsing(open_transport)

Walkthrough¶

This input is much simpler than the previous PWscf work, here the only novel thing you will have to learn is how to set a parent calculation.

As before, we write a script step-by-step.

We first load a couple of useful modules that you already met in the previous tutorial, and load the database settings:

#!/usr/bin/env python
from aiida import load_dbenv
load_dbenv()

from aiida.orm import Code
from aiida.orm import CalculationFactory, DataFactory

So, you were able to launch previously a pw.x calculation.

Code¶

Again, you need to have compiled the code on the cluster and configured a new code ph.x in AiiDA in the very same way you installed pw.x (see ....). Then we load the Code class-instance from the database:

codename = 'my-ph.x'
code = Code.get_from_string(codename)

Parameter¶

Just like the PWscf calculation, here we load the class ParameterData and we instanciate it in parameters. Again, ParameterData will simply represent a nested dictionary in the database, namelists at the first level, and then variables and values. But this time of course, we need to use the variables of PHonon!

ParameterData = DataFactory('parameter')
parameters = ParameterData(dict={
            'INPUTPH': {
                'tr2_ph' : 1.0e-8,
                'epsil' : True,
                'ldisp' : True,
                'nq1' : 1,
                'nq2' : 1,
                'nq3' : 1,
                }})

Calculation¶

Now we create the object PH-calculation. As for pw.x, we simply do:

calc = code.new_calc()

and we set the parameters of the scheduler (and just like the PWscf, this is a configuration valid for the PBSpro and slurm schedulers only, see Supported schedulers).

calc.set_max_wallclock_seconds(30*60) # 30 min
calc.set_resources({"num_machines": 1})

We then tell the calculation to use the code and the parameters that we prepared above:

calc.use_parameters(parameters)

Parent calculation¶

The phonon calculation needs to know on which PWscf do the perturbation theory calculation. From the database point of view, it means that the PHonon calculation is always a child of a PWscf. In practice, this means that when you want to impose this relationship, you decided to take the input parameters of the parent PWscf calculation, take its charge density and use them in the phonon run. That’s way we need to set the parent calculation.

You first need to remember the ID of the parent calculation that you launched before (let’s say it’s #6): so that you can load the class of a QE-PWscf calculation (with the CalculationFactory), and load the object that represent the QE-PWscf calculation with ID #6:

from aiida.orm import CalculationFactory
PwCalculation = CalculationFactory('quantumespresso.pw')
parent_id = 6
parentcalc = load_node(parent_id)

Now that we loaded the parent calculation, we can set the phonon calc to inherit the right information from it:

calc.set_parent_calc( parentcalc )

Note that in our database schema relations between two calculation objects are prohibited. The link between the two is indirect and is mediated by a third Data object, which represent the scratch folder on the remote cluster. Therefore the relation between the parent Pw and the child Ph appears like: Pw -> remotescratch -> Ph.

Execution¶

Now, everything is ready, and just like PWscf, you just need to store all the nodes and submit this input to AiiDA, and the calculation will launch!

calc.store_all()
calc.submit()

Script to execute¶

This is the script described in the tutorial above. You can use it, just remember to customize it using the right parent_id, the code, and the proper scheduler info.

#!/usr/bin/env python
from aiida import load_dbenv
load_dbenv()

from aiida.orm import Code
from aiida.orm import CalculationFactory, DataFactory

#####################
# ADAPT TO YOUR NEEDS
parent_id = 6
codename = 'my-ph.x'
#####################

code = Code.get_from_string(codename)

ParameterData = DataFactory('parameter')
parameters = ParameterData(dict={
            'INPUTPH': {
                'tr2_ph' : 1.0e-8,
                'epsil' : True,
                'ldisp' : True,
                'nq1' : 1,
                'nq2' : 1,
                'nq3' : 1,
                }})

QEPwCalc = CalculationFactory('quantumespresso.pw')
parentcalc = load_node(parent_id)

calc = code.new_calc()
calc.set_max_wallclock_seconds(30*60) # 30 min
calc.set_resources({"num_machines": 1})

calc.use_parameters(parameters)
calc.use_code(code)
calc.set_parent_calc(parentcalc)

calc.store_all()
print "created calculation with PK={}".format(calc.pk)
calc.submit()

Exception tolerant code¶

You can find a more sophisticated example, that checks the possible exceptions and prints nice error messages inside your AiiDA folder, under examples/submission/test_ph.py.

Walkthrough¶

Everything works like the PW calculation: you need to get the code from the database:

codename = 'my_cp'
code = Code.get_from_string(codename)

Then create the StructureData with the structure, and a ParameterData node for the inputs. This time, of course, you have to specify the correct variables for a cp.x calculation:

StructureData = DataFactory('structure')
alat = 4. # angstrom
cell = [[alat, 0., 0.,],
        [0., alat, 0.,],
        [0., 0., alat,],
       ]
s = StructureData(cell=cell)
s.append_atom(position=(0.,0.,0.),symbols=['Ba'])
s.append_atom(position=(alat/2.,alat/2.,alat/2.),symbols=['Ti'])
s.append_atom(position=(alat/2.,alat/2.,0.),symbols=['O'])
s.append_atom(position=(alat/2.,0.,alat/2.),symbols=['O'])
s.append_atom(position=(0.,alat/2.,alat/2.),symbols=['O'])

ParameterData = DataFactory('parameter')
parameters = ParameterData(dict={
          'CONTROL': {
              'calculation': 'cp',
              'restart_mode': 'from_scratch',
              'wf_collect': False,
              'iprint': 1,
              'isave': 100,
              'dt': 3.,
              'max_seconds': 25*60,
              'nstep': 10,
              },
          'SYSTEM': {
              'ecutwfc': 30.,
              'ecutrho': 240.,
              'nr1b': 24,
              'nr2b': 24,
              'nr3b': 24,
              },
          'ELECTRONS': {
              'electron_damping': 1.e-1,
              'electron_dynamics': 'damp',
              'emass': 400.,
              'emass_cutoff': 3.,
              },
          'IONS': {
              'ion_dynamics': 'none',
          }}).store()

We then create a new calculation with the proper settings:

calc = code.new_calc()
calc.set_max_wallclock_seconds(30*60) # 30 min
calc.set_resources({"num_machines": 1, "num_mpiprocs_per_machine": 16})

And we link the input data to the calculation (and therefore set the links in the database). The main difference here is that CP does not support k-points, so you should not (and cannot) link any kpoint as input:

calc.use_structure(s)
calc.use_code(code)
calc.use_parameters(parameters)

Finally, load the proper pseudopotentials using e.g. a pseudopotential family (see Pseudopotential families tutorial):

pseudo_family = 'lda_pslib'
calc.use_pseudos_from_family(pseudo_family)

and store everything and submit:

calc.store_all()
calc.submit()

And now, the calculation will be executed and saved in the database automatically.

Exception tolerant code¶

You can find a more sophisticated example, that checks the possible exceptions and prints nice error messages inside your AiiDA folder, under examples/submission/test_cp.py.

The CalculationResultManager¶

calc = load_node(YOURPK)

Each JobCalculation has a res attribute that is a CalculationResultManager object and gives direct access to parsed data.

To use it, you can just use then:

calc.res

that will however just return the class. You can however convert it to a list, to get all the possible keys that were parsed. For instance, if you type:

print list(calc.res)

you will get something like this:

[u'rho_cutoff', u'energy', u'energy_units', ...]

(the list of keys has been cut for clarity: you will get many more keys).

Once you know which keys have been parsed, you can access the parsed value simply as an attribute of the res ResultManager. For instance, to get the final total energy, you can use:

print calc.res.energy

that will print the total energy in units of eV, as also stated in the energy_units key:

print calc.res.energy_units

Similarly, you can get any other parsed value, for any code that provides a parser.

calc.res.

What is a pseudopotential family¶

As you might have seen in the previous PWscf tutorial, the procedure of attaching a pseudopotential file to each atomic species could be a bit tedious. In many situations, you will not produce a different pseudopotential file for every calculation you do. More likely, when you start a project you will stick to a pseudopotential file for as long as possible. Moreover, in a high-throughput calculation, you will like to do calculation over several elements keeping the same functional. That’s also part of the reason why there are several projects (like PSLibrary or GBRV to name a few), that intend to develop a set of pseudopotentials that covers most of the periodic table for different functionals.

That’s why we introduced the pseudopotential families. They are basically a set of pseudopotentials that are grouped together in a special type of AiiDA Group of nodes, with the requirement that at most one pseudopotential can be present for a given chemical element.

Of course, no requirements are enforced on the complete coverage of the periodic table (also because really complete pseudopotential sets for the whole periodic table do not exist). In other words, this means that you can create a pseudopotential family containing the pseudopotentials only for a few elements that you are interested in.

How to create a pseudopotential family¶

Let’s say for example that we want to create a family of LDA ultrasoft pseudopotentials. As the first step, you need to get all the pseudopotential files in a single folder. For your convenience, it is useful to use a common name for your files, for example with a structure like ‘Element.a-short-description.UPF’.

The utility to upload a family of pseudopotentials is accessed via verdi:

verdi data upf uploadfamily path/to/folder name_of_the_family "some description for your convenience"

where path/to/folder is the path to the folder where you collected all the UPF files that you want to add to the AiiDA database and to the family with name name_of_the_family, and the final parameter is a string that is set in the description field of the group.

After the upload (which may take some seconds, so please be patient) the upffamily will be ready to be used.

Note that if you pass as name_of_the_family a name that already exists, the pseudopotentials in the folder will be added to the existing group. The code will raise an error if you try to add two (different) pseudopotentials for the same element.

Get the list of existing families¶

If you want to know what are the pseudopotential families already existing in the DB, type:

verdi data upf listfamilies

Add a -d (or --with-description) flag if you want to read also the description of the family.

You can also filter the groups to get only a list of those containing a set of given elements using the -e option. For instance, if you want to get only the families containing the elements Ba, Ti and O, use:

verdi data upf listfamilies -e Ba Ti O

For more help on the command line options, type:

verdi data upf listfamilies -h

verdi calculation¶

• kill: stop the execution on the cluster of a calculation.
• logshow: shows the logs/errors produced by a calculation
• plugins: lists the supported calculation plugins
• inputcat: shows an input file of a calculation node.
• inputls: shows the list of the input files of a calculation node.
• list: list the AiiDA calculations. By default, lists only the running calculations.
• outputcat: shows an ouput file of a calculation node.
• outputls: shows the list of the output files of a calculation node.
• show: shows the database information related to the calculation: used code, all the input nodes and all the output nodes.
• gotocomputer: open a shell to the calc folder on the cluster
• label: view / set the label of a calculation
• description: view / set the description of a calculation

verdi code¶

• show: shows the information of the installed code.
• list: lists the installed codes
• hide: hide codes from verdi code list
• reveal: un-hide codes for verdi code list
• setup: setup a new code
• relabel: change the label (name) of a code. If you like to load codes based on their labels and not on their UUID’s or PK’s, take care of using unique labels!
• update: change (some of) the installation description of the code given at the moment of the setup.
• delete: delete a code from the database. Only possible for disconnected codes (i.e. a code that has not been used yet)

verdi comment¶

Manages the comments attached to a database node.

• add: add a new comment
• update: change an existing comment
• remove: remove a comment
• show: show the comments attached to a node.

verdi completioncommand¶

Prints the string to be copied and pasted to the bashrc in order to allow for autocompletion of the verdi commands.

verdi computer¶

• setup: creates a new computer object
• configure: set up some extra info that can be used in the connection with that computer.
• enable: to enable a computer. If the computer is disabled, the daemon will not try to connect to the computer, so it will not retrieve or launch calculations. Useful if a computer is under mantainance.
• rename: changes the name of a computer.
• disable: disable a computer (see enable for a larger description)
• show: shows the details of an installed computer
• list: list all installed computers
• delete: deletes a computer node. Works only if the computer node is a disconnected node in the database (has not been used yet)
• test: tests if the current user (or a given user) can connect to the computer and if basic operations perform as expected (file copy, getting the list of jobs in the scheduler queue, ...)

verdi daemon¶

Manages the daemon, i.e. the process that runs in background and that manages submission/retrieval of calculations.

• status: see the status of the daemon. Typically, it will either show Daemon not running or you will see two processes with state RUNNING.
• stop: stops the daemon
• configureuser: sets the user which is running the daemon. See the installation guide for more details.
• start: starts the daemon.
• logshow: show the last lines of the daemon log (use for debugging)
• restart: restarts the daemon.

verdi data¶

Manages database data objects.

• upf: handles the Pseudopotential Datas
  • listfamilies: list presently stored families of pseudopotentials
  • uploadfamily: install a new family (group) of pseudopotentials
  • import: create or return (if already present) a database node, having the contents of a supplied file
• structure: handles the StructureData
  • list: list currently saved nodes of StructureData kind
  • show: use a third-party visualizer (like vmd or xcrysden) to graphically show the StructureData
  • export: export the node as a string of a specified format
• parameter: handles the ParameterData objects
  • show: output the content of the python dictionary in different formats.
• cif: handles the CifData objects
  • list: list currently saved nodes of CifData kind
  • show: use third-party visualizer (like jmol) to graphically show the CifData
  • import: create or return (if already present) a database node, having the contents of a supplied file
  • export: export the node as a string of a specified format
• trajectory: handles the TrajectoryData objects
  • list: list currently saved nodes of TrajectoryData kind
  • show: use third-party visualizer (like jmol) to graphically show the TrajectoryData
  • export: export the node as a string of a specified format
• label: view / set the label of a data
• description: view / set the description of a data

verdi devel¶

Here there are some functions that are in the development stage, and that might eventually find their way outside of this placeholder. As such, they are buggy, possibly difficult to use, not necessarily documented, and they might be subject to non back-compatible changes.

verdi export¶

Export data from the AiiDA database to a file. See also verdi import to import this data on another database.

verdi group¶

• list: list all the groups in the database.

verdi import¶

Imports data (coming from other AiiDA databases) in the current database

verdi install¶

Used in the installation to configure the database. If it finds an already installed database, it updates the tables migrating them to the new schema.

verdi node¶

• repo: Show files and their contents in the local repository
• show: Show basic node information (PK, UUID, class, inputs and outputs)

verdi run¶

Run a python script for AiiDA. This is the command line equivalent of the verdi shell. Has also features of autogroupin: by default, every node created in one a call of verdi run will be grouped together.

verdi runserver¶

Starts a lightweight Web server for development and also serves static files. Currently in ongoing development.

verdi shell¶

Runs a Python interactive interpreter. Tries to use IPython or bpython, if one of them is available. Loads on start a good part of the AiiDA infrastructure.

verdi user¶

Manages the AiiDA users. Two valid subcommands.

• list: list existing users configured for your AiiDA installation.
• configure: configure a new AiiDA user.

verdi workflow¶

Manages the workflow. Valid subcommands:

• report: display the information on how the workflow is evolving.
• kill: kills a workflow.
• list: lists the workflows present in the database. By default, shows only the running ones.

Supported schedulers¶

The list below describes the supported schedulers, i.e. the batch job schedulers that manage the job queues and execution on any given computer.

Job resources¶

When asking a scheduler to allocate some nodes/machines for a given job, we have to specify some job resources (that typically include information as, for instance, the number of required nodes or the numbers of MPI processes per node).

Unfortunately, the way of specifying this piece of information is different on different clusters. Instead of having one only abstract class, we chose to adopt different subclasses, keeping in this way the specification of the resources as similar as possible to what the user would do when writing a scheduler script. Note that only one subclass can be used, given a specific scheduler.

The base class, from which all job resource subclasses inherit, is aiida.scheduler.datastructures.JobResource. All classes define at least one method, get_tot_num_mpiprocs(), that returns the total number of MPI processes requested.

from aiida.scheduler.datastructures import NodeNumberJobResource

calc = JobCalculation(computer=...) # select here a given computer configured
                                 # in AiiDA

# This assumes that the computer is configured to use a scheduler with
# job resources of type NodeNumberJobResource
calc.set_resources({"num_machines": 4, "num_mpiprocs_per_machine": 16})

res = NodeNumberJobResource()
res.num_machines = 4
res.num_mpiprocs_per_machine = 16

res = NodeNumberJobResource(num_machines=4, num_mpiprocs_per_machine=16)

calc.set_resources({"num_machines": 4, "num_mpiprocs_per_machine": 16})

res.num_machines * res.num_mpiprocs_per_machine = res.tot_num_mpiprocs

The verdi calculation command¶

The simplest way to check the state of submitted calculations is to use the verdi calculation list command from the command line. To get help on its use and command line options, run it with the -h or --help option:

verdi calculation list --help

Directly in python¶

If you prefer to have more flexibility or to check the state of a calculation programmatically, you can execute a script like the following, where you just need to specify the ID of the calculation you are interested in:

from aiida import load_dbenv
load_dbenv()

from aiida.orm import JobCalculation

## pk must be a valid integer pk
calc = load_node(pk)
## Alternatively, with the UUID (uuid must be a valid UUID string)
# calc = JobCalculation.get_subclass_from_uuid(uuid)
print "AiiDA state:", calc.get_state()
print "Last scheduler state seen by the AiiDA deamon:", calc.get_scheduler_state()

Note that, as specified in the comments, you can also get a code by knowing its UUID; the advantage is that, while the numeric ID will typically change after a sync of two databases, the UUID is a unique identifier and will be preserved across different AiiDA instances.

The verdi calculation gotocomputer command¶

Sometimes, it may be useful to directly go to the folder on which the calculation is running, for instance to check if the output file has been created.

In this case, it is possible to run:

verdi calculation gotocomputer CALCULATIONPK

where CALCULATIONPK is the PK of the calculation. This will open a new connection to the computer (either simply a bash shell or a ssh connection, depending on the transport) and directly change directory to the appropriate folder where the code is running.

Finding input and output nodes¶

Let’s start with a reference node that you loaded from the database, for example the node with PK 17:

n = load_node(17)

Now, we want to find the nodes which have a direct link to this node. There are several methods to extract this information (for developers see all the methods and their docstring: get_outputs, get_outputs_dict, get_inputdata_dict, c.get_inputs and c.get_inputs_dict). The most practical way to access this information, especially when working on the verdi shell, is by means of the inp and out methods.

The inp method is used to list and access the nodes with a direct link to n in input. The names of the input links can be printed by list(n.inp) or interactively by n.inp. + TAB. As an example, suppose that n has an input KpointsData object under the linkname kpoints. The command:

n.inp.kpoints

returns the KpointsData object.

Similar methods exists for the out method, which will display the names of links in output from n and can be used to access such output nodes. Suppose that n has an output FolderData with linkname retrieved, than the command:

n.out.retrieved

returns the FolderData object.

n.out.  >>
  * code
  * code_18
  * code_19

Directly querying in Django¶

If you know how AiiDA stores the data internally in the database, you can directly use Django to query the database (or even use directly SQL commands, if you really feel the urge to do so). Documentation on how queries work in Django can be found on the official Django documentation. The models can be found in aiida.djsite.db.models and is directly accessible as models in the verdi shell via verdi run.

Using the querytool¶

We provide a Python class (aiida.orm.querytool.QueryTool) to perform the most common types of queries (mainly on nodes, links and their attributes) through an easy Python class interface, without the need to know anything about the SQL query language.

To use it, in your script (or within the verdi shell) you need first to load the QueryTool class:

from aiida.orm.querytool import QueryTool

Then, create an instance of this class, which will represent your query (you need to create a new instance for each different query you want to execute):

q = QueryTool()

Now, you can call a set of methods on the q object to decide the filters you want to apply. The first type of filter one may want to apply is on the type of nodes you want to obtain (the QueryTool, in the current version, always queries only nodes in the DB). You can do so passing the correct Node subclass to the set_class() method, for instance: