AiiDA internals

Node

All nodes in an AiiDA provenance graph inherit from the Node class. Among those are the Data class, the ProcessNode class representing computations that transform data, and the Code class representing executables (and file collections that are used by calculations).

Immutability concept

A node can store information in attributes. Since AiiDA guarantees a certain level of provenance, these attributes become immutable as soon as the node is stored. This means that as soon as a node is stored, any attempt to alter its attributes, changing its value or deleting it altogether, shall be met with a raised exception. Certain subclasses of nodes need to adapt this behavior however, as for example in the case of the ProcessNode class (see calculation updatable attributes), but since the immutability of stored nodes is a core concept of AiiDA, this behavior is nonetheless enforced on the node level. This guarantees that any subclasses of the Node class will respect this behavior unless it is explicitly overriden.

Node methods

  • clean_value() takes a value and returns an object which can be serialized for storage in the database. Such an object must be able to be subsequently deserialized without changing value. If a simple datatype is passed (integer, float, etc.), a check is performed to see if it has a value of nan or inf, as these cannot be stored. Otherwise, if a list, tuple, dictionary, etc., is passed, this check is performed for each value it contains. This is done recursively, automatically handling the case of nested objects. It is important to note that iterable type objects are converted to lists during this process, and mappings are converted to normal dictionaries. For efficiency reasons, the cleaning of attribute values is delayed to the last moment possible. This means that for an unstored node, new attributes are not cleaned but simply set in the cache of the underlying database model. When the node is then stored, all attributes are cleaned in one fell swoop and if successful the values are flushed to the database. Once a node is stored, there no longer is such a cache and so the attribute values are cleaned straight away for each call. The same mechanism holds for the cleaning of the values of extras.

Node methods & properties

In the following sections, the most important methods and properties of the Node class will be described.

Node subclasses organization

The Node class has two important attributes:

  • _plugin_type_string characterizes the class of the object.

  • _query_type_string characterizes the class and all its subclasses (by pointing to the package or Python file that contain the class).

The convention for all the Node subclasses is that if a class B is inherited by a class A then there should be a package A under aiida/orm that has a file __init__.py and a B.py in that directory (or a B package with the corresponding __init__.py)

An example of this is the ArrayData and the KpointsData. ArrayData is placed in aiida/orm/data/array/__init__.py and KpointsData which inherits from ArrayData is placed in aiida/orm/data/array/kpoints.py

This is an implicit & quick way to check the inheritance of the Node subclasses.

General purpose methods

  • __init__(): Will construct a new unstored Node. Note that this cannot be used to load an existing node from the database.

  • ctime() and mtime() provide the creation and the modification time of the node.

  • computer() returns the computer associated to this node.

  • _validate() does a validation check for the node. This is important for Node subclasses where various attributes should be checked for consistency before storing.

  • user() returns the user that created the node.

  • uuid() returns the universally unique identifier (UUID) of the node.

Annotation methods

The Node can be annotated with labels, description and comments. The following methods can be used for the management of these properties.

Label management:

  • label returns the label of the node. It can also be used to change the label, e.g. mynode.label = "new label".

Description management:

  • description: returns the description of the node (more detailed than the label). It can also be used to change the description, e.g. mynode.description = "new description".

Comment management:

  • add_comment() adds a comment.

  • get_comments() returns a sorted list of the comments.

  • update_comment() updates the node comment. It can also be accessed through the CLI: verdi comment update.

  • remove_comment() removes the node comment. It can also be accessed through the CLI: verdi comment remove.

Folder management

Folder objects represent directories on the disk (virtual or not) where extra information for the node are stored. These folders can be temporary or permanent.

Store & deletion

  • store_all() stores all the input nodes, then it stores the current node and in the end, it stores the cached input links.

  • verify_are_parents_stored() checks that the parents are stored.

  • store() method checks that the node data is valid, then check if node’s parents are stored, then moves the contents of the temporary folder to the repository folder and in the end, it stores in the database the information that are in the cache. The latter happens with a database transaction. In case this transaction fails, then the data transfered to the repository folder are moved back to the temporary folder.

Folders

AiiDA uses Folder and its subclasses to add an abstraction layer between the functions and methods working directly on the file-system and AiiDA. This is particularly useful when we want to easily change between different folder options (temporary, permanent etc) and storage options (plain local directories, compressed files, remote files & directories etc).

Folder

This is the main class of the available Folder classes. Apart from the abstraction provided to the OS operations needed by AiiDA, one of its main features is that it can restrict all the available operations within a given folder limit. The available methods are:

RepositoryFolder

Objects of this class correspond to the repository folders. The RepositoryFolder specific methods are:

  • __init__() initializes the object with the necessary folder names and limits.

  • get_topdir() returns the top directory.

  • section() returns the section to which the folder belongs. This can be for the moment only node.

  • subfolder() returns the subfolder within the section/uuid folder.

  • uuid() the UUID of the corresponding node.

SandboxFolder

SandboxFolder objects correspond to temporary (“sandbox”) folders. The main methods are:

Data

ProcessNode

Navigating inputs and outputs

CalculationNode

Navigating inputs and outputs

  • inputs() returns a NodeLinksManager() object that can be used to access the node’s incoming INPUT_CALC links.

    The NodeLinksManager can be used to quickly go from a node to a neighboring node. For example:

    In [1]: # Let's load a node with a specific pk

In [2]: c = load_node(139168)

In [3]: c
Out[3]: <CpCalculation: uuid: 49084dcf-c708-4422-8bcf-808e4c3382c2 (pk: 139168)>

In [4]: # Let's traverse the inputs of this node.

In [5]: # By typing c.inputs.<TAB> we get all the input links

In [6]: c.inputs.
c.inputs.code                c.inputs.parent_calc_folder  c.inputs.pseudo_O            c.inputs.settings
c.inputs.parameters          c.inputs.pseudo_Ba           c.inputs.pseudo_Ti           c.inputs.structure

In [7]: # We may follow any of these links to access other nodes. For example, let's follow the parent_calc_folder

In [8]: c.inputs.parent_calc_folder
Out[8]: <RemoteData: uuid: becb4894-c50c-4779-b84f-713772eaceff (pk: 139118)>

In [9]: # Let's assign to r the node reached by the parent_calc_folder link

In [10]: r = c.inputs.parent_calc_folder

In [11]: r.inputs.__dir__()
Out[11]:
['__class__',
'__delattr__',
'__dict__',
'__dir__',
'__doc__',
'__format__',
'__getattr__',
'__getattribute__',
'__getitem__',
'__hash__',
'__init__',
'__iter__',
'__module__',
'__new__',
'__reduce__',
'__reduce_ex__',
'__repr__',
'__setattr__',
'__sizeof__',
'__str__',
'__subclasshook__',
'__weakref__',
'remote_folder']

    The .inputs manager for WorkflowNode and the .outputs manager both for CalculationNode and WorkflowNode work in the same way (see below).

  • outputs() returns a NodeLinksManager() object that can be used to access the node’s outgoing CREATE links.

Updatable attributes

The ProcessNode class is a subclass of the Node class, which means that its attributes become immutable once stored. However, for a Calculation to be runnable it needs to be stored, but that would mean that its state, which is stored in an attribute can no longer be updated. To solve this issue the Sealable mixin is introduced. This mixin can be used for subclasses of Node that need to have updatable attributes even after the node has been stored in the database. The mixin defines the _updatable_attributes tuple, which defines the attributes that are considered to be mutable even when the node is stored. It also allows the node to be sealed, after which even the updatable attributes become immutable.

WorkflowNode

Navigating inputs and outputs

Deprecated features, renaming, and adding new methods

In case a method is renamed or removed, this is the procedure to follow:

  1. (If you want to rename) move the code to the new function name. Then, in the docstring, add something like:

    .. versionadded:: 0.7
   Renamed from OLDMETHODNAME

  2. Don’t remove directly the old function, but just change the code to use the new function, and add in the docstring:

    .. deprecated:: 0.7
   Use :meth:`NEWMETHODNAME` instead.

    Moreover, at the beginning of the function, add something like:

    import warnings

# If we call this DeprecationWarning, pycharm will properly strike out the function
from aiida.common.warnings import AiidaDeprecationWarning as DeprecationWarning  # pylint: disable=redefined-builtin
warnings.warn("<Deprecation warning here - MAKE IT SPECIFIC TO THIS DEPRECATION, as it will be shown only once per different message>", DeprecationWarning)

# <REST OF THE FUNCTION HERE>

    (of course replace the parts between < > symbols with the correct strings).

    The advantage of the method above is:

    • pycharm will still show the method crossed out

    • Our AiidaDeprecationWarning does not inherit from DeprecationWarning, so it will not be “hidden” by python

    • User can disable our warnings (and only those) by using AiiDA properties with:

      verdi config warnings.showdeprecations False

Changing the config.json structure

In general, changes to config.json should be avoided if possible. However, if there is a need to modify it, the following procedure should be used to create a migration:

  1. Determine whether the change will be backwards-compatible. This means that an older version of AiiDA will still be able to run with the new config.json structure. It goes without saying that it’s preferable to change config.json in a backwards-compatible way.

  2. In aiida/manage/configuration/migrations/migrations.py, increase the CURRENT_CONFIG_VERSION by one. If the change is not backwards-compatible, set OLDEST_COMPATIBLE_CONFIG_VERSION to the same value.

  3. Write a function which transforms the old config dict into the new version. It is possible that you need user input for the migration, in which case this should also be handled in that function.

  4. Add an entry in _MIGRATION_LOOKUP where the key is the version before the migration, and the value is a ConfigMigration object. The ConfigMigration is constructed from your migration function, and the hard-coded values of CURRENT_CONFIG_VERSION and OLDEST_COMPATIBLE_CONFIG_VERSION. If these values are not hard-coded, the migration will break as soon as the values are changed again.

  5. Add tests for the migration, in aiida/backends/tests/manage/configuration/migrations/test_migrations.py. You can add two types of tests:

    • Tests that run the entire migration, using the check_and_migrate_config function. Make sure to run it with store=False, otherwise it will overwrite your config.json file. For these tests, you will have to update the reference files.

    • Tests that run a single step in the migration, using the ConfigMigration.apply method. This can be used if you need to test different edge cases of the migration.

There are examples for both types of tests.

Daemon and signal handling

While the AiiDA daemon is running, interrupt signals (SIGINT and SIGTERM) are captured so that the daemon can shut down gracefully. This is implemented using Python’s signal module, as shown in the following dummy example:

import signal

def print_foo(*args):
    print('foo')

signal.signal(signal.SIGINT, print_foo)

You should be aware of this while developing code which runs in the daemon. In particular, it’s important when creating subprocesses. When a signal is sent, the whole process group receives that signal. As a result, the subprocess can be killed even though the Python main process captures the signal. This can be avoided by creating a new process group for the subprocess, meaning that it will not receive the signal. To do this, you need to pass start_new_session=True to the subprocess function:

import os
import subprocess

print(subprocess.check_output('sleep 3; echo bar', start_new_session=True))