General architecture#
The storage of data is an important aspect of the AiiDA system. The design for this subsystem is illustrated below.
Separate data is stored per Profile
, forming a single provenance graph.
A Profile
instance represents a dictionary that includes the configuration details for accessing the storage for that profile, such as a database URI, etc.
Multiple Profile
can be stored in a Config
instance, which is stored in the configuration file (config.json
).
Within a single Python process, a single Manager
instance can be loaded, to manage access to a globally loaded Profile
and its StorageBackend
instance.
The storage API subsystem is based on an Object Relational Mapper (ORM) and is divided into two main parts: the frontend and the backend. The frontend is responsible for the user interface, and is agnostic of any particular storage technologies, and the backend is responsible for implementing interfaces with specific technologies (such as SQL databases).
Frontend ORM#
The frontend ORM comprises of a number of Collection
and Entity
subclasses, representing access to a single ORM type.
User
Represents the author of a particular entity.
Node
Represents a node in a provenance graph, containing data for a particular process (
ProcessNode
) or process input/output (Data
). Nodes are connected by links, that form an acyclic graph. Nodes also have aRepository
instance, which is used to store binary data of the node (see also Repository).Comment
Represents a comment on a node, by a particular user.
Log
Represents a log message on a
ProcessNode
, by a particular user.Group
Represents a group of nodes. A single node can be part of multiple groups (i.e. a one-to-many relationship).
Computer
Represents a compute resource on which a process is executed. A single computer can be attached to multiple
ProcessNode
(i.e. a one-to-many relationship).AuthInfo
Represents a authentication information for a particular computer and user.
The QueryBuilder
allows for querying of specific entities and their associated data.
Backend Implementations#
Backend implementations must implement the classes outlines in aiida.orm.implementation
.
There are currently two core backend implementations:
psql_dos
is implemented as the primary storage backend, see psql_dos format.sqlite_zip
is implemented as a storage backend for the AiiDA archive, see sqlite_zip (archive) format.
Storage maintenance and profile locking#
The maintain()
method is allows for maintenance operations on the storage (for example, to optimise memory usage), and is called by verdi storage maintain.
During “full” maintenance, to guarantee the safety of its procedures, it may be necessary that the storage is not accessed by other processes.
The :py:class`~aiida.manage.profile_access.ProfileAccessManager` allows for profile access requests, and locking of profiles during such procedures.
request_access()
is called within get_profile_storage()
.