aiida.manage.external package

User facing APIs to control AiiDA from the verdi cli, scripts or plugins

Submodules

Connect to an existing PostgreSQL cluster as the postgres superuser and execute SQL commands.

Note: Once the API of this functionality has converged, this module should be moved out of aiida-core and into a

separate package that can then be tested on multiple OS / postgres setups. Therefore, please keep this module entirely AiiDA-agnostic.

class aiida.manage.external.pgsu.PGSU(interactive=False, quiet=True, dbinfo=None, determine_setup=True)

Bases: object

Connect to an existing PostgreSQL cluster as the postgres superuser and execute SQL commands.

Tries to use psycopg2 with a fallback to psql subcommands (using sudo su to run as postgres user).

Simple Example:

postgres = PGSU()
postgres.execute("CREATE USER testuser PASSWORD 'testpw'")

Complex Example:

postgres = PGSU(interactive=True, dbinfo={'port': 5433})
postgres.execute("CREATE USER testuser PASSWORD 'testpw'")

Note: In postgresql

• you cannot drop databases you are currently connected to

• ‘template0’ is the unmodifiable template database (which you cannot connect to)

• ‘template1’ is the modifiable template database (which you can connect to)

__dict__ = mappingproxy({'__module__': 'aiida.manage.external.pgsu', '__doc__': '\n Connect to an existing PostgreSQL cluster as the `postgres` superuser and execute SQL commands.\n\n Tries to use psycopg2 with a fallback to psql subcommands (using ``sudo su`` to run as postgres user).\n\n Simple Example::\n\n postgres = PGSU()\n postgres.execute("CREATE USER testuser PASSWORD \'testpw\'")\n\n Complex Example::\n\n postgres = PGSU(interactive=True, dbinfo={\'port\': 5433})\n postgres.execute("CREATE USER testuser PASSWORD \'testpw\'")\n\n Note: In postgresql\n * you cannot drop databases you are currently connected to\n * \'template0\' is the unmodifiable template database (which you cannot connect to)\n * \'template1\' is the modifiable template database (which you can connect to)\n ', '__init__': <function PGSU.__init__>, 'execute': <function PGSU.execute>, 'set_setup_fail_callback': <function PGSU.set_setup_fail_callback>, 'determine_setup': <function PGSU.determine_setup>, '_no_setup_detected': <function PGSU._no_setup_detected>, 'is_connected': <property object>, '__dict__': <attribute '__dict__' of 'PGSU' objects>, '__weakref__': <attribute '__weakref__' of 'PGSU' objects>})

__init__(interactive=False, quiet=True, dbinfo=None, determine_setup=True)

Store postgres connection info.

Parameters

• interactive – use True for verdi commands

• quiet – use False to show warnings/exceptions

• dbinfo – psycopg dictionary containing keys like ‘host’, ‘user’, ‘port’, ‘database’

• determine_setup – Whether to determine setup upon instantiation. You may set this to False and use the ‘determine_setup()’ method instead.

__module__ = 'aiida.manage.external.pgsu'

__weakref__

list of weak references to the object (if defined)

_no_setup_detected()

Print a warning message and calls the failed setup callback

Returns

False, if no successful try.

determine_setup()

Determine how to connect as the postgres superuser.

Depending on how postgres is set up, psycopg2 can be used to create dbs and db users, otherwise a subprocess has to be used that executes psql as an os user with appropriate permissions.

Note: We aim to connect as a superuser (typically ‘postgres’) with privileges to manipulate (create/drop)

databases and database users.

Returns success

True, if connection could be established.

Rtype success

bool

execute(command, **kwargs)

Execute postgres command using determined connection mode.

Parameters

• command – A psql command line as a str

• kwargs – will be forwarded to _execute_… function

property is_connected

set_setup_fail_callback(callback)

Set a callback to be called when setup cannot be determined automatically

Parameters

callback – a callable with signature callback(interactive, dbinfo) that returns a dbinfo dictionary.

class aiida.manage.external.pgsu.PostgresConnectionMode

Bases: enum.IntEnum

Describe mode of connecting to postgres.

DISCONNECTED = 0

PSQL = 2

PSYCOPG = 1

__module__ = 'aiida.manage.external.pgsu'

aiida.manage.external.pgsu._execute_psql(command, user='postgres', quiet=True, interactive=False, **kwargs)

Executes an SQL command via psql as another system user in a subprocess.

Tries to “become” the user specified in kwargs (i.e. interpreted as UNIX system user) and run psql in a subprocess.

Parameters

• command – A psql command line as a str

• quiet – If True, don’t print warnings.

• interactive – If False, sudo won’t ask for a password and fail if one is required.

• kwargs – connection details to forward to psql, signature as in psycopg2.connect

aiida.manage.external.pgsu._execute_psyco(command, **kwargs)

executes a postgres commandline through psycopg2

Parameters

• command – A psql command line as a str

• kwargs – will be forwarded to psycopg2.connect

aiida.manage.external.pgsu._sudo_exists()

Check that the sudo command can be found

Returns

True if successful, False otherwise

aiida.manage.external.pgsu._try_connect_psycopg(**kwargs)

try to start a psycopg2 connection.

Returns

True if successful, False otherwise

aiida.manage.external.pgsu._try_subcmd(**kwargs)

try to run psql in a subprocess.

Returns

True if successful, False otherwise

aiida.manage.external.pgsu.escape_for_bash(str_to_escape)

This function takes any string and escapes it in a way that bash will interpret it as a single string.

Explanation:

At the end, in the return statement, the string is put within single quotes. Therefore, the only thing that I have to escape in bash is the single quote character. To do this, I substitute every single quote ‘ with ‘”’”’ which means:

First single quote: exit from the enclosing single quotes

Second, third and fourth character: “’” is a single quote character, escaped by double quotes

Last single quote: reopen the single quote to continue the string

Finally, note that for python I have to enclose the string ‘”’”’ within triple quotes to make it work, getting finally: the complicated string found below.

aiida.manage.external.pgsu.prompt_db_info(interactive, dbinfo)

Prompt interactively for postgres database connection details

Can be used as a setup fail callback for PGSU

Returns

dictionary with the following keys: host, port, database, user

Provides an API for postgres database maintenance tasks.

This API creates and drops postgres users and databases used by the verdi quicksetup commandline tool. It allows convenient access to this functionality from within python without knowing details about how postgres is installed by default on various systems. If the postgres setup is not the default installation, additional information needs to be provided.

class aiida.manage.external.postgres.Postgres(interactive=False, quiet=True, dbinfo=None, determine_setup=True)

Bases: aiida.manage.external.pgsu.PGSU

Adds convenience functions to pgsu.Postgres.

Provides conenience functions for

• creating/dropping users

• creating/dropping databases

etc. See pgsu.Postgres for implementation details.

Example:

postgres = Postgres()
postgres.create_dbuser('username', 'password')
if not postgres.db_exists('dbname'):
    postgres.create_db('username', 'dbname')

__module__ = 'aiida.manage.external.postgres'

check_db_name(dbname)

Looks up if a database with the name exists, prompts for using or creating a differently named one.

copy_db(src_db, dest_db, dbuser)

create_db(dbuser, dbname)

Create a database in postgres

Parameters

• dbuser – (str), Name of the user which should own the db.

• dbname – (str), Name of the database.

create_dbuser(dbuser, dbpass)

Create a database user in postgres

Parameters

• dbuser – (str), Name of the user to be created.

• dbpass – (str), Password the user should be given.

db_exists(dbname)

Check wether a postgres database with dbname exists

Parameters

dbname – Name of the database to check for

Returns

(bool), True if database exists, False otherwise

dbuser_exists(dbuser)

Find out if postgres user with name dbuser exists

Parameters

dbuser – (str) database user to check for

Returns

(bool) True if user exists, False otherwise

drop_db(dbname)

Drop a database in postgres

Parameters

dbname – (str), Name of the database.

drop_dbuser(dbuser)

Drop a database user in postgres

Parameters

dbuser – (str), Name of the user to be dropped.

classmethod from_profile(profile, **kwargs)

Create Postgres instance with dbinfo from AiiDA profile data.

Note: This only uses host and port from the profile, since the others are not going to be relevant for the

database superuser.

Parameters

• profile – AiiDA profile instance

• kwargs – keyword arguments forwarded to Postgres constructor

Returns

Postgres instance pre-populated with data from AiiDA profile

class aiida.manage.external.postgres.PostgresConnectionMode

Bases: enum.IntEnum

Describe mode of connecting to postgres.

DISCONNECTED = 0

PSQL = 2

PSYCOPG = 1

__module__ = 'aiida.manage.external.pgsu'

Components to communicate tasks to RabbitMQ.

exception aiida.manage.external.rmq.RemoteException

Bases: Exception

An exception occurred at the remote end of the call

__module__ = 'kiwipy.communications'

__weakref__

list of weak references to the object (if defined)

aiida.manage.external.rmq.CommunicationTimeout

alias of concurrent.futures._base.TimeoutError

exception aiida.manage.external.rmq.DeliveryFailed

Bases: Exception

Failed to deliver a message

__module__ = 'kiwipy.communications'

__weakref__

list of weak references to the object (if defined)

class aiida.manage.external.rmq.ProcessLauncher(loop=None, persister=None, load_context=None, loader=None)

Bases: plumpy.process_comms.ProcessLauncher

A sub class of plumpy.ProcessLauncher to launch a Process.

It overrides the _continue method to make sure the node corresponding to the task can be loaded and that if it is already marked as terminated, it is not continued but the future is reconstructed and returned

__module__ = 'aiida.manage.external.rmq'

_continue(communicator, pid, nowait, tag=None)

Continue the task.

Note that the task may already have been completed, as indicated from the corresponding the node, in which case it is not continued, but the corresponding future is reconstructed and returned. This scenario may occur when the Process was already completed by another worker that however failed to send the acknowledgment.

Parameters

• communicator – the communicator that called this method

• pid – the pid of the process to continue

• nowait – if True don’t wait for the process to finish, just return the pid, otherwise wait and return the results

• tag – the tag of the checkpoint to continue from

static handle_continue_exception(node, exception, message)

Handle exception raised in _continue call.

If the process state of the node has not yet been put to excepted, the exception was raised before the process instance could be reconstructed, for example when the process class could not be loaded, thereby circumventing the exception handling of the state machine. Raising this exception will then acknowledge the process task with RabbitMQ leaving an uncleaned node in the CREATED state for ever. Therefore we have to perform the node cleaning manually.

Parameters

• exception – the exception object

• message – string message to use for the log message