aiida.engine.daemon package

Submodules

Controls the daemon

class aiida.engine.daemon.client.ControllerProtocol(value)[source]

Bases: enum.Enum

The protocol to use to for the controller of the Circus daemon

IPC = 0
TCP = 1
__module__ = 'aiida.engine.daemon.client'
class aiida.engine.daemon.client.DaemonClient(profile: aiida.manage.configuration.profile.Profile)[source]

Bases: object

Extension of the Profile which also provides handles to retrieve profile specific properties related to the daemon client

DAEMON_ERROR_NOT_RUNNING = 'daemon-error-not-running'
DAEMON_ERROR_TIMEOUT = 'daemon-error-timeout'
_DAEMON_NAME = 'aiida-{name}'
_ENDPOINT_PROTOCOL = 0
__dict__ = mappingproxy({'__module__': 'aiida.engine.daemon.client', '__doc__': '\n Extension of the Profile which also provides handles to retrieve profile specific\n properties related to the daemon client\n ', 'DAEMON_ERROR_NOT_RUNNING': 'daemon-error-not-running', 'DAEMON_ERROR_TIMEOUT': 'daemon-error-timeout', '_DAEMON_NAME': 'aiida-{name}', '_ENDPOINT_PROTOCOL': <ControllerProtocol.IPC: 0>, '__init__': <function DaemonClient.__init__>, 'profile': <property object>, 'daemon_name': <property object>, 'cmd_string': <property object>, 'loglevel': <property object>, 'virtualenv': <property object>, 'circus_log_file': <property object>, 'circus_pid_file': <property object>, 'circus_port_file': <property object>, 'circus_socket_file': <property object>, 'circus_socket_endpoints': <property object>, 'daemon_log_file': <property object>, 'daemon_pid_file': <property object>, 'get_circus_port': <function DaemonClient.get_circus_port>, 'get_circus_socket_directory': <function DaemonClient.get_circus_socket_directory>, 'get_daemon_pid': <function DaemonClient.get_daemon_pid>, 'is_daemon_running': <property object>, 'delete_circus_socket_directory': <function DaemonClient.delete_circus_socket_directory>, 'get_available_port': <classmethod object>, 'get_controller_endpoint': <function DaemonClient.get_controller_endpoint>, 'get_pubsub_endpoint': <function DaemonClient.get_pubsub_endpoint>, 'get_stats_endpoint': <function DaemonClient.get_stats_endpoint>, 'get_ipc_endpoint': <function DaemonClient.get_ipc_endpoint>, 'get_tcp_endpoint': <function DaemonClient.get_tcp_endpoint>, 'client': <property object>, 'call_client': <function DaemonClient.call_client>, 'get_status': <function DaemonClient.get_status>, 'get_numprocesses': <function DaemonClient.get_numprocesses>, 'get_worker_info': <function DaemonClient.get_worker_info>, 'get_daemon_info': <function DaemonClient.get_daemon_info>, 'increase_workers': <function DaemonClient.increase_workers>, 'decrease_workers': <function DaemonClient.decrease_workers>, 'stop_daemon': <function DaemonClient.stop_daemon>, 'restart_daemon': <function DaemonClient.restart_daemon>, '__dict__': <attribute '__dict__' of 'DaemonClient' objects>, '__weakref__': <attribute '__weakref__' of 'DaemonClient' objects>})
__init__(profile: aiida.manage.configuration.profile.Profile)[source]

Construct a DaemonClient instance for a given profile

Parameters

profile – the profile instance aiida.manage.configuration.profile.Profile

__module__ = 'aiida.engine.daemon.client'
__weakref__

list of weak references to the object (if defined)

call_client(command: Dict[str, Any]) → Dict[str, Any][source]

Call the client with a specific command. Will check whether the daemon is running first by checking for the pid file. When the pid is found yet the call still fails with a timeout, this means the daemon was actually not running and it was terminated unexpectedly causing the pid file to not be cleaned up properly

Parameters

command – command to call the circus client with

Returns

the result of the circus client call

property circus_log_file
property circus_pid_file
property circus_port_file
property circus_socket_endpoints
property circus_socket_file
property client

Return an instance of the CircusClient with the endpoint defined by the controller endpoint, which used the port that was written to the port file upon starting of the daemon

N.B. This is quite slow the first time it is run due to the import of zmq.ssh in circus/utils.py in circus 0.15.0, which ultimately follows the import of CircusClient.

Returns

CircusClient instance

property cmd_string

Return the command string to start the AiiDA daemon

property daemon_log_file
property daemon_name

Get the daemon name which is tied to the profile name

property daemon_pid_file
decrease_workers(number: int) → Dict[str, Any][source]

Decrease the number of workers

Parameters

number – the number of workers to remove

Returns

the client call response

delete_circus_socket_directory()None[source]

Attempt to delete the directory used to store the circus endpoint sockets. Will not raise if the directory does not exist

classmethod get_available_port()[source]

Get an available port from the operating system

Returns

a currently available port

get_circus_port()int[source]

Retrieve the port for the circus controller, which should be written to the circus port file. If the daemon is running, the port file should exist and contain the port to which the controller is connected. If it cannot be read, a RuntimeError will be thrown. If the daemon is not running, an available port will be requested from the operating system, written to the port file and returned

Returns

the port for the circus controller

get_circus_socket_directory()str[source]

Retrieve the absolute path of the directory where the circus sockets are stored if the IPC protocol is used and the daemon is running. If the daemon is running, the sockets file should exist and contain the absolute path of the directory that contains the sockets of the circus endpoints. If it cannot be read, a RuntimeError will be thrown. If the daemon is not running, a temporary directory will be created and its path will be written to the sockets file and returned.

Note

A temporary folder needs to be used for the sockets because UNIX limits the filepath length to 107 bytes. Placing the socket files in the AiiDA config folder might seem like the more logical choice but that folder can be placed in an arbitrarily nested directory, the socket filename will exceed the limit. The solution is therefore to always store them in the temporary directory of the operation system whose base path is typically short enough as to not exceed the limit

Returns

the absolute path of directory to write the sockets to

get_controller_endpoint()[source]

Get the endpoint string for the circus controller. For the IPC protocol a profile specific socket will be used, whereas for the TCP protocol an available port will be found and saved in the profile specific port file

Returns

the endpoint string

get_daemon_info() → Dict[str, Any][source]

Get statistics about this daemon itself

Returns

the client call response If successful, will contain ‘info’ key

get_daemon_pid() → Optional[int][source]

Get the daemon pid which should be written in the daemon pid file specific to the profile

Returns

the pid of the circus daemon process or None if not found

get_ipc_endpoint(endpoint)[source]

Get the ipc endpoint string for a circus daemon endpoint for a given socket

Parameters

endpoint – the circus endpoint for which to return a socket

Returns

the ipc endpoint string

get_numprocesses() → Dict[str, Any][source]

Get the number of running daemon processes

Returns

the client call response If successful, will contain ‘numprocesses’ key

get_pubsub_endpoint()[source]

Get the endpoint string for the circus pubsub endpoint. For the IPC protocol a profile specific socket will be used, whereas for the TCP protocol any available port will be used

Returns

the endpoint string

get_stats_endpoint()[source]

Get the endpoint string for the circus stats endpoint. For the IPC protocol a profile specific socket will be used, whereas for the TCP protocol any available port will be used

Returns

the endpoint string

get_status() → Dict[str, Any][source]

Get the daemon running status

Returns

the client call response If successful, will will contain ‘status’ key

get_tcp_endpoint(port=None)[source]

Get the tcp endpoint string for a circus daemon endpoint. If the port is unspecified, the operating system will be asked for a currently available port.

Parameters

port – a port to use for the endpoint

Returns

the tcp endpoint string

get_worker_info() → Dict[str, Any][source]

Get workers statistics for this daemon

Returns

the client call response If successful, will contain ‘info’ key

increase_workers(number: int) → Dict[str, Any][source]

Increase the number of workers

Parameters

number – the number of workers to add

Returns

the client call response

property is_daemon_running

Return whether the daemon is running, which is determined by seeing if the daemon pid file is present

Returns

True if daemon is running, False otherwise

property loglevel
property profile
restart_daemon(wait: bool) → Dict[str, Any][source]

Restart the daemon

Parameters

wait – boolean to indicate whether to wait for the result of the command

Returns

the client call response

stop_daemon(wait: bool) → Dict[str, Any][source]

Stop the daemon

Parameters

wait – boolean to indicate whether to wait for the result of the command

Returns

the client call response

property virtualenv
aiida.engine.daemon.client.get_daemon_client(profile_name: Optional[str] = None)aiida.engine.daemon.client.DaemonClient[source]

Return the daemon client for the given profile or the current profile if not specified.

Parameters

profile_name – the profile name, will use the current profile if None

Returns

the daemon client

Raises

This file contains the main routines to submit, check and retrieve calculation results. These are general and contain only the main logic; where appropriate, the routines make reference to the suitable plugins for all plugin-specific operations.

aiida.engine.daemon.execmanager._find_data_node(inputs: Mapping[str, Any], uuid: str) → Optional[aiida.orm.nodes.node.Node][source]

Find and return the node with the given UUID from a nested mapping of input nodes.

Parameters
  • inputs – (nested) mapping of nodes

  • uuid – UUID of the node to find

Returns

instance of Node or None if not found

aiida.engine.daemon.execmanager._retrieve_singlefiles(job: aiida.orm.nodes.process.calculation.calcjob.CalcJobNode, transport: aiida.transports.transport.Transport, folder: aiida.common.folders.SandboxFolder, retrieve_file_list: List[Tuple[str, str, str]], logger_extra: Optional[dict] = None)[source]

Retrieve files specified through the singlefile list mechanism.

aiida.engine.daemon.execmanager.kill_calculation(calculation: aiida.orm.nodes.process.calculation.calcjob.CalcJobNode, transport: aiida.transports.transport.Transport)None[source]

Kill the calculation through the scheduler

Parameters
  • calculation – the instance of CalcJobNode to kill.

  • transport – an already opened transport to use to address the scheduler

aiida.engine.daemon.execmanager.retrieve_calculation(calculation: aiida.orm.nodes.process.calculation.calcjob.CalcJobNode, transport: aiida.transports.transport.Transport, retrieved_temporary_folder: str)None[source]

Retrieve all the files of a completed job calculation using the given transport.

If the job defined anything in the retrieve_temporary_list, those entries will be stored in the retrieved_temporary_folder. The caller is responsible for creating and destroying this folder.

Parameters
  • calculation – the instance of CalcJobNode to update.

  • transport – an already opened transport to use for the retrieval.

  • retrieved_temporary_folder – the absolute path to a directory in which to store the files listed, if any, in the retrieved_temporary_folder of the jobs CalcInfo

aiida.engine.daemon.execmanager.retrieve_files_from_list(calculation: aiida.orm.nodes.process.calculation.calcjob.CalcJobNode, transport: aiida.transports.transport.Transport, folder: str, retrieve_list: List[Union[str, Tuple[str, str, int], list]])None[source]

Retrieve all the files in the retrieve_list from the remote into the local folder instance through the transport. The entries in the retrieve_list can be of two types:

  • a string

  • a list

If it is a string, it represents the remote absolute filepath of the file. If the item is a list, the elements will correspond to the following:

  • remotepath

  • localpath

  • depth

If the remotepath contains file patterns with wildcards, the localpath will be treated as the work directory of the folder and the depth integer determines upto what level of the original remotepath nesting the files will be copied.

Parameters
  • transport – the Transport instance.

  • folder – an absolute path to a folder that contains the files to copy.

  • retrieve_list – the list of files to retrieve.

aiida.engine.daemon.execmanager.stash_calculation(calculation: aiida.orm.nodes.process.calculation.calcjob.CalcJobNode, transport: aiida.transports.transport.Transport)None[source]

Stash files from the working directory of a completed calculation to a permanent remote folder.

After a calculation has been completed, optionally stash files from the work directory to a storage location on the same remote machine. This is useful if one wants to keep certain files from a completed calculation to be removed from the scratch directory, because they are necessary for restarts, but that are too heavy to retrieve. Instructions of which files to copy where are retrieved from the stash.source_list option.

Parameters
  • calculation – the calculation job node.

  • transport – an already opened transport.

aiida.engine.daemon.execmanager.submit_calculation(calculation: aiida.orm.nodes.process.calculation.calcjob.CalcJobNode, transport: aiida.transports.transport.Transport)str[source]

Submit a previously uploaded CalcJob to the scheduler.

Parameters
  • calculation – the instance of CalcJobNode to submit.

  • transport – an already opened transport to use to submit the calculation.

Returns

the job id as returned by the scheduler submit_from_script call

aiida.engine.daemon.execmanager.upload_calculation(node: aiida.orm.nodes.process.calculation.calcjob.CalcJobNode, transport: aiida.transports.transport.Transport, calc_info: aiida.common.datastructures.CalcInfo, folder: aiida.common.folders.SandboxFolder, inputs: Optional[Mapping[str, Any]] = None, dry_run: bool = False)None[source]

Upload a CalcJob instance

Parameters
  • node – the CalcJobNode.

  • transport – an already opened transport to use to submit the calculation.

  • calc_info – the calculation info datastructure returned by CalcJob.presubmit

  • folder – temporary local file system folder containing the inputs written by CalcJob.prepare_for_submission

Function that starts a daemon runner.

async aiida.engine.daemon.runner.shutdown_runner(runner: aiida.engine.runners.Runner)None[source]

Cleanup tasks tied to the service’s shutdown.

aiida.engine.daemon.runner.start_daemon()None[source]

Start a daemon runner for the currently configured profile.