aiida.engine.processes.calcjobs package¶
Module for the CalcJob process and related utilities.
-
class
aiida.engine.processes.calcjobs.
CalcJob
(*args, **kwargs)[source]¶ Bases:
aiida.engine.processes.process.Process
Implementation of the CalcJob process.
-
_Process__called
= True¶
-
__abstractmethods__
= frozenset([])¶
-
__init__
(*args, **kwargs)[source]¶ Construct the instance only if it is a sub class of CalcJob otherwise raise InvalidOperation.
-
__module__
= 'aiida.engine.processes.calcjobs.calcjob'¶
-
_abc_cache
= <_weakrefset.WeakSet object>¶
-
_abc_negative_cache
= <_weakrefset.WeakSet object>¶
-
_abc_negative_cache_version
= 102¶
-
_abc_registry
= <_weakrefset.WeakSet object>¶
-
_node_class
¶ alias of
aiida.orm.nodes.process.calculation.calcjob.CalcJobNode
-
_spec
= <aiida.engine.processes.process_spec.CalcJobProcessSpec object>¶
-
_spec_type
¶ alias of
aiida.engine.processes.process_spec.CalcJobProcessSpec
-
link_label_retrieved
= 'retrieved'¶
-
on_terminated
()[source]¶ Cleanup the node by deleting the calulation job state.
Note
This has to be done before calling the super because that will seal the node after we cannot change it
-
parse
(retrieved_temporary_folder=None)[source]¶ Parse a retrieved job calculation.
This is called once it’s finished waiting for the calculation to be finished and the data has been retrieved.
-
presubmit
(folder)[source]¶ Prepares the calculation folder with all inputs, ready to be copied to the cluster.
Parameters: folder – a SandboxFolder, empty in input, that will be filled with calculation input files and the scheduling script. Return calcinfo: the CalcInfo object containing the information needed by the daemon to handle operations.
-
Submodules¶
Implementation of the CalcJob process.
-
class
aiida.engine.processes.calcjobs.calcjob.
CalcJob
(*args, **kwargs)[source]¶ Bases:
aiida.engine.processes.process.Process
Implementation of the CalcJob process.
-
_Process__called
= True¶
-
__abstractmethods__
= frozenset([])¶
-
__init__
(*args, **kwargs)[source]¶ Construct the instance only if it is a sub class of CalcJob otherwise raise InvalidOperation.
-
__module__
= 'aiida.engine.processes.calcjobs.calcjob'¶
-
_abc_cache
= <_weakrefset.WeakSet object>¶
-
_abc_negative_cache
= <_weakrefset.WeakSet object>¶
-
_abc_negative_cache_version
= 102¶
-
_abc_registry
= <_weakrefset.WeakSet object>¶
-
_node_class
¶ alias of
aiida.orm.nodes.process.calculation.calcjob.CalcJobNode
-
_spec
= <aiida.engine.processes.process_spec.CalcJobProcessSpec object>¶
-
_spec_type
¶ alias of
aiida.engine.processes.process_spec.CalcJobProcessSpec
-
link_label_retrieved
= 'retrieved'¶
-
on_terminated
()[source]¶ Cleanup the node by deleting the calulation job state.
Note
This has to be done before calling the super because that will seal the node after we cannot change it
-
parse
(retrieved_temporary_folder=None)[source]¶ Parse a retrieved job calculation.
This is called once it’s finished waiting for the calculation to be finished and the data has been retrieved.
-
presubmit
(folder)[source]¶ Prepares the calculation folder with all inputs, ready to be copied to the cluster.
Parameters: folder – a SandboxFolder, empty in input, that will be filled with calculation input files and the scheduling script. Return calcinfo: the CalcInfo object containing the information needed by the daemon to handle operations.
-
Module containing utilities and classes relating to job calculations running on systems that require transport.
-
class
aiida.engine.processes.calcjobs.manager.
JobsList
(authinfo, transport_queue)[source]¶ Bases:
object
A list of submitted jobs on a machine connected to by transport based on the authorisation information.
-
__dict__
= dict_proxy({'__module__': 'aiida.engine.processes.calcjobs.manager', '_get_jobs_with_scheduler': <function _get_jobs_with_scheduler>, '_ensure_updating': <function _ensure_updating>, '_get_jobs_from_scheduler': <function _get_jobs_from_scheduler>, 'get_last_updated': <function get_last_updated>, '_update_job_info': <function _update_job_info>, '__dict__': <attribute '__dict__' of 'JobsList' objects>, '_has_job_state_changed': <staticmethod object>, '_get_next_update_delay': <function _get_next_update_delay>, '_update_requests_outstanding': <function _update_requests_outstanding>, 'request_job_info_update': <function request_job_info_update>, 'get_minimum_update_interval': <function get_minimum_update_interval>, '__weakref__': <attribute '__weakref__' of 'JobsList' objects>, '__doc__': '\n A list of submitted jobs on a machine connected to by transport based on the\n authorisation information.\n ', '__init__': <function __init__>})¶
-
__init__
(authinfo, transport_queue)[source]¶ Parameters: - authinfo (
aiida.orm.AuthInfo
) – The authinfo used to check the jobs list - transport_queue – A transport queue
Type: - authinfo (
-
__module__
= 'aiida.engine.processes.calcjobs.manager'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
_ensure_updating
()[source]¶ Ensure that we are updating the job list from the remote resource. This will automatically stop if there are no outstanding requests.
-
_get_jobs_from_scheduler
(**kwargs)[source]¶ Get the current jobs list from the scheduler
Returns: A dictionary of {job_id: job info} Return type: dict
-
_get_jobs_with_scheduler
()[source]¶ Get all the jobs that are currently with scheduler for this authinfo
Returns: the list of jobs with the scheduler Return type: list
-
_get_next_update_delay
()[source]¶ Calculate when we are next allowed to call the scheduler get jobs command based on when we last called it, how long has elapsed and the minimum given update interval.
Returns: The delay (in seconds) for when it’s safe to call the get jobs command Return type: float
-
_update_job_info
(**kwargs)[source]¶ Update all of the job information objects for a given authinfo, that is to say for all the jobs on a particular machine for a particular user.
This will set the futures for all pending update requests where the corresponding job has a new status compared to the last update.
-
get_last_updated
()[source]¶ Get the timestamp of when the list was last updated as produced by time.time()
Returns: The last update point Return type: float
-
get_minimum_update_interval
()[source]¶ Get the minimum interval that can be expected between updates of the list :return: The minimum interval :rtype: float
-
request_job_info_update
(**kwds)[source]¶ Request job info about a job when it next changes it’s job state. If the job is not found in the jobs list at the update the future will resolve to None.
Parameters: job_id – The job identifier Returns: A future that will resolve to a JobInfo object when the job changes state
-
-
class
aiida.engine.processes.calcjobs.manager.
JobManager
(transport_queue)[source]¶ Bases:
object
A manager for jobs on a (usually) remote resource such as a supercomputer
-
__dict__
= dict_proxy({'__module__': 'aiida.engine.processes.calcjobs.manager', '__dict__': <attribute '__dict__' of 'JobManager' objects>, 'request_job_info_update': <function request_job_info_update>, '__weakref__': <attribute '__weakref__' of 'JobManager' objects>, '__doc__': '\n A manager for jobs on a (usually) remote resource such as a supercomputer\n ', '__init__': <function __init__>})¶
-
__module__
= 'aiida.engine.processes.calcjobs.manager'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
request_job_info_update
(**kwds)[source]¶ Get a future that will resolve to information about a given job. This is a context manager so that if the user leaves the context the request is automatically cancelled.
Returns: A tuple containing the JobInfo object and detailed job info. Both can be None. Return type: tornado.concurrent.Future
-
-
class
aiida.engine.processes.calcjobs.tasks.
Waiting
(process, done_callback, msg=None, data=None)[source]¶ Bases:
plumpy.process_states.Waiting
The waiting state for the CalcJob process.
-
__init__
(process, done_callback, msg=None, data=None)[source]¶ Parameters: state_machine ( StateMachine
) – The process this state belongs to
-
__module__
= 'aiida.engine.processes.calcjobs.tasks'¶
-
execute
(**kwargs)[source]¶ Execute the state, performing the actions that this state is responsible for. Return a state to transition to or None if finished.
-
interrupt
(reason)[source]¶ Interrupt the Waiting state by calling interrupt on the transport task InterruptableFuture.
-
-
aiida.engine.processes.calcjobs.tasks.
task_kill_job
(*args, **kwargs)[source]¶ Transport task that will attempt to kill a job calculation
The task will first request a transport from the queue. Once the transport is yielded, the relevant execmanager function is called, wrapped in the exponential_backoff_retry coroutine, which, in case of a caught exception, will retry after an interval that increases exponentially with the number of retries, for a maximum number of retries. If all retries fail, the task will raise a TransportTaskException
Parameters: - node – the node that represents the job calculation
- transport_queue – the TransportQueue from which to request a Transport
- cancellable (
aiida.engine.utils.InterruptableFuture
) – the cancelled flag that will be queried to determine whether the task was cancelled
Raises: Return if the tasks was successfully completed
Raises: TransportTaskException if after the maximum number of retries the transport task still excepted
-
aiida.engine.processes.calcjobs.tasks.
task_retrieve_job
(*args, **kwargs)[source]¶ Transport task that will attempt to retrieve all files of a completed job calculation
The task will first request a transport from the queue. Once the transport is yielded, the relevant execmanager function is called, wrapped in the exponential_backoff_retry coroutine, which, in case of a caught exception, will retry after an interval that increases exponentially with the number of retries, for a maximum number of retries. If all retries fail, the task will raise a TransportTaskException
Parameters: - node – the node that represents the job calculation
- transport_queue – the TransportQueue from which to request a Transport
- cancellable (
aiida.engine.utils.InterruptableFuture
) – the cancelled flag that will be queried to determine whether the task was cancelled
Raises: Return if the tasks was successfully completed
Raises: TransportTaskException if after the maximum number of retries the transport task still excepted
-
aiida.engine.processes.calcjobs.tasks.
task_submit_job
(*args, **kwargs)[source]¶ Transport task that will attempt to submit a job calculation
The task will first request a transport from the queue. Once the transport is yielded, the relevant execmanager function is called, wrapped in the exponential_backoff_retry coroutine, which, in case of a caught exception, will retry after an interval that increases exponentially with the number of retries, for a maximum number of retries. If all retries fail, the task will raise a TransportTaskException
Parameters: - node – the node that represents the job calculation
- transport_queue – the TransportQueue from which to request a Transport
- calc_info – the calculation info datastructure returned by CalcJobNode._presubmit
- script_filename – the job launch script returned by CalcJobNode._presubmit
- cancellable (
aiida.engine.utils.InterruptableFuture
) – the cancelled flag that will be queried to determine whether the task was cancelled
Raises: Return if the tasks was successfully completed
Raises: TransportTaskException if after the maximum number of retries the transport task still excepted
-
aiida.engine.processes.calcjobs.tasks.
task_update_job
(*args, **kwargs)[source]¶ Transport task that will attempt to update the scheduler status of the job calculation
The task will first request a transport from the queue. Once the transport is yielded, the relevant execmanager function is called, wrapped in the exponential_backoff_retry coroutine, which, in case of a caught exception, will retry after an interval that increases exponentially with the number of retries, for a maximum number of retries. If all retries fail, the task will raise a TransportTaskException
Parameters: - node (
aiida.orm.nodes.process.calculation.calcjob.CalcJobNode
) – the node that represents the job calculation - job_manager (
aiida.engine.processes.calcjobs.manager.JobManager
) – The job manager - cancellable (
aiida.engine.utils.InterruptableFuture
) – A cancel flag
Raises: Return containing True if the tasks was successfully completed, False otherwise
- node (
-
aiida.engine.processes.calcjobs.tasks.
task_upload_job
(*args, **kwargs)[source]¶ Transport task that will attempt to upload the files of a job calculation to the remote
The task will first request a transport from the queue. Once the transport is yielded, the relevant execmanager function is called, wrapped in the exponential_backoff_retry coroutine, which, in case of a caught exception, will retry after an interval that increases exponentially with the number of retries, for a maximum number of retries. If all retries fail, the task will raise a TransportTaskException
Parameters: - node – the node that represents the job calculation
- transport_queue – the TransportQueue from which to request a Transport
- calc_info – the calculation info datastructure returned by CalcJobNode._presubmit
- script_filename – the job launch script returned by CalcJobNode._presubmit
- cancellable (
aiida.engine.utils.InterruptableFuture
) – the cancelled flag that will be queried to determine whether the task was cancelled
Raises: Return if the tasks was successfully completed
Raises: TransportTaskException if after the maximum number of retries the transport task still excepted