aiida.schedulers package#

Module for classes and utilities to interact with cluster schedulers.

Subpackages#

Submodules#

Data structures used by Scheduler instances.

In particular, there is the definition of possible job states (job_states), the data structure to be filled for job submission (JobTemplate), and the data structure that is returned when querying for jobs in the scheduler (JobInfo).

class aiida.schedulers.datastructures.JobInfo(dictionary=None)[source]#

Bases: DefaultFieldsAttributeDict

Contains properties for a job in the queue. Most of the fields are taken from DRMAA v.2.

Note that default fields may be undefined. This is an expected behavior and the application must cope with this case. An example for instance is the exit_status for jobs that have not finished yet; or features not supported by the given scheduler.

Fields:

  • job_id: the job ID on the scheduler

  • title: the job title, as known by the scheduler

  • exit_status: the exit status of the job as reported by the operating system on the execution host

  • terminating_signal: the UNIX signal that was responsible for the end of the job.

  • annotation: human-readable description of the reason for the job being in the current state or substate.

  • job_state: the job state (one of those defined in aiida.schedulers.datastructures.JobState)

  • job_substate: a string with the implementation-specific sub-state

  • allocated_machines: a list of machines used for the current job. This is a list of aiida.schedulers.datastructures.MachineInfo objects.

  • job_owner: the job owner as reported by the scheduler

  • num_mpiprocs: the total number of requested MPI procs

  • num_cpus: the total number of requested CPUs (cores) [may be undefined]

  • num_machines: the number of machines (i.e., nodes), required by the job. If allocated_machines is not None, this number must be equal to len(allocated_machines). Otherwise, for schedulers not supporting the retrieval of the full list of allocated machines, this attribute can be used to know at least the number of machines.

  • queue_name: The name of the queue in which the job is queued or running.

  • account: The account/projectid in which the job is queued or running in.

  • qos: The quality of service in which the job is queued or running in.

  • wallclock_time_seconds: the accumulated wallclock time, in seconds

  • requested_wallclock_time_seconds: the requested wallclock time, in seconds

  • cpu_time: the accumulated cpu time, in seconds

  • submission_time: the absolute time at which the job was submitted, of type datetime.datetime

  • dispatch_time: the absolute time at which the job first entered the ‘started’ state, of type datetime.datetime

  • finish_time: the absolute time at which the job first entered the ‘finished’ state, of type datetime.datetime

__annotations__ = {}#
__module__ = 'aiida.schedulers.datastructures'#
_default_fields = ('job_id', 'title', 'exit_status', 'terminating_signal', 'annotation', 'job_state', 'job_substate', 'allocated_machines', 'job_owner', 'num_mpiprocs', 'num_cpus', 'num_machines', 'queue_name', 'account', 'qos', 'wallclock_time_seconds', 'requested_wallclock_time_seconds', 'cpu_time', 'submission_time', 'dispatch_time', 'finish_time')#
static _deserialize_date(value)[source]#

Deserialise a date :param value: The date vlue :return: The deserialised date

static _deserialize_job_state(job_state)[source]#

Return an instance of JobState from the job_state string.

static _serialize_date(value)[source]#

Serialise a data value :param value: The value to serialise :return: The serialised value

static _serialize_job_state(job_state)[source]#

Return the serialized value of the JobState instance.

_special_serializers = {'dispatch_time': 'date', 'finish_time': 'date', 'job_state': 'job_state', 'submission_time': 'date'}#
classmethod deserialize_field(value, field_type)[source]#

Deserialise the value of a particular field with a type :param value: The value :param field_type: The field type :return: The deserialised value

get_dict()[source]#

Serialise the current data into a dictionary that is JSON-serializable.

Returns:

A dictionary

classmethod load_from_dict(data)[source]#

Create a new instance loading the values from serialised data in dictionary form

Parameters:

data – The dictionary with the data to load from

classmethod load_from_serialized(data)[source]#

Create a new instance loading the values from JSON-serialised data as a string

Parameters:

data – The string with the JSON-serialised data to load from

serialize()[source]#

Serialize the current data (as obtained by self.get_dict()) into a JSON string.

Returns:

A string with serialised representation of the current data.

classmethod serialize_field(value, field_type)[source]#

Serialise a particular field value

Parameters:

  • value – The value to serialise

  • field_type – The field type

Returns:

The serialised value

class aiida.schedulers.datastructures.JobResource(dictionary=None)[source]#

Bases: DefaultFieldsAttributeDict

Data structure to store job resources.

Each Scheduler implementation must define the _job_resource_class attribute to be a subclass of this class. It should at least define the get_tot_num_mpiprocs method, plus a constructor to accept its set of variables.

Typical attributes are:

  • num_machines

  • num_mpiprocs_per_machine

or (e.g. for SGE)

  • tot_num_mpiprocs

  • parallel_env

The constructor should take care of checking the values. The init should raise only ValueError or TypeError on invalid parameters.

__abstractmethods__ = frozenset({'accepts_default_mpiprocs_per_machine', 'get_tot_num_mpiprocs', 'validate_resources'})#
__module__ = 'aiida.schedulers.datastructures'#
_abc_impl = <_abc._abc_data object>#
_default_fields = ()#
classmethod accepts_default_memory_per_machine()[source]#

Return True if this subclass accepts a default_memory_per_machine key, False otherwise.

abstract classmethod accepts_default_mpiprocs_per_machine()[source]#

Return True if this subclass accepts a default_mpiprocs_per_machine key, False otherwise.

abstract get_tot_num_mpiprocs()[source]#

Return the total number of cpus of this job resource.

classmethod get_valid_keys()[source]#

Return a list of valid keys to be passed to the constructor.

abstract classmethod validate_resources(**kwargs)[source]#

Validate the resources against the job resource class of this scheduler.

Parameters:

kwargs – dictionary of values to define the job resources

Raises:

ValueError – if the resources are invalid or incomplete

Returns:

optional tuple of parsed resource settings

class aiida.schedulers.datastructures.JobState(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: Enum

Enumeration of possible scheduler states of a CalcJob.

There is no FAILED state as every completed job is put in DONE, regardless of success.

DONE = 'done'#
QUEUED = 'queued'#
QUEUED_HELD = 'queued held'#
RUNNING = 'running'#
SUSPENDED = 'suspended'#
UNDETERMINED = 'undetermined'#
__module__ = 'aiida.schedulers.datastructures'#
class aiida.schedulers.datastructures.JobTemplate(dictionary=None)[source]#

Bases: DefaultFieldsAttributeDict

A template for submitting jobs to a scheduler.

This contains all required information to create the job header.

The required fields are: working_directory, job_name, num_machines, num_mpiprocs_per_machine, argv.

Fields:

  • shebang line: The first line of the submission script

  • submit_as_hold: if set, the job will be in a ‘hold’ status right after the submission

  • rerunnable: if the job is rerunnable (boolean)

  • job_environment: a dictionary with environment variables to set before the execution of the code.

  • environment_variables_double_quotes: if set to True, use double quotes instead of single quotes to escape the environment variables specified in job_environment.

  • working_directory: the working directory for this job. During submission, the transport will first do a ‘chdir’ to this directory, and then possibly set a scheduler parameter, if this is supported by the scheduler.

  • email: an email address for sending emails on job events.

  • email_on_started: if True, ask the scheduler to send an email when the job starts.

  • email_on_terminated: if True, ask the scheduler to send an email when the job ends. This should also send emails on job failure, when possible.

  • job_name: the name of this job. The actual name of the job can be different from the one specified here, e.g. if there are unsupported characters, or the name is too long.

  • sched_output_path: a (relative) file name for the stdout of this job

  • sched_error_path: a (relative) file name for the stdout of this job

  • sched_join_files: if True, write both stdout and stderr on the same file (the one specified for stdout)

  • queue_name: the name of the scheduler queue (sometimes also called partition), on which the job will be submitted.

  • account: the name of the scheduler account (sometimes also called projectid), on which the job will be submitted.

  • qos: the quality of service of the scheduler account, on which the job will be submitted.

  • job_resource: a suitable JobResource subclass with information on how many nodes and cpus it should use. It must be an instance of the aiida.schedulers.Scheduler.job_resource_class class. Use the Scheduler.create_job_resource method to create it.

  • num_machines: how many machines (or nodes) should be used

  • num_mpiprocs_per_machine: how many MPI procs should be used on each machine (or node).

  • priority: a priority for this job. Should be in the format accepted by the specific scheduler.

  • max_memory_kb: The maximum amount of memory the job is allowed to allocate ON EACH NODE, in kilobytes

  • max_wallclock_seconds: The maximum wall clock time that all processes of a job are allowed to exist, in seconds

  • custom_scheduler_commands: a string that will be inserted right after the last scheduler command, and before any other non-scheduler command; useful if some specific flag needs to be added and is not supported by the plugin

  • prepend_text: a (possibly multi-line) string to be inserted in the scheduler script before the main execution line

  • append_text: a (possibly multi-line) string to be inserted in the scheduler script after the main execution line

  • import_sys_environment: import the system environment variables

  • codes_info: a list of aiida.scheduler.datastructures.JobTemplateCodeInfo objects. Each contains the information necessary to run a single code. At the moment, it can contain:

    • cmdline_parameters: a list of strings with the command line arguments of the program to run. This is the main program to be executed. NOTE: The first one is the executable name. For MPI runs, this will probably be “mpirun” or a similar program; this has to be chosen at a upper level.

    • stdin_name: the (relative) file name to be used as stdin for the program specified with argv.

    • stdout_name: the (relative) file name to be used as stdout for the program specified with argv.

    • stderr_name: the (relative) file name to be used as stderr for the program specified with argv.

    • join_files: if True, stderr is redirected on the same file specified for stdout.

  • codes_run_mode: sets the run_mode with which the (multiple) codes have to be executed. For example, parallel execution:

    mpirun -np 8 a.x &
mpirun -np 8 b.x &
wait

    The serial execution would be without the &’s. Values are given by aiida.common.datastructures.CodeRunMode.

__annotations__ = {}#
__module__ = 'aiida.schedulers.datastructures'#
_default_fields = ('shebang', 'submit_as_hold', 'rerunnable', 'job_environment', 'environment_variables_double_quotes', 'working_directory', 'email', 'email_on_started', 'email_on_terminated', 'job_name', 'sched_output_path', 'sched_error_path', 'sched_join_files', 'queue_name', 'account', 'qos', 'job_resource', 'priority', 'max_memory_kb', 'max_wallclock_seconds', 'custom_scheduler_commands', 'prepend_text', 'append_text', 'import_sys_environment', 'codes_run_mode', 'codes_info')#
class aiida.schedulers.datastructures.JobTemplateCodeInfo(prepend_cmdline_params: list[str] = <factory>, cmdline_params: list[str] = <factory>, use_double_quotes: list[bool] = <factory>, wrap_cmdline_params: bool = False, stdin_name: None | str = None, stdout_name: None | str = None, stderr_name: None | str = None, join_files: bool = False)[source]#

Bases: object

Data structure to communicate to a Scheduler how a code should be run in submit script.

Scheduler.get_submit_script will pass a list of these objects to Scheduler._get_run_line which should build up the code execution line based on the parameters specified in this dataclass.

Parameters:

  • preprend_cmdline_params – list of unescaped command line arguments that are to be prepended to the executable.

  • cmdline_params – list of unescaped command line parameters.

  • use_double_quotes – list of two booleans. If true, use double quotes to escape command line arguments. The first value applies to prepend_cmdline_params and the second to cmdline_params.

  • wrap_cmdline_params – Boolean, by default False. If set to True, all the command line arguments, which includes the cmdline_params but also all file descriptor redirections (stdin, stderr and stdoout), should be wrapped in double quotes, turning it into a single command line argument. This is necessary to enable support for certain containerization technologies such as Docker.

  • stdin_name – filename of the the stdin file descriptor.

  • stdout_name – filename of the the stdout file descriptor.

  • stderr_name – filename of the the stderr file descriptor.

  • join_files – boolean, if true, stderr should be redirected to stdout.

__annotations__ = {'cmdline_params': 'list[str]', 'join_files': 'bool', 'prepend_cmdline_params': 'list[str]', 'stderr_name': 'None | str', 'stdin_name': 'None | str', 'stdout_name': 'None | str', 'use_double_quotes': 'list[bool]', 'wrap_cmdline_params': 'bool'}#
__dataclass_fields__ = {'cmdline_params': Field(name='cmdline_params',type='list[str]',default=<dataclasses._MISSING_TYPE object>,default_factory=<class 'list'>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'join_files': Field(name='join_files',type='bool',default=False,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'prepend_cmdline_params': Field(name='prepend_cmdline_params',type='list[str]',default=<dataclasses._MISSING_TYPE object>,default_factory=<class 'list'>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'stderr_name': Field(name='stderr_name',type='None | str',default=None,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'stdin_name': Field(name='stdin_name',type='None | str',default=None,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'stdout_name': Field(name='stdout_name',type='None | str',default=None,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'use_double_quotes': Field(name='use_double_quotes',type='list[bool]',default=<dataclasses._MISSING_TYPE object>,default_factory=<function JobTemplateCodeInfo.<lambda>>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'wrap_cmdline_params': Field(name='wrap_cmdline_params',type='bool',default=False,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD)}#
__dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,order=False,unsafe_hash=False,frozen=False)#
__dict__ = mappingproxy({'__module__': 'aiida.schedulers.datastructures', '__annotations__': {'prepend_cmdline_params': 'list[str]', 'cmdline_params': 'list[str]', 'use_double_quotes': 'list[bool]', 'wrap_cmdline_params': 'bool', 'stdin_name': 'None | str', 'stdout_name': 'None | str', 'stderr_name': 'None | str', 'join_files': 'bool'}, '__doc__': 'Data structure to communicate to a `Scheduler` how a code should be run in submit script.\n\n    `Scheduler.get_submit_script` will pass a list of these objects to `Scheduler._get_run_line` which\n    should build up the code execution line based on the parameters specified in this dataclass.\n\n    :param preprend_cmdline_params: list of unescaped command line arguments that are to be prepended to the executable.\n    :param cmdline_params: list of unescaped command line parameters.\n    :param use_double_quotes: list of two booleans. If true, use double quotes to escape command line arguments. The\n        first value applies to `prepend_cmdline_params` and the second to `cmdline_params`.\n    :param wrap_cmdline_params: Boolean, by default ``False``. If set to ``True``, all the command line arguments,\n        which includes the ``cmdline_params`` but also all file descriptor redirections (stdin, stderr and stdoout),\n        should be wrapped in double quotes, turning it into a single command line argument. This is necessary to enable\n        support for certain containerization technologies such as Docker.\n    :param stdin_name: filename of the the stdin file descriptor.\n    :param stdout_name: filename of the the `stdout` file descriptor.\n    :param stderr_name: filename of the the `stderr` file descriptor.\n    :param join_files: boolean, if true, `stderr` should be redirected to `stdout`.\n    ', 'wrap_cmdline_params': False, 'stdin_name': None, 'stdout_name': None, 'stderr_name': None, 'join_files': False, '__dict__': <attribute '__dict__' of 'JobTemplateCodeInfo' objects>, '__weakref__': <attribute '__weakref__' of 'JobTemplateCodeInfo' objects>, '__dataclass_params__': _DataclassParams(init=True,repr=True,eq=True,order=False,unsafe_hash=False,frozen=False), '__dataclass_fields__': {'prepend_cmdline_params': Field(name='prepend_cmdline_params',type='list[str]',default=<dataclasses._MISSING_TYPE object>,default_factory=<class 'list'>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'cmdline_params': Field(name='cmdline_params',type='list[str]',default=<dataclasses._MISSING_TYPE object>,default_factory=<class 'list'>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'use_double_quotes': Field(name='use_double_quotes',type='list[bool]',default=<dataclasses._MISSING_TYPE object>,default_factory=<function JobTemplateCodeInfo.<lambda>>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'wrap_cmdline_params': Field(name='wrap_cmdline_params',type='bool',default=False,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'stdin_name': Field(name='stdin_name',type='None | str',default=None,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'stdout_name': Field(name='stdout_name',type='None | str',default=None,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'stderr_name': Field(name='stderr_name',type='None | str',default=None,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'join_files': Field(name='join_files',type='bool',default=False,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD)}, '__init__': <function JobTemplateCodeInfo.__init__>, '__repr__': <function JobTemplateCodeInfo.__repr__>, '__eq__': <function JobTemplateCodeInfo.__eq__>, '__hash__': None, '__match_args__': ('prepend_cmdline_params', 'cmdline_params', 'use_double_quotes', 'wrap_cmdline_params', 'stdin_name', 'stdout_name', 'stderr_name', 'join_files')})#
__eq__(other)#

Return self==value.

__hash__ = None#
__init__(prepend_cmdline_params: list[str] = <factory>, cmdline_params: list[str] = <factory>, use_double_quotes: list[bool] = <factory>, wrap_cmdline_params: bool = False, stdin_name: None | str = None, stdout_name: None | str = None, stderr_name: None | str = None, join_files: bool = False) → None#
__match_args__ = ('prepend_cmdline_params', 'cmdline_params', 'use_double_quotes', 'wrap_cmdline_params', 'stdin_name', 'stdout_name', 'stderr_name', 'join_files')#
__module__ = 'aiida.schedulers.datastructures'#
__repr__()#

Return repr(self).

__weakref__#

list of weak references to the object

cmdline_params: list[str]#
join_files: bool = False#
prepend_cmdline_params: list[str]#
stderr_name: None | str = None#
stdin_name: None | str = None#
stdout_name: None | str = None#
use_double_quotes: list[bool]#
wrap_cmdline_params: bool = False#
class aiida.schedulers.datastructures.MachineInfo(dictionary=None)[source]#

Bases: DefaultFieldsAttributeDict

Similarly to what is defined in the DRMAA v.2 as SlotInfo; this identifies each machine (also called ‘node’ on some schedulers) on which a job is running, and how many CPUs are being used. (Some of them could be undefined)

  • name: name of the machine

  • num_cpus: number of cores used by the job on this machine

  • num_mpiprocs: number of MPI processes used by the job on this machine

__annotations__ = {}#
__module__ = 'aiida.schedulers.datastructures'#
_default_fields = ('name', 'num_mpiprocs', 'num_cpus')#
class aiida.schedulers.datastructures.NodeNumberJobResource(**kwargs)[source]#

Bases: JobResource

JobResource for schedulers that support the specification of a number of nodes and cpus per node.

__abstractmethods__ = frozenset({})#
__annotations__ = {'num_cores_per_machine': 'int', 'num_cores_per_mpiproc': 'int', 'num_machines': 'int', 'num_mpiprocs_per_machine': 'int'}#
__init__(**kwargs)[source]#

Initialize the job resources from the passed arguments.

Raises:

ValueError – if the resources are invalid or incomplete

__module__ = 'aiida.schedulers.datastructures'#
_abc_impl = <_abc._abc_data object>#
_default_fields = ('num_machines', 'num_mpiprocs_per_machine', 'num_cores_per_machine', 'num_cores_per_mpiproc')#
classmethod accepts_default_mpiprocs_per_machine()[source]#

Return True if this subclass accepts a default_mpiprocs_per_machine key, False otherwise.

get_tot_num_mpiprocs()[source]#

Return the total number of cpus of this job resource.

classmethod get_valid_keys()[source]#

Return a list of valid keys to be passed to the constructor.

classmethod validate_resources(**kwargs)[source]#

Validate the resources against the job resource class of this scheduler.

Parameters:

kwargs – dictionary of values to define the job resources

Returns:

attribute dictionary with the parsed parameters populated

Raises:

ValueError – if the resources are invalid or incomplete

class aiida.schedulers.datastructures.ParEnvJobResource(**kwargs)[source]#

Bases: JobResource

JobResource for schedulers that support the specification of a parallel environment and number of MPI procs.

__abstractmethods__ = frozenset({})#
__annotations__ = {'parallel_env': 'str', 'tot_num_mpiprocs': 'int'}#
__init__(**kwargs)[source]#

Initialize the job resources from the passed arguments (the valid keys can be obtained with the function self.get_valid_keys()).

Raises:

ValueError – if the resources are invalid or incomplete

__module__ = 'aiida.schedulers.datastructures'#
_abc_impl = <_abc._abc_data object>#
_default_fields = ('parallel_env', 'tot_num_mpiprocs')#
classmethod accepts_default_mpiprocs_per_machine()[source]#

Return True if this subclass accepts a default_mpiprocs_per_machine key, False otherwise.

get_tot_num_mpiprocs()[source]#

Return the total number of cpus of this job resource.

classmethod validate_resources(**kwargs)[source]#

Validate the resources against the job resource class of this scheduler.

Parameters:

kwargs – dictionary of values to define the job resources

Returns:

attribute dictionary with the parsed parameters populated

Raises:

ValueError – if the resources are invalid or incomplete

Implementation of Scheduler base class.

class aiida.schedulers.scheduler.Scheduler[source]#

Bases: object

Base class for a job scheduler.

__abstractmethods__ = frozenset({'_get_joblist_command', '_get_kill_command', '_get_submit_command', '_get_submit_script_header', '_parse_joblist_output', '_parse_kill_output', '_parse_submit_output'})#
__annotations__ = {'_features': 'dict[str, bool]', '_job_resource_class': 't.Type[JobResource] | None'}#
__dict__ = mappingproxy({'__module__': 'aiida.schedulers.scheduler', '__annotations__': {'_features': 'dict[str, bool]', '_job_resource_class': 't.Type[JobResource] | None'}, '__doc__': 'Base class for a job scheduler.', '_logger': <Logger aiida.scheduler (WARNING)>, '_features': {}, '_job_resource_class': None, '__str__': <function Scheduler.__str__>, 'preprocess_resources': <classmethod(<function Scheduler.preprocess_resources>)>, 'validate_resources': <classmethod(<function Scheduler.validate_resources>)>, '__init__': <function Scheduler.__init__>, 'get_short_doc': <classmethod(<function Scheduler.get_short_doc>)>, 'get_feature': <function Scheduler.get_feature>, 'logger': <property object>, 'job_resource_class': <aiida.common.lang.classproperty object>, 'create_job_resource': <classmethod(<function Scheduler.create_job_resource>)>, 'get_submit_script': <function Scheduler.get_submit_script>, '_get_submit_script_environment_variables': <function Scheduler._get_submit_script_environment_variables>, '_get_submit_script_header': <function Scheduler._get_submit_script_header>, '_get_submit_script_footer': <function Scheduler._get_submit_script_footer>, '_get_run_line': <function Scheduler._get_run_line>, '_get_joblist_command': <function Scheduler._get_joblist_command>, '_get_detailed_job_info_command': <function Scheduler._get_detailed_job_info_command>, 'get_detailed_job_info': <function Scheduler.get_detailed_job_info>, '_parse_joblist_output': <function Scheduler._parse_joblist_output>, 'get_jobs': <function Scheduler.get_jobs>, 'transport': <property object>, 'set_transport': <function Scheduler.set_transport>, '_get_submit_command': <function Scheduler._get_submit_command>, '_parse_submit_output': <function Scheduler._parse_submit_output>, 'submit_from_script': <function Scheduler.submit_from_script>, 'kill': <function Scheduler.kill>, '_get_kill_command': <function Scheduler._get_kill_command>, '_parse_kill_output': <function Scheduler._parse_kill_output>, 'parse_output': <function Scheduler.parse_output>, '__dict__': <attribute '__dict__' of 'Scheduler' objects>, '__weakref__': <attribute '__weakref__' of 'Scheduler' objects>, '__abstractmethods__': frozenset({'_parse_joblist_output', '_parse_submit_output', '_get_submit_script_header', '_parse_kill_output', '_get_kill_command', '_get_submit_command', '_get_joblist_command'}), '_abc_impl': <_abc._abc_data object>})#
__init__()[source]#
__module__ = 'aiida.schedulers.scheduler'#
__str__()[source]#

Return str(self).

__weakref__#

list of weak references to the object

_abc_impl = <_abc._abc_data object>#
_features: dict[str, bool] = {}#
_get_detailed_job_info_command(job_id: str) → dict[str, Any][source]#

Return the command to run to get detailed information for a given job.

This is typically called after the job has finished, to retrieve the most detailed information possible about the job. This is done because most schedulers just make finished jobs disappear from the qstat command, and instead sometimes it is useful to know some more detailed information about the job exit status, etc.

Raises:

aiida.common.exceptions.FeatureNotAvailable

abstract _get_joblist_command(jobs: list[str] | None = None, user: str | None = None) → str[source]#

Return the command to get the most complete description possible of currently active jobs.

Note

Typically one can pass only either jobs or user, depending on the specific plugin. The choice can be done according to the value returned by self.get_feature(‘can_query_by_user’)

Parameters:

  • jobs – either None to get a list of all jobs in the machine, or a list of jobs.

  • user – either None, or a string with the username (to show only jobs of the specific user).

abstract _get_kill_command(jobid: str) → str[source]#

Return the command to kill the job with specified jobid.

_get_run_line(codes_info: list[JobTemplateCodeInfo], codes_run_mode: CodeRunMode) → str[source]#

Return a string with the line to execute a specific code with specific arguments.

Parameters:

  • codes_info – a list of aiida.scheduler.datastructures.JobTemplateCodeInfo objects. Each contains the information needed to run the code. I.e. cmdline_params, stdin_name, stdout_name, stderr_name, join_files. See the documentation of JobTemplate and JobTemplateCodeInfo.

  • codes_run_mode – instance of aiida.common.datastructures.CodeRunMode contains the information on how to launch the multiple codes.

Returns:

string with format: [executable] [args] {[ < stdin ]} {[ < stdout ]} {[2>&1 | 2> stderr]}

abstract _get_submit_command(submit_script: str) → str[source]#

Return the string to execute to submit a given script.

Warning

the submit_script should already have been bash-escaped

Parameters:

submit_script – the path of the submit script relative to the working directory.

Returns:

the string to execute to submit a given script.

_get_submit_script_environment_variables(template: JobTemplate) → str[source]#

Return the part of the submit script header that defines environment variables.

Parameters:

template – a aiida.schedulers.datastrutures.JobTemplate instance.

Returns:

string containing environment variable declarations.

Return the submit script final part, using the parameters from the job template.

Parameters:

job_tmpl – a JobTemplate instance with relevant parameters set.

Returns:

string with the submission script footer.

abstract _get_submit_script_header(job_tmpl: JobTemplate) → str[source]#

Return the submit script header, using the parameters from the job template.

Parameters:

job_tmpl – a JobTemplate instance with relevant parameters set.

Returns:

string with the submission script header.

_job_resource_class: Type[JobResource] | None = None#
_logger = <Logger aiida.scheduler (WARNING)>#
abstract _parse_joblist_output(retval: int, stdout: str, stderr: str) → list[JobInfo][source]#

Parse the joblist output as returned by executing the command returned by _get_joblist_command method.

Returns:

list of JobInfo objects, one of each job each with at least its default params implemented.

abstract _parse_kill_output(retval: int, stdout: str, stderr: str) → bool[source]#

Parse the output of the kill command.

Returns:

True if everything seems ok, False otherwise.

abstract _parse_submit_output(retval: int, stdout: str, stderr: str) → str | ExitCode[source]#

Parse the output of the submit command returned by calling the _get_submit_command command.

Returns:

a string with the job ID or an exit code if the submission failed because the submission script is invalid and the job should be terminated.

classmethod create_job_resource(**kwargs)[source]#

Create a suitable job resource from the kwargs specified.

get_detailed_job_info(job_id: str) → dict[str, str | int][source]#

Return the detailed job info.

This will be a dictionary with the return value, stderr and stdout content returned by calling the command that is returned by _get_detailed_job_info_command.

Parameters:

job_id – the job identifier

Returns:

dictionary with retval, stdout and stderr.

get_feature(feature_name: str) → bool[source]#
get_jobs(jobs: list[str] | None = None, user: str | None = None, as_dict: bool = False) → list[JobInfo] | dict[str, JobInfo][source]#

Return the list of currently active jobs.

Note

typically, only either jobs or user can be specified. See also comments in _get_joblist_command.

Parameters:

  • jobs (list) – a list of jobs to check; only these are checked

  • user (str) – a string with a user: only jobs of this user are checked

  • as_dict (list) – if False (default), a list of JobInfo objects is returned. If True, a dictionary is returned, having as key the job_id and as value the JobInfo object.

Returns:

list of active jobs

classmethod get_short_doc()[source]#

Return the first non-empty line of the class docstring, if available.

get_submit_script(job_tmpl: JobTemplate) → str[source]#

Return the submit script as a string.

Parameters:

job_tmpl – a aiida.schedulers.datastrutures.JobTemplate instance.

The plugin returns something like

#!/bin/bash <- this shebang line is configurable to some extent scheduler_dependent stuff to choose numnodes, numcores, walltime, … prepend_computer [also from calcinfo, joined with the following?] prepend_code [from calcinfo] output of _get_script_main_content postpend_code postpend_computer

job_resource_class#

A class that, when used as a decorator, works as if the two decorators @property and @classmethod where applied together (i.e., the object works as a property, both for the Class and for any of its instance; and is called with the class cls rather than with the instance as its first argument).

kill(jobid: str) → bool[source]#

Kill a remote job and parse the return value of the scheduler to check if the command succeeded.

..note:

On some schedulers, even if the command is accepted, it may take some seconds for the job to actually
disappear from the queue.
Parameters:

jobid – the job ID to be killed

Returns:

True if everything seems ok, False otherwise.

property logger#

Return the internal logger.

parse_output(detailed_job_info: dict[str, str | int] | None = None, stdout: str | None = None, stderr: str | None = None) → ExitCode | None[source]#

Parse the output of the scheduler.

Parameters:

  • detailed_job_info – dictionary with the output returned by the Scheduler.get_detailed_job_info command. This should contain the keys retval, stdout and stderr corresponding to the return value, stdout and stderr returned by the accounting command executed for a specific job id.

  • stdout – string with the output written by the scheduler to stdout.

  • stderr – string with the output written by the scheduler to stderr.

Returns:

None or an instance of aiida.engine.processes.exit_code.ExitCode.

classmethod preprocess_resources(resources: dict[str, Any], default_mpiprocs_per_machine: int | None = None)[source]#

Pre process the resources.

Add the num_mpiprocs_per_machine key to the resources if it is not already defined and it cannot be deduced from the num_machines and tot_num_mpiprocs being defined. The value is also not added if the job resource class of this scheduler does not accept the num_mpiprocs_per_machine keyword. Note that the changes are made in place to the resources argument passed.

set_transport(transport: Transport)[source]#

Set the transport to be used to query the machine or to submit scripts.

This class assumes that the transport is open and active.

submit_from_script(working_directory: str, submit_script: str) → str | ExitCode[source]#

Submit the submission script to the scheduler.

Returns:

return a string with the job ID in a valid format to be used for querying.

property transport#

Return the transport set for this scheduler.

classmethod validate_resources(**resources)[source]#

Validate the resources against the job resource class of this scheduler.

Parameters:

resources – keyword arguments to define the job resources

Raises:

ValueError – if the resources are invalid or incomplete

exception aiida.schedulers.scheduler.SchedulerError[source]#

Bases: AiidaException

__annotations__ = {}#
__module__ = 'aiida.schedulers.scheduler'#
exception aiida.schedulers.scheduler.SchedulerParsingError[source]#

Bases: SchedulerError

__annotations__ = {}#
__module__ = 'aiida.schedulers.scheduler'#
On this page
Edit on GitHub
Show Source