How to run external codes¶

To run an external code with AiiDA, you will need to use an appropriate calculation plugin. This plugin must contain the instructions necessary for the engine to be able to:

Prepare the required input files inside of the folder in which the code will be executed
Run the code with the correct set of command line parameters

The following subsections will not only take you through the process of creating the calculation plugin and then using these to actually run the code. It will also show examples on how to implement tools that are commonly coupled with the running of a calculation, such as the parsing of outputs.

Some general guidelines to keep in mind are:

Check existing resources.

Before starting to write a plugin, check on the aiida plugin registry whether a plugin for your code is already available. If it is, there is maybe no need to write your own, and you can skip straight ahead to running the code.

Start simple.

Make use of existing classes like Dict, SinglefileData, … Write only what is necessary to pass information from and to AiiDA.

Don’t break data provenance.

Store at least what is needed for full reproducibility.

Expose the full functionality.

Standardization is good but don’t artificially limit the power of a code you are wrapping - or your users will get frustrated. If the code can do it, there should be some way to do it with your plugin.

Don’t rely on AiiDA internals. Functionality at deeper nesting levels is not considered part of the public API and may change between minor AiiDA releases, breaking your plugin.

Parse what you want to query for.

Make a list of which information to:

parse into the database for querying (Dict, …)

store in the file repository for safe-keeping (SinglefileData, …)

leave on the computer where the calculation ran (RemoteData, …)

To demonstrate how to create a plugin for an external code, we will use the trivial example of using the bash shell (/bin/bash) to sum two numbers by running the command: echo $(( numx + numy )). Here, the bash binary will be effectively acting as our Code executable, the input (aiida.in) will then be a file containing the command with the numbers provided by the user replaced, and the output (aiida.out) will be caught through the standard output. The final recipe to run this code will then be:

/bin/bash < aiida.in > aiida.out

Interfacing external codes¶

To provide AiiDA with the set of instructions, required to run a code, one should subclass the CalcJob class and implement the following two key methods:

define()

prepare_for_submission()

We will now show how each of these can be implemented.

Defining the specifications¶

The define method is where one specifies the different inputs that the caller of the CalcJob will have to provide in order to run the code, as well as the outputs that will be produced (exit codes will be discussed later). This is done through an instance of CalcJobProcessSpec, which, as can be seen in the snippet below, is passed as the spec argument to the define method. For the code that adds up two numbers, we will need to define those numbers as inputs (let’s call them x and y to label them) and the result as an output (sum). The snippet below shows one potential implementation, as it is included in aiida-core:

@classmethod
def define(cls, spec: CalcJobProcessSpec):
    """Define the process specification, including its inputs, outputs and known exit codes.

    :param spec: the calculation job process spec to define.
    """
    super().define(spec)
    spec.inputs['metadata']['options']['parser_name'].default = 'arithmetic.add'
    spec.inputs['metadata']['options']['input_filename'].default = 'aiida.in'
    spec.inputs['metadata']['options']['output_filename'].default = 'aiida.out'
    spec.inputs['metadata']['options']['resources'].default = {'num_machines': 1, 'num_mpiprocs_per_machine': 1}
    spec.input('x', valid_type=(orm.Int, orm.Float), help='The left operand.')
    spec.input('y', valid_type=(orm.Int, orm.Float), help='The right operand.')
    spec.output('sum', valid_type=(orm.Int, orm.Float), help='The sum of the left and right operand.')
    # start exit codes - marker for docs
    spec.exit_code(300, 'ERROR_NO_RETRIEVED_FOLDER', message='The retrieved output node does not exist.')
    spec.exit_code(310, 'ERROR_READING_OUTPUT_FILE', message='The output file could not be read.')
    spec.exit_code(320, 'ERROR_INVALID_OUTPUT', message='The output file contains invalid output.')
    spec.exit_code(410, 'ERROR_NEGATIVE_NUMBER', message='The sum of the operands is a negative number.')

The first line of the define implementation calls the method of the parent class CalcJob. This step is crucial as it will define inputs and outputs that are common to all CalcJob’s and failing to do so will leave the implementation broken. After the super call, we modify the default values for some of these inputs that are defined by the base class. Inputs that have already been defined can be accessed from the spec through the inputs attribute, which behaves like a normal dictionary.

After modifying the existing inputs, we define the inputs that are specific to this code. For this purpose we use the input() method, which does not modify the existing inputs, accessed through inputs, but defines new ones that will be specific to this implementation. You can also see that the definitions do not involve the assignment of a value, but only the passing of parameters to the method: a label to identify it, their valid types (in this case nodes of type Int) and a description. Finally, note that there is no return statement: this method does not need to return anything, since all modifications are made directly into the received spec object. You can check the Topics section about defining processes if you want more information about setting up your inputs and outputs (covering validation, dynamic number of inputs, etc.).

Preparing for submission¶

The prepare_for_submission() method is used for two purposes. Firstly, it should create the input files, based on the input nodes passed to the calculation, in the format that the external code will expect. Secondly, the method should create and return a CalcInfo instance that contains various instructions for the engine on how the code should be run. An example implementation, as shipped with aiida-core can be seen in the following snippet:

def prepare_for_submission(self, folder: Folder) -> CalcInfo:
    """Prepare the calculation for submission.

    Convert the input nodes into the corresponding input files in the format that the code will expect. In addition,
    define and return a `CalcInfo` instance, which is a simple data structure that contains information for the
    engine, for example, on what files to copy to the remote machine, what files to retrieve once it has completed,
    specific scheduler settings and more.

    :param folder: a temporary folder on the local file system.
    :returns: the `CalcInfo` instance
    """
    with folder.open(self.options.input_filename, 'w', encoding='utf8') as handle:
        handle.write('echo $(({x} + {y}))\n'.format(x=self.inputs.x.value, y=self.inputs.y.value))

    codeinfo = CodeInfo()
    codeinfo.code_uuid = self.inputs.code.uuid
    codeinfo.stdin_name = self.options.input_filename
    codeinfo.stdout_name = self.options.output_filename

    calcinfo = CalcInfo()
    calcinfo.codes_info = [codeinfo]
    calcinfo.retrieve_list = [self.options.output_filename]

    return calcinfo

Note that, unlike the define method, this one is implemented from scratch and so there is no super call. The external code that we are running with this CalcJob is bash and so to sum the input numbers x and y, we should write a bash input file that performs the summation, for example echo $((x + y)), where one of course has to replace x and y with the actual numbers. You can see how the snippet uses the folder argument, which is a Folder instance that represents a temporary folder on disk, to write the input file with the bash summation. It uses Python’s string interpolation to replace the x and y placeholders with the actual values that were passed as input, self.inputs.x and self.inputs.y, respectively.

Note

When the prepare_for_submission is called, the inputs that have been passed will have been validated against the specification defined in the define method and they can be accessed through the inputs attribute. This means that if a particular input is required according to the spec, you can safely assume that it will have been set and you do not need to check explicitly for its existence.

All the files that are copied into the sandbox folder will be automatically copied by the engine to the scratch directory where the code will be run. In this case we only create one input file, but you can create as many as you need, including subfolders if required.

Note

The input files written to the folder sandbox, will also be permanently stored in the file repository of the calculation node for the purpose of additional provenance guarantees. See the section on excluding files from provenance to learn how to prevent certain input files from being stored explicitly.

After having written the necessary input files, one should create the CodeInfo object, which can be used to instruct the engine on how to run the code. We assign the code_uuid attribute to the uuid of the Code node that was passed as an input, which can be retrieved through self.inputs.code. This is necessary such that the engine can retrieve the required information from the Code node, such as the full path of the executable. Note that we didn’t explicitly define this Code input in the define method, but this is one of the inputs defined in the base CalcJob class:

spec.input('code', valid_type=orm.Code, help='The `Code` to use for this job.')

After defining the UUID of the code node that the engine should use, we define the filenames where the stdin and stdout file descriptors should be redirected to. These values are taken from the inputs, which are part of the metadata.options namespace, for some of whose inputs we overrode the default values in the specification definition in the previous section. Note that instead of retrieving them through self.inputs.metadata['options']['input_filename'], one can use the shortcut self.options.input_filename as we do here. Based on this definition of the CodeInfo, the engine will create a run script that looks like the following:

#!/bin/bash

'[executable path in code node]' < '[input_filename]' > '[output_filename]'

The CodeInfo should be attached to the codes_info attribute of a CalcInfo object. A calculation can potentially run more than one code, so the CodeInfo object should be assigned as a list. Finally, we define the retrieve_list attribute, which is a list of filenames that the engine should retrieve from the running directory once the calculation job has finished. The engine will store these files in a FolderData node that will be attached as an output node to the calculation with the label retrieved. There are other file lists available that allow you to easily customize how to move files to and from the remote working directory in order to prevent the creation of unnecessary copies.

This was a minimal example of how to implement the CalcJob class to interface AiiDA with an external code. For more detailed information and advanced functionality on the CalcJob class, refer to the Topics section on defining calculations.

Parsing the outputs¶

The parsing of the output files generated by a CalcJob is optional and can be used to store (part of) their information as AiiDA nodes, which makes the data queryable and therefore easier to access and analyze. To enable CalcJob output file parsing, one should subclass the Parser class and implement the parse() method. The following is an example implementation, as shipped with aiida-core, to parse the outputs of the ArithmeticAddCalculation discussed in the previous section:

class ArithmeticAddParser(Parser):
    """Parser for an `ArithmeticAddCalculation` job."""

    def parse(self, **kwargs):
        """Parse the contents of the output files stored in the `retrieved` output node."""
        try:
            output_folder = self.retrieved
        except AttributeError:
            return self.exit_codes.ERROR_NO_RETRIEVED_FOLDER

        try:
            with output_folder.open(self.node.get_option('output_filename'), 'r') as handle:
                result = int(handle.read())
        except OSError:
            return self.exit_codes.ERROR_READING_OUTPUT_FILE
        except ValueError:
            return self.exit_codes.ERROR_INVALID_OUTPUT

        if result < 0:
            return self.exit_codes.ERROR_NEGATIVE_NUMBER

        self.out('sum', Int(result))

The output files generated by the completed calculation can be accessed from the retrieved output folder, which can be accessed through the retrieved property. It is an instance of FolderData and so provides, among other things, the open() method to open any file it contains. In this example implementation, we use it to open the output file, whose filename we get through the get_option() method of the corresponding calculation node, which we obtain through the node property of the Parser. We read the content of the file and cast it to an integer, which should contain the sum that was produced by the bash code. We catch any exceptions that might be thrown, for example when the file cannot be read, or if its content cannot be interpreted as an integer, and return an exit code. This method of dealing with potential errors of external codes is discussed in the section on handling parsing errors.

To attach the parsed sum as an output, use the out() method. The first argument is the name of the output, which will be used as the label for the link that connects the calculation and data node, and the second is the node that should be recorded as an output. Note that the type of the output should match the type that is specified by the process specification of the corresponding CalcJob. If any of the registered outputs do not match the specification, the calculation will be marked as failed.

To trigger the parsing using a Parser after a CalcJob has finished (such as the one described in the previous section) it should be defined in the metadata.options.parser_name input. If a particular parser should always be used by default for a given CalcJob, it can be defined as the default in the define method, for example:

@classmethod
def define(cls, spec):
    ...
    spec.inputs['metadata']['options']['parser_name'].default = 'arithmetic.add'

The default can be overridden through the inputs when launching the calculation job. Note, that one should not pass the Parser class itself, but rather the corresponding entry point name under which it is registered as a plugin. In other words, in order to use a Parser you will need to register it as explained in the how-to section on registering plugins.

Handling parsing errors¶

So far we have not spent too much attention on dealing with potential errors that might arise when running external codes. However, for many codes, there are lots of ways in which it can fail to execute nominally and produced the correct output. A Parser is the solution to detect these errors and report them to the caller through exit codes. These exit codes can be defined through the spec of the CalcJob that is used for that code, just as the inputs and output are defined. For example, the ArithmeticAddCalculation introduced in “Interfacing external codes”, defines the following exit codes:

spec.exit_code(300, 'ERROR_NO_RETRIEVED_FOLDER', message='The retrieved output node does not exist.')
spec.exit_code(310, 'ERROR_READING_OUTPUT_FILE', message='The output file could not be read.')
spec.exit_code(320, 'ERROR_INVALID_OUTPUT', message='The output file contains invalid output.')
spec.exit_code(410, 'ERROR_NEGATIVE_NUMBER', message='The sum of the operands is a negative number.')

Each exit_code defines an exit status (a positive integer), a label that can be used to reference the code in the parse method (through the self.exit_codes property, as seen below), and a message that provides a more detailed description of the problem. To use these in the parse method, you just need to return the corresponding exit code which instructs the engine to store it on the node of the calculation that is being parsed. The snippet of the previous section on parsing the outputs already showed two problems that are detected and are communicated by returning the corresponding the exit code:

try:
    with output_folder.open(self.node.get_option('output_filename'), 'r') as handle:
        result = int(handle.read())
except OSError:
    return self.exit_codes.ERROR_READING_OUTPUT_FILE
except ValueError:
    return self.exit_codes.ERROR_INVALID_OUTPUT

If the read() call fails to read the output file, for example because the calculation failed to run entirely and did not write anything, it will raise an OSError, which the parser catches and returns the ERROR_READING_OUTPUT_FILE exit code. Alternatively, if the file could be read, but it’s content cannot be interpreted as an integer, the parser returns ERROR_INVALID_OUTPUT. The Topics section on defining processes provides additional information on how to use exit codes.

Running external codes¶

To run an external code with AiiDA, you will need to use an appropriate calculation plugin that knows how to transform the input nodes into the input files that the code expects, copy everything in the code’s machine, run the calculation and retrieve the results. You can check the plugin registry to see if a plugin already exists for the code that you would like to run. If that is not the case, you can develop your own. After you have installed the plugin, you can start running the code through AiiDA. To check which calculation plugins you have currently installed, run:

$ verdi plugin list aiida.calculations

As an example, we will show how to use the arithmetic.add plugin, which is a pre-installed plugin that uses the bash shell to sum two integers. You can access it with the CalculationFactory:

from aiida.plugins import CalculationFactory
calculation_class = CalculationFactory('arithmetic.add')

Next, we provide the inputs for the code when running the calculation. Use verdi plugin to determine what inputs a specific plugin expects:

$ verdi plugin list aiida.calculations arithmetic.add
...
    Inputs:
           code:  required  Code        The `Code` to use for this job.
              x:  required  Int, Float  The left operand.
              y:  required  Int, Float  The right operand.
...

You will see that 3 inputs nodes are required: two containing the values to add up (x, y) and one containing information about the specific code to execute (code). If you already have these nodes in your database, you can get them by querying for them or using orm.load_node(<PK>). Otherwise, you will need to create them as shown below (note that you will need to already have the localhost computer configured, as explained in the previous how-to):

from aiida import orm
bash_binary = orm.Code(remote_computer_exec=[localhost, '/bin/bash'])
number_x = orm.Int(17)
number_y = orm.Int(11)

To provide these as inputs to the calculations, we will now use the builder object that we can get from the class:

calculation_builder = calculation_class.get_builder()
calculation_builder.code = bash_binary
calculation_builder.x = number_x
calculation_builder.y = number_y

Now everything is in place and ready to perform the calculation, which can be done in two different ways. The first one is blocking and will return a dictionary containing all the output nodes (keyed after their label, so in this case these should be: “remote_folder”, “retrieved” and “sum”) that you can safely inspect and work with:

from aiida.engine import run
output_dict = run(calculation_builder)
sum_result = output_dict['sum']

The second one is non blocking, as you will be submitting it to the daemon and control is immediately returned to the interpreter. The return value in this case is the calculation node that is stored in the database.

from aiida.engine import submit
calculation = submit(calculation_builder)

Note that, although you have access to the node, the underlying calculation process is not guaranteed to have finished when you get back control in the interpreter. You can use the verdi command line interface to monitor these processes:

$ verdi process list

Performing a dry-run¶

Additionally, you might want to check and verify your inputs before actually running or submitting a calculation. You can do so by specifying to use a dry_run, which will create all the input files in a local directory (submit_test/[date]-0000[x]) so you can inspect them before actually launching the calculation:

calculation_builder.metadata.dry_run = True
calculation_builder.metadata.store_provenance = False
run(calculation_builder)