Provides import/export functionalities.
To see history/git blame prior to the move to aiida.tools.importexport, explore tree: https://github.com/aiidateam/aiida-core/tree/eebef392c81e8b130834a92e1d7abf5e2e30b3ce Functionality: <tree>/aiida/orm/importexport.py Tests: <tree>/aiida/backends/tests/test_export_and_import.py
aiida.tools.importexport.
ArchiveExportError
Bases: aiida.tools.importexport.common.exceptions.ExportImportException
aiida.tools.importexport.common.exceptions.ExportImportException
Base class for all AiiDA export exceptions.
__module__
ArchiveImportError
Base class for all AiiDA import exceptions.
ArchiveMetadata
Bases: object
object
Class for storing metadata about this archive.
Required fields are necessary for importing the data back into AiiDA, whereas optional fields capture information about the export/migration process(es)
__annotations__
__dataclass_fields__
__dataclass_params__
__dict__
__eq__
Return self==value.
__hash__
__init__
Initialize self. See help(type(self)) for accurate signature.
__repr__
Return repr(self).
__weakref__
list of weak references to the object (if defined)
aiida_version
all_fields_info
conversion_info
entities_starting_set
export_version
graph_traversal_rules
include_comments
include_logs
unique_identifiers
ArchiveMigrationError
Base class for all AiiDA export archive migration exceptions.
ArchiveMigratorAbstract
Bases: abc.ABC
abc.ABC
An abstract base class to define an archive migrator.
__abstractmethods__
Initialise the migrator
filepath – the path to the archive file
version – the version of the archive file or, if None, the version will be auto-retrieved.
_abc_impl
filepath
Return the input file path.
migrate
Migrate the archive to another version
version – the version to migrate to
filename – the file path to migrate to. If None, the migrated archive will not be copied from the work_dir.
force – overwrite output file if it already exists
work_dir – The directory in which to perform the migration. If None, a temporary folder will be created and destroyed at the end of the process.
kwargs – key-word arguments specific to the concrete migrator implementation
path to the migrated archive or None if no migration performed (if filename is None, this will point to a path in the work_dir)
CorruptArchive: if the archive cannot be read
CorruptArchive
ArchiveMigrationError: if the archive cannot migrated to the requested version
ArchiveMigratorJsonBase
Bases: aiida.tools.importexport.archive.migrators.ArchiveMigratorAbstract
aiida.tools.importexport.archive.migrators.ArchiveMigratorAbstract
A migrator base for the JSON compressed formats.
_compress_archive_tar
Create a new zip compressed tar from a folder.
_compress_archive_zip
Create a new zip compressed zip from a folder.
_extract_archive
Extract the archive to a filepath.
_move_file
Move a file to a another path, deleting the target path first if it exists.
_perform_migration
Perform the migration(s) in the work directory, compress (if necessary), then move to the out_path (if not None).
_retrieve_version
Retrieve the version of the input archive.
ArchiveMigratorJsonTar
Bases: aiida.tools.importexport.archive.migrators.ArchiveMigratorJsonBase
aiida.tools.importexport.archive.migrators.ArchiveMigratorJsonBase
A migrator for a JSON tar compressed format.
ArchiveMigratorJsonZip
A migrator for a JSON zip compressed format.
ArchiveReaderAbstract
An abstract interface for AiiDA archive readers.
An ArchiveReader implementation is intended to be used with a context:
ArchiveReader
with ArchiveReader(filename) as reader: reader.entity_count('Node')
__enter__
__exit__
An archive reader
filename – the filename (possibly including the absolute path) of the file to import.
assert_within_context
Assert that the method is called within a context.
~aiida.common.exceptions.InvalidOperation: if not called within a context
check_version
Check the version compatibility of the archive.
~aiida.tools.importexport.common.exceptions.IncompatibleArchiveVersionError: If the version is not compatible
compatible_export_version
Return the export version that this reader is compatible with.
entity_count
Return the count of an entity or None if not contained in the archive.
entity_names
Return list of all entity names.
Return the export version.
CorruptArchive – If the version cannot be retrieved.
file_format_verbose
The file format name.
filename
Return the name of the file that is being read from.
iter_entity_fields
Iterate over entities and yield their pk and database fields.
iter_group_uuids
Iterate over group UUIDs and the a set of node UUIDs they contain.
iter_link_data
Iterate over links: {‘input’: <UUID>, ‘output’: <UUID>, ‘label’: <LABEL>, ‘type’: <TYPE>}
iter_node_repos
Yield temporary folders containing the contents of the repository for each node.
uuids – UUIDs of the nodes over whose repository folders to iterate
callback –
a callback to report on the process, callback(action, value), with the following callback signatures:
callback(action, value)
callback('init', {'total': <int>, 'description': <str>})
to signal the start of a process, its total iterations and description
callback('update', <int>)
to signal an update to the process and the number of iterations to progress
CorruptArchive – If the repository does not exist.
iter_node_uuids
Iterate over node UUIDs.
link_count
Return the count of links.
metadata
Return the full (validated) archive metadata.
node_repository
Return a temporary folder with the contents of the repository for a single node.
uuid – The UUID of the node
ArchiveWriterAbstract
An abstract interface for AiiDA archive writers.
Open the contextmanager
Initiate the writer.
filepath – the path to the file to export to.
kwargs – keyword arguments specific to the writer implementation.
_remove_filepath
To run before moving the final export from a temporary folder.
add_export_info
Record information about the export process.
This information can be accessed by writer.export_info, it is reset on entrance of the context manager.
writer.export_info
close
Finalise the export.
This method is called on exiting a context manager.
excepted – Whether
export_info
Return information stored during the export process.
Return the file format name.
Return the filepath to write to.
open
Setup the export process.
This method is called on entering a context manager.
write_entity_data
Write the data for a single DB entity.
name – the name of the entity (e.g. ‘NODE’)
pk – the primary key of the entity (unique for the current DB only)
id_key – the key within fields that denotes the unique identifier of the entity (unique for all DBs)
fields
fields – mapping of DB fields to store -> values
write_group_nodes
Write a mapping of a group to the nodes it contains.
uuid – the UUID of the group
node_uuids – the list of node UUIDs the group contains
write_link
Write a dictionary of information for a single provenance link.
data – {'input': <UUID_STR>, 'output': <UUID_STR>, 'label': <LABEL_STR>, 'type': <TYPE_STR>}
{'input': <UUID_STR>, 'output': <UUID_STR>, 'label': <LABEL_STR>, 'type': <TYPE_STR>}
write_metadata
Write the metadata of the export process.
write_node_repo_folder
Write a node repository to the archive.
path – The path to the repository folder on disk
overwrite – Allow to overwrite existing path in archive
CacheFolder
A class to encapsulate a folder path with cached read/writes.
The class can be used as a context manager, and will flush the cache on exit:
with CacheFolder(path) as folder: # these are stored in memory (no disk write) folder.write_text('path/to/file.txt', 'content') folder.write_json('path/to/data.json', {'a': 1}) # these will be read from memory text = folder.read_text('path/to/file.txt') text = folder.load_json('path/to/data.json') # all files will now have been written to disk
Enter the contextmanager.
Exit the contextmanager.
Initialise cached folder.
path – folder path to cache
encoding – encoding of text to read/write
_limit_cache
Ensure the cache does not exceed a set limit.
Content is uncached on a First-In-First-Out basis.
_write_object
Write an object from the cache to disk.
path – relative path of file
ctype – the type of the content
content – the content to write
flush
Flush the cache.
get_path
Return the path.
flush – flush the cache before returning
load_json
Load a json file from the cache folder.
Important: if the dict is returned directly from the cache, any mutations will affect the cached dict.
path – path relative to base folder
ensure_copy – ensure the dict is a copy of that from the cache
(from cache, the content) If from cache, mutations will directly affect the cache
read_text
write text from the cache or base folder.
remove_file
Remove a file from both the cache and base folder (if present).
write_json
Write dict to the folder, to be serialized as json.
The dictionary is stored in memory, until the cache is flushed, at which point the dictionary is serialized to json and written to disk.
write_text
write text to the cache.
Raised when an operation is applied to a corrupt export archive, e.g. missing files or invalid formats.
DanglingLinkError
Bases: aiida.tools.importexport.common.exceptions.MigrationValidationError
aiida.tools.importexport.common.exceptions.MigrationValidationError
Raised when an export archive is detected to contain dangling links when importing.
ExportFileFormat
Bases: str, enum.Enum
str
enum.Enum
Archive file formats
TAR_GZIPPED
ZIP
_generate_next_value_
_member_map_
_member_names_
_member_type_
alias of builtins.str
builtins.str
_value2member_map_
ExportImportException
Bases: aiida.common.exceptions.AiidaException
aiida.common.exceptions.AiidaException
Base class for all AiiDA export/import module exceptions.
ExportValidationError
Bases: aiida.tools.importexport.common.exceptions.ArchiveExportError
aiida.tools.importexport.common.exceptions.ArchiveExportError
Raised when validation fails during export, e.g. for non-sealed ProcessNode s.
ProcessNode
ImportUniquenessError
Bases: aiida.tools.importexport.common.exceptions.ArchiveImportError
aiida.tools.importexport.common.exceptions.ArchiveImportError
Raised when the user tries to violate a uniqueness constraint.
Similar to UniquenessError.
UniquenessError
ImportValidationError
Raised when validation fails during import, e.g. for parameter types and values.
IncompatibleArchiveVersionError
Raised when trying to import an export archive with an incompatible schema version.
MigrationValidationError
Bases: aiida.tools.importexport.common.exceptions.ArchiveMigrationError
aiida.tools.importexport.common.exceptions.ArchiveMigrationError
Raised when validation fails during migration of export archives.
ProgressBarError
Something is wrong with setting up the tqdm progress bar
ReaderJsonBase
Bases: aiida.tools.importexport.archive.readers.ArchiveReaderAbstract
aiida.tools.importexport.archive.readers.ArchiveReaderAbstract
A reader base for the JSON compressed formats.
FILENAME_DATA
FILENAME_METADATA
REPO_FOLDER
A reader for JSON compressed archives.
filename – the filename (possibly including the absolute path) of the file on which to export.
sandbox_in_repo – Create the temporary uncompressed folder within the aiida repository
_extract
Extract repository data to a temporary folder.
path_prefix – Only extract paths starting with this prefix.
TypeError – if parameter types are not respected
_get_data
Retrieve the data JSON.
_get_metadata
Retrieve the metadata JSON.
ReaderJsonFolder
Bases: aiida.tools.importexport.archive.readers.ReaderJsonBase
aiida.tools.importexport.archive.readers.ReaderJsonBase
A reader for a JSON plain folder format.
ReaderJsonTar
A reader for a JSON tar compressed format.
ReaderJsonZip
A reader for a JSON zip compressed format.
WriterJsonFolder
Bases: aiida.tools.importexport.archive.writers.ArchiveWriterAbstract
aiida.tools.importexport.archive.writers.ArchiveWriterAbstract
An archive writer, which writes database data as a single JSON and repository data in a folder system.
This writer is mainly intended for backward compatibility with export_tree.
folder – a folder to write the archive to.
filepath – dummy value not used
WriterJsonTar
The entire containing folder is then compressed as a tar file.
WriterJsonZip
An archive writer, which writes database data as a single JSON and repository data in a zipped folder system.
use_compression – Whether or not to deflate the objects inside the zip file.
cache_zipinfo – Cache the zip file index on disk during the write. This reduces the RAM usage of the process, but will make the process slower.
detect_archive_type
For back-compatibility, but should be replaced with direct comparison of classes.
in_path – the path to the file
the archive type identifier (currently one of ‘zip’, ‘tar.gz’, ‘folder’)
export
Export AiiDA data to an archive file.
Note, the logging level and progress reporter should be set externally, for example:
from aiida.common.progress_reporter import set_progress_bar_tqdm EXPORT_LOGGER.setLevel('DEBUG') set_progress_bar_tqdm(leave=True) export(...)
Deprecated since version 1.5.0: Support for the parameter silent will be removed in v2.0.0. Please set the log level and progress bar implementation independently.
Deprecated since version 1.5.0: Support for the parameter use_compression will be removed in v2.0.0. Please use writer_init={‘use_compression’: True}.
Deprecated since version 1.2.1: Support for the parameters what and outfile will be removed in v2.0.0. Please use entities and filename instead, respectively.
entities – a list of entity instances; they can belong to different models/entities.
file_format – ‘zip’, ‘tar.gz’ or ‘folder’ or a specific writer class.
overwrite – if True, overwrite the output file without asking, if it exists. If False, raise an ArchiveExportError if the output file already exists.
allowed_licenses – List or function. If a list, then checks whether all licenses of Data nodes are in the list. If a function, then calls function for licenses of Data nodes expecting True if license is allowed, False otherwise.
forbidden_licenses – List or function. If a list, then checks whether all licenses of Data nodes are in the list. If a function, then calls function for licenses of Data nodes expecting True if license is allowed, False otherwise.
include_comments – In-/exclude export of comments for given node(s) in entities. Default: True, include comments in export (as well as relevant users).
entities
include_logs – In-/exclude export of logs for given node(s) in entities. Default: True, include logs in export.
writer_init – Additional key-word arguments to pass to the writer class init
batch_size – batch database query results in sub-collections to reduce memory usage
traversal_rules – graph traversal rules. See aiida.common.links.GraphTraversalRules what rule names are toggleable and what the defaults are.
aiida.common.links.GraphTraversalRules
a dictionary of data regarding the export process (timings, etc)
ArchiveExportError – if there are any internal errors when exporting.
LicensingException – if any node is licensed under forbidden license.
extract_tar
Extract the nodes to be imported from a (possibly zipped) tar file.
infile (str) – file path
folder (SandboxFolder) – a temporary fodler used to extract the file tree
SandboxFolder
nodes_export_subfolder (str) – name of the subfolder for AiiDA nodes
check_files – list of files to check are present
silent (bool) – suppress progress bar
CorruptArchive – if the archive misses files or files have incorrect formats
extract_tree
Prepare to import nodes from plain file system tree by copying in the given sandbox folder.
Note
the contents of the unpacked archive directory are copied into the sandbox folder, because the files will anyway haven to be copied to the repository at some point. By copying the contents of the source directory now and continuing operation only on the sandbox folder, we do not risk to modify the source files accidentally. During import then, the node files from the sandbox can be moved to the repository, so they won’t have to be copied again in any case.
infile (str) – absolute filepath point to the unpacked archive directory
folder (SandboxFolder) – a temporary folder to which the archive contents are copied
extract_zip
Extract the nodes to be imported from a zip file.
folder (SandboxFolder) – a temporary folder used to extract the file tree
get_migrator
Return the available archive migrator classes.
get_reader
Return the available writer classes.
get_writer
import_data
Import exported AiiDA archive to the AiiDA database and repository.
Proxy function for the backend-specific import functions. If in_path is a folder, calls extract_tree; otherwise, tries to detect the compression format (zip, tar.gz, tar.bz2, …) and calls the correct function.
in_path
from aiida.common.progress_reporter import set_progress_bar_tqdm IMPORT_LOGGER.setLevel('DEBUG') set_progress_bar_tqdm(leave=True) import_data(...)
in_path (str) – the path to a file or folder that can be imported in AiiDA.
group (Group) – Group wherein all imported Nodes will be placed.
Group
extras_mode_existing (str) – 3 letter code that will identify what to do with the extras import. The first letter acts on extras that are present in the original node and not present in the imported node. Can be either: ‘k’ (keep it) or ‘n’ (do not keep it). The second letter acts on the imported extras that are not present in the original node. Can be either: ‘c’ (create it) or ‘n’ (do not create it). The third letter defines what to do in case of a name collision. Can be either: ‘l’ (leave the old value), ‘u’ (update with a new value), ‘d’ (delete the extra), or ‘a’ (ask what to do if the content is different).
extras_mode_new (str) – ‘import’ to import extras of new nodes or ‘none’ to ignore them
comment_mode (str) – Comment import modes (when same UUIDs are found). Can be either: ‘newest’ (will keep the Comment with the most recent modification time (mtime)) or ‘overwrite’ (will overwrite existing Comments with the ones from the import file).
New and existing Nodes and Links.
dict
ArchiveImportError – if there are any internal errors when importing.
null_callback
A null callback function.