aiida.repository package#

Module with resources dealing with the file repository.

Subpackages#

aiida.repository.backend package
- Submodules

Submodules#

Module with resources common to the repository.

class aiida.repository.common.File(name: str = '', file_type: aiida.repository.common.FileType = FileType.DIRECTORY, key: Optional[str] = None, objects: Optional[Dict[str, aiida.repository.common.File]] = None)[source]#

Bases: object

Data class representing a file object.

__dict__ = mappingproxy({'__module__': 'aiida.repository.common', '__doc__': 'Data class representing a file object.', '__init__': <function File.__init__>, 'from_serialized': <classmethod(<function File.from_serialized>)>, 'serialize': <function File.serialize>, 'name': <property object>, 'file_type': <property object>, 'is_file': <function File.is_file>, 'is_dir': <function File.is_dir>, 'key': <property object>, 'objects': <property object>, '__eq__': <function File.__eq__>, '__repr__': <function File.__repr__>, '__dict__': <attribute '__dict__' of 'File' objects>, '__weakref__': <attribute '__weakref__' of 'File' objects>, '__hash__': None, '__annotations__': {}})#

__eq__(other) → bool[source]#: Return whether this instance is equal to another file object instance.

__hash__ = None#

__init__(name: str = '', file_type: aiida.repository.common.FileType = FileType.DIRECTORY, key: Optional[str] = None, objects: Optional[Dict[str, aiida.repository.common.File]] = None) → None[source]#

Construct a new instance.

Parameters

name – The final element of the file path
file_type – Identifies whether the File is a file or a directory
key – A key to map the file to its contents in the backend repository (file only)
objects – Mapping of child names to child Files (directory only)

Raises

ValueError – If a key is defined for a directory, or objects are defined for a file

__module__ = 'aiida.repository.common'#

__repr__()[source]#: Return repr(self).

__weakref__#: list of weak references to the object (if defined)

property file_type: aiida.repository.common.FileType#: Return the file type of the file object.

classmethod from_serialized(serialized: dict, name='') → aiida.repository.common.File[source]#

Construct a new instance from a serialized instance.

Parameters: serialized – the serialized instance.
Returns: the reconstructed file object.

is_dir() → bool[source]#: Return whether this instance is a directory object.

is_file() → bool[source]#: Return whether this instance is a file object.

property key: Optional[str]#: Return the key of the file object.

property name: str#: Return the name of the file object.

property objects: Dict[str, aiida.repository.common.File]#: Return the objects of the file object.

serialize() → dict[source]#

Serialize the metadata into a JSON-serializable format.

Note

the serialization format is optimized to reduce the size in bytes.

Returns: dictionary with the content metadata.

class aiida.repository.common.FileType(value)[source]#

Bases: enum.Enum

Enumeration to represent the type of a file object.

DIRECTORY = 0#

FILE = 1#

__module__ = 'aiida.repository.common'#

Module for the implementation of a file repository.

class aiida.repository.repository.Repository(backend: Optional[aiida.repository.backend.abstract.AbstractRepositoryBackend] = None)[source]#

Bases: object

File repository.

This class provides an interface to a backend file repository instance, but unlike the backend repository, this class keeps a reference of the virtual file hierarchy. This means that through this interface, a client can create files and directories with a file hierarchy, just as they would on a local file system, except it is completely virtual as the files are stored by the backend which can store them in a completely flat structure. This also means that the internal virtual hierarchy of a Repository instance does not necessarily represent all the files that are stored by repository backend. The repository exposes a mere subset of all the file objects stored in the backend. This is why object deletion is also implemented as a soft delete, by default, where the files are just removed from the internal virtual hierarchy, but not in the actual backend. This is because those objects can be referenced by other instances.

__dict__ = mappingproxy({'__module__': 'aiida.repository.repository', '__doc__': 'File repository.\n\n This class provides an interface to a backend file repository instance, but unlike the backend repository, this\n class keeps a reference of the virtual file hierarchy. This means that through this interface, a client can create\n files and directories with a file hierarchy, just as they would on a local file system, except it is completely\n virtual as the files are stored by the backend which can store them in a completely flat structure. This also means\n that the internal virtual hierarchy of a ``Repository`` instance does not necessarily represent all the files that\n are stored by repository backend. The repository exposes a mere subset of all the file objects stored in the\n backend. This is why object deletion is also implemented as a soft delete, by default, where the files are just\n removed from the internal virtual hierarchy, but not in the actual backend. This is because those objects can be\n referenced by other instances.\n ', '_file_cls': <class 'aiida.repository.common.File'>, '__init__': <function Repository.__init__>, '__str__': <function Repository.__str__>, 'uuid': <property object>, 'is_initialised': <property object>, 'from_serialized': <classmethod(<function Repository.from_serialized>)>, 'reset': <function Repository.reset>, 'serialize': <function Repository.serialize>, 'flatten': <classmethod(<function Repository.flatten>)>, 'hash': <function Repository.hash>, '_pre_process_path': <staticmethod(<function Repository._pre_process_path>)>, 'backend': <property object>, 'set_backend': <function Repository.set_backend>, '_insert_file': <function Repository._insert_file>, 'create_directory': <function Repository.create_directory>, 'get_file_keys': <function Repository.get_file_keys>, 'get_object': <function Repository.get_object>, 'get_directory': <function Repository.get_directory>, 'get_file': <function Repository.get_file>, 'list_objects': <function Repository.list_objects>, 'list_object_names': <function Repository.list_object_names>, 'put_object_from_filelike': <function Repository.put_object_from_filelike>, 'put_object_from_file': <function Repository.put_object_from_file>, 'put_object_from_tree': <function Repository.put_object_from_tree>, 'is_empty': <function Repository.is_empty>, 'has_object': <function Repository.has_object>, 'open': <function Repository.open>, 'get_object_content': <function Repository.get_object_content>, 'delete_object': <function Repository.delete_object>, 'erase': <function Repository.erase>, 'clone': <function Repository.clone>, 'walk': <function Repository.walk>, 'copy_tree': <function Repository.copy_tree>, 'initialise': <function Repository.initialise>, 'delete': <function Repository.delete>, '__dict__': <attribute '__dict__' of 'Repository' objects>, '__weakref__': <attribute '__weakref__' of 'Repository' objects>, '__annotations__': {}})#

__init__(backend: Optional[aiida.repository.backend.abstract.AbstractRepositoryBackend] = None)[source]#

Construct a new instance with empty metadata.

Parameters: backend – instance of repository backend to use to actually store the file objects. By default, an instance of the SandboxRepositoryBackend will be created.

__module__ = 'aiida.repository.repository'#

__str__() → str[source]#: Return the string representation of this repository.

__weakref__#: list of weak references to the object (if defined)

_file_cls#: alias of aiida.repository.common.File

_insert_file(path: pathlib.PurePosixPath, key: str) → None[source]#

Insert a new file object in the object mapping.

Note

this assumes the path is a valid relative path, so should be checked by the caller.

Parameters

path – the relative path where to store the object in the repository.
key – fully qualified identifier for the object within the repository.

static _pre_process_path(path: Optional[Union[str, pathlib.PurePosixPath]] = None) → pathlib.PurePosixPath[source]#

Validate and convert the path to instance of pathlib.PurePosixPath.

This should be called by every method of this class before doing anything, such that it can safely assume that the path is a pathlib.PurePosixPath object, which makes path manipulation a lot easier.

Parameters: path – the path as a pathlib.PurePosixPath object or None.
Raises: TypeError – if the type of path was not a str nor a pathlib.PurePosixPath instance.

property backend: aiida.repository.backend.abstract.AbstractRepositoryBackend#

Return the current repository backend.

Returns: the repository backend.

clone(source: aiida.repository.repository.Repository) → None[source]#: Clone the contents of another repository instance.

copy_tree(target: Union[str, pathlib.Path], path: Optional[Union[str, pathlib.PurePosixPath]] = None) → None[source]#

Copy the contents of the entire node repository to another location on the local file system.

Note

If path is specified, only its contents are copied, and the relative path with respect to the root is discarded. For example, if path is relative/sub, the contents of sub will be copied directly to the target, without the relative/sub directory.

Parameters

target – absolute path of the directory where to copy the contents to.
path – optional relative path whose contents to copy.

Raises

TypeError – if target is of incorrect type or not absolute.
NotADirectoryError – if path does not reference a directory.

create_directory(path: Union[str, pathlib.PurePosixPath]) → aiida.repository.common.File[source]#

Create a new directory with the given path.

Parameters: path – the relative path of the directory.
Returns: the created directory.
Raises: TypeError – if the path is not a string or Path, or is an absolute path.

delete() → None[source]#: Delete the repository.

Important

This will not just delete the contents of the repository but also the repository itself and all of its assets. For example, if the repository is stored inside a folder on disk, the folder may be deleted.

delete_object(path: Union[str, pathlib.PurePosixPath], hard_delete: bool = False) → None[source]#

Soft delete the object from the repository.

Note

can only delete file objects, but not directories.

Parameters

path – the relative path of the object within the repository.
hard_delete – when true, not only remove the file from the internal mapping but also call through to the delete_object method of the actual repository backend.

Raises

TypeError – if the path is not a string or Path, or is an absolute path.
FileNotFoundError – if the file does not exist.
IsADirectoryError – if the object is a directory and not a file.
OSError – if the file could not be deleted.

erase() → None[source]#: Delete all objects from the repository.

classmethod flatten(serialized=typing.Optional[typing.Dict[str, typing.Any]], delimiter: str = '/') → Dict[str, Optional[str]][source]#

Flatten the serialized content of a repository into a mapping of path -> key or None (if folder).

Note, all folders are represented in the flattened output, and their path is suffixed with the delimiter.

Parameters

serialized – the serialized content of the repository.
delimiter – the delimiter to use to separate the path elements.

Returns

dictionary with the flattened content.

classmethod from_serialized(backend: aiida.repository.backend.abstract.AbstractRepositoryBackend, serialized: Dict[str, Any]) → aiida.repository.repository.Repository[source]#

Construct an instance where the metadata is initialized from the serialized content.

Parameters: backend – instance of repository backend to use to actually store the file objects.

get_directory(path: Optional[Union[str, pathlib.PurePosixPath]] = None) → aiida.repository.common.File[source]#

Return the directory object at the given path.

Parameters

path – the relative path of the directory.

Returns

the File representing the object located at the given relative path.

Raises

TypeError – if the path is not a string or Path, or is an absolute path.
FileNotFoundError – if no object exists for the given path.
NotADirectoryError – if the object at the given path is not a directory.

get_file(path: Union[str, pathlib.PurePosixPath]) → aiida.repository.common.File[source]#

Return the file object at the given path.

Parameters

path – the relative path of the file object.

Returns

the File representing the object located at the given relative path.

Raises

TypeError – if the path is not a string or Path, or is an absolute path.
FileNotFoundError – if no object exists for the given path.
IsADirectoryError – if the object at the given path is not a directory.

get_file_keys() → List[str][source]#

Return the keys of all file objects contained within this repository.

Returns: list of keys, which map a file to its content in the backend repository.

get_object(path: Optional[Union[str, pathlib.PurePosixPath]] = None) → aiida.repository.common.File[source]#

Return the object at the given path.

Parameters

path – the relative path where to store the object in the repository.

Returns

the File representing the object located at the given relative path.

Raises

TypeError – if the path is not a string or Path, or is an absolute path.
FileNotFoundError – if no object exists for the given path.

get_object_content(path: Union[str, pathlib.PurePosixPath]) → bytes[source]#

Return the content of a object identified by path.

Parameters

path – the relative path of the object within the repository.

Raises

TypeError – if the path is not a string or Path, or is an absolute path.
FileNotFoundError – if the file does not exist.
IsADirectoryError – if the object is a directory and not a file.
OSError – if the file could not be opened.

has_object(path: Union[str, pathlib.PurePosixPath]) → bool[source]#

Return whether the repository has an object with the given path.

Parameters: path – the relative path of the object within the repository.
Returns: True if the object exists, False otherwise.
Raises: TypeError – if the path is not a string or Path, or is an absolute path.

hash() → str[source]#

Generate a hash of the repository’s contents.

Warning

this will read the content of all file objects contained within the virtual hierarchy into memory.

Returns: the hash representing the contents of the repository.

initialise(**kwargs: Any) → None[source]#

Initialise the repository if it hasn’t already been initialised.

Parameters: kwargs – keyword argument that will be passed to the initialise call of the backend.

is_empty() → bool[source]#

Return whether the repository is empty.

Returns: True if the repository contains no file objects.

property is_initialised: bool#: Return whether the repository backend has been initialised.

list_object_names(path: Optional[Union[str, pathlib.PurePosixPath]] = None) → List[str][source]#

Return a sorted list of the object names contained in this repository, optionally in the given sub directory.

Parameters

path – the relative path of the directory.

Returns

a list of File named tuples representing the objects present in directory with the given path.

Raises

TypeError – if the path is not a string or Path, or is an absolute path.
FileNotFoundError – if no object exists for the given path.
NotADirectoryError – if the object at the given path is not a directory.

list_objects(path: Optional[Union[str, pathlib.PurePosixPath]] = None) → List[aiida.repository.common.File][source]#

Return a list of the objects contained in this repository sorted by name, optionally in given sub directory.

Parameters

path – the relative path of the directory.

Returns

a list of File named tuples representing the objects present in directory with the given path.

Raises

TypeError – if the path is not a string or Path, or is an absolute path.
FileNotFoundError – if no object exists for the given path.
NotADirectoryError – if the object at the given path is not a directory.

open(path: Union[str, pathlib.PurePosixPath]) → Iterator[BinaryIO][source]#

Open a file handle to an object stored under the given path.

Note

this should only be used to open a handle to read an existing file. To write a new file use the method put_object_from_filelike instead.

Parameters

path – the relative path of the object within the repository.

Returns

yield a byte stream object.

Raises

TypeError – if the path is not a string or Path, or is an absolute path.
FileNotFoundError – if the file does not exist.
IsADirectoryError – if the object is a directory and not a file.
OSError – if the file could not be opened.

put_object_from_file(filepath: Union[str, pathlib.PurePosixPath], path: Union[str, pathlib.PurePosixPath]) → None[source]#

Store a new object under path with contents of the file located at filepath on the local file system.

Parameters

filepath – absolute path of file whose contents to copy to the repository
path – the relative path where to store the object in the repository.

Raises

TypeError – if the path is not a string and relative path, or the handle is not a byte stream.

put_object_from_filelike(handle: BinaryIO, path: Union[str, pathlib.PurePosixPath]) → None[source]#

Store the byte contents of a file in the repository.

Parameters

handle – filelike object with the byte content to be stored.
path – the relative path where to store the object in the repository.

Raises

TypeError – if the path is not a string or Path, or is an absolute path.

put_object_from_tree(filepath: Union[str, pathlib.PurePosixPath], path: Optional[Union[str, pathlib.PurePosixPath]] = None) → None[source]#

Store the entire contents of filepath on the local file system in the repository with under given path.

Parameters

filepath – absolute path of the directory whose contents to copy to the repository.
path – the relative path where to store the objects in the repository.

Raises

TypeError – if the filepath is not a string or Path, or is a relative path.
TypeError – if the path is not a string or Path, or is an absolute path.

reset() → None[source]#

serialize() → Dict[str, Any][source]#

Serialize the metadata into a JSON-serializable format.

Returns: dictionary with the content metadata.

set_backend(backend: aiida.repository.backend.abstract.AbstractRepositoryBackend) → None[source]#

Set the backend for this repository.

Parameters: backend – the repository backend.
Raises: TypeError – if the type of the backend is invalid.

property uuid: Optional[str]#: Return the unique identifier of the repository backend or None if it doesn’t have one.

walk(path: Optional[Union[str, pathlib.PurePosixPath]] = None) → Iterable[Tuple[pathlib.PurePosixPath, List[str], List[str]]][source]#

Walk over the directories and files contained within this repository.

Note

the order of the dirname and filename lists that are returned is not necessarily sorted. This is in line with the os.walk implementation where the order depends on the underlying file system used.

Parameters: path – the relative path of the directory within the repository whose contents to walk.
Returns: tuples of root, dirnames and filenames just like os.walk, with the exception that the root path is always relative with respect to the repository root, instead of an absolute path and it is an instance of pathlib.PurePosixPath instead of a normal string