aiida.repository package#
Module with resources dealing with the file repository.
Subpackages#
Submodules#
Module with resources common to the repository.
- class aiida.repository.common.File(name: str = '', file_type: aiida.repository.common.FileType = FileType.DIRECTORY, key: Optional[str] = None, objects: Optional[Dict[str, aiida.repository.common.File]] = None)[source]#
Bases:
object
Data class representing a file object.
- __dict__ = mappingproxy({'__module__': 'aiida.repository.common', '__doc__': 'Data class representing a file object.', '__init__': <function File.__init__>, 'from_serialized': <classmethod(<function File.from_serialized>)>, 'serialize': <function File.serialize>, 'name': <property object>, 'file_type': <property object>, 'is_file': <function File.is_file>, 'is_dir': <function File.is_dir>, 'key': <property object>, 'objects': <property object>, '__eq__': <function File.__eq__>, '__repr__': <function File.__repr__>, '__dict__': <attribute '__dict__' of 'File' objects>, '__weakref__': <attribute '__weakref__' of 'File' objects>, '__hash__': None, '__annotations__': {}})#
- __hash__ = None#
- __init__(name: str = '', file_type: aiida.repository.common.FileType = FileType.DIRECTORY, key: Optional[str] = None, objects: Optional[Dict[str, aiida.repository.common.File]] = None) None [source]#
Construct a new instance.
- Parameters
name – The final element of the file path
file_type – Identifies whether the File is a file or a directory
key – A key to map the file to its contents in the backend repository (file only)
objects – Mapping of child names to child Files (directory only)
- Raises
ValueError – If a key is defined for a directory, or objects are defined for a file
- __module__ = 'aiida.repository.common'#
- __weakref__#
list of weak references to the object (if defined)
- property file_type: aiida.repository.common.FileType#
Return the file type of the file object.
- classmethod from_serialized(serialized: dict, name='') aiida.repository.common.File [source]#
Construct a new instance from a serialized instance.
- Parameters
serialized – the serialized instance.
- Returns
the reconstructed file object.
- property objects: Dict[str, aiida.repository.common.File]#
Return the objects of the file object.
- class aiida.repository.common.FileType(value)[source]#
Bases:
enum.Enum
Enumeration to represent the type of a file object.
- DIRECTORY = 0#
- FILE = 1#
- __module__ = 'aiida.repository.common'#
Module for the implementation of a file repository.
- class aiida.repository.repository.Repository(backend: Optional[aiida.repository.backend.abstract.AbstractRepositoryBackend] = None)[source]#
Bases:
object
File repository.
This class provides an interface to a backend file repository instance, but unlike the backend repository, this class keeps a reference of the virtual file hierarchy. This means that through this interface, a client can create files and directories with a file hierarchy, just as they would on a local file system, except it is completely virtual as the files are stored by the backend which can store them in a completely flat structure. This also means that the internal virtual hierarchy of a
Repository
instance does not necessarily represent all the files that are stored by repository backend. The repository exposes a mere subset of all the file objects stored in the backend. This is why object deletion is also implemented as a soft delete, by default, where the files are just removed from the internal virtual hierarchy, but not in the actual backend. This is because those objects can be referenced by other instances.- __dict__ = mappingproxy({'__module__': 'aiida.repository.repository', '__doc__': 'File repository.\n\n This class provides an interface to a backend file repository instance, but unlike the backend repository, this\n class keeps a reference of the virtual file hierarchy. This means that through this interface, a client can create\n files and directories with a file hierarchy, just as they would on a local file system, except it is completely\n virtual as the files are stored by the backend which can store them in a completely flat structure. This also means\n that the internal virtual hierarchy of a ``Repository`` instance does not necessarily represent all the files that\n are stored by repository backend. The repository exposes a mere subset of all the file objects stored in the\n backend. This is why object deletion is also implemented as a soft delete, by default, where the files are just\n removed from the internal virtual hierarchy, but not in the actual backend. This is because those objects can be\n referenced by other instances.\n ', '_file_cls': <class 'aiida.repository.common.File'>, '__init__': <function Repository.__init__>, '__str__': <function Repository.__str__>, 'uuid': <property object>, 'is_initialised': <property object>, 'from_serialized': <classmethod(<function Repository.from_serialized>)>, 'reset': <function Repository.reset>, 'serialize': <function Repository.serialize>, 'flatten': <classmethod(<function Repository.flatten>)>, 'hash': <function Repository.hash>, '_pre_process_path': <staticmethod(<function Repository._pre_process_path>)>, 'backend': <property object>, 'set_backend': <function Repository.set_backend>, '_insert_file': <function Repository._insert_file>, 'create_directory': <function Repository.create_directory>, 'get_file_keys': <function Repository.get_file_keys>, 'get_object': <function Repository.get_object>, 'get_directory': <function Repository.get_directory>, 'get_file': <function Repository.get_file>, 'list_objects': <function Repository.list_objects>, 'list_object_names': <function Repository.list_object_names>, 'put_object_from_filelike': <function Repository.put_object_from_filelike>, 'put_object_from_file': <function Repository.put_object_from_file>, 'put_object_from_tree': <function Repository.put_object_from_tree>, 'is_empty': <function Repository.is_empty>, 'has_object': <function Repository.has_object>, 'open': <function Repository.open>, 'get_object_content': <function Repository.get_object_content>, 'delete_object': <function Repository.delete_object>, 'erase': <function Repository.erase>, 'clone': <function Repository.clone>, 'walk': <function Repository.walk>, 'copy_tree': <function Repository.copy_tree>, 'initialise': <function Repository.initialise>, 'delete': <function Repository.delete>, '__dict__': <attribute '__dict__' of 'Repository' objects>, '__weakref__': <attribute '__weakref__' of 'Repository' objects>, '__annotations__': {}})#
- __init__(backend: Optional[aiida.repository.backend.abstract.AbstractRepositoryBackend] = None)[source]#
Construct a new instance with empty metadata.
- Parameters
backend – instance of repository backend to use to actually store the file objects. By default, an instance of the
SandboxRepositoryBackend
will be created.
- __module__ = 'aiida.repository.repository'#
- __weakref__#
list of weak references to the object (if defined)
- _file_cls#
alias of
aiida.repository.common.File
- _insert_file(path: pathlib.PurePosixPath, key: str) None [source]#
Insert a new file object in the object mapping.
Note
this assumes the path is a valid relative path, so should be checked by the caller.
- Parameters
path – the relative path where to store the object in the repository.
key – fully qualified identifier for the object within the repository.
- static _pre_process_path(path: Optional[Union[str, pathlib.PurePosixPath]] = None) pathlib.PurePosixPath [source]#
Validate and convert the path to instance of
pathlib.PurePosixPath
.This should be called by every method of this class before doing anything, such that it can safely assume that the path is a
pathlib.PurePosixPath
object, which makes path manipulation a lot easier.- Parameters
path – the path as a
pathlib.PurePosixPath
object or None.- Raises
TypeError – if the type of path was not a str nor a
pathlib.PurePosixPath
instance.
- property backend: aiida.repository.backend.abstract.AbstractRepositoryBackend#
Return the current repository backend.
- Returns
the repository backend.
- clone(source: aiida.repository.repository.Repository) None [source]#
Clone the contents of another repository instance.
- copy_tree(target: Union[str, pathlib.Path], path: Optional[Union[str, pathlib.PurePosixPath]] = None) None [source]#
Copy the contents of the entire node repository to another location on the local file system.
Note
If
path
is specified, only its contents are copied, and the relative path with respect to the root is discarded. For example, ifpath
isrelative/sub
, the contents ofsub
will be copied directly to the target, without therelative/sub
directory.- Parameters
target – absolute path of the directory where to copy the contents to.
path – optional relative path whose contents to copy.
- Raises
TypeError – if
target
is of incorrect type or not absolute.NotADirectoryError – if
path
does not reference a directory.
- create_directory(path: Union[str, pathlib.PurePosixPath]) aiida.repository.common.File [source]#
Create a new directory with the given path.
- Parameters
path – the relative path of the directory.
- Returns
the created directory.
- Raises
TypeError – if the path is not a string or
Path
, or is an absolute path.
- delete() None [source]#
Delete the repository.
Important
This will not just delete the contents of the repository but also the repository itself and all of its assets. For example, if the repository is stored inside a folder on disk, the folder may be deleted.
- delete_object(path: Union[str, pathlib.PurePosixPath], hard_delete: bool = False) None [source]#
Soft delete the object from the repository.
Note
can only delete file objects, but not directories.
- Parameters
path – the relative path of the object within the repository.
hard_delete – when true, not only remove the file from the internal mapping but also call through to the
delete_object
method of the actual repository backend.
- Raises
TypeError – if the path is not a string or
Path
, or is an absolute path.FileNotFoundError – if the file does not exist.
IsADirectoryError – if the object is a directory and not a file.
OSError – if the file could not be deleted.
- classmethod flatten(serialized=typing.Optional[typing.Dict[str, typing.Any]], delimiter: str = '/') Dict[str, Optional[str]] [source]#
Flatten the serialized content of a repository into a mapping of path -> key or None (if folder).
Note, all folders are represented in the flattened output, and their path is suffixed with the delimiter.
- Parameters
serialized – the serialized content of the repository.
delimiter – the delimiter to use to separate the path elements.
- Returns
dictionary with the flattened content.
- classmethod from_serialized(backend: aiida.repository.backend.abstract.AbstractRepositoryBackend, serialized: Dict[str, Any]) aiida.repository.repository.Repository [source]#
Construct an instance where the metadata is initialized from the serialized content.
- Parameters
backend – instance of repository backend to use to actually store the file objects.
- get_directory(path: Optional[Union[str, pathlib.PurePosixPath]] = None) aiida.repository.common.File [source]#
Return the directory object at the given path.
- Parameters
path – the relative path of the directory.
- Returns
the File representing the object located at the given relative path.
- Raises
TypeError – if the path is not a string or
Path
, or is an absolute path.FileNotFoundError – if no object exists for the given path.
NotADirectoryError – if the object at the given path is not a directory.
- get_file(path: Union[str, pathlib.PurePosixPath]) aiida.repository.common.File [source]#
Return the file object at the given path.
- Parameters
path – the relative path of the file object.
- Returns
the File representing the object located at the given relative path.
- Raises
TypeError – if the path is not a string or
Path
, or is an absolute path.FileNotFoundError – if no object exists for the given path.
IsADirectoryError – if the object at the given path is not a directory.
- get_file_keys() List[str] [source]#
Return the keys of all file objects contained within this repository.
- Returns
list of keys, which map a file to its content in the backend repository.
- get_object(path: Optional[Union[str, pathlib.PurePosixPath]] = None) aiida.repository.common.File [source]#
Return the object at the given path.
- Parameters
path – the relative path where to store the object in the repository.
- Returns
the File representing the object located at the given relative path.
- Raises
TypeError – if the path is not a string or
Path
, or is an absolute path.FileNotFoundError – if no object exists for the given path.
- get_object_content(path: Union[str, pathlib.PurePosixPath]) bytes [source]#
Return the content of a object identified by path.
- Parameters
path – the relative path of the object within the repository.
- Raises
TypeError – if the path is not a string or
Path
, or is an absolute path.FileNotFoundError – if the file does not exist.
IsADirectoryError – if the object is a directory and not a file.
OSError – if the file could not be opened.
- has_object(path: Union[str, pathlib.PurePosixPath]) bool [source]#
Return whether the repository has an object with the given path.
- Parameters
path – the relative path of the object within the repository.
- Returns
True if the object exists, False otherwise.
- Raises
TypeError – if the path is not a string or
Path
, or is an absolute path.
- hash() str [source]#
Generate a hash of the repository’s contents.
Warning
this will read the content of all file objects contained within the virtual hierarchy into memory.
- Returns
the hash representing the contents of the repository.
- initialise(**kwargs: Any) None [source]#
Initialise the repository if it hasn’t already been initialised.
- Parameters
kwargs – keyword argument that will be passed to the
initialise
call of the backend.
- is_empty() bool [source]#
Return whether the repository is empty.
- Returns
True if the repository contains no file objects.
- list_object_names(path: Optional[Union[str, pathlib.PurePosixPath]] = None) List[str] [source]#
Return a sorted list of the object names contained in this repository, optionally in the given sub directory.
- Parameters
path – the relative path of the directory.
- Returns
a list of File named tuples representing the objects present in directory with the given path.
- Raises
TypeError – if the path is not a string or
Path
, or is an absolute path.FileNotFoundError – if no object exists for the given path.
NotADirectoryError – if the object at the given path is not a directory.
- list_objects(path: Optional[Union[str, pathlib.PurePosixPath]] = None) List[aiida.repository.common.File] [source]#
Return a list of the objects contained in this repository sorted by name, optionally in given sub directory.
- Parameters
path – the relative path of the directory.
- Returns
a list of File named tuples representing the objects present in directory with the given path.
- Raises
TypeError – if the path is not a string or
Path
, or is an absolute path.FileNotFoundError – if no object exists for the given path.
NotADirectoryError – if the object at the given path is not a directory.
- open(path: Union[str, pathlib.PurePosixPath]) Iterator[BinaryIO] [source]#
Open a file handle to an object stored under the given path.
Note
this should only be used to open a handle to read an existing file. To write a new file use the method
put_object_from_filelike
instead.- Parameters
path – the relative path of the object within the repository.
- Returns
yield a byte stream object.
- Raises
TypeError – if the path is not a string or
Path
, or is an absolute path.FileNotFoundError – if the file does not exist.
IsADirectoryError – if the object is a directory and not a file.
OSError – if the file could not be opened.
- put_object_from_file(filepath: Union[str, pathlib.PurePosixPath], path: Union[str, pathlib.PurePosixPath]) None [source]#
Store a new object under path with contents of the file located at filepath on the local file system.
- Parameters
filepath – absolute path of file whose contents to copy to the repository
path – the relative path where to store the object in the repository.
- Raises
TypeError – if the path is not a string and relative path, or the handle is not a byte stream.
- put_object_from_filelike(handle: BinaryIO, path: Union[str, pathlib.PurePosixPath]) None [source]#
Store the byte contents of a file in the repository.
- Parameters
handle – filelike object with the byte content to be stored.
path – the relative path where to store the object in the repository.
- Raises
TypeError – if the path is not a string or
Path
, or is an absolute path.
- put_object_from_tree(filepath: Union[str, pathlib.PurePosixPath], path: Optional[Union[str, pathlib.PurePosixPath]] = None) None [source]#
Store the entire contents of filepath on the local file system in the repository with under given path.
- Parameters
filepath – absolute path of the directory whose contents to copy to the repository.
path – the relative path where to store the objects in the repository.
- Raises
- serialize() Dict[str, Any] [source]#
Serialize the metadata into a JSON-serializable format.
- Returns
dictionary with the content metadata.
- set_backend(backend: aiida.repository.backend.abstract.AbstractRepositoryBackend) None [source]#
Set the backend for this repository.
- Parameters
backend – the repository backend.
- Raises
TypeError – if the type of the backend is invalid.
- property uuid: Optional[str]#
Return the unique identifier of the repository backend or
None
if it doesn’t have one.
- walk(path: Optional[Union[str, pathlib.PurePosixPath]] = None) Iterable[Tuple[pathlib.PurePosixPath, List[str], List[str]]] [source]#
Walk over the directories and files contained within this repository.
Note
the order of the dirname and filename lists that are returned is not necessarily sorted. This is in line with the
os.walk
implementation where the order depends on the underlying file system used.- Parameters
path – the relative path of the directory within the repository whose contents to walk.
- Returns
tuples of root, dirnames and filenames just like
os.walk
, with the exception that the root path is always relative with respect to the repository root, instead of an absolute path and it is an instance ofpathlib.PurePosixPath
instead of a normal string