aiida.storage.sqlite_zip.migrations package#

Subpackages#

Submodules#

Upper level SQLAlchemy migration funcitons.

aiida.storage.sqlite_zip.migrations.env.run_migrations_online()[source]#

Run migrations in ‘online’ mode.

The connection should have been passed to the config, which we use to configue the migration context.

Migration from the “legacy” JSON format, to an sqlite database, and node uuid based repository to hash based.

aiida.storage.sqlite_zip.migrations.legacy_to_main._convert_datetime(key, value)[source]#
aiida.storage.sqlite_zip.migrations.legacy_to_main._create_directory(top_level: File, path: PurePosixPath) File[source]#

Create a new directory with the given path.

Parameters:

path – the relative path of the directory.

Returns:

the created directory.

aiida.storage.sqlite_zip.migrations.legacy_to_main._create_repo_metadata(paths: List[Tuple[str, str | None]]) Dict[str, Any][source]#

Create the repository metadata.

Parameters:

paths – list of (path, hashkey) tuples

Returns:

the repository metadata

aiida.storage.sqlite_zip.migrations.legacy_to_main._iter_entity_fields(data, name: str, node_repos: Dict[str, List[Tuple[str, str | None]]]) Iterator[Dict[str, Any]][source]#

Iterate through entity fields.

aiida.storage.sqlite_zip.migrations.legacy_to_main._json_to_sqlite(outpath: Path, data: dict, node_repos: Dict[str, List[Tuple[str, str | None]]], batch_size: int = 100) None[source]#

Convert a JSON archive format to SQLite.

aiida.storage.sqlite_zip.migrations.legacy_to_main.perform_v1_migration(inpath: Path, working: Path, new_zip: ZipPath, central_dir: Dict[str, Any], is_tar: bool, metadata: dict, data: dict) Path[source]#

Perform the repository and JSON to SQLite migration.

  1. Iterate though the repository paths in the archive

  2. If a file, hash its contents and, if not already present, stream it to the new archive

  3. Store a mapping of the node UUIDs to a list of (path, hashkey or None if a directory) tuples

Parameters:
  • inpath – the input path to the old archive

  • metadata – the metadata to migrate

  • data – the data to migrate

:returns:the path to the sqlite database file

Common variables

aiida.storage.sqlite_zip.migrations.utils.copy_tar_to_zip(inpath: Path, outpath: Path, path_callback: Callable[[Path, ZipPath], bool], *, compression: int = 6, overwrite: bool = True, title: str = 'Writing new zip file', info_order: Sequence[str] = ()) None[source]#

Create a new zip file from an existing tar file.

The tar file is first extracted to a temporary directory, and then the new zip file is created, with the path_callback allowing for per path modifications. The new zip file is first created in a temporary directory, and then moved to the desired location.

Parameters:
  • inpath – the path to the existing archive

  • outpath – the path to output the new archive

  • path_callback – a callback that is called for each path in the archive: (inpath, outpath) -> handled If handled is True, the path is assumed to already have been copied to the new zip file.

  • compression – the default compression level to use for the new zip file

  • overwrite – whether to overwrite the output file if it already exists

  • title – the title of the progress bar

  • info_orderZipInfo for these file names will be written first to the zip central directory. This allows for faster reading of these files, with archive_path.read_file_in_zip.

aiida.storage.sqlite_zip.migrations.utils.copy_zip_to_zip(inpath: Path, outpath: Path, path_callback: Callable[[ZipPath, ZipPath], bool], *, compression: int = 6, overwrite: bool = True, title: str = 'Writing new zip file', info_order: Sequence[str] = ()) None[source]#

Create a new zip file from an existing zip file.

All files/folders are streamed directly to the new zip file, with the path_callback allowing for per path modifications. The new zip file is first created in a temporary directory, and then moved to the desired location.

Parameters:
  • inpath – the path to the existing archive

  • outpath – the path to output the new archive

  • path_callback – a callback that is called for each path in the archive: (inpath, outpath) -> handled If handled is True, the path is assumed to already have been copied to the new zip file.

  • compression – the default compression level to use for the new zip file

  • overwrite – whether to overwrite the output file if it already exists

  • title – the title of the progress bar

  • info_orderZipInfo for these file names will be written first to the zip central directory. This allows for faster reading of these files, with archive_path.read_file_in_zip.

aiida.storage.sqlite_zip.migrations.utils.update_metadata(metadata, version)[source]#

Update the metadata with a new version number and a notification of the conversion that was executed.

Parameters:
  • metadata – the content of an export archive metadata.json file

  • version – string version number that the updated metadata should get

aiida.storage.sqlite_zip.migrations.utils.verify_metadata_version(metadata, version=None)[source]#

Utility function to verify that the metadata has the correct version number.

If no version number is passed, it will just extract the version number and return it.

Parameters:
  • metadata – the content of an export archive metadata.json file

  • version – string version number that the metadata is expected to have

This is the sqlite DB schema, coresponding to the main_0000 revision of the sqlite_zip backend, see: versions/main_0000_initial.py

For normal operation of the archive, we auto-generate the schema from the models in aiida.storage.psql_dos.models. However, when migrating an archive from the old format, we require a fixed revision of the schema.

The only difference between the PostGreSQL schema and SQLite one, is the replacement of JSONB with JSON, and UUID with CHAR(32).

class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbAuthInfo(**kwargs)[source]#

Bases: Base

Class that keeps the authentication data.

__init__(**kwargs)#

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7f3b7731dbd0; DbAuthInfo>#
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'#
__table__ = Table('db_dbauthinfo', MetaData(), Column('id', Integer(), table=<db_dbauthinfo>, primary_key=True, nullable=False), Column('aiidauser_id', Integer(), ForeignKey('db_dbuser.id'), table=<db_dbauthinfo>), Column('dbcomputer_id', Integer(), ForeignKey('db_dbcomputer.id'), table=<db_dbauthinfo>), Column('metadata', JSON(), table=<db_dbauthinfo>, default=CallableColumnDefault(<function dict>)), Column('auth_params', JSON(), table=<db_dbauthinfo>, default=CallableColumnDefault(<function dict>)), Column('enabled', Boolean(), table=<db_dbauthinfo>, default=ScalarElementColumnDefault(True)), schema=None)#
__table_args__ = (UniqueConstraint(Column('aiidauser_id', Integer(), ForeignKey('db_dbuser.id'), table=<db_dbauthinfo>), Column('dbcomputer_id', Integer(), ForeignKey('db_dbcomputer.id'), table=<db_dbauthinfo>)),)#
__tablename__ = 'db_dbauthinfo'#
_metadata#
_sa_class_manager = {'_metadata': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'aiidauser_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'auth_params': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'dbcomputer_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'enabled': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}#
aiidauser_id#
auth_params#
dbcomputer_id#
enabled#
id#
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbComment(**kwargs)[source]#

Bases: Base

Class to store comments.

__init__(**kwargs)#

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7f3b7731e9b0; DbComment>#
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'#
__table__ = Table('db_dbcomment', MetaData(), Column('id', Integer(), table=<db_dbcomment>, primary_key=True, nullable=False), Column('uuid', CHAR(length=32), table=<db_dbcomment>, nullable=False, default=CallableColumnDefault(<function get_new_uuid>)), Column('dbnode_id', Integer(), ForeignKey('db_dbnode.id'), table=<db_dbcomment>), Column('ctime', DateTime(timezone=True), table=<db_dbcomment>, default=CallableColumnDefault(<function now>)), Column('mtime', DateTime(timezone=True), table=<db_dbcomment>, default=CallableColumnDefault(<function now>)), Column('user_id', Integer(), ForeignKey('db_dbuser.id'), table=<db_dbcomment>), Column('content', Text(), table=<db_dbcomment>, default=ScalarElementColumnDefault('')), schema=None)#
__tablename__ = 'db_dbcomment'#
_sa_class_manager = {'content': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'ctime': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'dbnode_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'mtime': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'user_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'uuid': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}#
content#
ctime#
dbnode_id#
id#
mtime#
user_id#
uuid#
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbComputer(**kwargs)[source]#

Bases: Base

Class to store computers.

__init__(**kwargs)#

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7f3b7731f790; DbComputer>#
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'#
__table__ = Table('db_dbcomputer', MetaData(), Column('id', Integer(), table=<db_dbcomputer>, primary_key=True, nullable=False), Column('uuid', CHAR(length=32), table=<db_dbcomputer>, nullable=False, default=CallableColumnDefault(<function get_new_uuid>)), Column('label', String(length=255), table=<db_dbcomputer>, nullable=False), Column('hostname', String(length=255), table=<db_dbcomputer>, default=ScalarElementColumnDefault('')), Column('description', Text(), table=<db_dbcomputer>, default=ScalarElementColumnDefault('')), Column('scheduler_type', String(length=255), table=<db_dbcomputer>, default=ScalarElementColumnDefault('')), Column('transport_type', String(length=255), table=<db_dbcomputer>, default=ScalarElementColumnDefault('')), Column('metadata', JSON(), table=<db_dbcomputer>, default=CallableColumnDefault(<function dict>)), schema=None)#
__tablename__ = 'db_dbcomputer'#
_metadata#
_sa_class_manager = {'_metadata': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'description': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'hostname': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'label': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'scheduler_type': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'transport_type': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'uuid': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}#
description#
hostname#
id#
label#
scheduler_type#
transport_type#
uuid#
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbGroup(**kwargs)[source]#

Bases: Base

Class to store groups.

__init__(**kwargs)#

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7f3b773451b0; DbGroup>#
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'#
__table__ = Table('db_dbgroup', MetaData(), Column('id', Integer(), table=<db_dbgroup>, primary_key=True, nullable=False), Column('uuid', CHAR(length=32), table=<db_dbgroup>, nullable=False, default=CallableColumnDefault(<function get_new_uuid>)), Column('label', String(length=255), table=<db_dbgroup>, nullable=False), Column('type_string', String(length=255), table=<db_dbgroup>, default=ScalarElementColumnDefault('')), Column('time', DateTime(timezone=True), table=<db_dbgroup>, default=CallableColumnDefault(<function now>)), Column('description', Text(), table=<db_dbgroup>, default=ScalarElementColumnDefault('')), Column('extras', JSON(), table=<db_dbgroup>, nullable=False, default=CallableColumnDefault(<function dict>)), Column('user_id', Integer(), ForeignKey('db_dbuser.id'), table=<db_dbgroup>, nullable=False), schema=None)#
__table_args__ = (UniqueConstraint(Column('label', String(length=255), table=<db_dbgroup>, nullable=False), Column('type_string', String(length=255), table=<db_dbgroup>, default=ScalarElementColumnDefault(''))),)#
__tablename__ = 'db_dbgroup'#
_sa_class_manager = {'description': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'extras': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'label': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'time': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'type_string': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'user_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'uuid': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}#
description#
extras#
id#
label#
time#
type_string#
user_id#
uuid#
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbGroupNodes(**kwargs)[source]#

Bases: Base

Class to store join table for group -> nodes.

__init__(**kwargs)#

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7f3b77344d60; DbGroupNodes>#
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'#
__table__ = Table('db_dbgroup_dbnodes', MetaData(), Column('id', Integer(), table=<db_dbgroup_dbnodes>, primary_key=True, nullable=False), Column('dbnode_id', Integer(), ForeignKey('db_dbnode.id'), table=<db_dbgroup_dbnodes>, nullable=False), Column('dbgroup_id', Integer(), ForeignKey('db_dbgroup.id'), table=<db_dbgroup_dbnodes>, nullable=False), schema=None)#
__table_args__ = (UniqueConstraint(Column('dbgroup_id', Integer(), ForeignKey('db_dbgroup.id'), table=<db_dbgroup_dbnodes>, nullable=False), Column('dbnode_id', Integer(), ForeignKey('db_dbnode.id'), table=<db_dbgroup_dbnodes>, nullable=False)),)#
__tablename__ = 'db_dbgroup_dbnodes'#
_sa_class_manager = {'dbgroup_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'dbnode_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}#
dbgroup_id#
dbnode_id#
id#

Bases: Base

Class to store links between nodes.

__init__(**kwargs)#

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7f3b77374d90; DbLink>#
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'#
__table__ = Table('db_dblink', MetaData(), Column('id', Integer(), table=<db_dblink>, primary_key=True, nullable=False), Column('input_id', Integer(), ForeignKey('db_dbnode.id'), table=<db_dblink>, nullable=False), Column('output_id', Integer(), ForeignKey('db_dbnode.id'), table=<db_dblink>, nullable=False), Column('label', String(length=255), table=<db_dblink>, nullable=False, default=ScalarElementColumnDefault('')), Column('type', String(length=255), table=<db_dblink>, nullable=False), schema=None)#
__tablename__ = 'db_dblink'#
_sa_class_manager = {'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'input_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'label': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'output_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'type': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}#
id#
input_id#
label#
output_id#
type#
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbLog(**kwargs)[source]#

Bases: Base

Class to store logs.

__init__(**kwargs)#

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7f3b773461a0; DbLog>#
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'#
__table__ = Table('db_dblog', MetaData(), Column('id', Integer(), table=<db_dblog>, primary_key=True, nullable=False), Column('uuid', CHAR(length=32), table=<db_dblog>, nullable=False, default=CallableColumnDefault(<function get_new_uuid>)), Column('time', DateTime(timezone=True), table=<db_dblog>, default=CallableColumnDefault(<function now>)), Column('loggername', String(length=255), table=<db_dblog>, default=ScalarElementColumnDefault('')), Column('levelname', String(length=50), table=<db_dblog>, default=ScalarElementColumnDefault('')), Column('dbnode_id', Integer(), ForeignKey('db_dbnode.id'), table=<db_dblog>, nullable=False), Column('message', Text(), table=<db_dblog>, default=ScalarElementColumnDefault('')), Column('metadata', JSON(), table=<db_dblog>, default=CallableColumnDefault(<function dict>)), schema=None)#
__tablename__ = 'db_dblog'#
_metadata#
_sa_class_manager = {'_metadata': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'dbnode_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'levelname': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'loggername': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'message': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'time': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'uuid': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}#
dbnode_id#
id#
levelname#
loggername#
message#
time#
uuid#
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbNode(**kwargs)[source]#

Bases: Base

Class to store nodes.

__init__(**kwargs)#

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7f3b77346dd0; DbNode>#
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'#
__table__ = Table('db_dbnode', MetaData(), Column('id', Integer(), table=<db_dbnode>, primary_key=True, nullable=False), Column('uuid', CHAR(length=32), table=<db_dbnode>, nullable=False, default=CallableColumnDefault(<function get_new_uuid>)), Column('node_type', String(length=255), table=<db_dbnode>, nullable=False, default=ScalarElementColumnDefault('')), Column('process_type', String(length=255), table=<db_dbnode>), Column('label', String(length=255), table=<db_dbnode>, default=ScalarElementColumnDefault('')), Column('description', Text(), table=<db_dbnode>, default=ScalarElementColumnDefault('')), Column('ctime', DateTime(timezone=True), table=<db_dbnode>, default=CallableColumnDefault(<function now>)), Column('mtime', DateTime(timezone=True), table=<db_dbnode>, default=CallableColumnDefault(<function now>)), Column('attributes', JSON(), table=<db_dbnode>), Column('extras', JSON(), table=<db_dbnode>), Column('repository_metadata', JSON(), table=<db_dbnode>, nullable=False, default=CallableColumnDefault(<function dict>), server_default=DefaultClause('{}', for_update=False)), Column('dbcomputer_id', Integer(), ForeignKey('db_dbcomputer.id'), table=<db_dbnode>), Column('user_id', Integer(), ForeignKey('db_dbuser.id'), table=<db_dbnode>, nullable=False), schema=None)#
__tablename__ = 'db_dbnode'#
_sa_class_manager = {'attributes': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'ctime': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'dbcomputer_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'description': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'extras': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'label': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'mtime': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'node_type': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'process_type': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'repository_metadata': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'user_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'uuid': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}#
attributes#
ctime#
dbcomputer_id#
description#
extras#
id#
label#
mtime#
node_type#
process_type#
repository_metadata#
user_id#
uuid#
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbUser(**kwargs)[source]#

Bases: Base

Class to store users.

__init__(**kwargs)#

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7f3b77375bd0; DbUser>#
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'#
__table__ = Table('db_dbuser', MetaData(), Column('id', Integer(), table=<db_dbuser>, primary_key=True, nullable=False), Column('email', String(length=254), table=<db_dbuser>, nullable=False), Column('first_name', String(length=254), table=<db_dbuser>, default=ScalarElementColumnDefault('')), Column('last_name', String(length=254), table=<db_dbuser>, default=ScalarElementColumnDefault('')), Column('institution', String(length=254), table=<db_dbuser>, default=ScalarElementColumnDefault('')), schema=None)#
__tablename__ = 'db_dbuser'#
_sa_class_manager = {'email': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'first_name': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'institution': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'last_name': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}#
email#
first_name#
id#
institution#
last_name#