aiida.storage.sqlite_zip.migrations package

Submodules

Upper level SQLAlchemy migration funcitons.

aiida.storage.sqlite_zip.migrations.env.run_migrations_online()[source]

Run migrations in ‘online’ mode.

The connection should have been passed to the config, which we use to configue the migration context.

Migration from the “legacy” JSON format, to an sqlite database, and node uuid based repository to hash based.

aiida.storage.sqlite_zip.migrations.legacy_to_main._convert_datetime(key, value)[source]
aiida.storage.sqlite_zip.migrations.legacy_to_main._create_directory(top_level: aiida.repository.common.File, path: pathlib.PurePosixPath) aiida.repository.common.File[source]

Create a new directory with the given path.

Parameters

path – the relative path of the directory.

Returns

the created directory.

aiida.storage.sqlite_zip.migrations.legacy_to_main._create_repo_metadata(paths: List[Tuple[str, Optional[str]]]) Dict[str, Any][source]

Create the repository metadata.

Parameters

paths – list of (path, hashkey) tuples

Returns

the repository metadata

aiida.storage.sqlite_zip.migrations.legacy_to_main._iter_entity_fields(data, name: str, node_repos: Dict[str, List[Tuple[str, Optional[str]]]]) Iterator[Dict[str, Any]][source]

Iterate through entity fields.

aiida.storage.sqlite_zip.migrations.legacy_to_main._json_to_sqlite(outpath: pathlib.Path, data: dict, node_repos: Dict[str, List[Tuple[str, Optional[str]]]], batch_size: int = 100) None[source]

Convert a JSON archive format to SQLite.

aiida.storage.sqlite_zip.migrations.legacy_to_main.perform_v1_migration(inpath: pathlib.Path, working: pathlib.Path, new_zip: archive_path.zip_path.ZipPath, central_dir: Dict[str, Any], is_tar: bool, metadata: dict, data: dict) pathlib.Path[source]

Perform the repository and JSON to SQLite migration.

  1. Iterate though the repository paths in the archive

  2. If a file, hash its contents and, if not already present, stream it to the new archive

  3. Store a mapping of the node UUIDs to a list of (path, hashkey or None if a directory) tuples

Parameters
  • inpath – the input path to the old archive

  • metadata – the metadata to migrate

  • data – the data to migrate

:returns:the path to the sqlite database file

Common variables

aiida.storage.sqlite_zip.migrations.utils.copy_tar_to_zip(inpath: pathlib.Path, outpath: pathlib.Path, path_callback: Callable[[pathlib.Path, archive_path.zip_path.ZipPath], bool], *, compression: int = 6, overwrite: bool = True, title: str = 'Writing new zip file', info_order: Sequence[str] = ()) None[source]

Create a new zip file from an existing tar file.

The tar file is first extracted to a temporary directory, and then the new zip file is created, with the path_callback allowing for per path modifications. The new zip file is first created in a temporary directory, and then moved to the desired location.

Parameters
  • inpath – the path to the existing archive

  • outpath – the path to output the new archive

  • path_callback – a callback that is called for each path in the archive: (inpath, outpath) -> handled If handled is True, the path is assumed to already have been copied to the new zip file.

  • compression – the default compression level to use for the new zip file

  • overwrite – whether to overwrite the output file if it already exists

  • title – the title of the progress bar

  • info_orderZipInfo for these file names will be written first to the zip central directory. This allows for faster reading of these files, with archive_path.read_file_in_zip.

aiida.storage.sqlite_zip.migrations.utils.copy_zip_to_zip(inpath: pathlib.Path, outpath: pathlib.Path, path_callback: Callable[[archive_path.zip_path.ZipPath, archive_path.zip_path.ZipPath], bool], *, compression: int = 6, overwrite: bool = True, title: str = 'Writing new zip file', info_order: Sequence[str] = ()) None[source]

Create a new zip file from an existing zip file.

All files/folders are streamed directly to the new zip file, with the path_callback allowing for per path modifications. The new zip file is first created in a temporary directory, and then moved to the desired location.

Parameters
  • inpath – the path to the existing archive

  • outpath – the path to output the new archive

  • path_callback – a callback that is called for each path in the archive: (inpath, outpath) -> handled If handled is True, the path is assumed to already have been copied to the new zip file.

  • compression – the default compression level to use for the new zip file

  • overwrite – whether to overwrite the output file if it already exists

  • title – the title of the progress bar

  • info_orderZipInfo for these file names will be written first to the zip central directory. This allows for faster reading of these files, with archive_path.read_file_in_zip.

aiida.storage.sqlite_zip.migrations.utils.update_metadata(metadata, version)[source]

Update the metadata with a new version number and a notification of the conversion that was executed.

Parameters
  • metadata – the content of an export archive metadata.json file

  • version – string version number that the updated metadata should get

aiida.storage.sqlite_zip.migrations.utils.verify_metadata_version(metadata, version=None)[source]

Utility function to verify that the metadata has the correct version number.

If no version number is passed, it will just extract the version number and return it.

Parameters
  • metadata – the content of an export archive metadata.json file

  • version – string version number that the metadata is expected to have

This is the sqlite DB schema, coresponding to the main_0000 revision of the sqlite_zip backend, see: versions/main_0000_initial.py

For normal operation of the archive, we auto-generate the schema from the models in aiida.storage.psql_dos.models. However, when migrating an archive from the old format, we require a fixed revision of the schema.

The only difference between the PostGreSQL schema and SQLite one, is the replacement of JSONB with JSON, and UUID with CHAR(32).

class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbAuthInfo(**kwargs)[source]

Bases: sqlalchemy.orm.decl_api.Base

Class that keeps the authentication data.

__init__(**kwargs)

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7efc895ff190; DbAuthInfo>
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'
__table__ = Table('db_dbauthinfo', MetaData(), Column('id', Integer(), table=<db_dbauthinfo>, primary_key=True, nullable=False), Column('aiidauser_id', Integer(), ForeignKey('db_dbuser.id'), table=<db_dbauthinfo>), Column('dbcomputer_id', Integer(), ForeignKey('db_dbcomputer.id'), table=<db_dbauthinfo>), Column('metadata', JSON(), table=<db_dbauthinfo>, default=ColumnDefault(<function dict>)), Column('auth_params', JSON(), table=<db_dbauthinfo>, default=ColumnDefault(<function dict>)), Column('enabled', Boolean(), table=<db_dbauthinfo>, default=ColumnDefault(True)), schema=None)
__table_args__ = (UniqueConstraint(Column('aiidauser_id', Integer(), ForeignKey('db_dbuser.id'), table=<db_dbauthinfo>), Column('dbcomputer_id', Integer(), ForeignKey('db_dbcomputer.id'), table=<db_dbauthinfo>)),)
__tablename__ = 'db_dbauthinfo'
_metadata
_sa_class_manager = {'_metadata': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'aiidauser_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'auth_params': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'dbcomputer_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'enabled': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}
aiidauser_id
auth_params
dbcomputer_id
enabled
id
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbComment(**kwargs)[source]

Bases: sqlalchemy.orm.decl_api.Base

Class to store comments.

__init__(**kwargs)

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7efc895ffca0; DbComment>
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'
__table__ = Table('db_dbcomment', MetaData(), Column('id', Integer(), table=<db_dbcomment>, primary_key=True, nullable=False), Column('uuid', CHAR(length=32), table=<db_dbcomment>, nullable=False, default=ColumnDefault(<function get_new_uuid>)), Column('dbnode_id', Integer(), ForeignKey('db_dbnode.id'), table=<db_dbcomment>), Column('ctime', DateTime(timezone=True), table=<db_dbcomment>, default=ColumnDefault(<function now>)), Column('mtime', DateTime(timezone=True), table=<db_dbcomment>, default=ColumnDefault(<function now>)), Column('user_id', Integer(), ForeignKey('db_dbuser.id'), table=<db_dbcomment>), Column('content', Text(), table=<db_dbcomment>, default=ColumnDefault('')), schema=None)
__tablename__ = 'db_dbcomment'
_sa_class_manager = {'content': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'ctime': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'dbnode_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'mtime': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'user_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'uuid': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}
content
ctime
dbnode_id
id
mtime
user_id
uuid
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbComputer(**kwargs)[source]

Bases: sqlalchemy.orm.decl_api.Base

Class to store computers.

__init__(**kwargs)

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7efc8969ae50; DbComputer>
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'
__table__ = Table('db_dbcomputer', MetaData(), Column('id', Integer(), table=<db_dbcomputer>, primary_key=True, nullable=False), Column('uuid', CHAR(length=32), table=<db_dbcomputer>, nullable=False, default=ColumnDefault(<function get_new_uuid>)), Column('label', String(length=255), table=<db_dbcomputer>, nullable=False), Column('hostname', String(length=255), table=<db_dbcomputer>, default=ColumnDefault('')), Column('description', Text(), table=<db_dbcomputer>, default=ColumnDefault('')), Column('scheduler_type', String(length=255), table=<db_dbcomputer>, default=ColumnDefault('')), Column('transport_type', String(length=255), table=<db_dbcomputer>, default=ColumnDefault('')), Column('metadata', JSON(), table=<db_dbcomputer>, default=ColumnDefault(<function dict>)), schema=None)
__tablename__ = 'db_dbcomputer'
_metadata
_sa_class_manager = {'_metadata': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'description': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'hostname': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'label': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'scheduler_type': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'transport_type': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'uuid': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}
description
hostname
id
label
scheduler_type
transport_type
uuid
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbGroup(**kwargs)[source]

Bases: sqlalchemy.orm.decl_api.Base

Class to store groups.

__init__(**kwargs)

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7efc8958ae80; DbGroup>
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'
__table__ = Table('db_dbgroup', MetaData(), Column('id', Integer(), table=<db_dbgroup>, primary_key=True, nullable=False), Column('uuid', CHAR(length=32), table=<db_dbgroup>, nullable=False, default=ColumnDefault(<function get_new_uuid>)), Column('label', String(length=255), table=<db_dbgroup>, nullable=False), Column('type_string', String(length=255), table=<db_dbgroup>, default=ColumnDefault('')), Column('time', DateTime(timezone=True), table=<db_dbgroup>, default=ColumnDefault(<function now>)), Column('description', Text(), table=<db_dbgroup>, default=ColumnDefault('')), Column('extras', JSON(), table=<db_dbgroup>, nullable=False, default=ColumnDefault(<function dict>)), Column('user_id', Integer(), ForeignKey('db_dbuser.id'), table=<db_dbgroup>, nullable=False), schema=None)
__table_args__ = (UniqueConstraint(Column('label', String(length=255), table=<db_dbgroup>, nullable=False), Column('type_string', String(length=255), table=<db_dbgroup>, default=ColumnDefault(''))),)
__tablename__ = 'db_dbgroup'
_sa_class_manager = {'description': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'extras': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'label': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'time': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'type_string': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'user_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'uuid': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}
description
extras
id
label
time
type_string
user_id
uuid
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbGroupNodes(**kwargs)[source]

Bases: sqlalchemy.orm.decl_api.Base

Class to store join table for group -> nodes.

__init__(**kwargs)

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7efc896921c0; DbGroupNodes>
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'
__table__ = Table('db_dbgroup_dbnodes', MetaData(), Column('id', Integer(), table=<db_dbgroup_dbnodes>, primary_key=True, nullable=False), Column('dbnode_id', Integer(), ForeignKey('db_dbnode.id'), table=<db_dbgroup_dbnodes>, nullable=False), Column('dbgroup_id', Integer(), ForeignKey('db_dbgroup.id'), table=<db_dbgroup_dbnodes>, nullable=False), schema=None)
__table_args__ = (UniqueConstraint(Column('dbgroup_id', Integer(), ForeignKey('db_dbgroup.id'), table=<db_dbgroup_dbnodes>, nullable=False), Column('dbnode_id', Integer(), ForeignKey('db_dbnode.id'), table=<db_dbgroup_dbnodes>, nullable=False)),)
__tablename__ = 'db_dbgroup_dbnodes'
_sa_class_manager = {'dbgroup_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'dbnode_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}
dbgroup_id
dbnode_id
id

Bases: sqlalchemy.orm.decl_api.Base

Class to store links between nodes.

__init__(**kwargs)

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7efc895bd340; DbLink>
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'
__table__ = Table('db_dblink', MetaData(), Column('id', Integer(), table=<db_dblink>, primary_key=True, nullable=False), Column('input_id', Integer(), ForeignKey('db_dbnode.id'), table=<db_dblink>, nullable=False), Column('output_id', Integer(), ForeignKey('db_dbnode.id'), table=<db_dblink>, nullable=False), Column('label', String(length=255), table=<db_dblink>, nullable=False, default=ColumnDefault('')), Column('type', String(length=255), table=<db_dblink>, nullable=False), schema=None)
__tablename__ = 'db_dblink'
_sa_class_manager = {'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'input_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'label': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'output_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'type': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}
id
input_id
label
output_id
type
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbLog(**kwargs)[source]

Bases: sqlalchemy.orm.decl_api.Base

Class to store logs.

__init__(**kwargs)

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7efc8959ba90; DbLog>
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'
__table__ = Table('db_dblog', MetaData(), Column('id', Integer(), table=<db_dblog>, primary_key=True, nullable=False), Column('uuid', CHAR(length=32), table=<db_dblog>, nullable=False, default=ColumnDefault(<function get_new_uuid>)), Column('time', DateTime(timezone=True), table=<db_dblog>, default=ColumnDefault(<function now>)), Column('loggername', String(length=255), table=<db_dblog>, default=ColumnDefault('')), Column('levelname', String(length=50), table=<db_dblog>, default=ColumnDefault('')), Column('dbnode_id', Integer(), ForeignKey('db_dbnode.id'), table=<db_dblog>, nullable=False), Column('message', Text(), table=<db_dblog>, default=ColumnDefault('')), Column('metadata', JSON(), table=<db_dblog>, default=ColumnDefault(<function dict>)), schema=None)
__tablename__ = 'db_dblog'
_metadata
_sa_class_manager = {'_metadata': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'dbnode_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'levelname': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'loggername': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'message': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'time': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'uuid': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}
dbnode_id
id
levelname
loggername
message
time
uuid
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbNode(**kwargs)[source]

Bases: sqlalchemy.orm.decl_api.Base

Class to store nodes.

__init__(**kwargs)

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7efc895ac9d0; DbNode>
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'
__table__ = Table('db_dbnode', MetaData(), Column('id', Integer(), table=<db_dbnode>, primary_key=True, nullable=False), Column('uuid', CHAR(length=32), table=<db_dbnode>, nullable=False, default=ColumnDefault(<function get_new_uuid>)), Column('node_type', String(length=255), table=<db_dbnode>, nullable=False, default=ColumnDefault('')), Column('process_type', String(length=255), table=<db_dbnode>), Column('label', String(length=255), table=<db_dbnode>, default=ColumnDefault('')), Column('description', Text(), table=<db_dbnode>, default=ColumnDefault('')), Column('ctime', DateTime(timezone=True), table=<db_dbnode>, default=ColumnDefault(<function now>)), Column('mtime', DateTime(timezone=True), table=<db_dbnode>, default=ColumnDefault(<function now>)), Column('attributes', JSON(), table=<db_dbnode>), Column('extras', JSON(), table=<db_dbnode>), Column('repository_metadata', JSON(), table=<db_dbnode>, nullable=False, default=ColumnDefault(<function dict>), server_default=DefaultClause('{}', for_update=False)), Column('dbcomputer_id', Integer(), ForeignKey('db_dbcomputer.id'), table=<db_dbnode>), Column('user_id', Integer(), ForeignKey('db_dbuser.id'), table=<db_dbnode>, nullable=False), schema=None)
__tablename__ = 'db_dbnode'
_sa_class_manager = {'attributes': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'ctime': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'dbcomputer_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'description': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'extras': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'label': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'mtime': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'node_type': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'process_type': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'repository_metadata': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'user_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'uuid': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}
attributes
ctime
dbcomputer_id
description
extras
id
label
mtime
node_type
process_type
repository_metadata
user_id
uuid
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbUser(**kwargs)[source]

Bases: sqlalchemy.orm.decl_api.Base

Class to store users.

__init__(**kwargs)

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7efc895bdc10; DbUser>
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'
__table__ = Table('db_dbuser', MetaData(), Column('id', Integer(), table=<db_dbuser>, primary_key=True, nullable=False), Column('email', String(length=254), table=<db_dbuser>, nullable=False), Column('first_name', String(length=254), table=<db_dbuser>, default=ColumnDefault('')), Column('last_name', String(length=254), table=<db_dbuser>, default=ColumnDefault('')), Column('institution', String(length=254), table=<db_dbuser>, default=ColumnDefault('')), schema=None)
__tablename__ = 'db_dbuser'
_sa_class_manager = {'email': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'first_name': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'institution': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'last_name': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}
email
first_name
id
institution
last_name