aiida.storage.sqlite_zip.migrations package#

Subpackages#

Submodules#

Upper level SQLAlchemy migration funcitons.

aiida.storage.sqlite_zip.migrations.env.run_migrations_online()[源代码]#

Run migrations in ‘online’ mode.

The connection should have been passed to the config, which we use to configue the migration context.

Migration from the “legacy” JSON format, to an sqlite database, and node uuid based repository to hash based.

aiida.storage.sqlite_zip.migrations.legacy_to_main._convert_datetime(key, value)[源代码]#
aiida.storage.sqlite_zip.migrations.legacy_to_main._create_directory(top_level: File, path: PurePosixPath) File[源代码]#

Create a new directory with the given path.

参数:

path – the relative path of the directory.

返回:

the created directory.

aiida.storage.sqlite_zip.migrations.legacy_to_main._create_repo_metadata(paths: List[Tuple[str, str | None]]) Dict[str, Any][源代码]#

Create the repository metadata.

参数:

paths – list of (path, hashkey) tuples

返回:

the repository metadata

aiida.storage.sqlite_zip.migrations.legacy_to_main._iter_entity_fields(data, name: str, node_repos: Dict[str, List[Tuple[str, str | None]]]) Iterator[Dict[str, Any]][源代码]#

Iterate through entity fields.

aiida.storage.sqlite_zip.migrations.legacy_to_main._json_to_sqlite(outpath: Path, data: dict, node_repos: Dict[str, List[Tuple[str, str | None]]], batch_size: int = 100) None[源代码]#

Convert a JSON archive format to SQLite.

aiida.storage.sqlite_zip.migrations.legacy_to_main.perform_v1_migration(inpath: Path, working: Path, new_zip: ZipPath, central_dir: Dict[str, Any], is_tar: bool, metadata: dict, data: dict) Path[源代码]#

Perform the repository and JSON to SQLite migration.

  1. Iterate though the repository paths in the archive

  2. If a file, hash its contents and, if not already present, stream it to the new archive

  3. Store a mapping of the node UUIDs to a list of (path, hashkey or None if a directory) tuples

参数:
  • inpath – the input path to the old archive

  • metadata – the metadata to migrate

  • data – the data to migrate

:returns:the path to the sqlite database file

Common variables

aiida.storage.sqlite_zip.migrations.utils.copy_tar_to_zip(inpath: Path, outpath: Path, path_callback: Callable[[Path, ZipPath], bool], *, compression: int = 6, overwrite: bool = True, title: str = 'Writing new zip file', info_order: Sequence[str] = ()) None[源代码]#

Create a new zip file from an existing tar file.

The tar file is first extracted to a temporary directory, and then the new zip file is created, with the path_callback allowing for per path modifications. The new zip file is first created in a temporary directory, and then moved to the desired location.

参数:
  • inpath – the path to the existing archive

  • outpath – the path to output the new archive

  • path_callback – a callback that is called for each path in the archive: (inpath, outpath) -> handled If handled is True, the path is assumed to already have been copied to the new zip file.

  • compression – the default compression level to use for the new zip file

  • overwrite – whether to overwrite the output file if it already exists

  • title – the title of the progress bar

  • info_orderZipInfo for these file names will be written first to the zip central directory. This allows for faster reading of these files, with archive_path.read_file_in_zip.

aiida.storage.sqlite_zip.migrations.utils.copy_zip_to_zip(inpath: Path, outpath: Path, path_callback: Callable[[ZipPath, ZipPath], bool], *, compression: int = 6, overwrite: bool = True, title: str = 'Writing new zip file', info_order: Sequence[str] = ()) None[源代码]#

Create a new zip file from an existing zip file.

All files/folders are streamed directly to the new zip file, with the path_callback allowing for per path modifications. The new zip file is first created in a temporary directory, and then moved to the desired location.

参数:
  • inpath – the path to the existing archive

  • outpath – the path to output the new archive

  • path_callback – a callback that is called for each path in the archive: (inpath, outpath) -> handled If handled is True, the path is assumed to already have been copied to the new zip file.

  • compression – the default compression level to use for the new zip file

  • overwrite – whether to overwrite the output file if it already exists

  • title – the title of the progress bar

  • info_orderZipInfo for these file names will be written first to the zip central directory. This allows for faster reading of these files, with archive_path.read_file_in_zip.

aiida.storage.sqlite_zip.migrations.utils.update_metadata(metadata, version)[源代码]#

Update the metadata with a new version number and a notification of the conversion that was executed.

参数:
  • metadata – the content of an export archive metadata.json file

  • version – string version number that the updated metadata should get

aiida.storage.sqlite_zip.migrations.utils.verify_metadata_version(metadata, version=None)[源代码]#

Utility function to verify that the metadata has the correct version number.

If no version number is passed, it will just extract the version number and return it.

参数:
  • metadata – the content of an export archive metadata.json file

  • version – string version number that the metadata is expected to have

This is the sqlite DB schema, coresponding to the main_0000 revision of the sqlite_zip backend, see: versions/main_0000_initial.py

For normal operation of the archive, we auto-generate the schema from the models in aiida.storage.psql_dos.models. However, when migrating an archive from the old format, we require a fixed revision of the schema.

The only difference between the PostGreSQL schema and SQLite one, is the replacement of JSONB with JSON, and UUID with CHAR(32).

class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbAuthInfo(**kwargs)[源代码]#

基类:Base

Class that keeps the authentication data.

__init__(**kwargs)#

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7fc9f5e11cd0; DbAuthInfo>#
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'#
__table__ = Table('db_dbauthinfo', MetaData(), Column('id', Integer(), table=<db_dbauthinfo>, primary_key=True, nullable=False), Column('aiidauser_id', Integer(), ForeignKey('db_dbuser.id'), table=<db_dbauthinfo>), Column('dbcomputer_id', Integer(), ForeignKey('db_dbcomputer.id'), table=<db_dbauthinfo>), Column('metadata', JSON(), table=<db_dbauthinfo>, default=CallableColumnDefault(<function dict>)), Column('auth_params', JSON(), table=<db_dbauthinfo>, default=CallableColumnDefault(<function dict>)), Column('enabled', Boolean(), table=<db_dbauthinfo>, default=ScalarElementColumnDefault(True)), schema=None)#
__table_args__ = (UniqueConstraint(Column('aiidauser_id', Integer(), ForeignKey('db_dbuser.id'), table=<db_dbauthinfo>), Column('dbcomputer_id', Integer(), ForeignKey('db_dbcomputer.id'), table=<db_dbauthinfo>)),)#
__tablename__ = 'db_dbauthinfo'#
_metadata#
_sa_class_manager = {'_metadata': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'aiidauser_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'auth_params': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'dbcomputer_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'enabled': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}#
aiidauser_id#
auth_params#
dbcomputer_id#
enabled#
id#
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbComment(**kwargs)[源代码]#

基类:Base

Class to store comments.

__init__(**kwargs)#

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7fc9f5cd3ed0; DbComment>#
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'#
__table__ = Table('db_dbcomment', MetaData(), Column('id', Integer(), table=<db_dbcomment>, primary_key=True, nullable=False), Column('uuid', CHAR(length=32), table=<db_dbcomment>, nullable=False, default=CallableColumnDefault(<function get_new_uuid>)), Column('dbnode_id', Integer(), ForeignKey('db_dbnode.id'), table=<db_dbcomment>), Column('ctime', DateTime(timezone=True), table=<db_dbcomment>, default=CallableColumnDefault(<function now>)), Column('mtime', DateTime(timezone=True), table=<db_dbcomment>, default=CallableColumnDefault(<function now>)), Column('user_id', Integer(), ForeignKey('db_dbuser.id'), table=<db_dbcomment>), Column('content', Text(), table=<db_dbcomment>, default=ScalarElementColumnDefault('')), schema=None)#
__tablename__ = 'db_dbcomment'#
_sa_class_manager = {'content': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'ctime': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'dbnode_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'mtime': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'user_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'uuid': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}#
content#
ctime#
dbnode_id#
id#
mtime#
user_id#
uuid#
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbComputer(**kwargs)[源代码]#

基类:Base

Class to store computers.

__init__(**kwargs)#

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7fc9f5ccbad0; DbComputer>#
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'#
__table__ = Table('db_dbcomputer', MetaData(), Column('id', Integer(), table=<db_dbcomputer>, primary_key=True, nullable=False), Column('uuid', CHAR(length=32), table=<db_dbcomputer>, nullable=False, default=CallableColumnDefault(<function get_new_uuid>)), Column('label', String(length=255), table=<db_dbcomputer>, nullable=False), Column('hostname', String(length=255), table=<db_dbcomputer>, default=ScalarElementColumnDefault('')), Column('description', Text(), table=<db_dbcomputer>, default=ScalarElementColumnDefault('')), Column('scheduler_type', String(length=255), table=<db_dbcomputer>, default=ScalarElementColumnDefault('')), Column('transport_type', String(length=255), table=<db_dbcomputer>, default=ScalarElementColumnDefault('')), Column('metadata', JSON(), table=<db_dbcomputer>, default=CallableColumnDefault(<function dict>)), schema=None)#
__tablename__ = 'db_dbcomputer'#
_metadata#
_sa_class_manager = {'_metadata': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'description': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'hostname': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'label': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'scheduler_type': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'transport_type': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'uuid': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}#
description#
hostname#
id#
label#
scheduler_type#
transport_type#
uuid#
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbGroup(**kwargs)[源代码]#

基类:Base

Class to store groups.

__init__(**kwargs)#

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7fc9f5c2e450; DbGroup>#
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'#
__table__ = Table('db_dbgroup', MetaData(), Column('id', Integer(), table=<db_dbgroup>, primary_key=True, nullable=False), Column('uuid', CHAR(length=32), table=<db_dbgroup>, nullable=False, default=CallableColumnDefault(<function get_new_uuid>)), Column('label', String(length=255), table=<db_dbgroup>, nullable=False), Column('type_string', String(length=255), table=<db_dbgroup>, default=ScalarElementColumnDefault('')), Column('time', DateTime(timezone=True), table=<db_dbgroup>, default=CallableColumnDefault(<function now>)), Column('description', Text(), table=<db_dbgroup>, default=ScalarElementColumnDefault('')), Column('extras', JSON(), table=<db_dbgroup>, nullable=False, default=CallableColumnDefault(<function dict>)), Column('user_id', Integer(), ForeignKey('db_dbuser.id'), table=<db_dbgroup>, nullable=False), schema=None)#
__table_args__ = (UniqueConstraint(Column('label', String(length=255), table=<db_dbgroup>, nullable=False), Column('type_string', String(length=255), table=<db_dbgroup>, default=ScalarElementColumnDefault(''))),)#
__tablename__ = 'db_dbgroup'#
_sa_class_manager = {'description': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'extras': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'label': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'time': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'type_string': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'user_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'uuid': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}#
description#
extras#
id#
label#
time#
type_string#
user_id#
uuid#
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbGroupNodes(**kwargs)[源代码]#

基类:Base

Class to store join table for group -> nodes.

__init__(**kwargs)#

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7fc9f5cbef10; DbGroupNodes>#
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'#
__table__ = Table('db_dbgroup_dbnodes', MetaData(), Column('id', Integer(), table=<db_dbgroup_dbnodes>, primary_key=True, nullable=False), Column('dbnode_id', Integer(), ForeignKey('db_dbnode.id'), table=<db_dbgroup_dbnodes>, nullable=False), Column('dbgroup_id', Integer(), ForeignKey('db_dbgroup.id'), table=<db_dbgroup_dbnodes>, nullable=False), schema=None)#
__table_args__ = (UniqueConstraint(Column('dbgroup_id', Integer(), ForeignKey('db_dbgroup.id'), table=<db_dbgroup_dbnodes>, nullable=False), Column('dbnode_id', Integer(), ForeignKey('db_dbnode.id'), table=<db_dbgroup_dbnodes>, nullable=False)),)#
__tablename__ = 'db_dbgroup_dbnodes'#
_sa_class_manager = {'dbgroup_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'dbnode_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}#
dbgroup_id#
dbnode_id#
id#

基类:Base

Class to store links between nodes.

__init__(**kwargs)#

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7fc9f5c68750; DbLink>#
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'#
__table__ = Table('db_dblink', MetaData(), Column('id', Integer(), table=<db_dblink>, primary_key=True, nullable=False), Column('input_id', Integer(), ForeignKey('db_dbnode.id'), table=<db_dblink>, nullable=False), Column('output_id', Integer(), ForeignKey('db_dbnode.id'), table=<db_dblink>, nullable=False), Column('label', String(length=255), table=<db_dblink>, nullable=False, default=ScalarElementColumnDefault('')), Column('type', String(length=255), table=<db_dblink>, nullable=False), schema=None)#
__tablename__ = 'db_dblink'#
_sa_class_manager = {'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'input_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'label': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'output_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'type': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}#
id#
input_id#
label#
output_id#
type#
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbLog(**kwargs)[源代码]#

基类:Base

Class to store logs.

__init__(**kwargs)#

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7fc9f6defd10; DbLog>#
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'#
__table__ = Table('db_dblog', MetaData(), Column('id', Integer(), table=<db_dblog>, primary_key=True, nullable=False), Column('uuid', CHAR(length=32), table=<db_dblog>, nullable=False, default=CallableColumnDefault(<function get_new_uuid>)), Column('time', DateTime(timezone=True), table=<db_dblog>, default=CallableColumnDefault(<function now>)), Column('loggername', String(length=255), table=<db_dblog>, default=ScalarElementColumnDefault('')), Column('levelname', String(length=50), table=<db_dblog>, default=ScalarElementColumnDefault('')), Column('dbnode_id', Integer(), ForeignKey('db_dbnode.id'), table=<db_dblog>, nullable=False), Column('message', Text(), table=<db_dblog>, default=ScalarElementColumnDefault('')), Column('metadata', JSON(), table=<db_dblog>, default=CallableColumnDefault(<function dict>)), schema=None)#
__tablename__ = 'db_dblog'#
_metadata#
_sa_class_manager = {'_metadata': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'dbnode_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'levelname': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'loggername': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'message': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'time': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'uuid': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}#
dbnode_id#
id#
levelname#
loggername#
message#
time#
uuid#
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbNode(**kwargs)[源代码]#

基类:Base

Class to store nodes.

__init__(**kwargs)#

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7fc9f5c4b350; DbNode>#
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'#
__table__ = Table('db_dbnode', MetaData(), Column('id', Integer(), table=<db_dbnode>, primary_key=True, nullable=False), Column('uuid', CHAR(length=32), table=<db_dbnode>, nullable=False, default=CallableColumnDefault(<function get_new_uuid>)), Column('node_type', String(length=255), table=<db_dbnode>, nullable=False, default=ScalarElementColumnDefault('')), Column('process_type', String(length=255), table=<db_dbnode>), Column('label', String(length=255), table=<db_dbnode>, default=ScalarElementColumnDefault('')), Column('description', Text(), table=<db_dbnode>, default=ScalarElementColumnDefault('')), Column('ctime', DateTime(timezone=True), table=<db_dbnode>, default=CallableColumnDefault(<function now>)), Column('mtime', DateTime(timezone=True), table=<db_dbnode>, default=CallableColumnDefault(<function now>)), Column('attributes', JSON(), table=<db_dbnode>), Column('extras', JSON(), table=<db_dbnode>), Column('repository_metadata', JSON(), table=<db_dbnode>, nullable=False, default=CallableColumnDefault(<function dict>), server_default=DefaultClause('{}', for_update=False)), Column('dbcomputer_id', Integer(), ForeignKey('db_dbcomputer.id'), table=<db_dbnode>), Column('user_id', Integer(), ForeignKey('db_dbuser.id'), table=<db_dbnode>, nullable=False), schema=None)#
__tablename__ = 'db_dbnode'#
_sa_class_manager = {'attributes': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'ctime': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'dbcomputer_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'description': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'extras': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'label': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'mtime': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'node_type': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'process_type': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'repository_metadata': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'user_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'uuid': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}#
attributes#
ctime#
dbcomputer_id#
description#
extras#
id#
label#
mtime#
node_type#
process_type#
repository_metadata#
user_id#
uuid#
class aiida.storage.sqlite_zip.migrations.v1_db_schema.DbUser(**kwargs)[源代码]#

基类:Base

Class to store users.

__init__(**kwargs)#

A simple constructor that allows initialization from kwargs.

Sets attributes on the constructed instance using the names and values in kwargs.

Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.

__mapper__ = <Mapper at 0x7fc9f5c6bcd0; DbUser>#
__module__ = 'aiida.storage.sqlite_zip.migrations.v1_db_schema'#
__table__ = Table('db_dbuser', MetaData(), Column('id', Integer(), table=<db_dbuser>, primary_key=True, nullable=False), Column('email', String(length=254), table=<db_dbuser>, nullable=False), Column('first_name', String(length=254), table=<db_dbuser>, default=ScalarElementColumnDefault('')), Column('last_name', String(length=254), table=<db_dbuser>, default=ScalarElementColumnDefault('')), Column('institution', String(length=254), table=<db_dbuser>, default=ScalarElementColumnDefault('')), schema=None)#
__tablename__ = 'db_dbuser'#
_sa_class_manager = {'email': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'first_name': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'institution': <sqlalchemy.orm.attributes.InstrumentedAttribute object>, 'last_name': <sqlalchemy.orm.attributes.InstrumentedAttribute object>}#
email#
first_name#
id#
institution#
last_name#