aiida.backends.djsite.db.subtests.migrations package

Submodules

Tests for the migrations of the attributes, extras and settings from EAV to JSONB Migration 0037_attributes_extras_settings_json

class aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json.DbAttributeBaseClass[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json.DbMultipleValueAttributeBaseClass

Abstract base class for tables storing element-attribute-value data. Element is the dbnode; attribute is the key name. Value is the specific value to store.

This table had different SQL columns to store different types of data, and a datatype field to know the actual datatype.

Moreover, this class unpacks dictionaries and lists when possible, so that it is possible to query inside recursive lists and dicts.

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json'
__str__()[source]

Return str(self).

_subspecifier_field_name = 'dbnode'
classmethod set_value_for_node(dbnode, key, value, with_transaction=True, stop_if_existing=False)[source]

This is the raw-level method that accesses the DB. No checks are done to prevent the user from (re)setting a valid key. To be used only internally.

Todo

there may be some error on concurrent write; not checked in this unlucky case!

Parameters
  • dbnode – the dbnode for which the attribute should be stored; if an integer is passed, it will raise, since this functionality is not supported in the models for the migrations.

  • key – the key of the attribute to store; must be a level-zero attribute (i.e., no separators in the key)

  • value – the value of the attribute to store

  • with_transaction – if True (default), do this within a transaction, so that nothing gets stored if a subitem cannot be created. Otherwise, if this parameter is False, no transaction management is performed.

  • stop_if_existing – if True, it will stop with an UniquenessError exception if the key already exists for the given node. Otherwise, it will first delete the old value, if existent. The use with True is useful if you want to use a given attribute as a “locking” value, e.g. to avoid to perform an action twice on the same node. Note that, if you are using transactions, you may get the error only when the transaction is committed.

Raises

ValueError – if the key contains the separator symbol used internally to unpack dictionaries and lists (defined in cls._sep).

class aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json.DbAttributeFunctionality[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json.DbAttributeBaseClass

This class defines all the methods that are needed for the correct deserialization of given attribute dictionaries to the EAV table. It is a stripped-down Django EAV schema to the absolutely necessary methods for this deserialization.

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json'
class aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json.DbExtraFunctionality[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json.DbAttributeBaseClass

This class defines all the methods that are needed for the correct deserialization of given extras dictionaries to the EAV table. It is a stripped-down Django EAV schema to the absolutely necessary methods for this deserialization.

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json'
class aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json.DbMultipleValueAttributeBaseClass[source]

Bases: object

Abstract base class for tables storing attribute + value data, of different data types (without any association to a Node).

class Meta[source]

Bases: object

__dict__ = mappingproxy({'__module__': 'aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json', 'abstract': True, 'unique_together': (('key',),), '__dict__': <attribute '__dict__' of 'Meta' objects>, '__weakref__': <attribute '__weakref__' of 'Meta' objects>, '__doc__': None})
__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json'
__weakref__

list of weak references to the object (if defined)

abstract = True
unique_together = (('key',),)
__dict__ = mappingproxy({'__module__': 'aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json', '__doc__': '\n Abstract base class for tables storing attribute + value data, of\n different data types (without any association to a Node).\n ', '_sep': '.', 'Meta': <class 'aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json.DbMultipleValueAttributeBaseClass.Meta'>, '_subspecifier_field_name': None, 'subspecifier_pk': <property object>, 'validate_key': <classmethod object>, 'set_value': <classmethod object>, 'create_value': <classmethod object>, '__dict__': <attribute '__dict__' of 'DbMultipleValueAttributeBaseClass' objects>, '__weakref__': <attribute '__weakref__' of 'DbMultipleValueAttributeBaseClass' objects>})
__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json'
__weakref__

list of weak references to the object (if defined)

_sep = '.'
_subspecifier_field_name = None
classmethod create_value(key, value, subspecifier_value=None, other_attribs={})[source]

Create a new list of attributes, without storing them, associated with the current key/value pair (and to the given subspecifier, e.g. the DbNode for DbAttributes and DbExtras).

Note

No hits are done on the DB, in particular no check is done on the existence of the given nodes.

Parameters
  • key – a string with the key to create (can contain the separator cls._sep if this is a sub-attribute: indeed, this function calls itself recursively)

  • value – the value to store (a basic data type or a list or a dict)

  • subspecifier_value – must be None if this class has no subspecifier set (e.g., the DbSetting class). Must be the value of the subspecifier (e.g., the dbnode) for classes that define it (e.g. DbAttribute and DbExtra)

  • other_attribs – a dictionary of other parameters, to store only on the level-zero attribute (e.g. for description in DbSetting).

Returns

always a list of class instances; it is the user responsibility to store such entries (typically with a Django bulk_create() call).

classmethod set_value(key, value, with_transaction=True, subspecifier_value=None, other_attribs={}, stop_if_existing=False)[source]

Set a new value in the DB, possibly associated to the given subspecifier.

Note

This method also stored directly in the DB.

Parameters
  • key – a string with the key to create (must be a level-0 attribute, that is it cannot contain the separator cls._sep).

  • value – the value to store (a basic data type or a list or a dict)

  • subspecifier_value – must be None if this class has no subspecifier set (e.g., the DbSetting class). Must be the value of the subspecifier (e.g., the dbnode) for classes that define it (e.g. DbAttribute and DbExtra)

  • with_transaction – True if you want this function to be managed with transactions. Set to False if you already have a manual management of transactions in the block where you are calling this function (useful for speed improvements to avoid recursive transactions)

  • other_attribs – a dictionary of other parameters, to store only on the level-zero attribute (e.g. for description in DbSetting).

  • stop_if_existing – if True, it will stop with an UniquenessError exception if the new entry would violate an uniqueness constraint in the DB (same key, or same key+node, depending on the specific subclass). Otherwise, it will first delete the old value, if existent. The use with True is useful if you want to use a given attribute as a “locking” value, e.g. to avoid to perform an action twice on the same node. Note that, if you are using transactions, you may get the error only when the transaction is committed.

property subspecifier_pk

Return the subspecifier PK in the database (or None, if no subspecifier should be used)

classmethod validate_key(key)[source]

Validate the key string to check if it is valid (e.g., if it does not contain the separator symbol.).

Returns

None if the key is valid

Raises

aiida.common.ValidationError – if the key is not valid

class aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json.TestAttributesExtrasToJSONMigrationManyNodes(methodName='runTest')[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_common.TestMigrations

This test comparing to the previous one (TestAttributesExtrasToJSONMigrationSimple), it creates several nodes with different atributes and extras and checks their correct migration one-by-one.

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json'
migrate_from = '0036_drop_computer_transport_params'
migrate_to = '0037_attributes_extras_settings_json'
nodes_no_to_create = 20
nodes_to_verify = {}
setUpBeforeMigration()[source]

Anything to do before running the migrations, which should be implemented in test subclasses.

test_attributes_extras_migration_many()[source]

Verify that the attributes and extras were migrated correctly

class aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json.TestAttributesExtrasToJSONMigrationSimple(methodName='runTest')[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_common.TestMigrations

A “simple” test for the attributes and extra migration from EAV to JSONB. It stores a sample dictionary using the EAV deserialization of AiiDA Django for the attributes and extras. Then the test checks that they are corerctly converted to JSONB.

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json'
migrate_from = '0036_drop_computer_transport_params'
migrate_to = '0037_attributes_extras_settings_json'
nodes_to_verify = {}
setUpBeforeMigration()[source]

Anything to do before running the migrations, which should be implemented in test subclasses.

test_attributes_extras_migration()[source]

Verify that the attributes and extras were migrated correctly

class aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json.TestSettingsToJSONMigration(methodName='runTest')[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_common.TestMigrations

This test checks the correct migration of the settings. Setting records were used as an example from a typical settings table of Django EAV.

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_0037_attributes_extras_settings_json'
migrate_from = '0036_drop_computer_transport_params'
migrate_to = '0037_attributes_extras_settings_json'
setUpBeforeMigration()[source]

Anything to do before running the migrations, which should be implemented in test subclasses.

settings_info = {}
tearDown()[source]

Deletion of settings - this is needed because settings are not deleted by the typical test cleanup methods.

test_settings_migration()[source]

Verify that the settings were migrated correctly

Tests for the migrations of the attributes, extras and settings from EAV to JSONB Migration 0037_attributes_extras_settings_json

class aiida.backends.djsite.db.subtests.migrations.test_migrations_0038_data_migration_legacy_job_calculations.TestLegacyJobCalcStateDataMigration(methodName='runTest')[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_common.TestMigrations

Test the migration that performs a data migration of legacy JobCalcState.

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_0038_data_migration_legacy_job_calculations'
migrate_from = '0037_attributes_extras_settings_json'
migrate_to = '0038_data_migration_legacy_job_calculations'
setUpBeforeMigration()[source]

Anything to do before running the migrations, which should be implemented in test subclasses.

test_data_migrated()[source]

Verify that the process_state, process_status and exit_status are set correctly.

Tests for the migrations of legacy process attributes.

class aiida.backends.djsite.db.subtests.migrations.test_migrations_0040_data_migration_legacy_process_attributes.TestLegacyProcessAttributeDataMigration(methodName='runTest')[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_common.TestMigrations

Test the migration that performs a data migration of legacy JobCalcState.

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_0040_data_migration_legacy_process_attributes'
migrate_from = '0039_reset_hash'
migrate_to = '0040_data_migration_legacy_process_attributes'
setUpBeforeMigration()[source]

Anything to do before running the migrations, which should be implemented in test subclasses.

test_data_migrated()[source]

Verify that the correct attributes are removed.

The basic functionality for the migration tests

class aiida.backends.djsite.db.subtests.migrations.test_migrations_common.TestMigrations(methodName='runTest')[source]

Bases: aiida.backends.testbase.AiidaTestCase

This is the common test class that is used by all migration tests. It migrates to a given migration point, allows you to set up the database & AiiDA at that point with the necessary data and migrates then to the final migration point. In the end it forwards the database at the final migration (as it should be and found before the migration tests).

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_common'
_revert_database_schema()[source]

Bring back the DB to the correct state.

property app
load_node(pk)[source]
migrate_from = None
migrate_to = None
setUp()[source]

Go to a specific schema version before running tests.

setUpBeforeMigration()[source]

Anything to do before running the migrations, which should be implemented in test subclasses.

tearDown()[source]

At the end make sure we go back to the latest schema version.

This file contains the majority of the migration tests that are too short to go to a separate file.

class aiida.backends.djsite.db.subtests.migrations.test_migrations_many.TestCalcAttributeKeysMigration(methodName='runTest')[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_many.TestMigrationsModelModifierV0025

This test class checks that the migration 0023_calc_job_option_attribute_keys works as expected which migrates CalcJobNode attributes for metadata options whose key changed.

KEY_ENVIRONMENT_VARIABLES_NEW = 'environment_variables'
KEY_ENVIRONMENT_VARIABLES_OLD = 'custom_environment_variables'
KEY_PARSER_NAME_NEW = 'parser_name'
KEY_PARSER_NAME_OLD = 'parser'
KEY_PROCESS_LABEL_NEW = 'process_label'
KEY_PROCESS_LABEL_OLD = '_process_label'
KEY_RESOURCES_NEW = 'resources'
KEY_RESOURCES_OLD = 'jobresource_params'
__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_many'
migrate_from = '0022_dbgroup_type_string_change_content'
migrate_to = '0023_calc_job_option_attribute_keys'
setUpBeforeMigration()[source]

Anything to do before running the migrations, which should be implemented in test subclasses.

test_attribute_key_changes()[source]

Verify that the keys are successfully changed of the affected attributes.

class aiida.backends.djsite.db.subtests.migrations.test_migrations_many.TestDataMoveWithinNodeMigration(methodName='runTest')[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_common.TestMigrations

Check that backward migrations work also for the DbLog migration(s).

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_many'
migrate_from = '0024_dblog_update'
migrate_to = '0025_move_data_within_node_module'
setUpBeforeMigration()[source]

Anything to do before running the migrations, which should be implemented in test subclasses.

test_data_type_string()[source]

Verify that type string of the Data node was successfully adapted.

class aiida.backends.djsite.db.subtests.migrations.test_migrations_many.TestDbLogMigrationBackward(methodName='runTest')[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_common.TestMigrations

Check that backward migrations work also for the DbLog migration(s).

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_many'
migrate_from = '0024_dblog_update'
migrate_to = '0023_calc_job_option_attribute_keys'
setUpBeforeMigration()[source]

Anything to do before running the migrations, which should be implemented in test subclasses.

test_objpk_objname()[source]

This test verifies that the objpk and objname have the right values after a forward and a backward migration.

class aiida.backends.djsite.db.subtests.migrations.test_migrations_many.TestDbLogMigrationRecordCleaning(methodName='runTest')[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_common.TestMigrations

This test class checks that the migration 0024_dblog_update works as expected. That migration updates of the DbLog table and adds uuids

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_many'
migrate_from = '0023_calc_job_option_attribute_keys'
migrate_to = '0024_dblog_update'
setUpBeforeMigration()[source]

Anything to do before running the migrations, which should be implemented in test subclasses.

tearDown()[source]

Cleaning the DbLog, DbUser, DbWorkflow and DbNode records

test_dblog_calculation_node()[source]

Verify that after the migration there is only two log records left and verify that they corresponds to the CalculationNodes.

test_dblog_correct_export_of_logs()[source]

Verify that export log methods for legacy workflows, unknown entities and log records that don’t correspond to nodes, work as expected

test_dblog_unique_uuids()[source]

Verify that the UUIDs of the log records are unique

test_metadata_correctness()[source]

Verify that the metadata of the remaining records don’t have an objpk and objmetadata values.

class aiida.backends.djsite.db.subtests.migrations.test_migrations_many.TestDuplicateNodeUuidMigration(methodName='runTest')[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_common.TestMigrations

Test the migration that verifies that there are no duplicate UUIDs

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_many'
migrate_from = '0013_django_1_8'
migrate_to = '0014_add_node_uuid_unique_constraint'
setUpBeforeMigration()[source]

Anything to do before running the migrations, which should be implemented in test subclasses.

test_deduplicated_uuids()[source]

Verify that after the migration, all expected nodes are still there with unique UUIDs.

class aiida.backends.djsite.db.subtests.migrations.test_migrations_many.TestGroupRenamingMigration(methodName='runTest')[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_common.TestMigrations

This test class checks the migration 0022_dbgroup_type_string_change_content which updates the type_string column of the groups.

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_many'
migrate_from = '0021_dbgroup_name_to_label_type_to_type_string'
migrate_to = '0022_dbgroup_type_string_change_content'
setUpBeforeMigration()[source]

Anything to do before running the migrations, which should be implemented in test subclasses.

test_group_string_update()[source]

Test that the type_string were updated correctly

class aiida.backends.djsite.db.subtests.migrations.test_migrations_many.TestMigrationsModelModifierV0025(methodName='runTest')[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_common.TestMigrations

Sub class of TestMigrations that need to work on node attributes using the ModelModifierV0025.

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_many'
get_attribute(node, key, default=None)[source]
static get_node_array(node, name)[source]
set_attribute(node, key, value)[source]
set_node_array(node, name, array)[source]

Store a new numpy array inside a node. Possibly overwrite the array if it already existed.

Internally, it stores a name.npy file in numpy format.

Parameters
  • name – The name of the array.

  • array – The numpy array to store.

class aiida.backends.djsite.db.subtests.migrations.test_migrations_many.TestNoMigrations(methodName='runTest')[source]

Bases: aiida.backends.testbase.AiidaTestCase

Verify that no django migrations remain.

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_many'
test_no_remaining_migrations()[source]

Verify that no django migrations remain. Equivalent to python manage.py makemigrations –check

class aiida.backends.djsite.db.subtests.migrations.test_migrations_many.TestNodePrefixRemovalMigration(methodName='runTest')[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_common.TestMigrations

This test class checks that the migration 0028_remove_node_prefix works as expected.

That is the final data migration for Nodes after aiida.orm.nodes reorganization was finalized to remove the node. prefix

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_many'
migrate_from = '0027_delete_trajectory_symbols_array'
migrate_to = '0028_remove_node_prefix'
setUpBeforeMigration()[source]

Anything to do before running the migrations, which should be implemented in test subclasses.

test_data_node_type_string()[source]

Verify that type string of the nodes was successfully adapted.

class aiida.backends.djsite.db.subtests.migrations.test_migrations_many.TestParameterDataToDictMigration(methodName='runTest')[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_common.TestMigrations

This test class checks that the migration 0029_rename_parameter_data_to_dict works as expected.

This is a data migration for the renaming of ParameterData to Dict.

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_many'
migrate_from = '0028_remove_node_prefix'
migrate_to = '0029_rename_parameter_data_to_dict'
setUpBeforeMigration()[source]

Anything to do before running the migrations, which should be implemented in test subclasses.

test_data_node_type_string()[source]

Verify that type string of the nodes was successfully adapted.

class aiida.backends.djsite.db.subtests.migrations.test_migrations_many.TestResetHash(methodName='runTest')[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_common.TestMigrations

This test class checks that only the hash extra is removed.

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_many'
migrate_from = '0038_data_migration_legacy_job_calculations'
migrate_to = '0039_reset_hash'
setUpBeforeMigration()[source]

Anything to do before running the migrations, which should be implemented in test subclasses.

test_data_migrated()[source]

Verify that type string of the nodes was successfully adapted.

class aiida.backends.djsite.db.subtests.migrations.test_migrations_many.TestTextFieldToJSONFieldMigration(methodName='runTest')[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_common.TestMigrations

This test class checks that the migration 0033_replace_text_field_with_json_field works as expected.

That migration replaces the use of text fields to store JSON data with builtin JSONFields.

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_many'
migrate_from = '0032_remove_legacy_workflows'
migrate_to = '0033_replace_text_field_with_json_field'
setUpBeforeMigration()[source]

Anything to do before running the migrations, which should be implemented in test subclasses.

test_text_field_to_json_field_migration()[source]

Verify that the values in the text fields were maintained after migrating the field to JSONField.

class aiida.backends.djsite.db.subtests.migrations.test_migrations_many.TestTrajectoryDataMigration(methodName='runTest')[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_many.TestMigrationsModelModifierV0025

This test class checks that the migrations 0026_trajectory_symbols_to_attribute and 0027_delete_trajectory_symbols_array work as expected. These are data migrations for TrajectoryData nodes where symbol lists are moved from repository array to attributes.

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_many'
cells = array([[[2., 0., 0.], [0., 2., 0.], [0., 0., 2.]], [[3., 0., 0.], [0., 3., 0.], [0., 0., 3.]]])
migrate_from = '0025_move_data_within_node_module'
migrate_to = '0027_delete_trajectory_symbols_array'
positions = array([[[0. , 0. , 0. ], [0.5, 0.5, 0.5], [1.5, 1.5, 1.5]], [[0. , 0. , 0. ], [0.5, 0.5, 0.5], [1.5, 1.5, 1.5]]])
setUpBeforeMigration()[source]

Anything to do before running the migrations, which should be implemented in test subclasses.

stepids = array([60, 70])
test_trajectory_symbols()[source]

Check that the trajectories are migrated correctly

times = array([0.6, 0.7])
velocities = array([[[ 0. , 0. , 0. ], [ 0. , 0. , 0. ], [ 0. , 0. , 0. ]], [[ 0.5, 0.5, 0.5], [ 0.5, 0.5, 0.5], [-0.5, -0.5, -0.5]]])
class aiida.backends.djsite.db.subtests.migrations.test_migrations_many.TestUuidMigration(methodName='runTest')[source]

Bases: aiida.backends.djsite.db.subtests.migrations.test_migrations_common.TestMigrations

This test class checks the migration 0018_django_1_11 which switches from the django_extensions UUID field to the native UUIDField of django 1.11. It also introduces unique constraints on all uuid columns (previously existed only on dbnode).

__module__ = 'aiida.backends.djsite.db.subtests.migrations.test_migrations_many'
migrate_from = '0017_drop_dbcalcstate'
migrate_to = '0018_django_1_11'
setUpBeforeMigration()[source]

Anything to do before running the migrations, which should be implemented in test subclasses.

test_uuid_untouched()[source]

Verify that Node uuids remain unchanged.