Module that contains the db migrations.
aiida.backends.djsite.db.migrations.
DeserializationException
Bases: aiida.common.exceptions.AiidaException
aiida.common.exceptions.AiidaException
__module__
ModelModifierV0025
Bases: object
object
This class implements the legacy EAV model used originally instead of JSONB.
The original Django backend implementation used a custom entity-attribute-value table for the attributes and extras of a node. The logic was implemented in this class which was removed when the native JSONB field was used. However, for the migrations this code is still needed, that is why it is kept here.
AIIDA_ATTRIBUTE_SEP
__dict__
__init__
Initialize self. See help(type(self)) for accurate signature.
__weakref__
list of weak references to the object (if defined)
_sep
_subspecifier_field_name
apps
create_value
Create a new list of attributes, without storing them, associated with the current key/value pair (and to the given subspecifier, e.g. the DbNode for DbAttributes and DbExtras).
No hits are done on the DB, in particular no check is done on the existence of the given nodes.
key – a string with the key to create (can contain the separator self._sep if this is a sub-attribute: indeed, this function calls itself recursively)
value – the value to store (a basic data type or a list or a dict)
subspecifier_value – must be None if this class has no subspecifier set (e.g., the DbSetting class). Must be the value of the subspecifier (e.g., the dbnode) for classes that define it (e.g. DbAttribute and DbExtra)
other_attribs – a dictionary of other parameters, to store only on the level-zero attribute (e.g. for description in DbSetting).
always a list of class instances; it is the user responsibility to store such entries (typically with a Django bulk_create() call).
del_value
Delete a value associated with the given key (if existing).
No exceptions are raised if no entry is found.
key – the key to delete. Can contain the separator self._sep if you want to delete a subkey.
only_children – if True, delete only children and not the entry itself.
del_value_for_node
Delete an attribute from the database for the given dbnode.
no exception is raised if no attribute with the given key is found in the DB.
dbnode – the dbnode for which you want to delete the key.
key – the key to delete.
get_value_for_node
Get an attribute from the database for the given dbnode.
the value stored in the Db table, correctly converted to the right type.
AttributeError – if no key is found for the given dbnode
getvalue
This can be called on a given row and will get the corresponding value, casting it correctly.
set_basic_data_attributes
Set obj attributes if they are of basic Python types.
set_value
Set a new value in the DB, possibly associated to the given subspecifier.
This method also stored directly in the DB.
key – a string with the key to create (must be a level-0 attribute, that is it cannot contain the separator cls._sep).
with_transaction – True if you want this function to be managed with transactions. Set to False if you already have a manual management of transactions in the block where you are calling this function (useful for speed improvements to avoid recursive transactions)
stop_if_existing – if True, it will stop with an UniquenessError exception if the new entry would violate an uniqueness constraint in the DB (same key, or same key+node, depending on the specific subclass). Otherwise, it will first delete the old value, if existent. The use with True is useful if you want to use a given attribute as a “locking” value, e.g. to avoid to perform an action twice on the same node. Note that, if you are using transactions, you may get the error only when the transaction is committed.
set_value_for_node
This is the raw-level method that accesses the DB. No checks are done to prevent the user from (re)setting a valid key. To be used only internally.
there may be some error on concurrent write; not checked in this unlucky case!
dbnode – the dbnode for which the attribute should be stored; in an integer is passed, this is used as the PK of the dbnode, without any further check (for speed reasons)
key – the key of the attribute to store; must be a level-zero attribute (i.e., no separators in the key)
value – the value of the attribute to store
with_transaction – if True (default), do this within a transaction, so that nothing gets stored if a subitem cannot be created. Otherwise, if this parameter is False, no transaction management is performed.
stop_if_existing – if True, it will stop with an UniquenessError exception if the key already exists for the given node. Otherwise, it will first delete the old value, if existent. The use with True is useful if you want to use a given attribute as a “locking” value, e.g. to avoid to perform an action twice on the same node. Note that, if you are using transactions, you may get the error only when the transaction is committed.
ValueError – if the key contains the separator symbol used internally to unpack dictionaries and lists (defined in cls._sep).
subspecifier_pk
Return the subspecifier PK in the database (or None, if no subspecifier should be used)
subspecifiers_dict
Return a dict to narrow down the query to only those matching also the subspecifier.
validate_key
Validate the key string to check if it is valid (e.g., if it does not contain the separator symbol.).
None if the key is valid
aiida.common.ValidationError – if the key is not valid
_deserialize_attribute
Deserialize a single attribute.
mainitem – the main item (either the attribute itself for base types (None, string, …) or the main item for lists and dicts. Must contain the ‘key’ key and also the following keys: datatype, tval, fval, ival, bval, dval. NOTE that a type check is not performed! tval is expected to be a string, dval a date, etc.
subitems – must be a dictionary of dictionaries. In the top-level dictionary, the key must be the key of the attribute, stripped of all prefixes (i.e., if the mainitem has key ‘a.b’ and we pass subitems ‘a.b.0’, ‘a.b.1’, ‘a.b.1.c’, their keys must be ‘0’, ‘1’, ‘1.c’). It must be None if the value is not iterable (int, str, float, …). It is an empty dictionary if there are no subitems.
sep – a string, the separator between subfields (to separate the name of a dictionary from the keys it contains, for instance)
original_class – if these elements come from a specific subclass of DbMultipleValueAttributeBaseClass, pass here the class (note: the class, not the instance!). This is used only in case the wrong number of elements is found in the raw data, to print a more meaningful message (if the class has a dbnode associated to it)
original_pk – if the elements come from a specific subclass of DbMultipleValueAttributeBaseClass that has a dbnode associated to it, pass here the PK integer. This is used only in case the wrong number of elements is found in the raw data, to print a more meaningful message
lesserrors – If set to True, in some cases where the content of the DB is not consistent but data is still recoverable, it will just log the message rather than raising an exception (e.g. if the number of elements of a dictionary is different from the number declared in the ival field).
the deserialized value
aiida.backends.djsite.db.migrations.DeserializationException – if an error occurs
_deserialize_basic_type
Deserialize the basic python data types.
_update_schema_version
The update schema uses the current models (and checks if the value is stored in EAV mode or JSONB) to avoid to use the DbSettings schema that may change (as it changed with the migration of the settings table to JSONB).
_upgrade_schema_generation
current_schema_version
Migrate the current schema version.
deserialize_attributes
Deserialize the attributes from the format internally stored in the DB to the actual format (dictionaries, lists, integers, …
data – must be a dictionary of dictionaries. In the top-level dictionary, the key must be the key of the attribute. The value must be a dictionary with the following keys: datatype, tval, fval, ival, bval, dval. Other keys are ignored. NOTE that a type check is not performed! tval is expected to be a string, dval a date, etc.
a dictionary, where for each entry the corresponding value is returned, deserialized back to lists, dictionaries, etc. Example: if data = {'a': {'datatype': "list", "ival": 2, ...}, 'a.0': {'datatype': "int", "ival": 2, ...}, 'a.1': {'datatype': "txt", "tval": "yy"}], it will return {"a": [2, "yy"]}
data = {'a': {'datatype': "list", "ival": 2, ...}, 'a.0': {'datatype': "int", "ival": 2, ...}, 'a.1': {'datatype': "txt", "tval": "yy"}]
{"a": [2, "yy"]}
deserialize_dict
Deserialize a Python dictionary.
deserialize_list
Deserialize a Python list.
upgrade_schema_version
Database migration.
aiida.backends.djsite.db.migrations.0001_initial.
Migration
Bases: django.db.migrations.migration.Migration
django.db.migrations.migration.Migration
dependencies
operations
aiida.backends.djsite.db.migrations.0002_db_state_change.
fix_calc_states
Fix calculation states.
aiida.backends.djsite.db.migrations.0003_add_link_type.
aiida.backends.djsite.db.migrations.0004_add_daemon_and_uuid_indices.
aiida.backends.djsite.db.migrations.0005_add_cmtime_indices.
aiida.backends.djsite.db.migrations.0006_delete_dbpath.
aiida.backends.djsite.db.migrations.0007_update_linktypes.
aiida.backends.djsite.db.migrations.0008_code_hidden_to_extra.
aiida.backends.djsite.db.migrations.0009_base_data_plugin_type_string.
aiida.backends.djsite.db.migrations.0010_process_type.
aiida.backends.djsite.db.migrations.0011_delete_kombu_tables.
aiida.backends.djsite.db.migrations.0012_drop_dblock.
aiida.backends.djsite.db.migrations.0013_django_1_8.
Add a uniqueness constraint to the uuid column of DbNode table.
aiida.backends.djsite.db.migrations.0014_add_node_uuid_unique_constraint.
reverse_code
verify_node_uuid_uniqueness
Check whether the database contains nodes with duplicate UUIDS.
Note that we have to redefine this method from aiida.manage.database.integrity.verify_node_uuid_uniqueness because the migrations.RunPython command that will invoke this function, will pass two arguments and therefore this wrapper needs to have a different function signature.
IntegrityError if database contains nodes with duplicate UUIDS.
Invalidating node hash - User should rehash nodes for caching.
aiida.backends.djsite.db.migrations.0015_invalidating_node_hash.
Invalidating node hash - User should rehash nodes for caching
aiida.backends.djsite.db.migrations.0016_code_sub_class_of_data.
aiida.backends.djsite.db.migrations.0017_drop_dbcalcstate.
Migration for upgrade to django 1.11
aiida.backends.djsite.db.migrations.0018_django_1_11.
This migration switches from the django_extensions UUID field to the native UUIDField of django 1.11
It also introduces unique constraints on all uuid columns (previously existed only on dbnode).
_verify_uuid_uniqueness
Check whether the respective tables contain rows with duplicate UUIDS.
Note that we have to redefine this method from aiida.manage.database.integrity because the migrations.RunPython command that will invoke this function, will pass two arguments and therefore this wrapper needs to have a different function signature.
IntegrityError if database contains rows with duplicate UUIDS.
Migration to reflect the name change of the built in calculation entry points in the database.
aiida.backends.djsite.db.migrations.0019_migrate_builtin_calculations.
Migration to remove entry point groups from process type strings and prefix unknown types with a marker.
Migration after the provenance redesign
aiida.backends.djsite.db.migrations.0020_provenance_redesign.
Migration to effectuate changes introduced by the provenance redesign
This includes in order:
Rename the type column of process nodes Remove illegal links Rename link types
Rename the type column of process nodes
Remove illegal links
Rename link types
The exact reverse operation is not possible because the renaming of the type string of JobCalculation nodes is done in a lossy way. Originally this type string contained the exact sub class of the JobCalculation but in the migration this is changed to always be node.process.calculation.calcjob.CalcJobNode.. In the reverse operation, this can then only be reset to calculation.job.JobCalculation. but the information on the exact sub class is lost.
detect_unexpected_links
Scan the database for any links that are unexpected.
The checks will verify that there are no outgoing call or return links from calculation nodes and that if a workflow node has a create link, it has at least an accompanying return link to the same data node, or it has a call link to a calculation node that takes the created data node as input.
migrate_infer_calculation_entry_point
Set the process type for calculation nodes by inferring it from their type string.
Reversing the inference of the process type is not possible and not necessary.
Migration that renames name and type columns to label and type_string
aiida.backends.djsite.db.migrations.0021_dbgroup_name_to_label_type_to_type_string.
Migration after the update of group_types
aiida.backends.djsite.db.migrations.0022_dbgroup_type_string_change_content.
Migration of ProcessNode attributes for metadata options whose key changed.
aiida.backends.djsite.db.migrations.0023_calc_job_option_attribute_keys.
Renamed attribute keys:
custom_environment_variables -> environment_variables (CalcJobNode) jobresource_params -> resources (CalcJobNode) _process_label -> process_label (ProcessNode) parser -> parser_name (CalcJobNode)
custom_environment_variables -> environment_variables (CalcJobNode)
jobresource_params -> resources (CalcJobNode)
_process_label -> process_label (ProcessNode)
parser -> parser_name (CalcJobNode)
linkname_retrieved (We do not actually delete it just in case some relies on it)
Migration for the update of the DbLog table. Addition of uuids
aiida.backends.djsite.db.migrations.0024_dblog_update.
This migration updates the DbLog schema and adds UUID for correct export of the DbLog entries. More specifically, it adds UUIDS, it exports to files the not needed log entries (that correspond to legacy workflows and unknown entities), it creates a foreign key to the dbnode table, it transfers there the objpk data to the new dbnode column (just altering the objpk column and making it a foreign key when containing data, raised problems) and in the end objpk and objname columns are removed.
clean_dblog_metadata
Remove objpk and objname from the DbLog table metadata.
enrich_dblog_metadata
Add objpk and objname to the DbLog table metadata.
export_and_clean_workflow_logs
Export the logs records that correspond to legacy workflows and to unknown entities.
get_legacy_workflow_log_number
Get the number of the log records that correspond to legacy workflows
get_logs_with_no_nodes_number
Get the number of the log records that don’t correspond to a node
get_serialized_legacy_workflow_logs
Get the serialized log records that correspond to legacy workflows
get_serialized_logs_with_no_nodes
Get the serialized log records that don’t correspond to a node
get_serialized_unknown_entity_logs
Get the serialized log records that correspond to unknown entities
get_unknown_entity_log_number
Get the number of the log records that correspond to unknown entities
set_new_uuid
Set new UUIDs for all logs
Data migration for Data nodes after it was moved in the aiida.orm.node module changing the type string.
aiida.backends.djsite.db.migrations.0025_move_data_within_node_module.
Data migration for TrajectoryData nodes where symbol lists are moved from repository array to attribute.
This process has to be done in two separate consecutive migrations to prevent data loss in between.
aiida.backends.djsite.db.migrations.0026_trajectory_symbols_to_attribute.
Storing symbols in TrajectoryData nodes as attributes, while keeping numpy arrays. TrajectoryData symbols arrays are deleted in the next migration. We split the migration into two because every migration is wrapped in an atomic transaction and we want to avoid to delete the data while it is written in the database
create_trajectory_symbols_attribute
Create the symbols attribute from the repository array for all TrajectoryData nodes.
delete_trajectory_symbols_attribute
Delete the symbols attribute for all TrajectoryData nodes.
aiida.backends.djsite.db.migrations.0027_delete_trajectory_symbols_array.
Deleting duplicated information stored in TrajectoryData symbols numpy arrays
create_trajectory_symbols_array
Create the symbols array for all TrajectoryData nodes.
delete_trajectory_symbols_array
Delete the symbols array from all TrajectoryData nodes.
Final data migration for Nodes after aiida.orm.nodes reorganization was finalized to remove the node. prefix
aiida.backends.djsite.db.migrations.0028_remove_node_prefix.
Now all node sub classes live in aiida.orm.nodes so now the node. prefix can be removed.
Data migration for after ParameterData was renamed to Dict.
aiida.backends.djsite.db.migrations.0029_rename_parameter_data_to_dict.
Renaming DbNode.type to DbNode.node_type
aiida.backends.djsite.db.migrations.0030_dbnode_type_to_dbnode_node_type.
Remove DbComputer.enabled
aiida.backends.djsite.db.migrations.0031_remove_dbcomputer_enabled.
Remove legacy workflow.
aiida.backends.djsite.db.migrations.0032_remove_legacy_workflows.
export_workflow_data
Export existing legacy workflow data to a JSON file.
Replace use of text fields to store JSON data with builtin JSONField.
aiida.backends.djsite.db.migrations.0033_replace_text_field_with_json_field.
Drop the columns nodeversion and public from the DbNode model.
aiida.backends.djsite.db.migrations.0034_drop_node_columns_nodeversion_public.
Simplify the DbUser model.
aiida.backends.djsite.db.migrations.0035_simplify_user_model.
Simplify the DbUser model by dropping unused columns.
Drop the transport_params from the Computer database model.
aiida.backends.djsite.db.migrations.0036_drop_computer_transport_params.
Adding JSONB field for Node.attributes and Node.Extras
aiida.backends.djsite.db.migrations.0037_attributes_extras_settings_json.
This migration changes Django backend to support the JSONB fields. It is a schema migration that removes the DbAttribute and DbExtra tables and their reference to the DbNode tables and adds the corresponding JSONB columns to the DbNode table. It is also a data migration that transforms and adds the data of the DbAttribute and DbExtra tables to the JSONB columns to the DbNode table.
attributes_to_dict
Transform the attributes of a node into a dictionary. It assumes the key are ordered alphabetically, and that they all belong to the same node.
lazy_bulk_fetch
select_from_key
Return element of the dict to do the insertion on. If it is foo.1.bar, it will return d[“foo”][1]. If it is only foo, it will return d directly.
transition_attributes_extras
Migrate the DbAttribute & the DbExtras tables into the attributes and extras columns of DbNode.
transition_settings
Migrate the DbSetting EAV val into the JSONB val column of the same table.
Data migration for legacy JobCalculations.
These old nodes have already been migrated to the correct CalcJobNode type in a previous migration, but they can still contain a state attribute with a deprecated JobCalcState value and they are missing a value for the process_state, process_status, process_label and exit_status. The process_label is impossible to infer consistently in SQL so it will be omitted. The other will be mapped from the state attribute as follows:
Old state | Process state | Exit status | Process status ---------------------|----------------|-------------|---------------------------------------------------------- `NEW` | `killed` | `None` | Legacy `JobCalculation` with state `NEW` `TOSUBMIT` | `killed` | `None` | Legacy `JobCalculation` with state `TOSUBMIT` `SUBMITTING` | `killed` | `None` | Legacy `JobCalculation` with state `SUBMITTING` `WITHSCHEDULER` | `killed` | `None` | Legacy `JobCalculation` with state `WITHSCHEDULER` `COMPUTED` | `killed` | `None` | Legacy `JobCalculation` with state `COMPUTED` `RETRIEVING` | `killed` | `None` | Legacy `JobCalculation` with state `RETRIEVING` `PARSING` | `killed` | `None` | Legacy `JobCalculation` with state `PARSING` `SUBMISSIONFAILED` | `excepted` | `None` | Legacy `JobCalculation` with state `SUBMISSIONFAILED` `RETRIEVALFAILED` | `excepted` | `None` | Legacy `JobCalculation` with state `RETRIEVALFAILED` `PARSINGFAILED` | `excepted` | `None` | Legacy `JobCalculation` with state `PARSINGFAILED` `FAILED` | `finished` | 2 | - `FINISHED` | `finished` | 0 | - `IMPORTED` | - | - | -
Note the IMPORTED state was never actually stored in the state attribute, so we do not have to consider it. The old state attribute has to be removed after the data is migrated, because its value is no longer valid or useful.
Note: in addition to the three attributes mentioned in the table, all matched nodes will get Legacy JobCalculation as their process_label which is one of the default columns of verdi process list.
aiida.backends.djsite.db.migrations.0038_data_migration_legacy_job_calculations.
aiida.backends.djsite.db.migrations.0039_reset_hash.
notify_user
Data migration for some legacy process attributes.
Attribute keys that are renamed:
_sealed -> sealed
Attribute keys that are removed entirely:
_finished _failed _aborted _do_abort
_finished
_failed
_aborted
_do_abort
Finally, after these first migrations, any remaining process nodes that still do not have a sealed attribute, have it set to True. Excluding the nodes that have a process_state attribute of one of the active states: created; running; or waiting, because those are valid active processes that are not yet sealed.
aiida.backends.djsite.db.migrations.0040_data_migration_legacy_process_attributes.
Data migration for legacy process attributes.
Seal any process nodes that have not yet been sealed but should.
This should have been accomplished by the last step in the previous migration, but because the WHERE clause was incorrect, not all nodes that should have been targeted were included. The problem is with the statement:
attributes->>’process_state’ NOT IN (‘created’, ‘running’, ‘waiting’)
The problem here is that this will yield False if the attribute process_state does not even exist. This will be the case for legacy calculations like InlineCalculation nodes. Their node type was already migrated in 0020 but most of them will be unsealed.
aiida.backends.djsite.db.migrations.0041_seal_unsealed_processes.
Prepare the schema reset.
aiida.backends.djsite.db.migrations.0042_prepare_schema_reset.
Update all link labels with the value _return which is the legacy default single link label.
The old process functions used to use _return as the default link label, however, since labels that start or end with and underscore are illegal because they are used for namespacing.
aiida.backends.djsite.db.migrations.0043_default_link_label.
Migrate.
Migration after the Group class became pluginnable and so the group type_string changed.
aiida.backends.djsite.db.migrations.0044_dbgroup_type_string.
Migration after the update of group type_string
Migration to add the extras JSONB column to the DbGroup model.
aiida.backends.djsite.db.migrations.0045_dbgroup_extras.
Migrate to add the extras column to the dbgroup table.