This migration cleans the log records from non-Node entity records. It removes from the DbLog table the legacy workflow records and records that correspond to an unknown entity and places them to corresponding files.
This migration corresponds to the 0024_dblog_update Django migration.
Revision ID: 041a79fc615f Revises: 7ca08c391c49 Create Date: 2018-12-28 15:53:14.596810
aiida.backends.sqlalchemy.migrations.versions.041a79fc615f_dblog_cleaning.
downgrade
Downgrade function to the previous schema.
export_and_clean_workflow_logs
Export the logs records that correspond to legacy workflows and to unknown entities (place them to files and remove them from the DbLog table).
get_legacy_workflow_log_number
Get the number of the log records that correspond to legacy workflows
get_logs_with_no_nodes_number
Get the number of the log records that correspond to nodes that were deleted
get_serialized_legacy_workflow_logs
Get the serialized log records that correspond to legacy workflows
get_serialized_logs_with_no_nodes
Get the serialized log records that correspond to nodes that were deleted
get_serialized_unknown_entity_logs
Get the serialized log records that correspond to unknown entities
get_unknown_entity_log_number
Get the number of the log records that correspond to unknown entities
upgrade
Changing the log table columns to use uuid to reference remote objects and log entries. Upgrade function.
Drop the transport_params from the Computer database model.
Revision ID: 07fac78e6209 Revises: de2eaf6978b4 Create Date: 2019-02-16 15:32:42.745450
aiida.backends.sqlalchemy.migrations.versions.07fac78e6209_drop_computer_transport_params.
Migrations for the downgrade.
Migrations for the upgrade.
Correct the type string for the base data types
Revision ID: 0aebbeab274d Revises: 7a6587e16f4c Create Date: 2018-02-24 20:12:44.731358
aiida.backends.sqlalchemy.migrations.versions.0aebbeab274d_base_data_plugin_type_string.
Migration to add the extras JSONB column to the DbGroup model.
Revision ID: 0edcdd5a30f0 Revises: bf591f31dd12 Create Date: 2019-04-03 14:38:50.585639
aiida.backends.sqlalchemy.migrations.versions.0edcdd5a30f0_dbgroup_extras.
Downgrade: Drop the extras column from the ‘db_dbgroup’ table
Upgrade: Add the extras column to the ‘db_dbgroup’ table
Update all link labels with the value _return which is the legacy default single link label.
The old process functions used to use _return as the default link label, however, since labels that start or end with and underscore are illegal because they are used for namespacing.
Revision ID: 118349c10896 Revises: 91b573400be5 Create Date: 2019-11-21 09:43:45.006053
aiida.backends.sqlalchemy.migrations.versions.118349c10896_default_link_label.
Move trajectory symbols from repository array to attribute
Revision ID: 12536798d4d3 Revises: 37f3d4882837 Create Date: 2019-01-21 10:15:02.451308
aiida.backends.sqlalchemy.migrations.versions.12536798d4d3_trajectory_symbols_to_attribute.
Migration to reflect the name change of the built in calculation entry points in the database.
Revision ID: 140c971ae0a3 Revises: 162b99bca4a2 Create Date: 2018-12-06 12:42:01.897037
aiida.backends.sqlalchemy.migrations.versions.140c971ae0a3_migrate_builtin_calculations.
Drop the DbCalcState table
Revision ID: 162b99bca4a2 Revises: a603da2cc809 Create Date: 2018-11-14 08:37:13.719646
aiida.backends.sqlalchemy.migrations.versions.162b99bca4a2_drop_dbcalcstate.
Drop the columns nodeversion and public from the DbNode model.
Revision ID: 1830c8430131 Revises: 1b8ed3425af9 Create Date: 2019-05-27 15:35:37.404644
aiida.backends.sqlalchemy.migrations.versions.1830c8430131_drop_node_columns_nodeversion_public.
Remove legacy workflows
Revision ID: 1b8ed3425af9 Revises: 3d6190594e19 Create Date: 2019-04-03 17:11:44.073582
aiida.backends.sqlalchemy.migrations.versions.1b8ed3425af9_remove_legacy_workflows.
export_workflow_data
Export existing legacy workflow data to a JSON file.
json_serializer
JSON serializer for objects not serializable by default json code
Migration after the provenance redesign
Revision ID: 239cea6d2452 Revises: 140c971ae0a3 Create Date: 2018-12-04 21:14:15.250247
aiida.backends.sqlalchemy.migrations.versions.239cea6d2452_provenance_redesign.
detect_unexpected_links
Scan the database for any links that are unexpected.
The checks will verify that there are no outgoing call or return links from calculation nodes and that if a workflow node has a create link, it has at least an accompanying return link to the same data node, or it has a call link to a calculation node that takes the created data node as input.
The downgrade migration actions.
migrate_infer_calculation_entry_point
Set the process type for calculation nodes by inferring it from their type string.
The upgrade migration actions.
Data migration for legacy JobCalculations.
These old nodes have already been migrated to the correct CalcJobNode type in a previous migration, but they can still contain a state attribute with a deprecated JobCalcState value and they are missing a value for the process_state, process_status, process_label and exit_status. The process_label is impossible to infer consistently in SQL so it will be omitted. The other will be mapped from the state attribute as follows:
Old state | Process state | Exit status | Process status ---------------------|----------------|-------------|---------------------------------------------------------- `NEW` | `Killed` | `None` | Legacy `JobCalculation` with state `NEW` `TOSUBMIT` | `Killed` | `None` | Legacy `JobCalculation` with state `TOSUBMIT` `SUBMITTING` | `Killed` | `None` | Legacy `JobCalculation` with state `SUBMITTING` `WITHSCHEDULER` | `Killed` | `None` | Legacy `JobCalculation` with state `WITHSCHEDULER` `COMPUTED` | `Killed` | `None` | Legacy `JobCalculation` with state `COMPUTED` `RETRIEVING` | `Killed` | `None` | Legacy `JobCalculation` with state `RETRIEVING` `PARSING` | `Killed` | `None` | Legacy `JobCalculation` with state `PARSING` `SUBMISSIONFAILED` | `Excepted` | `None` | Legacy `JobCalculation` with state `SUBMISSIONFAILED` `RETRIEVALFAILED` | `Excepted` | `None` | Legacy `JobCalculation` with state `RETRIEVALFAILED` `PARSINGFAILED` | `Excepted` | `None` | Legacy `JobCalculation` with state `PARSINGFAILED` `FAILED` | `Finished` | 2 | - `FINISHED` | `Finished` | 0 | - `IMPORTED` | - | - | -
Note the IMPORTED state was never actually stored in the state attribute, so we do not have to consider it. The old state attribute has to be removed after the data is migrated, because its value is no longer valid or useful.
Note: in addition to the three attributes mentioned in the table, all matched nodes will get Legacy JobCalculation as their process_label which is one of the default columns of verdi process list.
Revision ID: 26d561acd560 Revises: 07fac78e6209 Create Date: 2019-06-22 09:55:25.284168
aiida.backends.sqlalchemy.migrations.versions.26d561acd560_data_migration_legacy_job_calculations.
Migrating ‘hidden’ properties from DbAttribute to DbExtra for code.Code. nodes
Revision ID: 35d4ee9a1b0e Revises: 89176227b25 Create Date: 2018-02-21 22:00:43.460534
aiida.backends.sqlalchemy.migrations.versions.35d4ee9a1b0e_code_hidden_attr_to_extra.
This migration adds uniqueness constraint to the UUID column.
Revision ID: 375c2db70663 Revises: ea2f50e7f615 Create Date: 2019-01-30 20:26:16.550071
aiida.backends.sqlalchemy.migrations.versions.375c2db70663_dblog_uuid_uniqueness_constraint.
Remove unique key constraint to the UUID column.
Add unique key constraint to the UUID column.
Make all uuid columns unique
Revision ID: 37f3d4882837 Revises: 6a5c2ea1439d Create Date: 2018-11-17 17:18:58.691209
aiida.backends.sqlalchemy.migrations.versions.37f3d4882837_make_all_uuid_columns_unique.
verify_uuid_uniqueness
Check whether the database contains duplicate UUIDS.
Note that we have to redefine this method from aiida.manage.database.integrity.verify_uuid_uniqueness because that uses the default database connection, while here the one created by Alembic should be used instead.
IntegrityError if database contains nodes with duplicate UUIDS.
Remove DbComputer.enabled
Revision ID: 3d6190594e19 Revises: 5a49629f0d45 Create Date: 2019-04-03 14:38:50.585639
aiida.backends.sqlalchemy.migrations.versions.3d6190594e19_remove_dbcomputer_enabled.
Adding indexes and constraints to the dbnode-dbgroup relationship table
Revision ID: 59edaf8a8b79 Revises: a514d673c163 Create Date: 2018-06-22 14:50:18.447211
aiida.backends.sqlalchemy.migrations.versions.59edaf8a8b79_adding_indexes_and_constraints_to_the_.
Adding indices on the input_id, output_id and type column of the DbLink table
Revision ID: 5a49629f0d45 Revises: 5ddd24e52864 Create Date: 2019-03-04 16:38:42.249231
aiida.backends.sqlalchemy.migrations.versions.5a49629f0d45_dblink_indices.
Invalidating node hash - User should rehash nodes for caching
Revision ID: 5d4d844852b6 Revises: 62fe0d36de90 Create Date: 2018-10-26 17:14:33.566670
aiida.backends.sqlalchemy.migrations.versions.5d4d844852b6_invalidating_node_hash.
drop the hashes also when downgrading
drop the hashes when upgrading
Renaming DbNode.type to DbNode.node_type
Revision ID: 5ddd24e52864 Revises: d254fdfed416 Create Date: 2019-02-22 17:09:57.715114
aiida.backends.sqlalchemy.migrations.versions.5ddd24e52864_dbnode_type_to_dbnode_node_type.
Final data migration for Nodes after aiida.orm.nodes reorganization was finalized to remove the node. prefix
Revision ID: 61fc0913fae9 Revises: ce56d84bcc35 Create Date: 2019-02-16 15:32:42.745450
aiida.backends.sqlalchemy.migrations.versions.61fc0913fae9_remove_node_prefix.
Add a unique constraint on the UUID column of the Node model
Revision ID: 62fe0d36de90 Revises: 59edaf8a8b79 Create Date: 2018-07-02 17:50:42.929382
aiida.backends.sqlalchemy.migrations.versions.62fe0d36de90_add_node_uuid_unique_constraint.
verify_node_uuid_uniqueness
Check whether the database contains nodes with duplicate UUIDS.
Note that we have to redefine this method from aiida.manage.database.integrity.verify_node_uuid_uniqueness because that uses the default database connection, while here the one created by Alembic should be used instead.
Data migration for Data nodes after it was moved in the aiida.orm.node module changing the type string.
Revision ID: 6a5c2ea1439d Revises: 375c2db70663 Create Date: 2019-01-18 19:44:32.156083
aiida.backends.sqlalchemy.migrations.versions.6a5c2ea1439d_move_data_within_node_module.
Add the process_type column to DbNode
Revision ID: 6c629c886f84 Revises: 0aebbeab274d Create Date: 2018-03-15 13:23:12.941148
aiida.backends.sqlalchemy.migrations.versions.6c629c886f84_process_type.
Deleting dbpath table and triggers
Revision ID: 70c7d732f1b2 Revises: Create Date: 2017-10-17 10:30:23.327195
aiida.backends.sqlalchemy.migrations.versions.70c7d732f1b2_delete_dbpath.
Unique constraints for the db_dbgroup_dbnodes table
Revision ID: 7a6587e16f4c Revises: 35d4ee9a1b0e Create Date: 2019-02-11 19:25:11.744902
aiida.backends.sqlalchemy.migrations.versions.7a6587e16f4c_unique_constraints_for_the_db_dbgroup_.
Remove unique constraints from the db_dbgroup_dbnodes table.
Add unique constraints to the db_dbgroup_dbnodes table.
Seal any process nodes that have not yet been sealed but should.
This should have been accomplished by the last step in the previous migration, but because the WHERE clause was incorrect, not all nodes that should have been targeted were included. The problem is with the statement:
attributes->>’process_state’ NOT IN (‘created’, ‘running’, ‘waiting’)
The problem here is that this will yield False if the attribute process_state does not even exist. This will be the case for legacy calculations like InlineCalculation nodes. Their node type was already migrated in 0020 but most of them will be unsealed.
Revision ID: 7b38a9e783e7 Revises: e734dd5e50d7 Create Date: 2019-10-28 13:22:56.224234
aiida.backends.sqlalchemy.migrations.versions.7b38a9e783e7_seal_unsealed_processes.
Migration of ProcessNode attributes for metadata options whose key changed.
Revision ID: 7ca08c391c49 Revises: e72ad251bcdb Create Date: 2019-01-15 15:03:43.876133
aiida.backends.sqlalchemy.migrations.versions.7ca08c391c49_calc_job_option_attribute_keys.
Renamed attribute keys:
custom_environment_variables -> environment_variables (CalcJobNode) jobresource_params -> resources (CalcJobNode) _process_label -> process_label (ProcessNode) parser -> parser_name (CalcJobNode)
custom_environment_variables -> environment_variables (CalcJobNode)
jobresource_params -> resources (CalcJobNode)
_process_label -> process_label (ProcessNode)
parser -> parser_name (CalcJobNode)
linkname_retrieved (We do not actually delete it just in case some relies on it)
Add indexes to dbworkflowdata table
Revision ID: 89176227b25 Revises: Create Date: 2017-11-03 11:06:00.327195
aiida.backends.sqlalchemy.migrations.versions.89176227b25_add_indexes_to_dbworkflowdata_table.
Prepare schema reset.
Revision ID: 91b573400be5 Revises: 7b38a9e783e7 Create Date: 2019-07-25 14:58:39.866822
aiida.backends.sqlalchemy.migrations.versions.91b573400be5_prepare_schema_reset.
Drop the DbLock model
Revision ID: a514d673c163 Revises: f9a69de76a9a Create Date: 2018-05-10 19:08:51.780194
aiida.backends.sqlalchemy.migrations.versions.a514d673c163_drop_dblock.
Correct the type string for the code class
Revision ID: a603da2cc809 Revises: 5d4d844852b6 Create Date: 2018-11-13 18:15:07.300709
aiida.backends.sqlalchemy.migrations.versions.a603da2cc809_code_sub_class_of_data.
Updating link types - This is a copy of the Django migration script
Revision ID: a6048f0ffca8 Revises: Create Date: 2017-10-17 10:51:23.327195
aiida.backends.sqlalchemy.migrations.versions.a6048f0ffca8_update_linktypes.
DbGroup class: Rename name with label and type with type_string
Revision ID: b8b23ddefad4 Revises: 239cea6d2452 Create Date: 2018-12-06 15:25:32.865136
aiida.backends.sqlalchemy.migrations.versions.b8b23ddefad4_dbgroup_name_to_label_type_to_type_string.
Migration after the Group class became pluginnable and so the group type_string changed.
Revision ID: bf591f31dd12 Revises: 118349c10896 Create Date: 2020-03-31 10:00:52.609146
aiida.backends.sqlalchemy.migrations.versions.bf591f31dd12_dbgroup_type_string.
Delete trajectory symbols array from the repository and the reference in the attributes
Revision ID: ce56d84bcc35 Revises: 12536798d4d3 Create Date: 2019-01-21 15:35:07.280805
aiida.backends.sqlalchemy.migrations.versions.ce56d84bcc35_delete_trajectory_symbols_array.
Data migration for after ParameterData was renamed to Dict.
Revision ID: d254fdfed416 Revises: 61fc0913fae9 Create Date: 2019-02-25 19:29:11.753089
aiida.backends.sqlalchemy.migrations.versions.d254fdfed416_rename_parameter_data_to_dict.
Drop various columns from the DbUser model.
These columns were part of the default Django user model
Revision ID: de2eaf6978b4 Revises: 1830c8430131 Create Date: 2019-05-28 11:15:33.242602
aiida.backends.sqlalchemy.migrations.versions.de2eaf6978b4_simplify_user_model.
Initial schema
Revision ID: e15ef2630a1b Revises: Create Date: 2017-06-28 17:12:23.327195
aiida.backends.sqlalchemy.migrations.versions.e15ef2630a1b_initial_schema.
DbGroup class: change type_string values
Revision ID: e72ad251bcdb Revises: b8b23ddefad4 Create Date: 2018-12-06 19:34:47.732890
aiida.backends.sqlalchemy.migrations.versions.e72ad251bcdb_dbgroup_class_change_type_string_values.
Data migration for some legacy process attributes.
Attribute keys that are renamed:
_sealed -> sealed
Attribute keys that are removed entirely:
_finished _failed _aborted _do_abort
_finished
_failed
_aborted
_do_abort
Finally, after these first migrations, any remaining process nodes that still do not have a sealed attribute and have it set to True. Excluding the nodes that have a process_state attribute of one of the active states created, running` or waiting, because those are actual valid active processes that are not yet sealed.
Revision ID: e734dd5e50d7 Revises: e797afa09270 Create Date: 2019-07-04 18:23:56.127994
aiida.backends.sqlalchemy.migrations.versions.e734dd5e50d7_data_migration_legacy_process_attributes.
Revision ID: e797afa09270 Revises: 26d561acd560 Create Date: 2019-07-01 19:39:33.605457
aiida.backends.sqlalchemy.migrations.versions.e797afa09270_reset_hash.
drop_hashes
Drop hashes of nodes.
Print warning only if the DB actually contains nodes.
This migration creates UUID column and populates it with distinct UUIDs
Revision ID: ea2f50e7f615 Revises: 041a79fc615f Create Date: 2019-01-30 19:22:50.984380
aiida.backends.sqlalchemy.migrations.versions.ea2f50e7f615_dblog_create_uuid_column.
Remove the UUID column
set_new_uuid
Set new and distinct UUIDs to all the logs
Add an UUID column an populate it with unique UUIDs
Delete the kombu tables that were used by the old Celery based daemon and the obsolete related timestamps
Revision ID: f9a69de76a9a Revises: 6c629c886f84 Create Date: 2018-05-10 15:07:59.235950
aiida.backends.sqlalchemy.migrations.versions.f9a69de76a9a_delete_kombu_tables.