Two years after the U.S. Food and Drug Administration (FDA) published its draft guidance on data integrity, the final guidance remains in the “forthcoming” stage. Meanwhile, regulatory guidelines for ensuring the completeness, consistency and accuracy of data still must be upheld. We are all well aware that there are numerous regulations surrounding data integrity, so I’ll forgo the trite references relating to corralling felines. Instead, I will suggest that a good approach to achieving data integrity compliance is to develop a practical data management strategy around data storage and migration.
Perhaps, we can agree that going digital with data management has been both beneficial and somewhat challenging. Certainly, the ability to automate data collection, storage and analysis, as well as immediately access any data in real time has been positive.
On the other hand, the various, continuously changing technologies for storing and migrating data have made maintaining data integrity a bit unwieldy. For example, as data storage infrastructures evolve, data stored for 10 years or longer may not be readable on newer versions of operating systems or applications. Still, data integrity guidelines require that every 1 and 0 of original data remain intact.
Data migration technologies also evolve. Therefore, keeping up your end of the data integrity compliance agreement means you routinely need to move the data you’re trying to keep safe and undisturbed, yet still accessible, to a new system.
Data integrity guidelines do not prescribe a specific method or technology for data storage. Still, two areas of focus for data storage compliance are integrity and retention.
Data Integrity – Data integrity is established where the data is stored and managed in its original form. The guidelines for maintaining the integrity of stored data include:
Data Retention – Data is retained for both short-term and long-term purposes. Retention dictates how long data must remain in storage, which is determined by the data lifecycle. Short-term retention is in the form of data backups and long-term storage refers to data that is archived. Data backup and data archive are often used interchangeably. However, they each have different roles in the data integrity scheme.
Data migration is the process of moving stored data and metadata from one storage system to another. Moving stored data is necessary due to storage technology becoming obsolete and no longer able to meet regulatory requirements. Also, data stored in one location can deteriorate over time. Unfortunately, data migration is not a risk-free endeavor. Some of the migration risks include:
Data loss – When data is migrated to a new system, some of the data may not move over from the source system.
Inconsistency – Even when data migration is done efficiently, the data in one column of the source database could appear in a different column in the new database.
Data corruption – Some of the data migrated from the source system may not be compatible with the new system software, which could result in errors or incomplete datasets.
Migration not performed in correct order – It is extremely important that data be migrated in the proper order as there are varied dependencies between the different processes. Skipping processes or performing them out of order could cause subsequent processes to fail.
System incompatibility – Some programs in the new system may not be compatible with the programs used to migrate data from the source system. This could lead to errors with the migrated data.
Risky or not, the need for data migration is unavoidable. However, many of the risks with migration can be avoided. A good way to mitigate the risks with data migration is to create a data migration strategy that includes testing and verification procedures that help maintain the integrity of data during and after migration.
A common challenge with data migration is companies have a large amount of data in storage. The first task of a migration strategy is to identify which data needs to be migrated. The following are suggestions for processes to include in your data migration strategy:
Data governance structure – When you have data moving from one location to another, it’s important to identify who has rights to access, edit or remove archived data. This information may be included in the metadata.
Understand the quality of existing data – Before you begin migrating data, spend some time assessing the quality of the data in the source system. Is the data complete? Does it comply with data integrity requirements? Will the data be readable on the new system?
Identify the data that needs to move – Not all archived data needs to be migrated. You can ease the complexity of data migration by clearly identifying and migrating only the data you need keep archived for data integrity compliance.
Protect data at rest and en route – To maintain data integrity and security, keep your data in read-only format throughout the retention timeframe and during migration.
Test migrated data – You can alleviate many data migration risks with data migration testing procedures. To validate the completeness, consistency and correctness of the migrated data, perform a validation test, which involves comparing the data of the source system and the new system using a predefined comparison criteria.
Verify data if format changes – It’s common for the data format to change during migration. A format change is OK. You just need to verify that the data itself remained the same and is still readable in the new data storage system.
When developing your data management strategy, it’s recommended that you devote significant attention to your data migration quality assurance and testing processes. It’s also important to fully document your data migration experience and outcome in the event you need tangible evidence to demonstrate data integrity compliance. Creating a well-structured plan as part of your strategy can be invaluable in your efforts to preserve the integrity of your data.