|Follow the audit trail one step at a time.
One of the most effective audits I ever participated in started with a systems-naive auditor asking a simple question: "Imagine I'm a sample arriving at your loading dock. What happens to me?" We proceeded down the trail together, following the sample through the processes of accessioning, analysis, reporting, and storage.
While you can start at any point in the process, if you're new to this, I recommend starting at the beginning. Figure out what should be happening and then find out what actually happens.
It is often most effective to start with what should be happening, mapping out the process and what's happening to the data. Remember that the most important "should" is the predicate rule. Be sure you understand the predicate rule requirements for the process and the data.
What should the process be?
SOPs describe management's intent for the
process. Start by reviewing SOPs and
sketching a trail map (in technical language,
make a process map). You don't need to
be proficient with flow charting -
you can do this with paper and pencil. The main things you want to identify are:
- What are the inputs?
- Who does what?
- What are the outputs?
- What is
true North? The
What should happen to the data?
As you map the SOPs, note:
- When data are created, modified, or deleted
- Who is authorized to perform these actions
Watch the trail for warning signs
What are some of the warning signs you might
Here's an example: During an audit of a data management department, you find instructions in an SOP for performing hard deletes of data entered into an EDC system by clinical investigator site staff. A hard delete often means that nothing about the data is left behind in the database. What do the predicate rules say? 21 CFR Part 312.62(b) requires the investigator to "prepare and maintain adequate and accurate case histories that record all observations." ICH E-6 Section 5.5.3(c) requires there be "no deletion of entered data." So when you see instructions for performing hard deletes, that's a warning sign.
Prepare some questions. Who can perform hard deletes? When would they perform a hard delete? Is there an audit trail of the delete, including former values, time, date, and user? How would the investigator know data had been deleted? How are hard deletes documented in the system requirements, specifications, and testing?
Time for a dose of reality!
It's not enough to examine what should be happening in an audit. You have to do a reality check, and that means interviews and observations. Even if you're doing a qualification audit, the provider should be able to give you a demonstration of their processes and systems.
What is the process?
Take a tour (not the warm, fuzzy 15 minute facility tour). This is your opportunity to engage with people responsible for executing the process and ask them to demonstrate how things really work. Have your map in hand and share it with the auditee. Ask for clarifications and take notes.
What is happening to the data?
During the tour, request user demonstrations of creating, modifying, and deleting data. Record the details: who did it and when? Ask for screen shots and reports from the system so you can verify what you saw matches what the system recorded. After the tour, cross check what you saw with the process requirements and the system requirements, specifications, and testing.
As your confidence grows, you can go deeper and deeper into areas like user roles, data review, and configuration settings.
It is not uncommon for reality to differ from the "shoulds." Sometimes users will have developed undocumented work-arounds for system or process limitations. Believe it or not, they're often just waiting for someone like you to come along and give them some traction to get the limitation fixed.
What are the Benefits?
In the 4 KISS Principles of Computer Validation blog post, I told you spending all your audit time up to your necks in validation binders was a bad thing. The good things about following the trail are
1. You will identify problems
I didn't say risks, did I?
A risk is the possibility that something bad will happen.
A problem is a risk that has materialized.
Your management needs and wants
to know about risks and the detective
and preventive controls that are
in place and that are missing. However,
it's even more important for them
to know what the problems are so
they can assure the appropriate corrective
and preventive action plans are put
2. Your audit team will be more effective
You company probably uses audit teams for provider and internal process audits. Audit teams often include auditors who are highly skilled in GLP and GCP predicate rule requirements and auditors who have systems compliance expertise.
You can each sit at your own little corner of the table and do your own things. Or, you can follow the trail together. Working as a team, your individual strengths will combine to make for a highly effective audit.
Audits should provide information for decisions. If you follow the trail, you will reach this destination.
Jamie Colgin is the President of Colgin Consulting, Inc., a consulting firm that helps pharmaceutical, biopharmaceutical, and contract research organizations assure the integrity of their preclinical and clinical data through integrated computer system compliance audits, mock inspections, remediation assistance, and on-the-job training.
With over 20 years of hands-on experience in statistics, computer system validation, audits, and monitoring, Jamie has developed validated SAS programs to support GLP studies, managed the retrospective evaluation of hundreds of GLP and GCP systems, written policies and procedures, and set new standards for communicating audit findings by using process flow diagrams.
Recipient of the prestigious Charles H. Butler Excellence in Teaching Award, Jamie can provide on-the-job training for your audit staff focused on updating skill sets and understanding regulatory trends.Prior to working in the pharmaceutical industry, Jamie proudly served with the US Peace Corps in the Philippines. She credits this experience for her resourceful nature.