Quality audits routinely require an examination of the organization’s processes, standard operating procedures (SOPs), and systems that ensure it meets the requirements from regulators and applicable quality management system (QMS) standards. These include the procedures and actions an organization needs to take with regards to data and its analytics. Use of statistical methods is also indicated with qualifying phrases like “as appropriate.”
Since many crucial decisions about a product or service are based on data, errors and omissions in this area have a significant impact on the users and customers. This article illuminates the need for more rigor on auditing of data, its analytics, and application of statistical methods. The need is universal across industry or service sectors, and regardless of the type of audit – certification, surveillance, re-certification, supplier audits, etc.
Much remains to be done.
A well-established and reputed medical device firm had to recall surgical instruments due to out of spec hardness. Their testing lab indicated certain lots of instruments had higher hardness and these were already in the field. It was discovered as part of random checks during quality compliance audits the company conducted periodically.
This created a risk of instrument breakage during use and the potential for ensuing complications of retrieving the broken pieces. The instruments were procured from various suppliers on their approved supplier list.
An examination of the supplier-provided certificates of conformance (CoCs) from the suspect batches indicated the hardness values were within specs. Now the question was if this was a false alarm that triggered an unnecessary recall. Or was the recall justified, and did the firm really avoid a potentially reputation-damaging outcome?
The leadership rightfully wanted to know the truth, what remediation would be needed, and how such errors could be avoided in future. Was there an issue with manufacturing data and analytics? Was it due to manufacturing data analytics software?
A cross-functional team undertook the investigation by examining internal processes for evaluating the hardness of the instruments and recording the same. Some members of the team visited the related suppliers to study their practices for manufacturing, testing, and documenting results. Below is a summary of what they found:
The results befuddled many in the leadership team. If measuring instruments were in decent shape, why did we have systemic and endemic errors? And if such errors were present, how were they missed in prior quality audits? It was time to peel the onion some more!
It was imperative to ask tough questions about the auditing process itself and to take a closer look at:
Invariably, compliance audits (and other types of audits) use checklists and questions based on the applicable requirements to determine compliance. In this case, there were bare-bones questions pertaining to data, manufacturing data analytics, and statistical methods. Questions like “Do you have a process for collecting, analyzing, and processing data to create quality metrics?” required a Yes/No response. They are of little help and not atypical.
Do the standards and regulations provide enough information in this regard to create a solid quality compliance audit checklist? Let us examine their requirements.
Since the case study is about a medical device, controls defined in 21 CFR Part 820 and ISO 13485 apply.
21 CFR Part 820 requires, where appropriate, each manufacturer shall:
Various sections of ISO 13485 require documented plans and/or procedures for use of appropriate statistical techniques in:
When it comes to data, the verbiage often used is “appropriate” or “where appropriate” as qualifiers. Further explanation is provided which indicates the default is to be taken as appropriate unless the manufacturer can document justification otherwise, or non-implementation could result in product related issues. This allows some room for judgement and opinion, both on the part of the auditor and the auditee.
Equally if not more important than manufacturing data analytics is data validation and data integrity. Lack of checks and balances in these areas increase subsurface risks – akin to the proverbial iceberg – where 90% of the risk lies below the waterline.
Couple this with the prevalent situation where a vast majority of auditors do not even know enough to ask the right questions about data, and its analysis, and you have the perfect recipe for “very weak” compliance audit.
Is there a solution? YES!
When it comes to focusing on data and analytics, the intent of standards and regulations is right –although there is room for improvement. Since auditors are at the frontlines – examining firsthand how data gets used, processed, and presented – can we equip them with better tools (questions and techniques) on this topic? Sure! While a comprehensive treatment of this topic will require a daylong workshop and practice, here are five questions that every auditor can use, and the rationale for each.
So, there you have it – Five things every auditor should do regarding data analysis.
Copyrights to the Author – 2023 Rai Chowdhary, The KPI System
Enjoying this blog? Learn More.
Quality Audit – A Tool for Continuous Improvement and ComplianceGet the guide