Measuring effectiveness of an internal audit program
I really want to ask people who have experience with internal auditing for GMP compliance.

What are some measures you have used to show effectiveness of an internal audit program?
In an external workshop on auditing, I have asked this question and the general average response was just schedule adherence and CAPA closures.

Are there other links to performance we can measure? Our team has thought about compliance to regulatory, no critical observations from regulatory audits, reduced overall number of regulatory observations. The quality mind set aspect is not measurable and collaboration between departments is also not measurable but is a very important aspect of internal auditing.

Any ideas or insights? I would also appreciate any reference material or articles I can read about on this topic. I did a search on the discussion board and came across some old posts with ASQ articles on improvement and performance. I will review those for some information.

Thank you in advance.
6 Replies
Shirley

For general measures on our process, yes we use the schedule adherence and CA closure rates as indicators.  I have also instituted a weighted scoring method for each of our audits, based on the checklist for each process (in excel).  It is fairly simple and we are looking to improve it to be more robust.  For each process we look at each step and we determine a weighting for each step, based on whether it is a major step, minor step, trivial, requires peer review and Configuration Management, whether there has been an issue in the past with compliance, etc.  Then as we audit, each step is rated as to its compliance (1), whether it is a major finding (0), minor (.5), OFI(1), negative observation(.75) or positive observation (1) .  The weighting for the step multiplied by the compliance score then produces a score for that step.  All the step scores are then added and divided by the total possible weight and we get a 1-10 compliance/effectiveness score for the process.  We monitor that score from audit to audit to show compliance improvement.  Our next step is to rate efficiency into the scoring.  Not sure how yet.
As an addendum,  think about how they score figure skating and gymnastics.  Each move is weighted and figures into the total score.  When the organization wants to see more or less of certain moves, they increase or decrease the weight in the scoring.
Thank you Jerry for your replies. I will ponder on your points and share the insights with my team members. I liked the analogy of figure skating! I understand what you mean!
I have used a scoring method for auditing. Basically each audit would have a set of standard questions (optional questions could be included) based on the process documented inputs, outputs and controls along with the ISO or industrial requirements. Each question for each audit is given a score : for example 1 = Does not comply (Audit finding)  2= Needs Improvement (observation) 3= Complies. The form can shows the scores for each audit and measure weather the process is getting better or worse. 


The scores can be  summed to provide an overall score and the area can can also be plotted.  Below screen shot for an example. 


Duke Okes 1670
101 Posts
How about something along the lines of:
- What is the potential magnitude of failures that have been prevented as a result of audit findings & corrective action
- What were the magnitudes of some failures that audits did not detect?

Both of these focus on the effectiveness of the audit process.
Be careful about regarding measures of effectiveness that are really just measures of implementation.  Two ways to really measure the effectiveness of an internal assessment program is to measure the incidence of repeat issues, and how well you are doing on external assessments.  Repeat incidents should go down, and external assessment ratings should go up.  The reason why you should focus on these two items? - the purpose of the internal assessment program is continuous improvement - if you find issues before they get big, drill to root cause and eliminate them, they shouldn't come up again.  Also, your internal program should be finding issues before an external auditor does - therefore the external audit evaluations should go up over time.