Quality System Effectiveness In Consulting Engineering


Featured Article From the QMD Management Elements and Methods CMC
By Marg Latham, P.Eng., CMC

Why Measure QMS Effectiveness


Marg Latham is a professional engineer and certified management consultant. She is President of Aqua Libra Consulting Ltd., a management consulting firm. Since starting her business in 2010, she has helped public and private engineering organizations improve professional practice and quality management. From 2000 to 2009, Marg was a Vice President with consulting firms, UMA and AECOM where she implemented management systems to improve client delivery. She spent the first 20 years of her career managing institutional, residential and infrastructure projects in Toronto, Winnipeg and Vancouver. In June 2019, she became Chair of the Board of the Canadian Centre for Women in Science, Engineering, Trades and Technology (WinSETT) and actively volunteers her time to increase the number of women in SETT careers.
Why is measuring the effectiveness of a Quality Management System (QMS) important? Developing and implementing a QMS takes time and money. Although improving quality by making this investment may seem like a ‘no-brainer’ to a quality professional, the executive team and particularly, the CFO will want to know the business case. Why invest in a QMS versus other strategic and seemingly equally important initiatives? And quite frankly, if implementing a QMS is not going to improve the bottom line, why invest limited resources in it? Purists may say that these types of discussions should not enter discussions about improving quality. However, without them, the implementation of a QMS may end up being a costly, bureaucratic exercise with no buy-in by users that does not produce better quality.

A frequent comment from those faced with the implementation of a new quality management system, especially one that is to be certified under ISO 9001, is that it will be too costly. It is true that an investment is needed to develop and introduce a QMS where none exists or to improve a QMS that is ineffective. However, that investment should more than pay for itself if made wisely.

What questions should your QMS business case answer:
  • Why should the organization implement a QMS? What objectives will it achieve for the organization? What are the risks of not managing the organization’s quality more effectively? How do these objectives align with the organization’s strategic plan and priorities?
  • What are the quality issues that implementing a QMS will help to resolve? What evidence exists to support that these issues are occurring? How much is poor quality costing the organization?
  • Why are these quality issues occurring? What practices and processes need to be developed and implemented or improved to prevent the issues from occurring?
  • What steps and resources are required? How much time and money will it take to develop or revamp and implement the QMS?
  • How will the organization know that the QMS is achieving the planned objectives?
  • What is the expected return on investment?
With the answers to these questions, you will have built a business case for why the organization should authorize and fund the development or improvement of a QMS, and a plan for developing, implementing and measuring the effectiveness of that QMS.

So, How to Begin

There is a quote, often attributed to Lewis Carroll in Alice in Wonderland, ‘if you don’t know where you are going, any road will get you there’. It is also true that, if you don’t know where you are now, it is hard to tell whether you are making progress. To evaluate the effectiveness of a Quality Management System (QMS), an organization needs to know what it wants to achieve by implementing its QMS and the current state of its quality. To know what you are trying to improve, first you need to know what is wrong with the current level of quality.

Calculating the cost of poor quality seemed like the place to start to influence the leadership team of an engineering company to develop and implement a company-wide quality management system back in the late 1990s. The bad quality issues we would want to improve seemed obvious – client complaints, rework, write-downs, claims, late delivery – all we needed was to know their current state. Unfortunately, like most financial management systems of the day, these quality issues were never the intended focus of these management systems. Finding or implementing ways to assess the current state and calculate the cost of poor quality turned out to be a challenge. The issues were:
  • No consistent way of collecting client feedback about our performance. In fact, for the most part, the firm did not ask clients how well the firm had delivered their project.
  • Rework seemed to be nearly impossible to track without adding more effort that it was worth. We had no idea how much rework was occurring and whether it was charged to clients or written off.
  • Write-downs occurred or were taken for numerous reasons. No one was focused on why a project did not achieve its planned profitability. Being more granular in knowing why write-downs occurred seemed to be an important piece of the puzzle, but the reasons were not determined or tracked.
  • Claims may come in months or years after a project is complete. They may be based on earlier practices that have been changed. These issues made it difficult to determine a dollar value or number of claims for a current year.
  • Completion dates and project schedules were tracked at a project level if at all.
Even though getting at the performance in these areas was difficult, totalling the combined costs of write-downs and claims for the current year was shocking. The amount was greater than the dollar amount of our bottom-line profit for the year. It indicated that it was worth investing time and money to identify the what, why and how to improve.

Seeking Feedback from Clients

To be a successful consulting engineering firm, or for that matter any professional services firm, you must be effectively delivering solutions that your clients value. How satisfied are your clients and how effectively are you delivering your services? If you do not know the answer to these questions, your quality management or lack of it may be costing your firm business and profitability. Lost opportunities and clients, damaged reputation, project write-downs, reduced profit margins or costly claims are all potential outcomes of not meeting your client’s requirements and expectations.

Start by finding out how your clients perceive your firm’s performance. Meet with clients. Ask open ended questions about how your organization is delivering the client’s projects. Use a standard set of questions about performance issues so that you get some consistent information. Find out how you are performing in areas that are important to your clients. Capture exact quotes. Be sure to ask your clients how they rate your firm compared with your competition and why. Review the information you capture from these interviews and look for common themes that identify areas to investigate or improve. Do your people take the time to understand what your clients require? Do they confirm that these requirements have been met before issuing deliverables to clients? Have your people listened and responded to clients in a timely manner? Was the project completed to meet their budget and schedule?
The information gathered will be invaluable in determining where to focus process improvement efforts to better satisfy clients.

What do the Trends Indicate and Why?

What else will tell you where to invest in better quality? The answer lies in areas where poor quality is currently having a negative impact on your profitability. Review project financial reports. Assess whether your projects are achieving planned profitability. Identify common themes such as sizes and types of projects that regularly fail to achieve planned profit or project managers with poor track records.

By the time you have completed your review of client and financial data, you will likely be seeing some common themes that flag areas in need of improvement. However, these trends at a macro level will not necessarily tell you what to change to improve your quality. You need to dig deeper. What are the root causes? Look at some projects that reflect the problems identified in client interviews and financial results and conduct a root cause analysis. Conducting this type of analysis on a sample of your problem projects will give you a good sense of where to start to improve your quality management.

I was asked by a transportation sector leader to help identify why his group was experiencing significant project write-downs. I worked with him to facilitate a series of root cause analyses on a sampling of problem projects. We found that most of the problems began with a poorly executed proposal process. Many of the proposals did not include a detailed task list and documented assumptions to identify out of scope work. Improving the proposal process had the potential to resolve over half of the problem projects. Clearly stating what the client’s the scope of work included, what was not included and any assumptions that could affect the outcome means less misunderstandings with clients and goes a long way to better satisfying them.

In another instance a review of claims and client complaints indicated a high incidence of design review issues. A root cause analysis of some showed two common issues. The teams were not confirming all design inputs and not reviewing input data to confirm that it was current, accurate, complete and adequate for the design. A simple process and tools dramatically decreased the incidence of related claims and complaints within its first two years of use. This compared with several claims with potentially high settlements for work completed in the two years prior to the process change.
Focusing on confirming and documenting input requirements and consistently carrying out quality reviews during the design process to confirm that these requirements were being met significantly reduced project write-downs and provided an immediate return on the investment in process improvement.

In another instance, concerns were raised about the ability of project managers to manage the firm’s scope of work on projects effectively. It would have been easy to blame the project managers. However, an assessment determined the need for some specific project management tools and training.

Putting effective quality management in place should not be costly nor should it handcuff competent professionals. Finding the root cause of costly poor quality will provide you with the focus for initial quality management controls and form the basis of assessing the effectiveness of your QMS. How much emphasis, effort and controls that you put in place in other areas will depend on the level of skill of your professionals and the risk to the firm.

Improving Practices and Changing Behaviour

Developing policies and procedures into a new or redeveloped QMS based on the identified root causes you are trying to address is a start. However, in and of itself, the new QMS will not suddenly result in different actions and behaviours. Publishing the QMS must be accompanied by:
  • A communication plan before and after the launch to let users know the why, what, when and how of this new endeavour. By the time the new or redeveloped QMS is launched, users should have a complete understanding of what to expect and why the change is important to the firm and to them.
  • A training plan to make sure users understand the new practices and know how to use the accompanying tools effectively.
  • A means for users to take ownership, provide feedback and suggest improvements.
  • Objectives, goals, measures and tracking to identify areas for improvement and provide the impetus for people to adopt the new practices.
These actions will help to make the QMS implementation successful. The last bullet will provide the basis to assess the effectiveness of the QMS. The business case that we spoke of developing early on in this article will provide the objectives and some guidance about how you were going to assess effectiveness.

Developing a Quality Scorecard

I like to create a scorecard that sets measurable goals for each objective, determines the current state for each goal and then tracks results quarterly or annually.

Goals and the metrics supporting them should be a balance of leading indicators that measure actions or behaviour changes and lagging indicators that measure the results.

As an example, in determining the current state of our quality, we noted that many projects proceeded without contracts that documented the scope of work and terms of the agreement. The lack of an agreement was leading to some of the quality issues we were identifying later in the project. Although the long-term change was to reduce those issues and improve quality, the immediate change in behaviour was to ensure that we had executed agreements in place before any work began. Regional controllers began to refuse to set up jobs in accounting until they received a copy of the signed agreement. Centrally we captured and tracked our progress that showed dramatic improvement over a few months.

Begin where you can have and see progress while not losing sight of the longer-term objectives.

In Conclusion

Measuring the effectiveness of a Quality Management System is fundamental to being able to achieve better quality, continually improve practices and ultimately, to achieve a return on any investment your organization makes in developing, improving and implementing a QMS. To create an effective means to measure starts when you begin developing your QMS and takes forethought about what you want to achieve. As the late Steven Covey states in the title on one of his many books, “Begin with the End in Mind”.
News Quality Management Division 04/12/2021 12:21pm CDT


No Data Available