Toward a better meritocracy

ASQ is a meritocracy. We measure skill via testing and certifications, and those with high skill are recognized, acclaimed, and rewarded. In essence we, as part of ASQ, are gatekeepers. But how good are we at gatekeeping? Are we as good as we think? Can we do better?

Malcolm Gladwell is a well-known deep thinker. Recently he did a talk for Google's Zeitgeist in December 2022. I watched his talk and the points he made, and I want to see how his thoughts on gatekeeping might apply to ASQ. You can view his talk via this link:

Here is a summary of Gladwell’s criticism of gatekeeping and meritocracies, and my thoughts on how each point applies to ASQ:

1 Real world after-the-fact data does not support prior estimated success.

Have we (ASQ) established that after someone has tested and become certified their performance is better than before? I'm not talking about salary surveys. What people are paid changes due to many factors some of which may not be valid indicators of their actual contributions to organizational success.

Perhaps a review by peers of their contributions before and after their certification could provide at least a subjective measure of actual contribution. Hard to do, expensive, and anecdotal, but certainly something closer to measuring actual improvement due to the certification. And for those who object to subjective metrics, many of our most important metrics we rely upon are subjective. One example: how much does your mother love you?

2 Testing is not geared toward actual needed skills applied later.

We use multiple choice tests to determine if someone is to be certified. But what skills are most valuable? Let’s use communication as an example. Most of our communication is verbal, a lot is written, and all of it involves understanding by other people. Multiple choice testing is a measure of reading, remembering, and at best some individual mental integration. Do we test for true active communication within groups? In past years the testing for the Certified Manager of Quality/ Organizational Excellence (CMQ/OE) included a structured response where examinees were given a scenario and asked to write their own ideas of how to approach a solution. I maintain that even if this was hard to grade, it was a good way to determine the skill in written communication the examinee possessed.

We can record a person’s verbal presentation today. Could a video not help in assessing an examinee’s speaking skills? It’s just a thought.

3 Complex work is done best by a team so do not measure individual performance.

We should all know about W. Edwards Deming’s views on measuring only individual performance (the red bead experiment). As far as I know, no ASQ examination looks at how an examinee performs with a team. I believe the ASQ team award process has been discontinued, but even in its past incarnation I do not know how that process measured the actual success in Forming, Storming, Norming, and Performing (from American psychologist Bruce Tuckman in 1965). If anyone knows of a good test of this process, please let us all know about it.

4 There is no process for fixing bad gatekeeping.

If our measures of quality skills are weak, just how do we go about fixing them? A Linked-In post? How about volunteering for a committee at ASQ to change the testing? To be honest, some of the criticisms noted above are not new and have been raised in those committees. This has not resulted in significant improvement. As a matter of fact, the testing sems to be weaker now than in the past. (See CMQ/OE above). And why is this true?

5 Institutional inertia and entrenched power and position.

Gladwell’s final point is this. When something seems to work, and all profit by it, there is enormous inertia. We lean on old ways since they are known and comfortable. W. Edwards Deming was not accepted by most US executives as he did not follow the established leadership. He said things that most leaders rejected. Look at ASQ’s founding thought leaders and ask, how would they react to the lack of improvement in certifications?

Now I have been picking on ASQ, but we in quality should not think that we are so terrible. Indeed, Gladwell’s video talks about many other fields: doctors, teachers, and so on. These thoughts are not particular to any one field. It is my belief that these tendencies are part of human nature and transcend culture and time.

To end my rant, what should be done about this? And who will step up to do it?

3 Replies

@Douglas Wood I can appreciate this discussion you brought forward. The ASQ tests do measure technical knowledge and not soft skills. There are other organizations who have tried a combination approach but it is more expensive and hard to manage.

For example, Certified Quality Auditor is a multiple choice test. There are other certifying bodies offering a lead auditor certification which requires a multiple day standards course (costs thousands of dollars) and used to require an in-person witnessed audit where you had to pay for travel and expenses for an assessor. Very expensive and hard to maintain.

I think most organizations know that hiring someone with a certification ensures that someone has a basic BOK in the subject area, but doesn’t guarantee soft skills or organizational fit. You have to measure that through your interview process.

The process ASQ uses measures that the technical knowledge improved, so they are measuring the skills the test covers. That seems reasonable.

Points are well taken about what we test and do not test- gatekeeping. What to do indeed. A manual/course might teach concepts, principles and process/how to do something so we meet some kind of common threshold/body of knowledge to belong /get it. But I never met a manual that taught critical thinking and critical listening create... to develop powers of sense and to perceive see what is and and what is not and adapt. This takes curiosity, experience, trial and error, success and failure, and mentorship. Then there are the soft skills of interpersonal relationships. Not at all certain that all of this can be measured or tested.

With this in mind and just for fun. Once I picked up on an idea from an HRM specialist and used it to interview potential hires. It went like this. Please listen carefully. A sailor left his ship docked in port and went to the nearest seafood restaurant. He went in, sat down, and the waiter came over to present a menu. The sailor simply asked. Do you serve albatross soup? The waiter replied, yes we do .. one of our specialties. Bring me that, said the sailor. When it came, the sailor took one spoonful. Stopped and turned pale. Took a second taste. Paused. Put down the spoon, took out a pistol and shot himself in the head. The question is why? To solve, you get to ask questions to which I will only answer yes or no.

This tested ability to listen, to adapt to the unusual, to deal with stress, to think and question logically, and even to smile/cope nicely when you might think this interviewing person is weird - who does this in a job interview!

@Douglas Wood

Outstanding summary of the quandary. I am reminded of the quote attributed to Einstein, which goas along the lines of “not everything that counts can be counted, and not everything that can be counted counts.”

To me the challenge with testing somebody's capability is assessing the integration and application of the skill set - which written tests really cannot assess.