Josh,
I'm flabbergasted by the finding! I assume there is a gage control system that would be available to confirm/refute that there are missing gages (and calibration records). I assume the auditor didn't do “due diligence” to confirm his suspicion.
Also, depending on the number gages in your system and quality of your gage calibration/tracking system, being concerned about 1 issue may constitute a weakness, but not a full-out failure of the system.
I faced a similar situation with an auditor on their own for the first time; They found 1 gage in a work area of 27 gages sampled to be late for calibration and declared a calibration system of 1000+ gages non-conforming in a ISO9001 audit (machine shop operation, non- aerospace), without looking at the control system, or sampling another cell in the plant.
The finding was withdrawn by the auditor's employer when we protested.