Measurement fundamentalsReading time 13 min

A gauge that reads to 0.001 does not mean you can measure to 0.001

“The tolerance is ±0.01, are our calipers good enough?” The answer is usually no, and the reason is not the resolution.

01Four words that get used interchangeably

Term	Definition	In plain language
Resolution	The smallest increment the instrument can display	How much one step on the screen is worth
Accuracy	How close a reading is to the true value	Is it systematically off
Precision / repeatability	The spread of repeated readings under the same conditions	Will ten readings agree
Uncertainty	A quantified statement of the interval you can trust	How far can you believe this number

Resolution is the best-looking figure on a datasheet, and the least meaningful.

A digital caliper reading to 0.01 mm may have a measurement uncertainty above 0.03 mm once measuring force, Abbe error, temperature and the operator's feel are included. None of those error sources appears on the display.

The Abbe principle

Abbe's principle says the measuring axis should lie in line with the length being measured. A caliper breaks it: the scale sits below while the jaws reach out, so any tilt of the jaws is amplified into error. That is why a micrometer, which obeys the principle, is far more accurate than a caliper of the same resolution.

02GR&R: the variation the measurement system adds

Gauge repeatability and reproducibility splits measurement variation in two. Repeatability is the spread when one person measures one part with one instrument. Reproducibility is the difference between operators measuring the same part.

%GR&R	Verdict	Action
< 10%	Acceptable	Fit for process control
10% ～ 30%	Conditionally acceptable	Decide on the importance of the application and the cost of improvement
> 30%	Not acceptable	The measurement system must be improved

AIAG MSA acceptance thresholds. %GR&R is measurement variation as a proportion of the tolerance band or of total variation.

03Where each instrument belongs

Instrument	Typical resolution	Typical uncertainty	Typical use
Vernier caliper	0.02 / 0.01 mm	±0.03 mm	Quick checks on the floor, coarse tolerances
External micrometer	0.001 mm	±0.004 mm	Outside diameters, plate thickness, IT7 and looser
Lever dial indicator	0.001 mm	±0.003 mm (comparative)	Runout, parallelism, setting up
Gauge blocks (grade 0)	—	±0.1 μm class	Calibration reference
Optical comparator / vision system	0.001 mm	±0.002 mm	Two-dimensional profiles, many features at once
Coordinate measuring machine	0.0001 mm	MPE varies by model	Three-dimensional features, geometric tolerances
Roundness tester	0.01 μm	Sub-micron	Roundness, cylindricity

The figures are typical ranges; the real values depend on model, range and environment.

Contact and non-contact

Contact methods are unaffected by surface colour or gloss and have mature traceability, but every point has to be touched, complex profiles are slow, and measuring force deflects thin or soft parts.
Vision systems capture hundreds of dimensions at once and suit sheet and stamped parts, but they cannot see height, and edge detection depends on surface condition and lighting.
Laser and confocal reach into deep holes and cope with high-contrast surfaces, but distort on steep slopes and transparent materials.

04Temperature: the variable that gets forgotten

The international reference temperature for measurement is 20 °C. Steel expands about 11.7 μm/(m·K) and aluminium about 23 μm/(m·K). A 500 mm aluminium part warming from 20 °C to 28 °C grows by roughly 92 μm, far more than most fit tolerances.

Materials	Expansion coefficient (μm/m·K)	Change on a 500 mm part, 8 °C rise
Carbon steel	11.7	47 μm
Stainless 304	17.3	69 μm
Aluminium alloys	23.1	92 μm
Brass	19.0	76 μm
Cast iron	10.5	42 μm

Which is why precision measurement needs a temperature-controlled room, and why the part must be allowed to reach room temperature before it is measured.

05How to choose what to measure with

Start with the feature. Two-dimensional profile or three-dimensional form? Are there geometric tolerances that need datums?
Then look at the tolerance band. Aim for measurement variation below one tenth of the band.
Then throughput. Sampling or 100% inspection? How many a day? What is the takt?
Finally traceability. Does the customer want an inspection report, or a calibration certificate?

The answer usually converges on its own: three-dimensional, tight tolerance, sampled → CMM; two-dimensional, high volume, 100% → vision; a quick check on the floor → calipers and micrometers will always have their place.