Skip to content
PreciSim
physics

Ground truth and tolerances

What "truth" means here, how error is computed, where tolerances come from — and why a small residual is not an accurate calibration.

What truth is

On the virtual machine, truth is the set of parameters used to generate the scene, not an approximation recovered from it. The starred fields in the machine description are the truth:

Chain Truth field Value on demo-xy-galvo
(1) Pixel size cameras[].pixelSizeUm 1.7236 µm/px
(1) Mounting angle cameras[].rotationDeg −0.80°
(2) Camera-actuator offset scanHeads[].offsetXMm/offsetYMm 0.412 / 55.180 mm
(3) Lens distortion cameras[].k1/k2 0 / 0 (use the test-distortion machine to practise)
(4) Hand-eye the 2x2 Jacobian formed by pixel size and mounting angle —
(5) Galvo field scanHeads[].scaleX/scaleY/k1/skew 1.0012 / 0.9993 / 1.8e−5 / 3.0e−4
(6) Laser focus Z scanHeads[].focusZMm 112.500 mm
(7) Camera focus Z workingDistanceMm + zMountOffsetMm 110 + 2.347 = 112.347 mm

So truth scoring is not "solving it again with another algorithm." It reconciles your result against the generator's input. That is the one decisive advantage a virtual machine has over real hardware.

On real hardware there is no truth, TruthCompares is empty, and the criterion falls back to residuals. In the code these are the same path, not two implementations — so the way of reading results you practise on the virtual machine transfers directly.

Residual is not error

The single most useful sentence in this documentation.

Definition What it tells you
Residual the model's fit error over the data used in the solve how self-consistent model and data are
Error the difference between your result and the truth whether the calibration is accurate

A small residual only says "your model explains the data you collected". If the data itself is systematically flawed — distortion uncalibrated, points crowded in the centre of the field, span too small — the residual will be small and the error large.

The easiest way to produce this: shrink the sampling grid and run chain (1). The residual RMS barely moves while the error multiplies.

Three residual statistics, all of them

One residual number cannot be interpreted, so every chain reports three (ResidualStats):

Statistic What it tells you Compensatable?
Raw maximum how bad the worst point is —
De-translated what remains after removing the overall translation a translation can be absorbed by a station offset
Leave-one-out recomputed with each point held out; predictive power detects overfitting

A constant translation and a random scatter are indistinguishable inside one RMS, yet the remedies are opposite: the first is one offset, the second means changing the process or the hardware.

Where tolerances come from

All of them live in CalibTolerance in core/src/PreciSim.Core/Constants.cs — never as magic numbers in the chains:

Chain Tolerance
(1) Pixel size 0.002 µm/px
(1) Mounting angle 0.02°
(2) Camera-actuator offset 0.01 mm
(3) Lens distortion 0.3 px RMS residual
(4) Hand-eye 0.005 mm residual
(5) Galvo field 0.010 mm de-translated residual
(6) Laser focus 0.02 mm
(7) Camera focus Z 0.05 mm

Those correspond to demo-xy-galvo. On real hardware they should be derived from an error budget, not copied:

  1. Write down the figure you must guarantee, e.g. "positioning accuracy ±0.05 mm (3 sigma)";
  2. List every contributor: calibration, motion, vision repeatability, part tolerance, thermal drift;
  3. Combine them in RSS — they are independent;
  4. Allocate a budget to each so that 3*sigma_total <= 0.05 mm.
sigma_total = sqrt(sigma_calib^2 + sigma_motion^2 + sigma_vision^2 + sigma_part^2 + sigma_thermal^2)

Read the shares, not the absolute values. If vision repeatability is half the variance, further work on the calibration algorithm buys little; better illumination and larger dots buy a lot. The main value of an error budget is telling you what not to optimise.

Taking this to real hardware

There is no truth there, but the method survives; only the verification changes:

  • Use a more accurate instrument as reference (dial gauge, gauge blocks, laser interferometer, a better target);
  • Use consistency instead of accuracy: repeat the same position 30 times and look at the spread;
  • Use a closed-loop check: after calibrating, perform a move of known size and measure the result.

The point of the virtual machine is to let you practise the method while the answer is known, so that on real hardware you have the judgement to ask whether a number is plausible.

Last updated: Sep 23, 2026
Was this page helpful?