Our first definition of resolution, and the one that has found general acceptance for instrument performance, is “the smallest difference between measurements that indicates that the corresponding measurands are different”, which normally uses single measurements of each measurand. Resolution defined this way will be limited primarily by the precision of the measurement. Consider first the case where two measurands x1 and x2 are the same. If measurements of each (M1 and M2) have a random error characterized by 𝜎M and the random errors in repeated measurements are independent, then the expected standard deviation arising from random error in the difference y between the two measurements is 𝜎y = √2𝜎M. The two measurements will differ by less than 𝜎y in about 68.3% of cases, as is appropriate for a confidence level corresponding to one standard deviation, but they will differ by more than 𝜎M, the instrument precision, in about 48% of cases. A measured difference of 𝜎M is therefore an unreliable indicator that the measurands are different; a measured difference of at least √2𝜎M is needed to indicate that the measurands are different. Figure 1 shows the probability distribution functions for the two measurements (as the blue line) and that for the difference (as the red line).
To find the resolution, a different limit is needed. In this case we want to find the minimum separation between measurands y' = x1' - x2' that leads to a statistically significant likelihood that measurements will show the two measurands to be different. (Here the primed quantities refer to the measurands and unprimed quantities to the measurements.) We need measurements separated by at least √2𝜎x to be confident, on the basis of measurements with precision 𝜎M, that the measurands are different. We therefore need to find the minimum separation between measurands required to give measurements separated by at least √2𝜎x in at least 68.3% of cases. Then, with one-standard-deviation confidence, we can use the measurements to conclude that the measurands are different.
The separation at which the probability “P2” of finding a difference outside √2𝜎x is about 68.3%. The probability for a separation between measurements of d = 𝜎x is only about 43%, so a separation between measurands equal to the precision of the sensor the probability is still far below the desired limit.
In order to achieve a probability P2 = 68.3% that two measurands are different with a one-standard deviation confidence limit, the separation between measurements must be about twice the precision, or d = 2.06 𝜎x.
Sometimes it is desirable to quote a resolution that gives confidence at the two-standard-deviation limit (the 95% confidence limit) that the measurands are different. In this case, the separation d of the measurements must be about five times the precision.