Meteorological Instrument Performance Characteristics

Instrument Performance Characteristics » Precision and Resolution » Resolution Definition #1

Our first definition of resolution, and the one that has found general acceptance for instrument performance, is “the smallest difference between measurements that indicates that the corresponding measurands are different”, which normally uses single measurements of each measurand. Resolution defined this way will be limited primarily by the precision of the measurement. Consider first the case where two measurands x1 and x2 are the same. If measurements of each (M1 and M2) have a random error characterized by 𝜎M and the random errors in repeated measurements are independent, then the expected standard deviation arising from random error in the difference y between the two measurements is 𝜎y = √2𝜎M. The two measurements will differ by less than 𝜎y in about 68.3% of cases, as is appropriate for a confidence level corresponding to one standard deviation, but they will differ by more than 𝜎M, the instrument precision, in about 48% of cases. A measured difference of 𝜎M is therefore an unreliable indicator that the measurands are different; a measured difference of at least √2𝜎M is needed to indicate that the measurands are different. Figure 1 shows the probability distribution functions for the two measurements (as the blue line) and that for the difference (as the red line).

Probability distributions for two measurements, both with measurand 0 (blue line), and the probability distribution for the measured difference (red line).

Probability distributions for two measurements, both with measurand 0 (blue line), and the probability distribution for the measured difference (red line). The assumed standard deviation corresponding to random error is 1. Image from NCAR/EOL.

To find the resolution, a different limit is needed. In this case we want to find the minimum separation between measurands y' = x1' - x2' that leads to a statistically significant likelihood that measurements will show the two measurands to be different. (Here the primed quantities refer to the measurands and unprimed quantities to the measurements.) We need measurements separated by at least √2𝜎x to be confident, on the basis of measurements with precision 𝜎M, that the measurands are different. We therefore need to find the minimum separation between measurands required to give measurements separated by at least √2𝜎x in at least 68.3% of cases. Then, with one-standard-deviation confidence, we can use the measurements to conclude that the measurands are different.

The separation at which the probability “P2” of finding a difference outside √2𝜎x is about 68.3%. The probability for a separation between measurements of d = 𝜎x is only about 43%, so a separation between measurands equal to the precision of the sensor the probability is still far below the desired limit.

The Probability Density Function for a separation d between two measurands equal to the precision, which yields a probability of only 43 percent (P2) that the measurements will be different by 2 times the precision.

The Probability Density Function for a separation d between two measurands equal to the precision 𝜎x, which yields a probability of only 43 percent (P2) that the measurements will be different by √2𝜎x. Image from NCAR/EOL.

In order to achieve a probability P2 = 68.3% that two measurands are different with a one-standard deviation confidence limit, the separation between measurements must be about twice the precision, or d = 2.06 𝜎x.

The Probability Density Function for a separation d between two measurements equal to about twice the precision 2.06 times sigma, which yields a probability of only 68.3 percent (P2) that the measurands are likely to be different.

The Probability Density Function for a separation d between two measurements equal to about twice the precision 2.06 𝜎x, which yields a probability of only 68.3 percent (P2) that the measurands are likely to be different. Image from NCAR/EOL.

Sometimes it is desirable to quote a resolution that gives confidence at the two-standard-deviation limit (the 95% confidence limit) that the measurands are different. In this case, the separation d of the measurements must be about five times the precision.

The Probability Density Function for a separation d between two measurements equal to about five times the precision 5.2 times sigma, which yields a probability of 95 percent (P2) that the measurands are different.

The Probability Density Function for a separation d between two measurements equal to about five times the precision 5.2 𝜎x, which yields a probability of 95 percent (P2) that the measurands are different. Image from NCAR/EOL.