Meteorological Instrument Performance Characteristics

### Instrument Performance Characteristics » Precision-Resolution Relationship

#### Resolution Interaction 1 Two Gaussian probability distributions with unit standard deviation, centered on x1 and x2, which differ by d. The red curve is a Gaussian distribution with standard deviation of √2 and “P” listed in the plot is the fraction of measurements of x2 that differ more than √2 from x1. The measurements of x2 that contribute to this fraction are shown as the thicker green line. The plot shows the fraction that differ from a fixed value of x1 at the center of the distribution, but this result should be integrated over possible values of x1. “P2” is the result of that integration, so this is the value to consider when assessing confidence estimates. Image from NCAR/EOL.

Precision is the component of uncertainty arising from the random error in a measurement. An estimation of the precision is obtained from the root-mean-square (RMS) error determined from the unbiased standard deviation.

Use the images to explore the relationships between precision and resolution, then answer the following questions.

##### Probability Distribution 1 Two Gaussian probability distributions with unit standard deviation, centered on x1 and x2, which differ by d = 2.06. The red curve is a Gaussian distribution with standard deviation of √2 and “P” listed in the plot is the fraction of measurements of x2 that differ more than 0.95𝜎 from x1, or the limits on the red curve that include 50% of the distribution. The measurements of x2 that contribute to this fraction are shown as the thicker green line. The plot shows the fraction that differ from a fixed value of x1 at the center of the distribution, but this result should be integrated over possible values of x1. “P2” is the result of that integration, so this is the value to consider when assessing confidence estimates. Image from NCAR/EOL.

##### Probability Distribution 2 As in the previous figure but with measurand separation 2.60 and integration limits that correspond to 68.3% of the area under the red line, or ± √2𝜎. Image from NCAR/EOL.

##### Probability Distribution 3 As in the previous figures but with measurand separation 4.02 and integration limits that correspond to 95.4% of the area under the red line, or ± 2√2𝜎. Image from NCAR/EOL.

##### Probability Distribution 4 As in the previous figures but with measurand separation 5.20 and integration limits that correspond to 95.4% of the area under the red line, or ± 2√2𝜎. Image from NCAR/EOL.

#### Question

If 𝜎 is the instrument precision (at the one-standard deviation limit), what separation between measurands is required for the difference between measurements to be at least √2𝜎 for at least 80% of the pairs of measurements?

With a separation of 2.6𝜎x or 2.6 times the precision, the measurements will differ by √2𝜎 in 80% of the cases. In this case, the integrations are done outside the √2𝜎 limits. Probability Distributions 1 and 3 use integration limits determined by 𝜎 or 2√2𝜎, respectively, and Probability Distribution 4 represents a higher likelihood and also a too-large integration limit.

#### Resolution Interaction 2

##### Probability Distribution 1 Two Gaussian probability distributions with unit standard deviation, centered on x1 and x2, which differ by d = 3.74. The red curve is a Gaussian distribution with standard deviation of and “P” listed in the plot is the fraction of measurements of x2 that differ more than √2𝜎 from x1, or the limits on the red curve that include 68.3% of the distribution. The measurements of x2 that contribute to this fraction are shown as the thicker green line. The plot shows the fraction that differ from a fixed value of x1 at the center of the distribution, but this result should be integrated over possible values of x1. “P2” is the result of that integration, so this is the value to consider when assessing confidence estimates. Image from NCAR/EOL.

##### Probability Distribution Function 2 As in the previous tab figure except that the measurand separation is 5.20 and the integration limits are ± 2√2𝜎 as needed to include 95.4% of the area under the red curve. Image from NCAR/EOL.

##### Probability Distribution Function 3 As in the preceding figure except that the measurand separation is 3.28 and the integration limits are ± 0.95𝜎 as needed to include 50% of the area under the red curve. Image from NCAR/EOL.

#### Question

For measurement pairs to differ by at least 2√2𝜎 in 99% of cases, as would be needed for 95% confidence that the measurands are different, what separation between measurands would be required?

The correct answer is b, 5.2 𝜎.

Measurands would have to be different by this separation distance 99% of the time to be able to report findings to the 95% confidence limit. For a separation distance between measurands of 3.74 and 3.28, the confidence limits corresponding to 99% of measurements are 68.3% and 50%, respectively. As in the preceding figures but for a separation between measurands of d = 1 and integration limits for the red curve that cover 50% of the distribution. Image by NCAR/EOL.

#### Question

For measurands separated by the instrument precision, what is the likelihood (in %; i.e. enter a number like 50 for 50%) that pairs of measurements will differ by less than the true separation? Report your answer to the nearest whole number.

The correct answer is 40-42%. The approximate answer can be found using the figure below, which shows the probability distributions for a separation of d = 1 and integration limits covering 50% of the red distribution. Notice that the red vertical lines are very close to the ±d limits from the center of x1. The integration of the regions outside ±d then give about 0.60, so the fraction *inside* is 0.4. This is only approximate because the limits are not exactly ±d, but this will give an answer within a few percent of the right answer.

Answer the following three questions using this statement:

A transducer produces a voltage in the range from 0-10 V and is connected to an analog-to-digital converter covering that full range that produces a 10-bit output.

#### Question 1 of 3

What is the number of available digital increments covering the full range?

Computers count by base 2. The smallest increment of data on a computer is a bi. A bit is a binary digit, that is, it can hold only one of two values: 0 or 1, corresponding to the electrical values of off or on, respectively. A 10-bit output will, therefore, have 210 available digital increments over the full range of 10 V. Answers a and b are incorrect. Answer a is the number of bits in a byte and answer b is for an 8-bit output.

#### Question 2 of 3

What is the finest resolution that this instrument can have?

The finest resolution is equal to the range divided by the number of increments over that range, or 10 V / 1024 = 0.00977 V = 9.77 mV. Answer a is incorrect because it is the square root of 10 V / 1024, and answer c is 10V / 29 in mV, and has no relevancy to a 10-bit output.