Meteorological Instrument Performance Characteristics

### Instrument Performance Characteristics Β» Precision-Resolution Relationship

#### Resolution Interaction 1

Two Gaussian probability distributions with unit standard deviation, centered on x1 and x2, which differ by d. The red curve is a Gaussian distribution with standard deviation of √2 and βPβ listed in the plot is the fraction of measurements of x2 that differ more than √2 from x1. The measurements of x2 that contribute to this fraction are shown as the thicker green line. The plot shows the fraction that differ from a fixed value of x1 at the center of the distribution, but this result should be integrated over possible values of x1. βP2β is the result of that integration, so this is the value to consider when assessing confidence estimates. Image from NCAR/EOL.

Precision is the component of uncertainty arising from the random error in a measurement. An estimation of the precision is obtained from the root-mean-square (RMS) error determined from the unbiased standard deviation.

Use the images to explore the relationships between precision and resolution, then answer the following questions.

##### Probability Distribution 1

Two Gaussian probability distributions with unit standard deviation, centered on x1 and x2, which differ by d = 2.06. The red curve is a Gaussian distribution with standard deviation of √2 and βPβ listed in the plot is the fraction of measurements of x2 that differ more than 0.95π from x1, or the limits on the red curve that include 50% of the distribution. The measurements of x2 that contribute to this fraction are shown as the thicker green line. The plot shows the fraction that differ from a fixed value of x1 at the center of the distribution, but this result should be integrated over possible values of x1. βP2β is the result of that integration, so this is the value to consider when assessing confidence estimates. Image from NCAR/EOL.

##### Probability Distribution 2

As in the previous figure but with measurand separation 2.60 and integration limits that correspond to 68.3% of the area under the red line, or Β± √2π. Image from NCAR/EOL.

##### Probability Distribution 3

As in the previous figures but with measurand separation 4.02 and integration limits that correspond to 95.4% of the area under the red line, or Β± 2√2π. Image from NCAR/EOL.

##### Probability Distribution 4

As in the previous figures but with measurand separation 5.20 and integration limits that correspond to 95.4% of the area under the red line, or Β± 2√2π. Image from NCAR/EOL.

#### Question

If π is the instrument precision (at the one-standard deviation limit), what separation between measurands is required for the difference between measurements to be at least √2π for at least 80% of the pairs of measurements?

With a separation of 2.6πx or 2.6 times the precision, the measurements will differ by √2π in 80% of the cases. In this case, the integrations are done outside the √2π limits. Probability Distributions 1 and 3 use integration limits determined by π or 2√2π, respectively, and Probability Distribution 4 represents a higher likelihood and also a too-large integration limit.

#### Resolution Interaction 2

##### Probability Distribution 1

Two Gaussian probability distributions with unit standard deviation, centered on x1 and x2, which differ by d = 3.74. The red curve is a Gaussian distribution with standard deviation of and βPβ listed in the plot is the fraction of measurements of x2 that differ more than √2π from x1, or the limits on the red curve that include 68.3% of the distribution. The measurements of x2 that contribute to this fraction are shown as the thicker green line. The plot shows the fraction that differ from a fixed value of x1 at the center of the distribution, but this result should be integrated over possible values of x1. βP2β is the result of that integration, so this is the value to consider when assessing confidence estimates. Image from NCAR/EOL.

##### Probability Distribution Function 2

As in the previous tab figure except that the measurand separation is 5.20 and the integration limits are Β± 2√2π as needed to include 95.4% of the area under the red curve. Image from NCAR/EOL.

##### Probability Distribution Function 3

As in the preceding figure except that the measurand separation is 3.28 and the integration limits are Β± 0.95π as needed to include 50% of the area under the red curve. Image from NCAR/EOL.

#### Question

For measurement pairs to differ by at least 2√2π in 99% of cases, as would be needed for 95% confidence that the measurands are different, what separation between measurands would be required?

The correct answer is b, 5.2 π.

Measurands would have to be different by this separation distance 99% of the time to be able to report findings to the 95% confidence limit. For a separation distance between measurands of 3.74 and 3.28, the confidence limits corresponding to 99% of measurements are 68.3% and 50%, respectively.

As in the preceding figures but for a separation between measurands of d = 1 and integration limits for the red curve that cover 50% of the distribution. Image by NCAR/EOL.

#### Question

For measurands separated by the instrument precision, what is the likelihood (in %; i.e. enter a number like 50 for 50%) that pairs of measurements will differ by less than the true separation? Report your answer to the nearest whole number.

The correct answer is 40-42%. The approximate answer can be found using the figure below, which shows the probability distributions for a separation of d = 1 and integration limits covering 50% of the red distribution. Notice that the red vertical lines are very close to the Β±d limits from the center of x1. The integration of the regions outside Β±d then give about 0.60, so the fraction *inside* is 0.4. This is only approximate because the limits are not exactly Β±d, but this will give an answer within a few percent of the right answer.

Answer the following three questions using this statement:

A transducer produces a voltage in the range from 0-10 V and is connected to an analog-to-digital converter covering that full range that produces a 10-bit output.

#### Question 1 of 3

What is the number of available digital increments covering the full range?

Computers count by base 2. The smallest increment of data on a computer is a bi. A bit is a binary digit, that is, it can hold only one of two values: 0 or 1, corresponding to the electrical values of off or on, respectively. A 10-bit output will, therefore, have 210 available digital increments over the full range of 10 V. Answers a and b are incorrect. Answer a is the number of bits in a byte and answer b is for an 8-bit output.

#### Question 2 of 3

What is the finest resolution that this instrument can have?

The finest resolution is equal to the range divided by the number of increments over that range, or 10 V / 1024 = 0.00977 V = 9.77 mV. Answer a is incorrect because it is the square root of 10 V / 1024, and answer c is 10V / 29 in mV, and has no relevancy to a 10-bit output.