The following interactive tool can be used to explore the relationship between precision and resolution. After viewing the tool, answer the questions to test your understanding of the material. If you are viewing a downloaded version of this lesson, please see the "Printable Lesson" for the ShinyApp exercise information.

If π is the instrument precision (at the one-standard deviation limit), what separation between measurands is required for the difference between measurements to be at least √2π for at least 80% of the pairs of measurements?

The correct answer is b.

With a separation of 2.6π_{x} or 2.6 times the precision, the measurements will differ by √2π in 80% of the cases. To see this case, select the 68.3% checkbox (which determines the √2π limits) and adjust the slider until P2 = 0.8. Other answers correspond to lower (answer a) or higher (answers c and d) probabilities than 80%.

For measurement pairs to differ by at least 2√2π in 95% of cases, as would be needed for 95% confidence that the measurands are different, what separation between measurands would be required?

The correct answer is b.

The correct answer is b, 5.2 π. To see this, select the β95.4%β button and adjust the slider to the point where P2 = 0.95. Answers a and c correspond to significantly lower likelihoods that the measurements would differ by 2√2π .

For measurands separated by the instrument precision, what is the likelihood that pairs of measurements will differ by less than the true separation? Report your answer to the nearest whole number.

(Type your answer in the box, then click Done.)

The correct answer is 42%. The answer can be found by checking 50% and using d = 1. Notice that the red vertical lines are very close to the +/-d limits from the center of x1. The integration of the regions **outside** +/-d then give about 0.60, so the fraction **inside** is 0.4. This is only approximate because the limits are not exactly +/-d, but this will give an answer within a few percent of the right answer.

Answer the following three questions using this statement:

**A transducer produces a voltage in the range from 0-10 V and is connected to an analog-to-digital converter covering that full range that produces a 10-bit output.**

What is the number of available digital increments covering the full range?

The correct answer is c.

Computers count by base 2. The smallest increment of data on a computer is a bit. A bit is a binary digit, that is, it can hold only one of two values: 0 or 1, corresponding to the electrical values of off or on, respectively. A 10-bit output will, therefore, have 2^{10} available digital increments over the full range of 10 V. Answers a and b are incorrect. Answer a is the number of bits in a byte and answer b is for an 8-bit output.

What is the smallest resolution that this instrument can have?

The correct answer is b.

The finest resolution is equal to the range divided by the number of increments over that range, or 5.8 mV. Answer a is incorrect because it is the square root of 10 V / 1024, and answer c is 10 V / 2^{9} in mV, and 2^{9} has no relevance to a 10-bit output. Note that answer b is correct in reference to resolution definition #1. In this definition, precision is
Delta/sqrt(12). Delta for 10-bit resolution of 10 V is about 0.01 V. Precision then is (10/1024)/sqrt(12)=2.82 mV. From definition #1, resolution is about 2.06*precision, so the resolution is 5.8 mV.
If we use resolution definition #2, the answer will be 9.77 mV.

What is the instrument precision in the absence of other (larger) random-error contributions to the precision?

The correct answer is a.

The precision is given by π = (difference in measurand corresponding to a one-bit change in digital output) / number of bits = 9.77 mV / √12 = 2.82 mV.

Precision is the component of uncertainty arising from the random error in a measurement. An estimation of the precision is obtained from the root-mean-square (RMS) error determined from the unbiased standard deviation.

Use the images to explore the relationships between precision and resolution, then answer the following questions.

If π is the instrument precision (at the one-standard deviation limit), what separation between measurands is required for the difference between measurements to be at least √2π for at least 80% of the pairs of measurements?

The correct answer is b.

With a separation of 2.6π_{x} or 2.6 times the precision, the measurements will differ by √2π in 80% of the cases. In this case, the integrations are done outside the √2π limits. Probability Distributions 1 and 3 use integration limits determined by π or 2√2π, respectively, and Probability Distribution 4 represents a higher likelihood and also a too-large integration limit.

For measurement pairs to differ by at least 2√2π in 99% of cases, as would be needed for 95% confidence that the measurands are different, what separation between measurands would be required?

The correct answer is b, 5.2 π.

Measurands would have to be different by this separation distance 99% of the time to be able to report findings to the 95% confidence limit. For a separation distance between measurands of 3.74 and 3.28, the confidence limits corresponding to 99% of measurements are 68.3% and 50%, respectively.

For measurands separated by the instrument precision, what is the likelihood (in %; i.e. enter a number like 50 for 50%) that pairs of measurements will differ by less than the true separation? Report your answer to the nearest whole number.

The correct answer is 40-42%. The approximate answer can be found using the figure below, which shows the probability distributions for a separation of d = 1 and integration limits covering 50% of the red distribution. Notice that the red vertical lines are very close to the Β±d limits from the center of x1. The integration of the regions **outside** Β±d then give about 0.60, so the fraction *inside* is 0.4. This is only approximate because the limits are not exactly Β±d, but this will give an answer within a few percent of the right answer.

Answer the following three questions using this statement:

**A transducer produces a voltage in the range from 0-10 V and is connected to an analog-to-digital converter covering that full range that produces a 10-bit output.**

What is the number of available digital increments covering the full range?

The correct answer is c.

Computers count by base 2. The smallest increment of data on a computer is a bi. A bit is a binary digit, that is, it can hold only one of two values: 0 or 1, corresponding to the electrical values of off or on, respectively. A 10-bit output will, therefore, have 2^{10} available digital increments over the full range of 10 V. Answers a and b are incorrect. Answer a is the number of bits in a byte and answer b is for an 8-bit output.

What is the finest resolution that this instrument can have?

The correct answer is b.

The finest resolution is equal to the range divided by the number of increments over that range, or 10 V / 1024 = 0.00977 V = 9.77 mV. Answer a is incorrect because it is the square root of 10 V / 1024, and answer c is 10V / 2^{9} in mV, and has no relevancy to a 10-bit output.

What is the instrument precision in the absence of other (larger) random-error contributions to the precision?

The correct answer is a.

The precision is given by π = (difference in measurand corresponding to a one-bit change in digital output) / √12 = 2.82 mV.

For more information about resolution, see Appendix 1.