Index of /blog/post-3/

previous page

Learning from Mistakes: Error Propagation

This blog post serves as an introduction to statistical analysis which can be a useful tool for application developers to make meaningful statements about the accuracy, precision and uncertainty of laboratory devices and the measurements they yield.

Self-Reflection

When growing up I could have never imagined that I would end up interning as a developer at a company in Estonia. There was certainly a time in my life when I tried a little bit of everything, which over the years helped me appreciate many different areas of specializations that you sometimes come into contact with during your professional career. I am convinced that it takes a certain kind of mindest to excel in this field, but at the moment I cannot speak to it any further since I am not there yet. However, if there's one thing I'd like to know more about, it's without a question math. That's also why I was a little bit disappointed at the beginning that they don't teach you math in vocational school. Enterprise resource planning (ERP) software often deals with large amounts of quantitative data, which inspired me to take a look into statistical methods.

Dealing with Uncertainties

In a laboratory environment there are many distinctive challenges involved when it comes to collecting and analyzing quantitative data. The term quantitative data encompasses numerical data, ordinal data as well as nominal data. In short, anything that can be reasonably thought of as factual information falls under this category [1]. Knowing the differences between all these types of data is important in order to determine what kind of tools you need to employ to collect and analyze your observations correctly. The process of systematically categorizing and illustrating observations can be attributed to the branch of descriptive statistics. This paves the way for a researcher to draw conclusions through inductive reasoning by using processes and techniques from the realm of inferential statistics [2].

The process of taking measurements can introduce different kind of uncertainties that should be accounted for in your final report. To help you guide through this process, you should think about these three questions before you start to collect measurement data:

In response to the first question you should be able to say something about the statistical measurement uncertainty of your device (this is different from the systematic uncertainty which is constant). A scale, for instance, can have an offset of 2mg due to a zero point error. The second question requires you to think about the human-device interaction; analog devices are particularly affected by this. As for the third question, if repeatibility cannot be procured or doesn't make sense to follow through with (e.g. when you measure the resistance of a resistor with a multimeter), assign the resolution of your measuring device as uncertainty in your equations. If that's not the case, use the variance to quantify uncertainty. Both situations are examples of the intrinsic randomness of events that can take place in any experiment that involves continuous data.

A note on terminology: avoid the term human error in your reports when you don't actually mean it and try to be specific about the cause of uncertainty at hand; errors are synonymous to mistakes and can invalidate your analysis completely in the worst case scenario. The expression measure twice, cut once also applies to sampling data!

Accuracy vs Precision

Note that the terms statistical and systematic uncertainties are related to the idea of accuracy and precision. While these words are sometimes use interchangeably in casual speech, they actually mean something very different in a scientific context: accuracy is a measure that describes how close a measurement \(x\) is to its corresponding universally accepted (true) value \(x_T\), whereas precision speaks to the distance between individual measurement points. In our calculations, we should always round to the same precision as the least precise number before we can report a result with confidence, i.e. we should always work with the most imprecise measurements. To illustrate this definition by example, let's make the silly assumption that the standard math library in Python contains the true value of \(\pi\). In this case we want to figure out the accuracy of \(x\) when we assign it the number 3.14. This relationship can be expressed as a ratio function for estimates lower than or equal to the true value; however this doesn't work very well for overestimates for obvious reasons. Hence,

$$a:\mathbb{R}\to[0,1]: a(x)=1-\frac{|x-x_T|}{x_T}$$

We can convert this formula into a function that accurately rounds the result with the appropriate number of significant figures: 

  import math
  from decimal import Decimal

  def accuracy(x: float, x_T: float) -> float:
    precision = len(Decimal(str(x)).as_tuple().digits)
    return round(1 - abs(x - x_T) / x_T * 100, precision)

  result = accuracy(3.14, math.pi)

  # 99.95%
  print(f"{result}%")

Interestingly enough we just discovered that rounding PI to 2 significant digits only introduced an uncertainty of 0.05%, which from a programmers point of view is accurate enough to cover most basic calculation's needs.

However, this kind of approach assumes that we know the true value of \(x\) in advance and that the true value is a rational number. On top of that, a more realistic example would take into account that we have to put multiple different measurements into relationship with each other, which raises the question how the uncertainty propagates into the final result?

Don't be Mean: Variance and Uncertainty

The arithmetic mean \(\mu\) (sometimes also referred to as the average value \(\bar{x}_n\)) in a population \(X\) of size \(n\) is defined by

$$\mu=\frac{1}{n} \sum_{i=1}^n x_i$$

This is the measure of central tendency you are probably most familiar with already. See below a short list of properties to help you interpret this value appropriately:

The standard deviation (for individual elements in a population) is defined as the square root of variance \(\sigma\):

$$\sigma_n^2=\frac{1}{n-1} \sum_{i=1}^n (x_i - \mu)^2$$

Notice that we lose a degree of freedom in the standard deviation to ensure that the this equation results in an unbiased estimate of the variance [3]. To determine the standard deviation of the mean (which is equivalent to the notion of uncertainty that we developed earlier), we can use the formula

$$\bar{\sigma}_n=\frac{\sigma_n}{\sqrt{n}}$$

Example: Given the population \(X=\{71\text{mm},72\text{mm},72\text{mm},73\text{mm},71\text{mm}\}\), calculate the mean and its corresponding standard deviation. Translating this problem into Python yields: 

  from math import sqrt
  from statistics import mean, stdev

  X = [71, 72, 72, 73, 71]

  x̄ = mean(X)
  u = stdev(X) / sqrt(len(X))

  print(f"({round(x̄, 1)}±{round(u, 1)})mm")

Hence, we can report the final result as \(x = (71.8 \pm 0.4)\text{mm}\). It is important to note that the mean should be rounded to one more significant figure than occurs in the population's lowest precision.

Propagation of Uncertainty

In statistics, the propagation of uncertainties is defined as the effect of variable's uncertainties on the uncertainty of a function \(f\) based on them. When we assume that these variables \(x_1,x_2,\dots,x_n\) are independent of each other, their function's uncertainty \(\sigma_f\) reduces [4] to the simplified variance formula

$$\sigma_f = \sqrt{\sum_{i=1}^n \left(\frac{\partial f}{\partial x_i}\sigma_{x,i}\right)^2}$$

Any time a calculations requires more than one variable to solve, we need to determine the propagation of uncertainty to properly determine the uncertainty of a multivariable function. This technique also assumes that the relative uncertainty of each individual \(x_i\) is comparetively small.

Example 1: Let \(x,y\) be continous variables equiped with independent random uncertainties. In this context independence means that an uncertainty in \(x\) doesn't affect \(y\), and vice versa. The term random uncertainty further implies that the uncertainties are not of systematical nature. Then, given a function \(f\) defined by

$$f:\mathbb{R}\to\mathbb{R}: f(x,y)-x^2-y^2+6$$

(which describes a downward opening elliptic paraboloid) we can determine its standard deviation with

$$ \begin{align*} \sigma_f^2 &= \left(\frac{\partial f}{\partial x}\sigma_x\right)^2 + \left(\frac{\partial f}{\partial y}\sigma_y\right)^2 \\ &= (-2x\sigma_x)^2 + (-2y\sigma_y)^2 \\ &= 4\left[(x\sigma_x)^2 + (y\sigma_y)^2\right] \end{align*} $$

Hence, we can report the final result as \(z = f(x) \pm \sigma_f\).

Example 2: Let \(f(x,y,z)=x+y+z\). Then it follows almost immediately that \(\sigma_f=\sqrt{\sigma_x^2+\sigma_y^2+\sigma_z^2}\).

Final Thoughts

We only scratched the surface on what's possible this time. Statistics is a vast field and offers much more to explore, and it certainly helps to have a basic understanding of probability theory in order to advance further in our studies. As someone who primarily engages with software development, it makes a lot of fun to bring these equations to life with code.

Further Reading