Suppose we wish to estimate the mean of a sample drawn from a normal distribution. In the Bayesian approach, we know the prior distribution for the mean (it could be a non-informative prior) and then we update this with our observations to create the posterior, the latter giving us improved information about the distribution of the mean. In symbols

Typically, the samples are chosen to be independent, and all of the data is used to perform the update but, given independence, there is no particular reason to do that, updates can performed one at a time and the result is the same; nor is the order of update important. Being a bit imprecise, we have

The standard notation in Bayesian statistics is to denote the parameters of interest as and the observations as . For reasons that will become apparent in later blog posts, let us change notation and label the parameters as and the observations as .

Let us take a very simple example of a prior where is known and then sample from a normal distribution with mean and variance for the -th sample where is known (normally we would not know the variance but adding this generality would only clutter the exposition unnecessarily).

The likelihood is then

As we have already noted, instead of using this with the prior to calculate the posterior, we can update the prior with each observation separately. Suppose that we have obtained the posterior given samples (we do not know this is normally distributed yet but we soon will):

Then we have

Writing

and then completing the square we also obtain

# More Formally

Now let’s be a bit more formal about conditional probability and use the notation of -algebras to define and where , is as before and . We have previously calculated that and that and the tower law for conditional probabilities then allows us to conclude . By Jensen’s inequality, we have

Hence is bounded in and therefore converges in and almost surely to . The noteworthy point is that if if and only if converges to 0. Explicitly we have

which explains why we took the observations to have varying and known variances. You can read more in Williams’ book (Williams 1991).

# A Quick Check

We have reformulated our estimation problem as a very simple version of the celebrated Kalman filter. Of course, there are much more interesting applications of this but for now let us try “tracking” the sample from the random variable.

```
> {-# OPTIONS_GHC -Wall #-}
> {-# OPTIONS_GHC -fno-warn-name-shadowing #-}
> {-# OPTIONS_GHC -fno-warn-type-defaults #-}
> {-# OPTIONS_GHC -fno-warn-unused-do-bind #-}
> {-# OPTIONS_GHC -fno-warn-missing-methods #-}
> {-# OPTIONS_GHC -fno-warn-orphans #-}
```

```
> module FunWithKalmanPart1 (
> obs
> , nObs
> , estimates
> , uppers
> , lowers
> ) where
>
> import Data.Random.Source.PureMT
> import Data.Random
> import Control.Monad.State
```

```
> var, cSquared :: Double
> var = 1.0
> cSquared = 1.0
>
> nObs :: Int
> nObs = 100
```

```
> createObs :: RVar (Double, [Double])
> createObs = do
> x <- rvar (Normal 0.0 var)
> ys <- replicateM nObs $ rvar (Normal x cSquared)
> return (x, ys)
>
> obs :: (Double, [Double])
> obs = evalState (sample createObs) (pureMT 2)
>
> updateEstimate :: (Double, Double) -> (Double, Double) -> (Double, Double)
> updateEstimate (xHatPrev, varPrev) (y, cSquared) = (xHatNew, varNew)
> where
> varNew = recip (recip varPrev + recip cSquared)
> xHatNew = varNew * (y / cSquared + xHatPrev / varPrev)
>
> estimates :: [(Double, Double)]
> estimates = scanl updateEstimate (y, cSquared) (zip ys (repeat cSquared))
> where
> y = head $ snd obs
> ys = tail $ snd obs
>
> uppers :: [Double]
> uppers = map (\(x, y) -> x + 3 * (sqrt y)) estimates
>
> lowers :: [Double]
> lowers = map (\(x, y) -> x - 3 * (sqrt y)) estimates
```

# Bibliography

Williams, David. 1991. *Probability with Martingales*. Cambridge University Press.

If you haven’t looked at them already, you may want to check out Unscented Kalman Filters (and the Unscented Transform in general.)

Thanks – I have a reference for them but so far only glanced at it. I was going to do particle filters next but maybe I can compare the two approaches.