Calculating a moving average
Posted by mop Sun, 18 Jul 2004 17:41:00 GMT
What I need is an efficient generalized approach to calculating a moving average. I hope to measure the time required to complete an operation (many ops per second) and record a moving average of these times for at least two periods. I expect that the periods may be sufficiently large to cause storage issues if all measurements for the period are stored.
As an example, lets say that 100 operations are performed per second, and that I’ll be calculating a 30 minute moving average. That’s 180,000 data points. Of course, I don’t need to store all the measurements in order to calculate a simple mean average, just keep a running sum and a count. Now let’s say I want to record this moving average once every ten minutes -- storing a sum and a count is no longer enough.
There are a few ways I can think of to deal with the problem:
- change the definition of the moving average to include a fixed number of measurements instead of a fixed period
- make an assumption about the number of measurements in a thirty minute period, then the average could be approximated for each new data point using μ = (μ⋅n + x)/(n + 1)
- again, make an assumption about the number of measurements in a period, but adjust the assumption as the rate of operations changes
- store all the data points for at least thirty minutes ;-)
The first option is appealing for a number of reasons: it is simple, accurate, and allows for more more in-depth statistical measurements. It does however call for a judicious choice of the sample size. In our example, we likely would not choose a sample size equivalent to 30 minutes of measurements. Assuming that 30 minutes was an arbitrary measure (it is) I’ll use an equally arbitrary sample size.
