4.3.4 Bias
The bias of an estimator H is the expected value of the estimator less the value θ being estimated:
[4.6]
If an estimator has a zero bias, we say it is unbiased. Otherwise, it is biased. Let’s calculate the bias of the sample mean estimator [4.4]:
[4.7]
[4.8]
[4.9]
[4.10]
[4.11]
where μ is the mean E(X) being estimated. The sample mean estimator is unbiased.
4.3.5 Standard error
The standard error of an estimator is its standard deviation:
[4.12]
Let’s calculate the standard error of the sample mean estimator [4.4]:
[4.13]
[4.14]
[4.15]
[4.16]
[4.17]
[4.18]
where σ is the standard deviation std(X) being estimated. We don’t know the standard deviation σ of X, but we can approximate the standard error based upon some estimated value s for σ. Irrespective of the value of σ, the standard error decreases with the square root of the sample size m. Quadrupling the sample size halves the standard error.
4.3.6 Mean Squared Error
We seek estimators that are unbiased and have minimal standard error. Sometimes these goals are incompatible. Consider Exhibit 4.2, which indicates PDFs for two estimators of a parameter θ. One is unbiased. The other is biased but has a lower standard error. Which estimator should we use?
Mean squared error (MSE) combines the notions of bias and standard error. It is defined as
[4.19]
Since we have already determined the bias and standard error of estimator [4.4], calculating its mean squared error is easy:
[4.20]
[4.21]
[4.22]
Faced with alternative estimators for a given parameter, it is generally reasonable to use the one with the smallest MSE.