11.6 Shortcomings of Historical Simulation
In value-at-risk measures that employ a standard Monte Carlo transformation procedure, there is an inference procedure, perhaps using UWMA or EWMA. There is also a procedure that generates a realization {1r[1], 1r[2], … , 1r[m]} of a sample for use in the Monte Carlo analysis. That procedure uses a random number generator.
Historical simulation replaces both of those functions with raw historical data. Doing so introduces two problems, which we describe below. Those problems go hand in hand: addressing one tends to exacerbate the other.
11.6.1 Large Standard Errors
Historical simulation is a form of Monte Carlo analysis. As such, it entails standard error. Because a realization {1r[1], 1r[2], … , 1r[m]} of a sample for 1R is constructed directly from historical data, sample sizes m tend to be small. This produces large standard errors.
Historical simulation is routinely performed with historical samples of size m = 100. On the other extreme, two years of data might be the most that could reasonably be used. With approximately 250 trading days in a year, that translates into a sample size m of 500. With mirror values, that becomes 1000. In Exhibit 10.5, we assesses what would be the standard error of a Monte Carlo analysis of value-at-risk for several hypothetical portfolios, assuming a sample size of m = 1000. The standard errors vary, but they are routinely around 5% or more. If sample size drops to the more common m = 100, that 5% rises to almost 16%!
With a standard Monte Carlo transformation procedure, value-at-risk can be calculated with a small standard error. This is because the sample size m can be made as large as desired. The only limiting factor is the processing time required to perform the Monte Carlo analysis. Processing time used to be a serious limitation, but as the cost of processing power drops, it is becoming less so. Using variance reduction, or the similar technique of selective valuation of realizations, standard errors can be further cut, usually dramatically. None of these techniques are compatible with historical simulation.
11.6.2 Stale Historical Data
Markets change, sometimes gradually, and sometimes suddenly. Factors that cause them to change include:
- new technology,
- regulatory changes,
- altered perceptions in the wake of a scandal or crisis, or
- economic expansion or decline.
In rare cases, market data that is just a few months old may not be reflective of the same market today. This places historical simulations that use a year or more of historical data at a significant disadvantage compared to other value-at-risk measures: the data is often stale. A solution is to calculate historicalvalue-at-risk using only the most recent data, but doing so exacerbates standard error.