OLS Stress Test

A textbook interval looks fine until the errors are heavy, skewed, or louder when X is large. I wanted to see which interval still covers the truth.

The problem

Ordinary least squares assumes a well-behaved error. The interval students are shown is the one that interval is. The situation that breaks it is ordinary: a few wild points, a skew, a variance that grows as X grows. The person it frustrates is anyone who trusted the textbook band on data that does not look like the textbook.

It was worth a Monte Carlo because one unlucky sample can make any interval look wise. I wanted the coverage, over and over, next to intervals that are built for the mess.

How I built it

The model is Y = 2 + 0.5 X plus an error. X is standard normal, so the design is not the thing under test. The errors are. They are normal, a scaled t, variance that grows with the absolute value of X, a 5 percent contamination, or a shifted exponential. Each of those is a different way the textbook assumption fails.

Every sample gets four intervals: ordinary least squares, the HC3 sandwich, a percentile bootstrap, and a pivotal bootstrap. HC3 is there because heteroskedasticity is one of the rows. The two bootstraps are there because a sandwich is not the only repair. 5,000 replications and 2,000 bootstrap draws, at sample sizes from 30 to 500. joblib runs them. statsmodels fits the OLS.

Coverage is the only score I cared about. A narrow interval that misses the true slope is not a better interval. The plots are there so a row can be read as a curve across n, not as one sample size that happened to look good.

Challenges and what I learned

On the heteroskedastic row, OLS coverage stays about 0.77 at every n. HC3 moves from 0.919 at n = 30 to 0.950 at n = 500. A bigger sample did not rescue the textbook interval. That is the result I keep: more data, same broken assumption, same miss. The sandwich moves. OLS does not.