Law of
Large Numbers
Under appropriate conditions, an average built from many observations settles near the expected value. The theorem describes long-run concentration - not short-run compensation, certainty, or a promise that every sample behaves well.
Watch one path wobble, then settle.
Each dot is the running proportion of successes in independent Bernoulli trials. Change the true probability, sample size, or seed. The shaded band is a visual guide based on one standard deviation of the sample proportion, not a confidence guarantee for the displayed path.
Early swings are large because each observation carries substantial weight. Later observations move the average less.
A convergence statement under specified assumptions.
Independent trials do not remember a losing streak.
Systematic bias can survive any sample size.
It is a law about sequences of random variables.
Let X1, X2, ... be independent and identically distributed random variables with finite expected value mu. Their sample average is Sn/n = (X1 + ... + Xn)/n. A law of large numbers says that, as n grows, this average converges to mu in a specified probabilistic sense.
The familiar i.i.d. condition is sufficient, not the most general possible condition. There are versions for dependent or non-identically distributed variables, but they require their own hypotheses. The theorem never licenses averaging arbitrary observations merely because there are many of them.
Large numbers control random fluctuation around a stable target. They do not identify the target, validate the sampling process, or repair a changing world.
Weak and strong laws reach the same value differently.
Convergence in probability
P(|X-bar_n - mu| > epsilon) -> 0For every positive tolerance epsilon, the probability that the average is outside that tolerance tends to zero.
Almost-sure convergence
P(lim X-bar_n = mu) = 1With probability one, an infinite sample path eventually approaches mu and remains arbitrarily close.
Probability-one events may still exclude exceptional paths. And neither form tells you that a particular finite sample has already become accurate.
Independent noise is diluted; shared error is not.
For independent observations with finite variance, averaging divides variance by n, so the standard deviation of the mean falls like 1/sqrt(n). Chebyshev's inequality then bounds the chance of a large deviation and gives a direct proof of a weak law.
Ten times more data does not generally make standard error ten times smaller; it makes it about sqrt(10) times smaller. This square-root rate is why high precision can require very large samples.
Convergence is a destination; analysis needs a travel time.
P(|X-bar - mu| at least epsilon) is at most sigma squared/(n epsilon squared). Broadly applicable, often conservative.
Provides exponentially shrinking tail bounds when independent observations stay within known limits.
Under suitable conditions, the standardized sample mean approaches a normal distribution. This complements rather than replaces the LLN.
Repeated sampling can reveal skew, rare events, slow convergence, and the consequences of the actual stopping rule.
More rows amplify whatever generated them.
Do observations share shocks?
Clusters, repeated users, spatial correlation, and time dependence reduce the effective information in a nominal sample.
Is the same process operating?
Changing policy, seasonality, learning, drift, or feedback can make one fixed expectation a poor target.
Who can enter the sample?
Coverage and selection bias do not vanish with n. A huge convenience sample can precisely estimate the wrong population.
Does the mean exist?
Heavy-tailed distributions can converge slowly; in cases such as the Cauchy distribution, the ordinary expectation is undefined.
Are values recorded faithfully?
Calibration error, miscoding, survivorship, and proxy mismatch create systematic error that averaging preserves.
Why did collection end?
Repeatedly checking and stopping on a favorable result changes the inferential problem, even when each observation is random.
Where the law helps - and what it leaves unresolved.
Aggregate claims become more predictable.
If individual risks are sufficiently stable and weakly dependent, average loss can concentrate. Catastrophes and common shocks violate the simple independent-risk picture, so diversification needs dependence modeling.
LESSON: pool idiosyncratic risk; model systemic riskDefect rates stabilize across production.
Repeated inspection estimates a stable process rate. A process shift makes the historical average lag reality; control charts therefore look for changes, not just larger totals.
LESSON: convergence assumes a process worth averagingAverage treatment outcomes can settle.
Random assignment supports causal comparison, while sample size reduces sampling noise. Interference, attrition, peeking, and changing traffic still require design and analysis.
LESSON: randomization and LLN solve different problemsSimulation averages approximate expectations.
Independent simulated draws can estimate an integral or expected payoff. Rare-event probabilities may converge too slowly for naive sampling and need variance-reduction methods.
LESSON: asymptotic validity does not guarantee efficiencyThe law has no memory and no moral force.
"Five tails make heads due."
For independent fair tosses, the next toss remains 50/50. A future proportion may move closer to one half because the existing imbalance is diluted, not because later tosses compensate.
"Everything evens out quickly."
There is no theorem promising balance over a short, chosen window. Long runs and large deviations remain possible.
"Millions of rows remove uncertainty."
Sampling error can shrink while bias, confounding, measurement error, and dependence dominate.
"The mean describes everyone."
A stable aggregate can hide heterogeneous subgroups, multimodality, inequality, and tail risk.
From games of chance to modern probability.
Ars Conjectandi, published posthumously, proves that observed proportions concentrate near underlying probabilities in repeated trials.
Develops and names a "law of large numbers" while extending probability methods to broader settings.
Uses an inequality based on variance to establish a general weak law under tractable conditions.
Almost-sure convergence is formalized and strong-law results reach their modern form.
Before invoking large numbers, write the sampling contract.
Which population quantity is the expectation meant to represent?
What process produced observations, and can it drift?
What clusters, networks, or common shocks reduce effective sample size?
Are moments finite, and can rare values dominate the mean?
What bound, interval, or simulation describes current uncertainty?
Which errors will remain even as n becomes enormous?
Sources and further reading.
Foundational texts and modern probability treatments are separated from popular summaries.
- Jacob Bernoulli (1713) - Ars ConjectandiThe posthumously published foundational treatment containing Bernoulli's theorem on repeated trials.Smithsonian Libraries digital primary source
- Berkeley Statistics 201A - Probability lecture notesA graduate treatment of modes of convergence, weak laws, concentration, and related probability foundations.University of California, Berkeley
- Philip B. Stark - The Long Run and the Expected ValueAn accessible but precise account of expected value, repeated trials, and long-run concentration.University of California, Berkeley
- MIT OpenCourseWare - Probabilistic Systems AnalysisCourse materials covering laws of large numbers, central limit behavior, and probabilistic modeling.Massachusetts Institute of Technology
- Rick Durrett (2010) - Probability: Theory and ExamplesA rigorous reference for convergence concepts and strong-law results under broader conditions.Cambridge University Press