Empirical Distribution Functions and Weighted Observations

Why Empirical Distribution Functions Matter

An empirical distribution function turns a finite set of recorded values into a distribution that can be queried. For example, it can answer what fraction of observed losses are at most a threshold, what fraction lie in an interval, or what observed loss corresponds to a specified percentile. The function is a step function: its jumps occur at recorded values, and repeated values create larger jumps.

When records represent unequal amounts of exposure, treating every row equally may not represent the intended population. A record with three policy-years of exposure may contribute three times the mass of a record with one policy-year, provided the data and weighting scheme support that interpretation. The calculations are straightforward; the important judgment is what the observations and weights represent.

Empirical Distribution Function for Equal-Weight Observations

Definition: For observed values $x_1,\ldots,x_n$, each assigned equal weight, the empirical distribution function is $$ F_n(x)=\frac{1}{n}\sum_{j=1}^{n}\mathbf{1}(x_j\le x), $$ where $\mathbf{1}(x_j\le x)$ equals 1 when $x_j\le x$, and 0 otherwise. Thus $F_n(x)$ is the fraction of observations that are less than or equal to $x$.

The empirical distribution function is nondecreasing, takes values between 0 and 1, and includes an observation equal to $x$ in $F_n(x)$. Its jump at a value is the fraction of observations equal to that value. Sorting the data makes these properties easier to see, but sorting does not change the observations or their empirical distribution.

The endpoint convention matters when converting the function into an interval probability. For example, the fraction of observations in $(a,b]$ is $F_n(b)-F_n(a)$: the observations at $a$ are counted in both terms and cancel, while those at $b$ are counted only in the first term. Other endpoint choices require the corresponding strict or inclusive counts.

\widehat{P}(a<X\le b)=F_n(b)-F_n(a)

Here $\widehat{P}$ denotes a probability calculated under the empirical distribution, not a claim that the finite data are the full underlying population. When there are ties, all observations at a tied value contribute to the same jump.

Worked Example: Equal-Weight Losses and an Empirical Quantile

A hypothetical insurer records five losses, in thousands of dollars: $1,4,4,7,10$. Treat the five records as equally weighted. Find $F_n(4)$, the empirical probability that $4<X\le10$, and the empirical 80th percentile. State the empirical distribution at the observed values as a check.

Method and assumptions. Each observed loss has weight $1$, so the total weight is $n=5$. The empirical probabilities describe the five records with equal mass; no smoothing or interpolation between observed values is used.

Step 1: Calculate $F_n(4)$. Three observations, $1,4,4$, are at most 4. Therefore, $$ F_n(4)=\frac{3}{5}=0.6000. $$

Step 2: Calculate the interval probability. The observations satisfying $4<X\le10$ are $7$ and $10$. Equivalently, $F_n(10)-F_n(4)=1-\frac{3}{5}=\frac{2}{5}=0.4000$.

Step 3: Find the 80th percentile. Use the empirical quantile convention $Q(p)=\inf\{x:F_n(x)\ge p\}$. The cumulative fractions at the ordered values are $1/5$ at 1, $3/5$ at 4, $4/5$ at 7, and $1$ at 10. The first value whose cumulative fraction is at least $0.80$ is 7, so $Q(0.80)=7$ thousand dollars.

Check. At 7, $F_n(7)=4/5=0.80$, while for any $x<7$ sufficiently close to 7, $F_n(x)=3/5<0.80$. Thus the stated quantile follows the generalized-inverse definition. This method avoids interpolating to a value that was not observed.

Quantiles from a Step Function

Definition: For $0<p\le1$, the empirical $p$-quantile under the generalized-inverse convention is $$ Q_n(p)=\inf\{x:F_n(x)\ge p\}. $$ It is the smallest value at which the cumulative empirical mass reaches or exceeds $p$.

For equal-weight data, arrange the observations in nondecreasing order. The $p$-quantile is the first ordered value at which the cumulative count, divided by $n$, is at least $p$. This definition handles ties and percentile levels that fall between jumps without inventing a value between observations. In the loss-risk setting, this is the empirical counterpart of the quantile convention used for VaR.

The definition also explains why equality at the percentile threshold matters. If the cumulative mass at an observation is exactly $p$, that observation is the quantile; one does not advance to the next observed value. If a particular application specifies a different sample-quantile interpolation rule, that rule must be stated explicitly rather than silently substituted.

Weighted Empirical Distribution Functions

Definition: Suppose record $j$ has value $x_j$ and nonnegative weight $w_j$, with total weight $W=\sum_{j=1}^{n}w_j>0$. The weighted empirical distribution function is $$ F_w(x)=\frac{\sum_{j=1}^{n}w_j\mathbf{1}(x_j\le x)}{W}. $$ The record at $x_j$ contributes mass $w_j/W$. A zero-weight record contributes no mass.

A weight may represent exposure, a frequency in a replicated data set, or another specified amount of represented population. Its interpretation comes from the data design, not from the formula. If weights are exposure amounts, the weighted empirical distribution estimates the distribution for the exposure represented by those records only when the observations are appropriate representatives of that exposure. Weighting alone does not correct selection bias, missing observations, or differing risk characteristics.

Multiplying every weight by the same positive constant leaves $F_w$ unchanged, because the same factor appears in both numerator and denominator. Weights therefore determine relative mass. Weighted quantiles use the same generalized-inverse rule as before: find the smallest observed value at which the cumulative normalized weight reaches the desired percentile.

Worked Example: Unequal Weights, Ties, and a Weighted Median

Four hypothetical records have values $2,8,8,15$, with respective weights $1,1,3,1$. Find the weighted median, the weighted probability $P(8\le X<15)$, and the weighted probability $P(X\le8)$. Give probabilities to four decimal places.

Method and assumptions. Treat each weight as represented mass and normalize by the total weight. The two records at 8 must both contribute to events containing 8.

Step 1: Find the total weight and aggregate tied mass. $$ W=1+1+3+1=6. $$ At value 2 the mass is 1; at value 8 the combined mass is $1+3=4$; and at value 15 the mass is 1.

Step 2: Find the weighted median. The cumulative mass at 2 is $1/6$, which is below $0.5$. At 8 it is $(1+4)/6=5/6$, which is at least $0.5$. Therefore the smallest value reaching the 50th percentile is 8, so the weighted median is 8.

Step 3: Calculate $P(8\le X<15)$. Only the mass at 8 is included. The mass at 15 is excluded by the strict upper endpoint. Hence $$ \widehat{P}(8\le X<15)=\frac{1+3}{6}=\frac{4}{6}=0.6667. $$

Step 4: Calculate $P(X\le8)$. This includes the mass at 2 and all mass at 8: $$ F_w(8)=\frac{1+1+3}{6}=\frac{5}{6}=0.8333. $$ Check. The cumulative masses at 2, 8, and 15 are $1/6,5/6,6/6$, respectively, ending at 1 as a distribution function must. The method avoids counting tied records as only one observation or treating their weights as interchangeable with a count of rows.

Why the Weighted Formula Works

The weighted empirical distribution can be justified by viewing each record as representing its assigned share of the total mass. Normalize each weight to $w_j/W$. These shares are nonnegative and sum to one, so they define a discrete probability distribution concentrated on the observed values. The probability of an event is the sum of the shares for records whose values belong to that event.

Normalize the weights.
Assign record $j$ mass $w_j/W$. Since $W=\sum_j w_j>0$, these masses sum to one.

Identify the event records.
For the event $X\le x$, include exactly the records satisfying $x_j\le x$.

Sum their normalized mass.
The resulting probability is $\sum_{j:x_j\le x}w_j/W$, which is precisely $F_w(x)$.

This argument also gives interval probabilities: sum the normalized weights of records in the interval, taking care to honor each endpoint. Equivalently, differences of distribution-function values work when their endpoint conventions match the desired event. For example, $P(a<X\le b)=F_w(b)-F_w(a)$. A probability involving $a\le X$ requires accounting for any mass exactly at $a$.

Worked Example: Exposure-Weighted Loss Thresholds

A hypothetical portfolio has four loss records, in dollars, with the following represented exposures in policy-years: loss $1{,}200$ has weight 2, loss $2{,}500$ has weight 1, loss $6{,}000$ has weight 3, and loss $10{,}000$ has weight 2. Assume these records are suitable representatives of the exposure assigned to them. Find the exposure-weighted probability that a loss is at least $2{,}500$, the weighted 75th percentile, and the probability that a loss is strictly greater than $2{,}500$.

Method and assumptions. Use policy-years as relative empirical mass, rather than treating the four rows as equally important. The total represented exposure is $$ W=2+1+3+2=8\text{ policy-years}. $$

Step 1: Calculate the probability of a loss at least $2{,}500$. The records at $2{,}500$, $6{,}000$, and $10{,}000$ have total weight $1+3+2=6$. Thus $$ \widehat{P}(X\ge2{,}500)=\frac{6}{8}=0.7500. $$

Step 2: Find the weighted 75th percentile. In increasing order, the cumulative weights are 2 at $1{,}200$, 3 at $2{,}500$, 6 at $6{,}000$, and 8 at $10{,}000$. Dividing by 8 gives cumulative proportions $0.25,0.375,0.75,1$. The first loss whose cumulative proportion is at least $0.75$ is $6{,}000$, so $Q_w(0.75)=\$6{,}000$.

Step 3: Calculate the probability of a loss strictly greater than $2{,}500$. Exclude the weight at $2{,}500$; only the weights 3 and 2 at the larger losses count. Therefore $$ \widehat{P}(X>2{,}500)=\frac{3+2}{8}=\frac{5}{8}=0.6250. $$ Check. The difference $0.7500-0.6250=0.1250$ equals the mass at exactly $2{,}500$, namely $1/8$. This confirms that the distinction between “at least” and “strictly greater than” has been handled correctly.

Common Mistakes and Practical Checks

A frequent error is to use the number of records as the denominator when the intended distribution is weighted. In the four-record example with values $2,8,8,15$ and weights $1,1,3,1$, counting the two rows with value 8 and dividing by four gives $2/4=0.5000$ for $8\le X<15$. That is the wrong calculation for the weighted distribution: it assigns equal mass to rows despite the specified unequal weights. The correct calculation is $(1+3)/6=4/6=0.6667$. The weighted denominator is the sum of weights, and the numerator is the sum of weights for records in the event.

Other checks help catch errors before reporting a result:

Confirm the event endpoints explicitly. “At most” includes equality; “less than” does not.
Aggregate weights at tied values when reading jumps or locating quantiles.
Check that all weights are nonnegative and the total weight is positive.
Check that the empirical distribution is nondecreasing and reaches 1 at or above the largest positive-weight observation.
For a quantile, verify that the cumulative mass at the reported value reaches the requested level and that earlier values do not.
State what a weight represents. An exposure-weighted empirical distribution is not automatically representative of exposures that were not observed or appropriately modeled.

Empirical distributions are useful because they preserve the observed values and make probability and quantile calculations transparent. They do not, by themselves, smooth the data or describe values outside the observed range. Their interpretation is only as sound as the observation design, the weight definition, and the endpoint convention used in the calculation.

Check Your Understanding

Use the generalized-inverse quantile convention. Give probabilities to four decimal places where appropriate.

Equal-weight losses are $3,3,9,12$. Find $F_n(9)$, the empirical probability $P(3<X\le12)$, and the empirical 75th percentile. (Answer: $0.7500$; $0.5000$; $9$.)
Values $1,5,5,11$ have weights $2,1,2,1$, respectively. Find $P(5\le X<11)$ and the weighted median. (Answer: $0.5000$; $5$.)
Values $100,200,500$ have weights $0.5,1.5,2$, respectively. Find the weighted 50th percentile and $P(X>200)$. (Answer: $200$; $0.5000$.)
Equal-weight observations are $4,4,7,12,12,20$. Find $P(7<X\le12)$ and the empirical 80th percentile. (Answer: $0.3333$; $12$.)
Losses $0,100,300$ have exposure weights $3,2,1$, respectively. Find the exposure-weighted probability $P(X\ge100)$ and the weighted 60th percentile. (Answer: $0.5000$; $100$.)

What You'll Learn