4 What are plausible values?
Unlike variables such as age or gender, proficiency in reading or mathematics cannot be observed directly. Instead, it is inferred from students’ responses to assessment items using Item Response Theory (IRT).
Want to learn more? Click here to explore how Item Response Theory (IRT) is used to estimate student proficiency from assessment responses.
Each student’s observed responses provide only an imperfect indication of their underlying proficiency.
Why not assign a single score?
One possibility would be to estimate a single proficiency score for each student.
However, doing so treats proficiency as though it were measured without error. In reality, there is uncertainty around each student’s estimated proficiency.
Ignoring this uncertainty leads to:
- underestimated standard errors;
- biased estimates of population distributions;
- incorrect inference for subgroup comparisons and regression analyses.
A plausible value is a random draw from the estimated distribution of proficiency for a student, conditional on:
- the student’s item responses; and
- characteristics of the students, their teachers and their schools.
This conditioning on background variables is important because plausible values are generated partly using questionnaire responses. This means that some degree of dependency is introduced between the achievement variables and the background variables. For standard descriptive and associational analyses this is exactly what the methodology is designed for and should not cause concern. However, analysts building complex structural models should be aware of this feature.
Rather than assigning a single score, AMPL provides \(5\) plausible values for each student. The table below shows the \(5\) plausible values for reading and mathematics proficiency for the first two students in the data.
| Reading | Student 1 | Student 2 | Mathematics | Student 1 | Student 2 |
|---|---|---|---|---|---|
| pv1_read | 0.769 | -1.638 | pv1_math | 0.228 | -2.625 |
| pv2_read | 0.800 | -0.881 | pv2_math | 0.463 | -1.674 |
| pv3_read | 1.054 | -1.383 | pv3_math | 0.892 | -1.763 |
| pv4_read | 1.198 | -1.116 | pv4_math | 0.600 | -2.215 |
| pv5_read | 0.948 | -1.126 | pv5_math | 0.769 | -1.637 |
These plausible values should be viewed as multiple imputations of latent proficiency
How are plausible values combined?
Suppose we wish to estimate a population parameter \(Q\) such as a mean mathematics score, using \(M\) plausible values.
First, the statistic is estimated separately using each plausible value:
\[ Q_1, Q_2, Q_3, Q_4, Q_5 \]
where \(Q_m\) is the estimate obtained using plausible value \(m\).
Step 1: Calculate the final point estimate
The final estimate is simply the average across the plausible values: \[ \bar{Q} = \frac{1}{M} \sum_{m=1}^{M} Q_m \] For AMPL: \[ \bar{Q} = \frac{Q_1 + Q_2 + Q_3 + Q_4 + Q_5}{5} \] For example, if the five plausible-value means were:
| Plausible value | Mean |
|---|---|
| PV1 | -1.452 |
| PV2 | -1.454 |
| PV3 | -1.447 |
| PV4 | -1.456 |
| PV5 | -1.448 |
then \[ \bar{Q} = \frac{-1.452 -1.454 -1.447 -1.456 -1.448 }{5} = -1.541 \] Step 2: Calculate the imputation variance
The imputation variance measures how much the estimates differ across plausible values: \[ B = \frac{1}{M-1} \sum_{m=1}^{M} (Q_m - \bar{Q})^2 \] where - \(B\) = imputation variance; - \(M\) = number of plausible values; - \(Q_m\) = estimate from plausible value \(m\). For AMPL: $$
B = _{m=1}^{5} (Q_m - {Q})^2 $$ If all five plausible values produce very similar estimates, \(B\) will be small. Larger values of \(B\) indicate greater uncertainty arising from the measurement of proficiency.
Step 3: Calculate the sampling variance
For each plausible value, the paired jackknife replicate weights are used to calculate a sampling variance:
\[ U_1, U_2, U_3, U_4, U_5 \]
The average sampling variance is
\[ \bar{U} = \frac{1}{M} \sum_{m=1}^{M} U_m \]
where \(U_m\) is the jackknife variance estimate obtained for plausible value \(m\).
Step 4: Calculate the total variance
The final variance combines both sources of uncertainty:
- Sampling uncertainty (from replicate weights)
- Measurement uncertainty (from plausible values)
Using Rubin’s multiple-imputation formula:
\[ T = \bar{U} + \left(1 + \frac{1}{M}\right)B \]
For AMPL:
\[ T = \bar{U} + 1.2B \]
The factor \((1 + 1/M) = (1 + 1/5) = 1.2\) slightly inflates the imputation variance to account for the fact that only a finite number of plausible values (\(M=5\)) is used. With more plausible values, this factor would approach 1.
The standard error is:
\[ SE = \sqrt{T} \]
Interpretation
The total variance has two components:
\[ T = \underbrace{\bar{U}}_{\text{sampling variance}} + \underbrace{\left(1 + \frac{1}{M}\right)B}_{\text{imputation variance}} \]
The sampling variance reflects uncertainty due to observing only a sample of students rather than the entire population. The imputation variance reflects uncertainty in students’ latent proficiency estimates.
4.1 Worked example: mean reading proficiency in The Gambia
The following example applies the four steps to estimate the weighted mean reading proficiency among Grade 3 students in The Gambia. For each plausible value, the weighted mean (\(Q_m\)) was calculated using the final weight fwgt, and the jackknife sampling variance (\(U_m\)) was calculated using the paired jackknife replicate weights rwgt1–rwgt135 (see the Stata and R chapters for the replicate-weight procedure). Scores are on the Learning Progressions Scale (LPS).
Step 1: Point estimate
For each plausible value, calculate the weighted mean. The final point estimate is the average of the five means:
| Plausible value | Weighted mean (\(Q_m\)) |
|---|---|
| pv1_read | −1.361849 |
| pv2_read | −1.356652 |
| pv3_read | −1.361081 |
| pv4_read | −1.357076 |
| pv5_read | −1.358671 |
| Final estimate (\(\bar{Q}\)) | −1.359066 |
\[ \bar{Q} = \frac{-1.361849 + -1.356652 + -1.361081 + -1.357076 + -1.358671}{5} = -1.359066 \]
Step 2: Sampling variance
For each plausible value, calculate the jackknife sampling variance from the 135 replicate weights:
\[ U_m = \sum_{r=1}^{135}(\hat{\theta}_{m(r)} - \hat{\theta}_m)^2 \]
(with AMPLab’s JK2 variance factor equal to 1). The average sampling variance is:
| Plausible value | Sampling variance (\(U_m\)) |
|---|---|
| pv1_read | 0.0008892 |
| pv2_read | 0.0008369 |
| pv3_read | 0.0009350 |
| pv4_read | 0.0008816 |
| pv5_read | 0.0008522 |
| Average (\(\bar{U}\)) | 0.0008790 |
\[ \bar{U} = \frac{0.0008892 + 0.0008369 + 0.0009350 + 0.0008816 + 0.0008522}{5} = 0.0008790 \]
Step 3: Imputation variance
Calculate the squared deviation of each plausible-value mean from \(\bar{Q}\), sum the deviations, and divide by \(M - 1 = 4\):
| Plausible value | \(Q_m\) | \((Q_m - \bar{Q})^2\) |
|---|---|---|
| pv1_read | −1.361849 | 0.00000775 |
| pv2_read | −1.356652 | 0.00000583 |
| pv3_read | −1.361081 | 0.00000406 |
| pv4_read | −1.357076 | 0.00000396 |
| pv5_read | −1.358671 | 0.00000016 |
| Sum | 0.00002175 |
\[ B = \frac{1}{4}\left(0.00002175\right) = 0.00000544 \]
The five plausible-value means are very close to one another, so the imputation variance is small.
Step 4: Total variance and standard error
Combine sampling and measurement uncertainty using Rubin’s formula:
\[ T = \bar{U} + \left(1 + \frac{1}{M}\right)B = \bar{U} + 1.2B \]
\[ T = 0.0008790 + 1.2 \times 0.00000544 = 0.0008855 \]
\[ SE = \sqrt{T} = \sqrt{0.0008855} = 0.02976 \]
Summary
| Quantity | Symbol | Value |
|---|---|---|
| Point estimate | \(\bar{Q}\) | −1.359 |
| Average sampling variance | \(\bar{U}\) | 0.000879 |
| Imputation variance | \(B\) | 0.000005 |
| Total variance | \(T\) | 0.000885 |
| Standard error | \(SE\) | 0.0298 |
In this example, sampling uncertainty (\(\bar{U} = 0.000879\)) accounts for almost all of the total variance. The imputation variance (\(B = 0.000005\)) is small because the \(5\) plausible values produce very similar mean estimates, but it is still included in the final standard error.