Correcting Skewed Survey Samples With Post-Stratification, Raking, and Propensity Scores

Raw survey data is almost always wrong. If a researcher polls 1,000 people at random, they will inevitably over-sample older demographics who have more free time, under-sample younger ones, and miss specific minority groups entirely. To extract truth from skewed data, statisticians do not just count the answers; they weigh them. This mathematical correction is the invisible engine powering modern data analysis, ensuring that a flawed sample can still accurately reflect a complex population.[1][2]

The mechanism of weighting is straightforward in concept: it assigns a numerical multiplier to every respondent. If young men make up 10% of the actual population but only 5% of a survey's sample, each young man's answer is multiplied by two. Conversely, if older women are overrepresented, their answers are multiplied by a fraction. A perfectly representative respondent carries a base design weight of exactly 1.0. The complexity arises in how researchers calculate these multipliers across dozens of intersecting demographic traits.[1][2]

The three primary methods used to calculate these multipliers are post-stratification, raking, and propensity score adjustment. Each solves a different mathematical crisis in data collection, and each carries distinct limitations regarding how much variance it introduces into the final dataset. The choice of method dictates the integrity of the finding, determining whether a dataset accurately reflects reality or simply amplifies a hidden bias.[1][3][7]

Post-stratification is the most intuitive and historically common method. It divides the population into mutually exclusive, cross-tabulated buckets—such as "women aged 18-24 with a college degree"—and scales the sample to match the known census data for that exact specific bucket. When the sample is large and the variables are few, post-stratification provides an exact, mathematically ideal match to the target population.[3][4][5]

However, post-stratification is mathematically fragile because it suffers from the "curse of dimensionality." If a researcher tries to balance a survey on age, gender, education, race, and income simultaneously, they create thousands of micro-buckets. If even one of those buckets has zero respondents in the sample, the math breaks down entirely. You cannot multiply zero by any weight to reach a target population. This cell sparsity failure threshold forces researchers to either collapse categories—losing nuance—or abandon the method entirely.[3][5][7]

When post-stratification collapses under the weight of too many variables, statisticians turn to raking, formally known as Iterative Proportional Fitting. Instead of looking at intersecting micro-buckets, raking looks at the margins. It adjusts the weights for age, then adjusts for gender, then adjusts for education independently, rather than requiring a respondent to perfectly represent an intersection of all three.[1][3]

Because adjusting for education might throw the previously calculated age weights slightly out of balance, the raking algorithm loops back and repeats the process. It iterates over and over until all the marginal distributions converge on the target population numbers. This allows researchers to balance 10 or more variables simultaneously without needing respondents in every possible cross-tabulated cell.[1][5]

The trade-off is that raking assumes the variables do not heavily interact in unexpected ways. If the relationship between age and income is highly skewed in the sample compared to the general population, raking on the independent margins might not fix the internal distortion. The algorithm guarantees the overall percentage of young people and the overall percentage of wealthy people will match the census, but it does not guarantee the percentage of wealthy young people will be correct.[1][3][7]

As modern surveys increasingly rely on non-probability samples—such as opt-in internet panels where there is no known sampling frame—traditional demographic weighting is no longer sufficient. This is where Propensity Score Adjustment (PSA) enters the methodology. PSA shifts the mathematical focus from *who* the respondent is to *how likely* they were to take the survey in the first place.[4][6]

Using logistic regression or machine learning models, PSA calculates a probability score for every individual based on a wide array of covariates. It then applies Inverse Probability of Treatment Weighting (IPTW). Respondents who possess traits that made them statistically unlikely to participate are given higher weights, effectively simulating a random sample out of a non-random opt-in pool.[6]

Simulation studies demonstrate that propensity score estimation methods are highly effective at reducing bias in complex observational data. By modeling the mechanism of non-response directly, PSA can handle highly fragmented, high-dimensional data that would instantly break traditional post-stratification and confuse iterative raking algorithms.[4][6]

Yet, propensity score adjustment has a fatal, unavoidable limitation: it only works if the researchers have data on the specific variables that drive participation. If people opt into a survey based on an unmeasured trait—such as baseline political trust or inherent risk tolerance—PSA cannot correct the bias. It can only balance the data on the covariates the researcher had the foresight to measure.[4][6][7]

Ultimately, weighting is a balancing act between bias and variance. Pushing weights too high to correct a severe imbalance reduces bias but inflates the margin of error, making the final data highly sensitive to the answers of a few heavily-weighted individuals. Understanding whether a dataset used raking, post-stratification, or propensity scores is essential to understanding the limits of its claims.[1][2][7]