Evidence Pack: The Accuracy of BISG and Algorithmic Demographic Imputation

When a federal regulator or civil rights litigator suspects a lender of discriminatory practices, they face an immediate data void: federal law generally prohibits creditors from collecting the race or ethnicity of non-mortgage applicants. To decide whether to launch an enforcement action or file a lawsuit, these authorities must mathematically reconstruct the missing demographics before the statute of limitations expires. For more than a decade, the standard tool for this reconstruction has been Bayesian Improved Surname Geocoding (BISG), an algorithm that calculates the probability of an individual's race by combining the demographic distribution of their last name with the racial makeup of their neighborhood. But as the volume of automated lending and algorithmic screening grows, the mathematical limits of BISG are increasingly dictating which disparities are caught and which remain hidden.[1]

The stakes for accuracy are immense. Since formally adopting the method in 2014, the Consumer Financial Protection Bureau (CFPB) has levied tens of millions of dollars in fines based on disparities identified through BISG. Yet independent audits and newer machine-learning models reveal that the legacy algorithm systematically undercounts specific minority populations, particularly Black and Multiracial individuals. As data scientists introduce new techniques—ranging from first-name enhancements to large language models (LLMs)—the regulatory consensus is fracturing over whether to stick with a transparent but flawed legacy model or adopt opaque but highly accurate AI alternatives.[1][2][3][5]

The mechanics of standard BISG are straightforward and rely entirely on publicly available data. The algorithm first queries the U.S. Census Bureau's surname list, which catalogs the racial breakdown of every last name occurring more than 100 times nationwide. It then cross-references the individual's address against the racial demographics of their specific Census block group. Using Bayes' theorem, the algorithm updates the surname-based probability with the geographic data. The CFPB concluded in its foundational methodology report that "an integrated approach to building a proxy is more accurate than either surname or geographic data individually."[1]

In aggregate, the algorithm performs exceptionally well. The RAND Corporation, which pioneered the method, notes that "BISG estimates are strongly predictive of self-reported race and ethnicity for the four largest racial and ethnic groups in the U.S." RAND reports a Concordance Statistic (C-statistic) of 0.94 for Asian/Pacific Islander and Hispanic individuals, and 0.93 for Black and White individuals. A C-statistic of 0.5 indicates random guessing, while 1.0 represents perfect prediction, meaning that in more than 90% of cases, a randomly chosen individual of a specific race will have a higher BISG probability for that race than a randomly chosen individual of a different race.[1]

However, concordance measures how well the model ranks individuals, not how accurately it classifies them in discrete terms. When researchers apply threshold rules—assigning an individual to the race with the highest probability—the error rates diverge sharply by demographic group. A 2026 evaluation using balanced samples from Florida and North Carolina voter files found that standard BISG achieved an overall discrete classification accuracy of just 68.2%. The errors are not distributed equally; they disproportionately affect minority individuals living in diverse or gentrifying neighborhoods.[3]

The primary mathematical vulnerability of BISG is its core assumption of conditional independence. The algorithm assumes that once an individual's race is known, their surname and their neighborhood are statistically independent. In reality, this assumption routinely fails. A Black family and a White family sharing the same surname do not distribute themselves across neighborhoods randomly; historical segregation and economic factors heavily influence residential patterns. When the algorithm ignores this joint distribution, it systematically misclassifies minorities living in predominantly White, wealthier neighborhoods as White.[2][3][5]

This income bias inherent in BISG has profound implications for fair lending enforcement. If a Black applicant with excellent credit living in a high-income neighborhood is denied a loan, BISG is highly likely to classify that applicant as White. Consequently, the denial is recorded as a White denial rather than a Black denial, artificially depressing the measured rate of discrimination against minority borrowers. Civil rights advocates argue that this mathematical blind spot allows lenders to escape scrutiny for algorithmic redlining.[5]

To address these shortfalls, data scientists have developed Bayesian Improved First Name Surname Geocoding (BIFSG). By incorporating first-name demographic data, BIFSG adds a third dimension to the probability calculation. Validation studies using mortgage applicant data show that BIFSG outperforms standard BISG across all major racial categories. Crucially, the largest accuracy gains occur for non-Hispanic Black individuals, the exact group for which the legacy BISG model performs the weakest.[2]

Despite the documented improvements, regulatory adoption of BIFSG has been slow. The CFPB continues to rely primarily on the standard BISG methodology for its supervisory and enforcement actions. The hesitation stems partly from the lack of a comprehensive, publicly available first-name demographic dictionary comparable to the Census Bureau's surname list, forcing researchers to rely on proprietary or piecemeal datasets. Regulators prioritize reproducibility and transparency, often preferring a slightly less accurate public model over a highly accurate proprietary one.[1][2][5]

The landscape of demographic imputation is now being disrupted by large language models. A January 2026 preprint demonstrated that off-the-shelf LLMs, such as Gemini 3 Flash and GPT-4o, can infer ethnicity from names with an accuracy that significantly exceeds standard BISG, without requiring any additional training data. By implicitly learning the associations between names and demographic characteristics across vast text corpora, these models bypass the need for static Census dictionaries.[3]

When tested on the same balanced voter file samples, LLM-based classification achieved an accuracy of 84.7%, outperforming the 68.2% accuracy of standard BISG. Furthermore, enabling extended reasoning capabilities within the LLMs improved accuracy by an additional 1 to 3 percentage points. When metadata such as party registration was included in the prompt, the accuracy climbed to 86.7%. Most importantly, the LLM approach successfully reduced the income bias that plagues BISG, correctly identifying minority individuals in wealthier neighborhoods.[3]

The LLM approach also solves a critical limitation of BISG: geographic constraint. Because BISG relies on U.S. Census data, it is useless for international disparity analysis. LLMs, however, can infer ethnicity and demographic categories globally. Validation tests using Lebanese voter registrations, Indian parliamentary records, and Chilean voter rolls demonstrated that LLMs successfully recover known population distributions in countries where naming conventions are distinctive, enabling disparity research in regions with no equivalent to the U.S. Census.[3]

Private industry is also pushing beyond legacy algorithms. Credit underwriting firms have developed proprietary neural networks designed specifically to outperform BISG in fair lending analysis. One such model, the Zest AI Race Predictor, was tested against Florida voter data and correctly identified African American individuals 74% of the time, compared to just 47% for standard BISG on the same dataset. For Hispanic individuals, the proprietary model achieved 87% accuracy versus BISG's 77%.

The divergence between regulatory tools and state-of-the-art data science creates a legal and compliance paradox. Financial institutions testing their own portfolios for disparate impact may use advanced neural networks that reveal significant racial disparities. However, if the CFPB audits that same institution using legacy BISG, the regulator's math might show no disparity at all—or vice versa. This mathematical mismatch leaves compliance officers guessing which version of reality will hold up in court.[1][5]

The accuracy of demographic imputation dictates the boundaries of civil rights enforcement in the algorithmic age. As long as federal law restricts the direct collection of race and ethnicity data in non-mortgage lending, proxies will remain the only lens through which systemic discrimination can be viewed. The transition from standard BISG to first-name enhancements, neural networks, or LLMs is not merely a technical upgrade; it determines how many marginalized individuals the justice system can mathematically see. Until regulators update their mathematical baselines, the gap between the discrimination that exists and the discrimination the government can prove will continue to widen.[1][5]