Testing and Correcting Gender Bias in Healthcare AI

The same distinction applies to corrective methods. Adding records, resampling underrepresented groups, generating synthetic examples, or changing model-training procedures may improve performance, but the intent to reduce bias is not evidence that bias has been reduced. Each intervention must be evaluated using clinically relevant measures, across appropriate subgroups, on data independent of model development, and eventually in the environment where the system will be used.

This shifts the objective from broadly promoting “fair AI” to defining measurable equity. Developers and healthcare organizations need to know which populations are represented, where evidence remains weak, which errors occur most often, who bears the consequences of those errors, and whether performance changes after deployment. Without that evidence, claims of fairness remain largely aspirational.

The phrase “missing data” can describe several distinct problems. A population may be numerically underrepresented because its members were less likely to participate in research, access care, receive testing, or have their records digitized. Relevant information may never have been collected, or it may have been recorded inconsistently across sites. Patients may appear in a data set but be assigned to categories that obscure meaningful differences — including sex and gender. Their diagnoses and outcomes may also reflect disparities embedded in the healthcare system that produced the data.

These problems matter because AI models do not learn disease independently of the clinical record. They learn relationships among the variables, labels, images, outcomes, and decisions available in that record. If women were less likely to receive a diagnosis, were diagnosed later, or were evaluated using criteria developed primarily in men, the resulting data set may encode those patterns as though they were neutral features of the disease.

Medical imaging provides a clear example of how representation can affect model performance. In an analysis of chest X-ray classifiers trained under different demographic conditions, models generally performed less accurately for the group that was underrepresented in the training data. The result does not establish that every imbalance produces the same effect in every clinical application, but it demonstrates that data set composition can create measurable performance differences between groups.³

Underrepresentation, however, is only one source of distortion. Data can also be incomplete because care itself is uneven. Some patients may have fewer encounters, delayed referrals, less access to specialists, or lower rates of diagnostic testing. Others may have extensive records because they are already receiving intensive care. A large data set assembled from routine clinical information may therefore reflect patterns of healthcare utilization as much as underlying biology.

Labels introduce another layer of risk. Many supervised machine learning systems are trained using an assigned outcome, such as whether disease is present, whether hospitalization occurred, or whether a clinician made a particular diagnosis. Those labels may appear objective even when they depend on access to testing, physician judgment, coding practices, or historical definitions of disease. If women are more likely to have symptoms dismissed or attributed to another cause, the absence of a diagnosis in the record may not mean the absence of disease.

Categorization can compound the problem. Sex and gender are often recorded inconsistently, used interchangeably, or compressed into a single binary field. In other cases, demographic information may be missing, inferred from other variables, or aggregated to simplify analysis. Such practices can make subgroup evaluation easier to perform statistically while making the results less meaningful clinically.

A data set can therefore include equal numbers of women and men and still fail to represent either group accurately. Numerical balance cannot correct biased labels, missing variables, inconsistent measurement, or differences in the care pathways that generated the records. Standards for healthcare AI data sets increasingly emphasize transparency about who is included, why data were collected, how labels were assigned, how missing information was handled, and which populations may face a greater risk of harm.²

The practical implication is that developers must investigate the history of the data, not merely its dimensions. A model trained on millions of records is not necessarily more equitable than one trained on thousands. Scale can increase statistical power, but it can also reproduce a flawed pattern with greater confidence.

When an existing data set does not capture the information required to answer a clinical question, the strongest remedy may be to collect better data prospectively. This begins before model training, with a clear explanation of which sex-related biological attributes, gender-related social factors, or other patient characteristics are relevant to the intended use.

Sex and gender should not be treated as interchangeable terms. Sex generally refers to biological attributes, while gender encompasses socially constructed identities, roles, behaviors, and relationships. Both may influence health, but they do so through different pathways, and the appropriate variables depend on the question being studied. A model concerned with drug metabolism may require different information from one evaluating access to care, symptom reporting, or treatment adherence.⁴

A field labeled “sex” may also be too ambiguous to support the analysis a developer intends to perform. Depending on how the information was collected, it could refer to sex assigned at birth, a current legal designation, self-reported sex, or another concept. It cannot automatically be assumed to describe anatomy, hormone status, reproductive history, pregnancy, or gender identity. Researchers should define which dimensions they need, why they matter, and how they will be measured.⁵

Prospective planning also makes it possible to ensure that the study can support subgroup analysis. If a development program waits until the final evaluation to consider sex, it may discover that relevant groups are too small, variables are too poorly defined, or outcomes are too sparse to support reliable conclusions. Guidance on sex as a biological variable places consideration of sex across the research life cycle, including study design, data collection, analysis, reporting, interpretation, validation, and generalizability.⁶

Better prospective data will not solve the immediate limitations of every existing data set, so developers also need methods for curating and supplementing available information. That work should begin with documentation. Data set developers should describe where the data originated, which populations contributed, which groups are absent or sparsely represented, how variables were defined, how missing values were handled, and whether labels may reflect disparities in care. They should also distinguish the population represented by the data set from the broader population in which the model may ultimately be used.

Data linkage may recover information distributed across separate systems, although it can introduce its own governance, interoperability, and matching challenges. Targeted collection can fill specific gaps when a group is underrepresented or when existing records lack a clinically relevant variable. Resampling can alter the weight given to different groups during training, but it cannot create information that was never captured or correct inaccurate labels.

Synthetic data offer another possible strategy. Generative models can create artificial examples designed to resemble records or images from real patients, potentially expanding the representation of groups for which data are scarce. In several medical imaging tasks, synthetic augmentation improved classifier performance and reduced some subgroup disparities under distribution shifts. Those findings show that generative methods may help under defined experimental conditions, particularly when the available data are imbalanced.⁷

Synthetic data must nevertheless be treated as an augmentation tool rather than a substitute for representative clinical evidence. A generative model learns from its source data. If those data omit important features, contain biased labels, or fail to reflect the diversity of the intended population, the synthetic output may reproduce those limitations. Artificially increasing the number of examples from a group does not guarantee that those examples capture the group’s clinical variability.

For that reason, improvements achieved with synthetic augmentation must be tested on independent real-world data. A model should not be declared more equitable because it performs better on synthetic examples generated from the same underlying information used during development. The relevant question is whether the intervention improves clinically meaningful performance for real patients who were not part of the model-building process.

Similar caution applies to other technical remedies. Reweighting changes the influence of individual training examples, typically giving greater weight to underrepresented groups or outcomes so that they contribute more strongly to model learning. Adversarial training pairs the primary predictive model with a second model that tries to identify a protected characteristic, such as sex or gender, from its outputs; the primary model is then trained to make accurate predictions while limiting the information available to that adversary. Federated learning allows multiple institutions to train a shared model without transferring all patient-level data into a central repository, potentially broadening the range of populations represented while preserving local control of sensitive information. Each approach may have useful roles in particular settings, but none should be presented as an automatic correction for gender bias. A technically sophisticated intervention can still optimize the wrong outcome, conceal a disparity behind an average, or improve performance for one population while degrading it for another. The remedy must be judged by its results.^8,9

Data set composition and model fairness are closely related, but they are not equivalent. A balanced development set can reduce one obvious source of bias while leaving others intact. A model may still perform differently across groups because the available variables do not capture the same clinical information, the labels have different levels of accuracy, disease prevalence differs, or the relationship between the measured features and the outcome is not uniform.

The distinction becomes especially important when developers report only overall accuracy or a single summary measure. Aggregate results combine performance across all patients. Strong results in a large or easier-to-classify group can offset poor results in a smaller group, leaving the total performance apparently acceptable.

A sex-stratified analysis of machine learning models for liver disease prediction illustrates this problem. Models with overall accuracies above 70% produced higher false-negative rates among female patients across the classifiers evaluated. Some models that appeared effective by aggregate measures also showed among the largest differences in false-negative performance between the sexes.¹⁰

A false negative in a disease-prediction model means that the system fails to identify a patient who has the condition. Depending on how the model is used, that error could delay additional testing, referral, diagnosis, or treatment. A performance disparity may therefore have clinical importance even when it has only a modest effect on the model’s overall accuracy.

The study also demonstrates why better representation must be connected to better outcomes. Balancing the number of women and men in the training set would be useful only if it reduced the relevant performance gap without producing unacceptable new errors. The evaluation should examine what changed, for whom, and with what clinical consequences.

Different corrective methods may also produce different definitions of success. One intervention might equalize sensitivity while reducing specificity. Another might improve average performance for women while leaving a particularly vulnerable subgroup behind. A third might improve calibration while producing little change in classification errors. These outcomes cannot be reduced to a single universal fairness score.

Developers must instead define the clinical purpose of the model and identify which errors matter most in that context. A screening tool designed to identify patients for further evaluation may place particular emphasis on sensitivity and false negatives. A system that recommends an invasive procedure may need to control false positives more strictly. A risk score used to guide treatment should provide estimates that correspond closely to observed outcomes across relevant groups.

Fairness claims should therefore be tied to a clearly described use case, population, threshold, and set of consequences. A model is not equitable in the abstract. It performs more or less equitably for particular patients, under particular conditions, when used to support a particular decision.

A rigorous evaluation should begin with conventional measures of model performance but report them separately for relevant subgroups. Depending on the application, these measures may include sensitivity, specificity, positive predictive value, negative predictive value, false-positive rates, false-negative rates, discrimination, calibration, and performance at the decision threshold that will be used in practice. Draft life cycle guidance for AI-enabled medical devices from the U.S. Food and Drug Administration (FDA) recommends describing data set representativeness, prespecifying performance measures, evaluating relevant patient subgroups, and reporting counts of true-positive, false-positive, true-negative, and false-negative results where appropriate.¹¹

No single measure provides a complete picture. Sensitivity describes how often a model identifies patients who have the condition, while specificity describes how often it correctly identifies those who do not. Positive and negative predictive values reflect the probability that a result corresponds to the patient’s actual status, and they may change when disease prevalence differs between populations or settings.

Calibration asks a different question: whether predicted probabilities correspond to observed outcomes. A model may rank patients correctly from lower to higher risk while systematically overestimating or underestimating the absolute risk for one group. If treatment decisions depend on a risk threshold, poor calibration can alter who receives an intervention even when the model’s overall discrimination appears strong.

Error rates should be interpreted according to their clinical consequences. False negatives can contribute to missed disease, delayed treatment, or false reassurance. False positives can produce anxiety, unnecessary procedures, additional cost, and exposure to treatment risks. The appropriate balance depends on the application, and developers should explain why the chosen measures and thresholds fit the intended use.

Research on chest radiograph algorithms further demonstrates the importance of examining underdiagnosis across populations rather than relying only on aggregate metrics. Models can show differences in false-negative performance among demographic and underserved groups, raising the possibility that an apparently effective tool may distribute its errors unevenly.¹²

Subgroup analysis should also extend beyond a single female–male comparison when the use case and data permit. Women do not form a clinically uniform group. Age, race, ethnicity, socioeconomic status, disability, geography, pregnancy, and other biological or social factors may influence disease, access to care, measurement quality, and model performance. Sex- and gender-based analysis therefore includes consideration of how these factors interact rather than treating them as isolated characteristics.⁴

Intersectional evaluation must still be designed carefully. Dividing a data set into many small groups can generate unstable estimates, wide uncertainty, and apparent differences that may not be reproducible. Developers should identify clinically relevant subgroups in advance where possible, ensure adequate sample sizes, report uncertainty, and avoid presenting exploratory findings as conclusive.

The absence of sufficient data is itself an important result. If a model cannot be evaluated reliably in pregnant patients, older women, a racial or ethnic subgroup, or another population expected to encounter the tool, that limitation should be stated clearly. Uncertainty should not be converted into an assumption of equivalent performance.

Evaluation should also compare interventions against a defined baseline. If a developer introduces synthetic augmentation, resampling, or another mitigation method, the analysis should show whether subgroup sensitivity, calibration, and error rates improved, whether overall performance changed, and whether the intervention created new disparities elsewhere. This allows fairness to be treated as an empirical result rather than a property inferred from the method selected.

Even detailed subgroup testing can provide false confidence if the same institutions, populations, or data-generation processes are represented throughout model development and evaluation. Internal testing can show how a model performs on data similar to those it has already encountered, but it offers limited evidence that the model will generalize to a different hospital, patient population, device, or workflow.

External validation evaluates performance using patients whose data were not used to develop the model. Strong external studies may also include sites, geographic regions, clinical settings, or equipment that differ from those represented in training. This separation helps reveal whether the model has learned robust clinical relationships or patterns specific to the development environment.¹¹

External validation should reproduce the subgroup analyses used during development. A model that performed similarly in women and men at one site may encounter different documentation practices, prevalence, referral patterns, or demographic compositions elsewhere. Aggregate performance at the new site could remain stable while a subgroup disparity emerges.

Validation should also reflect the intended workflow. A model evaluated retrospectively on a curated data set may perform differently when embedded in routine care, where data arrive may at different times, variables may be missing, clinicians respond to recommendations, and patients move through complex pathways. The practical effect of an AI tool depends on both the model and the system in which it operates.

Healthcare organizations adopting externally developed AI should evaluate whether the supporting evidence matches their patients and the clinical environment. Procurement review should include the composition of the training and validation data sets, definitions of demographic variables, subgroup performance, intended users, decision thresholds, known limitations, update procedures, and the vendor’s plan for monitoring performance after implementation.

Local evaluation does not necessarily mean repeating every development study. The appropriate level of testing will depend on the model’s risk, purpose, evidence base, and degree of similarity between the validation setting and the adopting institution. A low-risk administrative tool may require a different process from a system used to influence diagnosis or treatment. The central requirement is that the health system understands where the evidence applies and where important uncertainty remains.

Local oversight can also address tools that change after deployment or operate outside traditional models of medical product review. Hospitals and clinics are positioned to observe whether the system functions within their workflows, whether clinicians use it as intended, and whether particular patient groups experience unexpected errors. Proposals for local AI oversight typically emphasize the need to complement centralized regulation with governance at the institutions where algorithms affect care.¹³

Responsibility should be assigned before problems occur. Developers may control model architecture and updates, while health systems control implementation, workflow integration, access, and local monitoring. Clinicians observe individual failures, but they may not see broader patterns without a reporting system. Governance structures should define who reviews subgroup performance, who investigates discrepancies, and who has authority to restrict, recalibrate, or suspend the tool.

Validation provides evidence about performance at a particular time, using particular data. It does not guarantee that those results will persist. Patient populations change, clinical practices evolve, equipment is replaced, documentation standards shift, and new treatments alter the relationship between predictors and outcomes. A model that performed equitably during validation can develop disparities later if those changes affect groups differently.

This phenomenon is often described as distribution shift. The data encountered after deployment no longer match the data on which the model was developed or validated. The shift may involve obvious technical changes, such as a new imaging device, or subtler changes in patient demographics, disease prevalence, comorbidities, or referral patterns.

Research on postmarket surveillance has tested methods for detecting changes in medical AI inputs, including shifts in patient sex and ethnicity. The results suggest that demographic shifts can be more difficult to detect than visually apparent changes in imaging data and that different monitoring methods vary in their sensitivity to particular types of change.¹⁴

Monitoring should therefore include both the model’s inputs and its outcomes. A health system can track whether the patient population is changing, whether missing data are becoming more common, and whether the distribution of key variables differs from the validated population. It should also monitor sensitivity, specificity, calibration, and error rates when reliable outcome data become available.

Subgroup performance remains essential after launch. An overall decline may be easy to detect, but a disparity affecting a smaller population can remain hidden within stable aggregate results. Monitoring plans should identify the groups and measures that matter for the intended use, establish a schedule for review, and define thresholds that trigger investigation.

The response to deterioration will depend on its cause and severity. Some models may require recalibration to reflect a change in prevalence. Others may need retraining on more recent or locally representative data. A workflow problem may call for additional training or a change in how recommendations are presented. In more serious cases, use may need to be restricted or suspended until the risk is understood.

A clinical AI quality-improvement framework must also ensure that performance information reaches the people responsible for care. Clinicians should know when a system’s evidence is outdated, when subgroup results are uncertain, and when an update changes expected behavior. Governance teams need records of model versions, changes, incidents, investigations, and corrective actions. Continuing monitoring, revision, communication, and possible suspension should be treated as normal components of AI quality management rather than exceptional responses to failure.¹⁵

Updates themselves require equity testing. Retraining a model on newer data can improve average accuracy while worsening performance for a subgroup. A change intended to correct one disparity may create another. Each material update should therefore undergo an evaluation proportionate to its potential clinical effect, including comparison with the previous version across relevant groups.

The path from missing data to measurable equity has no single endpoint. Better representation can reduce important gaps, but it cannot erase the history embedded in clinical records. Technical mitigation can improve performance, but it cannot be assumed to work outside the setting in which it was developed. Validation can reveal disparities, but it captures only a moment in a changing healthcare environment.

A credible claim of equitable healthcare AI must rest on a chain of evidence: clearly defined variables, transparent data provenance, representative development and validation populations, clinically meaningful subgroup metrics, independent testing, local oversight, and continuing surveillance. The objective is not to produce a data set that looks balanced or a model that satisfies a single mathematical definition of fairness. It is to determine, with evidence, whether the system supports safe and effective care for the patients expected to rely on it, and to keep asking that question throughout its useful life.

1. Alvaro, David. “The Gender Bias Built Into AI — And Its Threat to Women’s Health.” Pharma’s Almanac. 14 Apr. 2025.

2. Arora, Anmol, et al. “The Value of Standards for Health Datasets in Artificial Intelligence-Based Applications.” Nature Medicine. 29: 2929–2938 (2023).

3. Larrazabal, Agostina J, et al. “Gender Imbalance in Medical Imaging Datasets Produces Biased Classifiers for Computer-Aided Diagnosis.” Proceedings of the National Academy of Sciences of the United States of America. 117: 12592–12594 (2020).

4. “How to Integrate Sex and Gender into Research.” Canadian Institutes of Health Research. 21 Aug. 2019.

5. Lowik, AJ, et al. “Meet the Methods Series: Measuring Research Participants’ Sexes and Genders.” Canadian Institutes of Health Research. 26 Nov. 2025.

6. “Sex as a Biological Variable.” NIH Office of Research on Women’s Health. 17 Oct. 2025.

7. Ktena, Ira, et al. “Generative Models Improve Fairness of Medical Classifiers under Distribution Shifts.” Nature Medicine. 30: 1166–1173 (2024).

8. Zhang, Brian Hu, Blake Lemoine, and Margaret Mitchell. “Mitigating Unwanted Biases with Adversarial Learning.” Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. 335–340 (2018).

9. Teo, Zhen Ling, et al. “Federated Machine Learning in Healthcare: A Systematic Review on Clinical Applications and Technical Architecture.” Cell Reports Medicine. 5: 101481 (2024).

10. Straw, Isabel, and Honghan Wu. “Investigating for Bias in Healthcare Algorithms: A Sex-Stratified Analysis of Supervised Machine Learning Models in Liver Disease Prediction.” BMJ Health & Care Informatics. 29: e100457 (2022).

11. Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations: Draft Guidance for Industry and Food and Drug Administration Staff. U.S. Food and Drug Administration.7 Jan. 2025.

12. Seyyed-Kalantari, Laleh, et al. “Underdiagnosis Bias of Artificial Intelligence Algorithms Applied to Chest Radiographs in Under-Served Patient Populations.” Nature Medicine. 27: 2176–2182 (2021).

13. Herman, Daniel S, Jenna T Reece, and Gary E Weissman. “Lessons for Local Oversight of AI in Medicine from the Regulation of Clinical Laboratory Testing.” npj Digital Medicine. 7: 359 (2024).

14. Koch, Lisa M, Christian F Baumgartner, and Philipp Berens. “Distribution Shift Detection for the Postmarket Surveillance of Medical AI Algorithms: A Retrospective Simulation Study.” npj Digital Medicine. 7: 120 (2024).

15. Feng, Jean, et al. “Clinical Artificial Intelligence Quality Improvement: Towards Continual Monitoring and Updating of AI Algorithms in Healthcare.” npj Digital Medicine. 5: 66 (2022).