AI Risks and Benefits Scale

Abstract

The AI Risks and Benefits Scale (ARBS; Kerstan, Bienefeld, & Grote, 2024) is a psychometric instrument designed to evaluate cognitive appraisals of risks and benefits associated with the integration of artificial intelligence (AI) in healthcare. Developed at the Department of Management, Technology, and Economics at ETH Zurich, the scale captures general user sentiments toward clinical AI technologies without restricting evaluation to a single diagnostic platform or clinical specialty. The instrument comprises 19 self-report items distributed across two primary subscales: Risk Perceptions (10 items) and Benefit Perceptions (9 items). Respondents assess each item using a 7-point Likert response format ranging from 1 (Very unlikely) to 7 (Very likely) in response to the common stem: “In your opinion, how likely are the following risks or benefits to occur? AI in healthcare might…”

Psychometric evaluation demonstrates strong internal consistency, with Cronbach’s alpha coefficients of α = 0.87 for the Risk Perceptions subscale and α = 0.84 for the Benefit Perceptions subscale. Exploratory factor analysis (EFA) supports a bifurcated structure for risk facets alongside a coherent unidimensional structure for perceived benefits. Subsequent confirmatory factor analysis (CFA) confirms that modeling risks and benefits as distinct, correlated latent dimensions provides superior fit over unidimensional formulations (χ² = 1008.46, df = 547, p < .001, χ²/df = 1.84, CFI = 0.91, TLI = 0.90, RMSEA = 0.04, SRMR = 0.06). The ARBS offers healthcare administrators, human-computer interaction (HCI) researchers, and organizational psychologists a validated tool to measure public and patient readiness, anticipate algorithmic resistance, and examine the psychological determinants governing human-AI teaming in clinical environments.

Keywords

Artificial Intelligence, Healthcare Delivery, Risk Perceptions, Benefit Perceptions, Algorithmic Aversion, Human-Computer Interaction, Psychometrics, Medical Decision-Making, Patient Acceptance, Technology Assessment

Authors

The AI Risks and Benefits Scale was developed by researchers affiliated with the Chair of Work and Organizational Psychology within the Department of Management, Technology, and Economics at ETH Zurich (Switzerland):

• Sophie Kerstan — Department of Management, Technology, and Economics, ETH Zurich, Zurich, Switzerland. Correspondence email: [email protected].
• Nadine Bienefeld — Senior Researcher and Lecturer, Work and Organizational Psychology, Department of Management, Technology, and Economics, ETH Zurich, Zurich, Switzerland.
• Gudela Grote — Professor of Work and Organizational Psychology, Department of Management, Technology, and Economics, ETH Zurich, Zurich, Switzerland.

Purpose

The transition toward digital medicine relies heavily on machine learning algorithms for diagnostics, prognostic modeling, triage, and automated therapeutic recommendations. Despite technical accuracy exceeding or matching human clinicians in restricted benchmark tasks, clinical implementation frequently encounters hesitation from patients and healthcare providers. The AI Risks and Benefits Scale was developed to quantify prospective users’ subjective likelihood assessments regarding positive and negative outcomes of healthcare AI deployment.

Prior psychological investigations often assessed technology acceptance through broad frameworks such as the Technology Acceptance Model (TAM) or the Unified Theory of Acceptance and Use of Technology (UTAUT). While useful, these generic frameworks often aggregate affective reactions and perceived usefulness into high-level constructs that obscure domain-specific healthcare dynamics, such as fear of dehumanization, algorithmic diagnostic errors, data breaches, and medical depersonalization. The ARBS addresses these considerations by sampling explicit concerns identified in clinical literature (e.g., Blease et al., 2019) and testing their prevalence across non-specialist populations.

Researchers and practitioners use the ARBS for several distinct purposes:

• Predicting Provider and Patient Uptake: Disentangling risk perception from benefit perception allows researchers to determine which factor acts as the stronger driver or barrier in clinical decision-making, such as opting for an AI-generated versus a human-generated treatment regimen.
• Guiding Ethical Deployment and Public Communication: Healthcare organizations can administer the ARBS to identify community-level concerns (e.g., job displacement versus data insecurity), permitting targeted transparency campaigns and ethical policy formulation prior to clinical rollouts.
• Assessing Interventions: Educational modules, participatory algorithmic design sessions, and interactive clinical demonstrations can be systematically evaluated using pre- and post-intervention ARBS administrations to track shifts in risk-benefit cognitive trade-offs.

Psychological Construct

The core construct captured by the ARBS is the dual-dimensional cognitive appraisal of healthcare artificial intelligence. Grounded in the behavioral economics concept of risk-benefit calculus, individuals evaluate emerging, uncertain technologies not through an integrated, unidimensional evaluation, but by weighing distinct, coexisting vectors of prospective harm against prospective utility.

1. Risk Perceptions

Perceived risk reflects an individual’s subjective expectation of suffering loss, harm, or unintended negative consequences following the systemic introduction of AI into healthcare delivery. The ARBS operationalizes risk perceptions along several key facets:

• Epistemic and Diagnostic Inaccuracy: Beliefs that algorithmic architectures can generate false outputs, miscalculate vital indicators, or produce unreliable recommendations (e.g., Item R8: “… increase medical errors”; Item R9: “… provide unreliable information”; Item R10: “… measure patient parameters inaccurately”).
• Depersonalization and Dehumanization: Concerns that automated systems evaluate patients through rigid numerical abstractions, disregarding individual context, somatic variability, and empathic interpersonal connection (e.g., Item R1: “… fail to recognize the uniqueness of each patient’s condition”; Item R7: “… dehumanize care”).
• Systemic and Ethical Vulnerabilities: Apprehensions regarding organizational vulnerability, including malicious cyber-intrusions, systemic software failure, and broader ethical ambiguities (e.g., Item R3: “… increase patient data security breaches”; Item R4: “… lead to ethical problems”; Item R5: “… introduce new bugs and equipment failures”).
• Societal and Occupational Disruption: Appraisals regarding deskilling, clinical disempowerment, and workforce displacement (e.g., Item R2: “… result in a loss of jobs for healthcare professionals”; Item R6: “… result in overreliance on technology”).

2. Benefit Perceptions

Perceived benefit represents the subjective probability that AI integration will augment clinical outcomes, improve healthcare access, and enhance therapeutic efficiency. These items reflect expectations of operational optimization and augmented human capability:

• Diagnostic and Predictive Augmentation: Confidence that computational pattern recognition can identify latent pathologies and prevent adverse outcomes earlier than traditional procedures (e.g., Item B1: “… reduce medical errors”; Item B3: “… facilitate the prediction of negative health events”; Item B8: “… help in diagnosing health problems”).
• Therapeutic Optimization: Expectations of individualized pharmacotherapy and continuous efficacy tracking (e.g., Item B2: “… enable a more personalized care”; Item B7: “… help in monitoring treatment efficiency”; Item B9: “… help in choosing adequate treatments”).
• Systemic Accessibility and Communication: Beliefs that algorithmic processing alleviates administrative bottlenecks, democratizes clinical access, and bridges information asymmetry (e.g., Item B4: “… improve information sharing between patients and healthcare providers”; Item B5: “… improve the accessibility of care”; Item B6: “… provide more accurate health data measurements”).

Theoretical Framework

The construction of the ARBS is informed by psychological, organizational, and behavioral theories explaining human interactions with complex automated agents:

Risk-Benefit Calculus and Dual-Process Reasoning

Classical behavioral decision theory posits that under conditions of risk and ambiguity, individuals engage in an intuitive cognitive calculus weighing anticipated rewards against potential threats. Rather than operating as pure bipolar opposites along a single continuum—where a benefit is merely the mathematical negation of a risk—cognitive research demonstrated by Slovic et al. (2004) indicates that affective reactions cause risk and benefit judgments to be distinct cognitive representations. In public perception, technologies such as nuclear power, gene editing, and medical artificial intelligence regularly evoke simultaneous high-risk and high-benefit evaluations.

Algorithmic Aversion versus Algorithmic Appreciation

The ARBS draws on social-cognitive literature surrounding algorithm aversion (Dietvorst, Simmons, & Massey, 2015). Even when an algorithmic system demonstrates lower statistical error rates than human professionals, human judges often exhibit heightened sensitivity to errors produced by machines. Because human decision-makers forgive fallible peers more readily than flawed code, measuring domain-specific risk assessments (e.g., technical glitches, loss of unique context) is essential for predicting whether a user will accept or reject automated medical triage or treatment.

Trust in Automation

According to Lee and See’s (2004) conceptual framework on trust in automation, human reliance on algorithmic agency depends on three core dimensions: ability (technical competence), integrity (fidelity to ethical standards), and purpose (benevolent intent). The ARBS reflects these dimensions across its items. For instance, risk items address deficiencies in competence (Item R9, Item R10) and integrity (Item R3, Item R4), whereas benefit items align with perceived clinical efficacy and operational benevolence (Item B1, Item B5).

Validity

The structural and construct validity of the ARBS was established using an adult sample recruited via Prolific to examine choice preferences between human medical practitioners and automated algorithmic recommendations (Kerstan et al., 2024).

Construct and Structural Validity

Construct validity is substantiated by the scale’s internal factor structure. Factor-analytic modeling confirmed that risk perceptions and benefit perceptions represent empirically distinct, negatively correlated dimensions rather than a unitary, bipolar continuum. A four-factor comparative model that forced risk and benefit items onto a combined construct demonstrated a statistically significant decrement in model fit, supporting the theoretical independence of positive and negative appraisal mechanisms in clinical automation.

Discriminant and Convergent Validity

Item pairs addressing inverse concepts (such as Item R8: “… increase medical errors” and Item B1: “… reduce medical errors”) demonstrated appropriate divergent loading patterns rather than collinearity, showing that respondents differentiate between the presence of specific harms and the absence of specific benefits. The items accounted for substantial variance across general health attitudes without exhibiting redundancy with broader constructs such as general generalized optimism or broad computer anxiety.

Predictive and Criterion Validity

Kerstan et al. (2024) demonstrated the predictive utility of the ARBS in experimental choice paradigms. High scores on the Risk Perceptions subscale significantly predicted consumer preference for human clinicians over AI diagnostic tools, even when the algorithmic option was presented with superior benchmark accuracy. Conversely, elevated scores on the Benefit Perceptions subscale predicted willingness to adhere to algorithmic therapeutic suggestions and supported the implementation of remote autonomous monitoring.

Reliability

The reliability of the ARBS has been evaluated using classical test theory metrics for internal consistency across its primary subscales:

• Risk Perceptions Subscale: The 10-item risk inventory exhibited strong internal consistency, yielding a Cronbach’s alpha of α = 0.87. Corrected item-total correlations across the risk indicators remained consistently above acceptable psychometric thresholds (r > 0.40), confirming that despite addressing diverse domains (such as cybersecurity, job loss, and clinical error), the items reflect a coherent latent perception of technological risk.
• Benefit Perceptions Subscale: The 9-item benefit inventory demonstrated a Cronbach’s alpha of α = 0.84. The exclusion of one initial pilot item during scale refinement yielded a clean, highly cohesive factor characterized by strong inter-item covariation across diagnostic, accessibility, and personalization parameters.

The standard error of measurement (SEM) was observed to be low across both dimensions, supporting the instrument’s capacity to detect inter-individual variations in risk-benefit perceptions in experimental and observational settings.

Factor Analysis

The latent structure of the ARBS was tested using sequential exploratory and confirmatory factor analyses during its initial validation.

Exploratory Factor Analysis (EFA)

Initial factor structure discovery was conducted on the candidate item pool:

• Risk Item Structure: EFA on the risk items revealed an underlying two-factor structure. One factor clustered around technical and clinical safety threats (e.g., inaccurate parameter measurement, software bugs, elevated medical error rates), while the second factor reflected socio-ethical and interpersonal threats (e.g., loss of jobs, dehumanization of care, ethical conflicts, data privacy breaches).
• Benefit Item Structure: EFA on the benefit items indicated an essentially unidimensional solution, where diagnostic precision, access improvement, and therapeutic monitoring loaded on a unified general benefit factor. One initial item was eliminated during development because it split-loaded across secondary noise dimensions, resulting in the final 9-item benefit structure.

Confirmatory Factor Analysis (CFA)

Confirmatory factor analysis was subsequently conducted to test the structural integrity of the measurement model within a comprehensive five-factor structural equation model capturing clinical choice preferences. The model yielded acceptable goodness-of-fit indices:

• Chi-Square (χ²): 1008.46 (df = 547, p < .001)
• Relative Chi-Square (χ²/df): 1.84 (well within the conventional target threshold of < 3.0)
• Comparative Fit Index (CFI): 0.91
• Tucker-Lewis Index (TLI): 0.90
• Root Mean Square Error of Approximation (RMSEA): 0.04 (90% CI [0.036, 0.044])
• Standardized Root Mean Square Residual (SRMR): 0.06

To examine whether the separation of risk and benefit into independent factors was statistically necessary, a nested four-factor model collapsing risk and benefit items into a single bipolar factor was tested. The nested model produced a marked degradation in fit, confirming that risk and benefit perceptions operate as distinct latent dimensions.

Instrument / Measurement Tool

Below is a summary of the technical specifications, administrative protocol, and scoring instructions for the AI Risks and Benefits Scale:

• Test Type: Standardized self-report inventory / psychometric questionnaire.
• Format: Electronic or paper-and-pencil questionnaire; optimized for online assessment batteries and clinical surveys.
• Item Count: 19 items total (Risk Perceptions: 10 items; Benefit Perceptions: 9 items).
• Response Scale: 7-point Likert scale: 1 = Very unlikely, 2 = Unlikely, 3 = Somewhat unlikely, 4 = Neutral, 5 = Somewhat likely, 6 = Likely, 7 = Very likely.
• Prompt Stem: “In your opinion, how likely are the following risks or benefits to occur? AI in healthcare might…”
• Administration Time: Approximately 3 to 5 minutes.
• Target Population: Adults (18 years and older); applicable to both patient populations and general consumer samples.
• Scoring Protocol:
  • Subscale Scores: Calculate the mean or sum of all items belonging to each respective subscale.
    • Perceived Risk Score: Mean of items R1 through R10 (range: 1.00 – 7.00). Higher scores denote elevated threat appraisal regarding medical AI integration.
    • Perceived Benefit Score: Mean of items B1 through B9 (range: 1.00 – 7.00). Higher scores denote stronger positive expectations of medical AI efficacy.
  • No Composite Score: Because factor analysis demonstrates that risks and benefits constitute separate constructs, calculating a single aggregated composite score is psychometrically discouraged. Instead, researchers should model the subscales as separate predictors or compute an explicit risk-benefit differential variable (e.g., Benefit Mean − Risk Mean) if a trade-off index is theoretically warranted.
  • Missing Data Handling: Standard mean replacement across subscale items is permissible if missingness does not exceed 10% of items per subscale.

Permissions & Fee and Test Year

The AI Risks and Benefits Scale was published in 2024 by Sophie Kerstan, Nadine Bienefeld, and Gudela Grote from the Department of Management, Technology, and Economics at ETH Zurich. The instrument was developed under an open-access model and distributed under a Creative Commons License, permitting academic and institutional researchers to use, modify, and administer the items without licensing fees for non-commercial research purposes. Commercial deployments, clinical workflow integrations, and software embeddings should cite the primary authors and contact the corresponding author, Sophie Kerstan ([email protected]), for institutional permission.

References

• Blease, C., Kaptchuk, T. J., Bernstein, M. H., Mandl, K. D., Halamka, J. D., & DesRoches, C. M. (2019). Artificial intelligence and the future of primary care: Exploratory survey of UK general practitioners. Journal of Medical Internet Research, 21(3), e12802. https://doi.org/10.2196/12802
• Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114–126. https://doi.org/10.1037/xge0000033
• Kerstan, S., Bienefeld, N., & Grote, G. (2024). AI Risks and Benefits Scale: Measuring risk-benefit perceptions of AI-based technologies in healthcare. Department of Management, Technology, and Economics, Work and Organizational Psychology, ETH Zurich.
• Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. https://doi.org/10.1518/hfes.46.1.50_30392
• Slovic, P., Finucane, M. L., Peters, E., & MacGregor, D. G. (2004). Risk as feelings. Health Psychology, 23(2), 311–322. https://doi.org/10.1037/0278-6133.23.2.311

Items of the Scale