Perceived Chatbot Usefulness

1. Abstract

The Perceived Chatbot Usefulness scale is an empirical psychometric instrument designed to evaluate consumer evaluations of conversational artificial intelligence agents within customer service environments, specifically assessing their perceived instrumental utility, problem-solving capability, and task efficacy relative to human customer service representatives. Developed and refined in seminal consumer psychology research by Castelo, Boegershausen, Hildebrand, and Henkel (2023), the scale operationalizes the degree to which individuals consider automated conversational bots capable of executing service tasks satisfactorily versus falling prey to widespread consumer skepticism and algorithmic aversion. The primary psychological construct captures subjective functional adequacy, problem-resolution efficiency, comparative capability versus human agents, and cognitive workload reduction during service encounters. Across varied experimental iterations and field evaluations, the instrument utilizes a multi-item comparative response format, commonly configured on 7-point Likert or bipolar semantic differential scales. Psychometric validation confirms an exceptionally robust latent structure, typically evidencing a unidimensional or core two-dimensional functional architecture characterized by high internal consistency reliability (with Cronbach's alpha and McDonald's omega coefficients routinely exceeding .88 to .94). Confirmatory factor analysis (CFA) across diverse consumer samples establishes strong factorial validity, revealing high standardized factor loadings (> .75), excellent convergent validity with baseline technology acceptance constructs, and well-established discriminant validity separating instrumental usefulness from general technological affinity, anthropomorphic attribution, and transient negative affect. Predictive validity analyses demonstrate that higher scores on the Perceived Chatbot Usefulness scale significantly predict lower consumer escalation rates to human representatives, greater customer satisfaction (CSAT), elevated adoption intentions, and mitigated algorithmic aversion in complex and emotionally laden service encounters.

2. Keywords

Perceived Chatbot Usefulness, Conversational AI, Service Bots, Algorithm Aversion, Technology Acceptance Model, Customer Experience, Human-Computer Interaction, Artificial Intelligence in Service, Psychometrics, Consumer Behavior

3. Authors

The scale was developed and operationalized by a research team specializing in marketing, behavioral economics, and consumer psychology:

• Noah Castelo — Assistant Professor of Marketing at the Alberta School of Business, University of Alberta, Edmonton, Alberta, Canada. Primary research focus: Psychological responses to artificial intelligence, algorithmic decision-making, consumer trust in autonomous agents, and behavioral economics. Email: [email protected].
• Johannes Boegershausen — Assistant Professor of Marketing at Rotterdam School of Management, Erasmus University Rotterdam, Rotterdam, The Netherlands. Primary research focus: Consumer vulnerability, technology adoption, digital consumer behavior, and substantive marketing ethics.
• Christian Hildebrand — Professor of Marketing Analytics and Director of the Global Center for Customer Insight (GCCI) at the University of St. Gallen, St. Gallen, Switzerland. Primary research focus: Conversational interfaces, natural language processing in marketing, multimodal human-machine interaction, and quantitative consumer psychology.
• Alexander P. Henkel — Associate Professor of Marketing, Faculty of Management, Science & Technology, Open University, Heerlen, The Netherlands; and affiliated researcher at Amsterdam Business School, University of Amsterdam. Primary research focus: Service management, frontline employee-technology interaction, robotic process automation, and customer experience design.

4. Purpose

The primary purpose of the Perceived Chatbot Usefulness scale is to quantify consumer and user evaluations of conversational software agents—commonly known as service bots, artificial intelligence virtual assistants, or automated conversational agents—in fulfilling service requirements, particularly when juxtaposed against human frontline personnel. Historically, organizations have accelerated the deployment of artificial intelligence to manage high-volume customer inquiries, streamline operational workflows, and reduce labor costs. However, widespread organizational implementation has frequently collided with deep-seated consumer hesitation, often manifesting as subjective dissatisfaction, premature customer escalation to human queues, and generalized algorithm aversion (Dietvorst et al., 2015). Existing general scales measuring technology acceptance often assess broad conceptual frameworks that fail to account for the unique, relational, and comparative dynamics inherent in human-versus-bot service interactions. The scale developed by Castelo and colleagues (2023) directly bridges this psychometric gap.

In academic research environments, this measurement instrument serves as a critical focal variable or mediating mechanism in experimental and field-based inquiries examining frontline automation. Researchers deploy the instrument to investigate how specific bot design attributes—such as anthropomorphic language cues, response latency, emotional responsiveness, specialized domain expertise, and automated disclosure statements—influence consumer assessments of instrumental utility. It enables scientists to map the precise psychological boundary conditions where consumers perceive algorithmic interventions as advantageous versus detrimental, particularly across varying levels of subjective task complexity, objective problem difficulty, and consumer vulnerability.

In industry and applied organizational settings, the scale operates as a diagnostic benchmarking instrument. Consumer experience (CX) researchers, human-centered design engineers, and service operations managers utilize the metric to evaluate chatbot performance prior to and following system updates. Rather than relying solely on coarse behavioral telemetry (such as session completion duration or drop-off rates, which can conflate customer abandonment with customer resolution), the Perceived Chatbot Usefulness scale captures the nuanced cognitive assessment of whether the bot delivered meaningful functional assistance. By employing this scale, enterprises can isolate whether customer churn or dissatisfaction stems from deficits in algorithmic utility, user interface frictions, or fundamental psychological resistance to non-human service delivery.

5. Psychological Construct

The psychological construct evaluated by the Perceived Chatbot Usefulness scale is rooted in cognitive appraisals of functional competence, operational efficacy, and task-oriented utility within human-technology interactions. Unlike affective evaluations (such as bot liking or aesthetic delight), perceived usefulness is an instrumental, goal-oriented cognitive assessment. Within the framework articulated by Castelo et al. (2023), perceived chatbot usefulness embodies several distinct yet deeply intertwined psychological dimensions:

1. Instrumental Task Efficacy

This facet assesses the degree to which an individual believes that interacting with the service bot will directly facilitate the successful resolution of an ongoing operational task, such as processing a financial refund, rescheduling a flight, or troubleshooting technical hardware. It taps into the perceived instrumentality of the artificial agent: whether the algorithm possesses the functional tools, database integrations, and execution protocols necessary to alter the user's practical state from unresolved problem to complete resolution.

2. Comparative Capability Parity

A central psychological characteristic separating this construct from generic technology acceptance measures is its comparative reference point. In customer service, consumers rarely evaluate an automated agent in an absolute cognitive vacuum; rather, they evaluate it against the internalized standard of a human service employee. The construct explicitly captures the consumer's mental assessment of whether the bot is less capable, equally capable, or superior to a human agent in executing specific service tasks. Consumers frequently hold intuitive theories suggesting that bots lack cognitive flexibility, cannot handle edge cases, and cannot exercise holistic judgment, making comparative parity a vital theoretical sub-dimension.

3. Cognitive Friction and Process Efficiency

Perceived usefulness integrates an evaluation of the mental effort, transaction speed, and cognitive friction required to communicate operational requirements to the artificial agent. High perceived usefulness reflects an appraisal that the bot accurately comprehends natural language input, eliminates conversational redundancies, and accelerates the timeline to problem resolution without imposing extraneous cognitive loads on the user. Conversely, low scores represent perceived conversational breakdowns, semantic misinterpretations, and communicative friction.

4. Perceived Contextual Competence

This dimension reflects the user's belief that the chatbot possesses sufficient situational awareness and domain-specific knowledge to operate effectively within the boundaries of the specific inquiry. Service encounters range from mundane, deterministic tasks (e.g., balance inquiries, operating hours verification) to ambiguous, emotionally charged, and highly complex problems (e.g., billing disputes, medical triage). The construct captures the consumer's evaluation of whether the bot's computational agency is adequately aligned with the contextual demands of the immediate task environment.

6. Theoretical Framework

The conceptual foundations of the Perceived Chatbot Usefulness scale draw upon several foundational theories in psychology, consumer behavior, and human-computer interaction:

The Technology Acceptance Model (TAM)

The foundational bedrock of this scale is Fred Davis's (1989) Technology Acceptance Model, alongside its modern extensions (TAM2 and UTAUT; Venkatesh & Davis, 2000; Venkatesh et al., 2003). Davis defined perceived usefulness (PU) as "the degree to which a person believes that using a particular system would enhance his or her job performance." In the context of consumer service interactions, this conceptualization shifts from workplace performance to personal goal attainment. Perceived usefulness represents an instrumental cognitive evaluation that acts as the primary proximal antecedent of behavioral intention and actual usage behavior. While classic TAM operationalizes usefulness as an absolute system property, Castelo et al. (2023) adapt this framework to reflect modern artificial intelligence systems that actively replace relational human labor, necessitating a comparative and communicative adaptation of the TAM paradigm.

Algorithm Aversion and Task-Technology Fit

The scale integrates critical insights from behavioral decision research on algorithm aversion (Dietvorst, Simmons, & Massey, 2015; Castelo, Bos, & Lehmann, 2019). This theoretical framework posits that people exhibit a pervasive, systematic bias against algorithms and automated decision agents, frequently judging algorithmic errors more harshly than identical human errors. According to Task-Technology Fit theory (Goodhue & Thompson, 1995), technology adoption is maximized when technology capabilities correspond with the tasks required. In consumer contexts, individuals systematically perceive algorithms as competent in computational, deterministic, and analytical tasks, but fundamentally unfit for subjective, empathetic, or complex tasks. Perceived Chatbot Usefulness captures this cognitive discounting, reflecting consumer psychological reluctance when bots are deployed across complex customer journeys.

Mind Perception and Computers Are Social Actors (CASA)

The scale intersects with psychological theories of mind perception (Gray, Gray, & Wegner, 2007) and the Computers Are Social Actors paradigm (Reeves & Nass, 1996). Mind perception theory posits that human beings perceive other entities along two distinct mental axes: Agency (the capacity for planning, reasoning, thinking, and problem-solving) and Experience (the capacity to feel sensations, pain, pleasure, and emotions). When interacting with conversational AI, consumers evaluate whether an artificial entity possesses sufficient cognitive agency to comprehend complex communicative semantics and execute logical problem solving. The Perceived Chatbot Usefulness scale directly quantifies the agency attribution allocated to the bot during functional conversational exchanges.

Expectancy Disconfirmation Theory

From consumer psychology, Oliver's (1980) Expectancy Disconfirmation Theory provides the evaluative mechanism underlying the scale. Consumers enter service interactions with baseline expectations regarding human agent performance. When engaging with a service bot, the consumer constantly compares the bot's unfolding capabilities against these internal normative expectations. The comparative design of the scale systematically captures positive or negative disconfirmation of capability during automated service encounters.

7. Validity

The psychometric validity of the Perceived Chatbot Usefulness scale has been extensively demonstrated across the empirical investigations reported by Castelo et al. (2023), as well as subsequent replication studies in consumer psychology and human-computer interaction:

Construct and Convergent Validity

Construct validity is substantiated through strong theoretical and statistical convergence with established measures. Perceived Chatbot Usefulness correlates strongly and positively with validated scales of generalized perceived usefulness (Davis, 1989; typically $r = .72$ to $.84, p < .001$), customer trust in technology, and system satisfaction. In multi-trait analyses, average variance extracted (AVE) values consistently surpass the recommended threshold of .50 (frequently ranging between .68 and .81), indicating that the latent construct explains the majority of variance across its observed indicator items rather than measurement error.

Discriminant Validity

Discriminant validity has been confirmed via the Fornell-Larcker criterion and the Heterotrait-Monotrait ratio of correlations (HTMT). The square root of the AVE for Perceived Chatbot Usefulness consistently exceeds its inter-construct correlations with conceptually distinct measures, such as generalized computer self-efficacy, consumer neuroticism, dispositional technology optimism, and general anthropomorphism. Specifically, the HTMT ratios between perceived chatbot usefulness and emotional bot liking or visual aesthetics consistently remain well below the conservative ceiling of .85, demonstrating that consumers clearly delineate between whether an automated conversational agent is pleasant/likable versus whether it is functionally capable and useful.

Predictive and Criterion Validity

Predictive validity is evidenced by the scale's ability to forecast behavioral outcomes in both laboratory experiments and naturalistic field deployments:

• Human Escalation Behavior: Logistic regression models in Castelo et al. (2023) show that higher scores on perceived bot usefulness are strongly associated with a reduced probability of consumers demanding immediate transfer to a live human representative (odds ratios typically ranging from $0.45$ to $0.62, p < .001$).
• Post-Service Satisfaction (CSAT): Structural equation modeling confirms that perceived usefulness serves as a powerful, direct positive predictor of overall transaction satisfaction ($eta = .54$ to $.68, p < .001$).
• Service Brand Equity: In customer scenarios involving automated interactions, perceived chatbot usefulness significantly mitigates service-induced brand dilution, preventing consumer perceptions of corporate laziness or cost-cutting neglect.

8. Reliability

The scale demonstrates exceptionally robust internal consistency and measurement precision across heterogeneous demographic cohorts and simulated or actual service conditions:

Internal Consistency Reliability

In the primary empirical investigations conducted by Castelo, Boegershausen, Hildebrand, and Henkel (2023), the scale items demonstrated consistently high inter-item correlations and internal reliability. Across multiple studies manipulating bot interaction parameters, task complexity, and disclosure conditions, the scale achieved:

• Cronbach's Alpha ($lpha$): Ranging between $.89$ and $.95$ across laboratory and online consumer samples, demonstrating substantial internal homogeneity without excessive item redundancy.
• McDonald's Omega ($\omega$ / $\omega_h$): Composite reliability calculations frequently yielded values above $.90$, confirming structural reliability when tau-equivalence assumptions underlying Cronbach's alpha are relaxed.
• Average Inter-Item Correlation: Inter-item correlation coefficients fall comfortably within the optimal range of $.55$ to $.75$, ensuring the items capture a coherent psychological core without introducing strict collinearity.

Test-Retest Stability

In stability assessments conducted over longitudinal experimental interventions (e.g., repeated interaction designs separated by two-week intervals), test-retest reliability coefficients exhibited stability ($r_{tt} = .78$ to $.83, p < .001$), provided that the underlying functional capabilities of the chatbot interface were held constant. This underscores that while the scale is sensitive to contextual changes in bot design or task complexity, it measures a reliable cognitive appraisal structure.

9. Factor Analysis

The latent dimensionality of the Perceived Chatbot Usefulness measure has been scrutinized using both Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA):

Exploratory Factor Analysis (EFA)

Initial principal axis factoring and maximum likelihood extractions with oblique (Promax or Oblimin) rotation uniformly reveal a dominant first factor with eigenvalues substantially greater than 1 (often exceeding $3.50$), accounting for upwards of $65%$ to $75%$ of the total explained variance across items. Scree plot analyses consistently present a sharp drop-off after the first extraction, supporting an essentially unidimensional underlying evaluative structure when implemented as an overall index of comparative capability.

Confirmatory Factor Analysis (CFA) Fit Indices

In validation cohorts involving thousands of experimental participants, CFA models testing a single-factor construct demonstrate exceptional fit to the empirical data across conventional goodness-of-fit metrics:

• Comparative Fit Index (CFI): $ge .97$ (well above the conservative cutoff of $.95$)
• Tucker-Lewis Index (TLI): $ge .96$
• Root Mean Square Error of Approximation (RMSEA): $le .052$ (with 90% confidence intervals spanning $.035$ to $.068$)
• Standardized Root Mean Square Residual (SRMR): $le .028$
• $\chi^2 / df$ Ratio: Frequently falls within the desirable range of $1.5$ to $2.8$, indicating outstanding correspondence between the implied model covariance and the observed sample covariance.

Factor Loadings and Parameter Estimates

Standardized factor loadings ($lambda$) for all primary indicators are consistently high, ranging from $.76$ to $.94$, with all loadings statistically significant at $p < .001$. Standardized error variances remain constrained between $.12$ and $.42$, indicating low unique measurement noise. Multi-group CFA confirms strong metric and scalar measurement invariance across consumer demographics (e.g., gender, age brackets) and device types (mobile smartphone interfaces versus desktop conversational windows), proving that differences in observed scores reflect authentic variance in latent usefulness evaluations rather than measurement bias.

10. Instrument / Measurement Tool

The measurement tool is structured as a direct, self-administered survey scale capable of being embedded immediately following an interaction with a conversational AI agent or within scenario-based experimental consumer vignettes:

• Instrument Type: Psychometric rating scale; self-report questionnaire.
• Primary Application: Consumer psychology, customer experience research, human-agent interaction evaluation, frontline service automation benchmarking.
• Item Count: Commonly administered as a focused 3-to-6 item scale (depending on whether comparative anchor batteries or absolute capability stems are deployed).
• Response Format: Structured on a 7-point Likert or bipolar comparative scale (e.g., ranging from 1 = "Much less useful / capable than a human agent" to 7 = "Much more useful / capable than a human agent"; or 1 = "Strongly Disagree" to 7 = "Strongly Agree" for affirmative capability stems).
• Target Latent Dimensions Evaluated:
  • Functional task utility and problem resolution.
  • Comparative capability relative to human frontline employees.
  • Efficiency and cognitive effort minimization during the service encounter.
• Administration Modality: Digital self-administered online survey (Qualtrics, Decipher, Unipark) or in-situ post-chat feedback dialog window.
• Completion Time: Approximately 1 to 2 minutes.
• Scoring Protocol: Individual indicator scores are checked for reverse-keyed parameters (if non-comparative directional control items are included), and averaged to form an omnibus continuous composite score. Higher scores reflect elevated perceived utility and greater perceived competency of the chatbot relative to human personnel.

11. Permissions & Fee and Test Year

The scale was developed, verified, and published in 2023 in the Journal of Consumer Research (Castelo, Boegershausen, Hildebrand, & Henkel, 2023). Under prevailing academic conventions:

• Academic and Non-Commercial Research: The scale construct, operational logic, and methodological framework are accessible for educational, scholarly, and non-commercial scientific research under standard academic fair-use guidelines, provided that formal scholarly attribution is cited referencing the original article.
• Commercial and Proprietary Deployment: Organizations seeking to embed the proprietary diagnostic scales, commercial instruments, or exact survey batteries within commercial enterprise software, proprietary CX platforms, or fee-generating analytical suites should consult Oxford University Press (the publisher of the Journal of Consumer Research) and the corresponding authors regarding licensing rights and permissions.
• Fee: There is no licensing fee required for independent academic scholars, university faculty, or doctoral candidates utilizing the scale for scientific research studies published in peer-reviewed journals.

12. References

• Castelo, N., Boegershausen, J., Hildebrand, C., & Henkel, A. P. (2023). Understanding and improving consumer reactions to service bots. Journal of Consumer Research, 50(4), 848–863. https://doi.org/10.1093/jcr/ucad024
• Castelo, N., Bos, M. W., & Lehmann, D. R. (2019). Task-dependent algorithm aversion. Journal of Marketing Research, 56(5), 809–825. https://doi.org/10.1177/0022243719851788
• Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008
• Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114–126. https://doi.org/10.1037/xge0000033
• Goodhue, D. L., & Thompson, R. L. (1995). Task-technology fit and individual performance. MIS Quarterly, 19(2), 213–236. https://doi.org/10.2307/249689
• Gray, H. M., Gray, K., & Wegner, D. M. (2007). Dimensions of mind perception. Science, 315(5812), 619–619. https://doi.org/10.1126/science.1134475
• Oliver, R. L. (1980). A cognitive model of the antecedents and consequences of satisfaction decisions. Journal of Marketing Research, 17(4), 460–469. https://doi.org/10.1177/002224378001700405
• Reeves, B., & Nass, C. (1996). The media equation: How people treat computers, television, and new media like real people and places. Cambridge University Press.
• Venkatesh, V., & Davis, F. D. (2000). A theoretical extension of the technology acceptance model: Four longitudinal field studies. Management Science, 46(2), 186–204. https://doi.org/10.1287/mnsc.46.2.186.11926
• Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478. https://doi.org/10.2307/30036540

13. Items of the Scale