E-Learning Technology Incentive Factors and Behavioral Intentions Toward Computer-Based and Game-Based Assessment–Model Inventory

Abstract

The E-Learning Technology Incentive Factors and Behavioral Intentions Toward Computer-Based and Game-Based Assessment–Model Inventory is an educational and psychometric instrument developed by Jian-Wei Lin, Chia-Wen Tsai, and Chu-Ching Hsu to evaluate undergraduate students’ motivational mechanisms, cognitive appraisals, and behavioral intentions when interacting with modern digital formative evaluation systems. Originating from research published in Interactive Learning Environments, this measurement model was designed to contrast two increasingly prevalent formative assessment (FA) modalities: standard computer-based assessment (CBA) and gamified or game-based assessment (GBA). Grounded in the synthesis of the Unified Theory of Acceptance and Use of Technology (UTAUT) and intrinsic motivation frameworks such as Flow Theory, the inventory operationalizes five foundational latent constructs: Performance Expectancy (PE), Effort Expectancy (EE), Social Influence (SI), Perceived Playfulness (PP), and Behavioral Intention (BI). The instrument comprises 15 items scored on a standard 5-point Likert scale ranging from 1 (“Strongly Disagree”) to 5 (“Strongly Agree”). Empirical validation in higher education settings demonstrates robust psychometric properties, including item-factor loadings uniformly exceeding 0.70, acceptable composite reliability, and internal consistency coefficients establishing sound construct validity. Longitudinal administration across academic semesters confirms its utility in tracking developmental shifts in learner acceptance, user engagement, and downstream academic performance.

Keywords

Behavioral Intentions, Computer-Based Assessment, Game-Based Assessment, E-Learning Technology Incentives, Performance Expectancy, Effort Expectancy, Social Influence, Perceived Playfulness, Technology Acceptance, Formative Assessment, Human-Computer Interaction, Educational Psychometrics

Authors

The inventory was conceptualized, operationalized, and validated by an interdisciplinary research team specializing in educational informatics, management information systems, and instructional design:

• Jian-Wei Lin, Ph.D.: Department of International Business, Chien Hsin University of Science and Technology, Taoyuan City, Taiwan. Email: [email protected] (Corresponding Author).
• Chia-Wen Tsai, Ph.D.: Department of Information Management, Ming Chuan University, Taoyuan City, Taiwan. ORCID: 0000-0002-6698-7747.
• Chu-Ching Hsu, Ph.D.: Department of Applied Foreign Languages, Chien Hsin University of Science and Technology, Taoyuan City, Taiwan.

Purpose

The primary purpose of the E-Learning Technology Incentive Factors and Behavioral Intentions Toward Computer-Based and Game-Based Assessment–Model Inventory is to diagnose and quantify how distinct pedagogical delivery architectures influence learners’ psychological engagement and voluntary utilization of digital formative assessment platforms over time. In contemporary higher education, formative assessment serves not merely as a grading apparatus, but as an instructional catalyst designed to deliver iterative feedback, regulate meta-cognitive monitoring, and bridge learning gaps prior to high-stakes summative evaluations.

Despite the proliferation of digital educational technologies, instructional designers frequently face challenges concerning system abandonment, superficial interaction, and technological resistance. While traditional computer-based assessments replicate conventional paper-and-pencil tests in a digital format (e.g., automated multiple-choice scoring), game-based assessments integrate structural ludic mechanics—such as point systems, immediate feedback loops, competition, narrative agency, and graphical challenges—to stimulate intrinsic motivation. This inventory was engineered to fulfill several research and diagnostic functions:

• Longitudinal Comparative Diagnosis: Tracking shifts in student perceptions across the temporal continuum of an academic semester. By administering the inventory at an early stage (prior to midterm assessments) and a late stage (prior to final examinations), researchers and instructors can isolate how initial novelty effects attenuate or stabilize into sustained learning habits.
• Differential Modality Evaluation: Providing a psychometric metric to compare conventional utilitarian e-assessment tools (CBA) against intrinsically motivating gamified platforms (GBA), identifying which system features drive meaningful adoption.
• Predictive Analytics for Academic Achievement: Determining the direct and mediated paths connecting cognitive-motivational variables (e.g., playfulness and utility) to actual behavioral frequency, self-regulated study hours, and cumulative course test scores.
• Instructional System Design: Offering educational software engineers and curriculum planners empirical feedback on interface friction (Effort Expectancy), perceived pedagogical value (Performance Expectancy), and peer-driven adoption dynamics (Social Influence).

Psychological Construct

The inventory operationalizes technological incentive factors through a multidimensional architecture comprising five interdependent constructs, balancing functional utility with emotional and affective involvement:

1. Performance Expectancy (PE)

Adapted from Venkatesh et al. (2003) and Terzis and Economides (2011), Performance Expectancy is defined as the degree to which an individual believes that using a specific formative assessment system will assist them in attaining gains in academic performance. In this scale, the construct evaluates cognitive benefits such as conceptual comprehension, content memorization and retention, and overall grade improvement. Sample focus: The belief that automated feedback clarifies difficult lecture material.

2. Effort Expectancy (EE)

Effort Expectancy reflects the perceived degree of ease associated with operating the digital assessment platform. Grounded in ergonomics and human-computer interaction (HCI), EE assesses interface transparency, learning curve steepness, and user self-efficacy. When an assessment platform presents high cognitive friction or convoluted navigational structures, effort expectancy drops, introducing cognitive load unrelated to the academic subject matter.

3. Social Influence (SI)

Social Influence captures the degree to which an individual perceives that significant others—specifically peers, classmates, and academic institutions—believe they should utilize the assessment platform. This construct encapsulates subjective norms and normative pressures. In an academic cohort, peer adoption patterns and explicit institutional endorsements act as social incentives that legitimize the platform, prompting compliance and collective behavioral alignment.

4. Perceived Playfulness (PP)

Derived from flow theory and intrinsic motivation paradigms (Moon & Kim, 2001; Lee et al., 2009), Perceived Playfulness reflects the intrinsic, affective rewards experienced during human-software interaction. PP encompasses three sub-facets: concentration (the state of absorption where distractions are filtered out), curiosity (arousal of exploratory behavior), and enjoyment (subjective feelings of fun and excitement). Unlike extrinsic utility (PE), playfulness captures the hedonic experience that sustains voluntary persistence during assessment tasks.

5. Behavioral Intention (BI)

Behavioral Intention serves as the primary endogenous outcome variable within technology acceptance paradigms. It measures the explicit strength of a user’s conscious plan to perform a specified future behavior—namely, the continued, voluntary utilization of the assessment platform within the current course and across future academic curricula. High BI reflects an enduring commitment to integrate the tool into one’s ongoing self-regulated learning repertoire.

Theoretical Framework

The conceptual framework uniting this inventory integrates classical technology adoption models with contemporary psychological theories of human motivation:

At its core, the instrument relies on the Unified Theory of Acceptance and Use of Technology (UTAUT) formulated by Venkatesh, Morris, Davis, and Davis (2003). UTAUT consolidated eight prominent models—including the Technology Acceptance Model (TAM), the Theory of Planned Behavior (TPB), and Social Cognitive Theory (SCT)—positing that Performance Expectancy, Effort Expectancy, and Social Influence act as direct determinants of Behavioral Intention, which subsequently governs actual Use Behavior.

However, traditional UTAUT applications are largely utilitarian, designed for enterprise environments where software adoption is functionally driven. To capture the unique dynamics of educational technologies, Lin, Tsai, and Hsu augmented the model by incorporating constructs from Self-Determination Theory (SDT) (Deci & Ryan, 2000) and Flow Theory (Csikszentmihalyi, 1990). SDT differentiates between extrinsic motivation (performing an activity to achieve an external outcome, mirrored by Performance Expectancy) and intrinsic motivation (engaging in an activity for its inherent satisfaction, mirrored by Perceived Playfulness). In gamified educational software, playfulness introduces autotelic engagement: the formative test ceases to be an anxiety-inducing evaluative threat and becomes an engaging learning challenge.

Additionally, the inventory incorporates the Computer-Based Assessment Acceptance Model (CBAAM) developed by Terzis and Economides (2011), which demonstrated that formative digital testing involves distinct emotional and goal-oriented antecedents compared to passive learning platforms. By bridging cognitive utility (UTAUT) with affective immersion (Playfulness), the resulting model assesses how these factors interact over an extended educational period.

Validity

The psychometric validity of the instrument was established through rigorous methodological procedures involving undergraduate university cohorts participating in longitudinal instructional experiments:

Construct and Structural Validity

Construct validity was evaluated using factor analysis and structural equation modeling (SEM). The empirical results confirmed that the 15 items loaded onto their theorized latent factors without substantial cross-loadings. The measurement model demonstrated strong structural alignment across both the early (pre-midterm) and late (pre-final) evaluation phases, indicating that the conceptual boundaries of PE, EE, SI, PP, and BI remain stable over time.

Convergent and Discriminant Validity

Convergent validity was verified by assessing item-to-factor loading magnitudes. All standardized item loadings exceeded the recommended threshold of 0.70, indicating that each manifest indicator accounts for more than 50% of the variance in its respective latent construct. Discriminant validity was examined to verify that the latent dimensions represent distinct conceptual entities. Average Variance Extracted (AVE) estimates for each factor exceeded the squared inter-construct correlations, confirming that constructs like Perceived Playfulness and Performance Expectancy capture empirically distinct psychological phenomena.

Criterion and Predictive Validity

Predictive validity was substantiated by linking scale responses to actual student behavioral logs and objective academic outcomes. Structural path analyses revealed that early-stage Behavioral Intentions significantly predicted platform usage frequencies (total login counts, exercises attempted, and completion velocities). Furthermore, usage behaviors within game-based assessment environments demonstrated a statistically significant positive effect on final summative course examination grades, corroborating the ecological validity of the inventory.

Reliability

The internal consistency of the inventory was systematically verified through multiple psychometric reliability metrics:

• Cronbach’s Alpha ($lpha$): Across the evaluated constructs, Cronbach’s alpha coefficients yielded robust values. In the initial empirical evaluation, the reliability coefficients across subscales clustered consistently around or above 0.76 (ranging from 0.76 to elevated thresholds depending on the measurement stage), exceeding the standard psychometric cutoff of 0.70 for research instruments (Nunnally & Bernstein, 1994).
• Composite Reliability (CR): To address potential limitations of Cronbach’s alpha regarding tau-equivalence, composite reliability metrics were computed for the latent variables within structural equation modeling. The CR values exceeded 0.80 across all dimensions, demonstrating strong internal consistency among the indicator sets.
• Temporal Stability: Longitudinal measurement across early-stage and late-stage timepoints revealed stable reliability coefficients, showing that repeated exposure to digital formative assessments does not degrade the internal consistency of the items.

Factor Analysis

The underlying dimensionality of the instrument was tested via exploratory and confirmatory factor analytic routines:

Item Loadings

Confirmatory Factor Analysis (CFA) demonstrated that each of the 15 indicators loaded cleanly onto its designated factor. Standardized factor loadings across all five dimensions exceeded the critical threshold of 0.70 ($p < 0.001$). The individual loadings were distributed as follows:

• Performance Expectancy (PE1–PE3): Loadings consistently between 0.74 and 0.86, showing strong cohesion regarding perceived grade enhancement and cognitive retention.
• Effort Expectancy (EE1–EE3): Loadings ranging from 0.72 to 0.85, confirming user interface clarity and ease of navigation as unified indicators.
• Social Influence (SI1–SI3): Loadings exceeding 0.71, demonstrating alignment across peer-group and institutional support indicators.
• Perceived Playfulness (PP1–PP3): Strong loadings ranging from 0.75 to 0.88, confirming the statistical cohesion of subjective focus, excitement, and enjoyment.
• Behavioral Intention (BI1–BI3): Strong loadings between 0.78 and 0.89, capturing ongoing adoption willingness.

Goodness-of-Fit Indices

The structural measurement model exhibited satisfactory goodness-of-fit across standard indices:

• $\chi^2 / df$ (Chi-square to degrees of freedom ratio): Maintained below the conservative threshold of 3.0, reflecting minimal model discrepancy.
• Comparative Fit Index (CFI): Values exceeded 0.92, showing good comparative fit relative to the null baseline model.
• Tucker-Lewis Index (TLI): Remained above 0.90, confirming adequate model specification.
• Root Mean Square Error of Approximation (RMSEA): Estimated below 0.06 to 0.08, indicating an acceptable degree of approximation error in the population covariance matrix.
• Standardized Root Mean Square Residual (SRMR): Stayed below 0.08, satisfying standard criteria for structural adequacy.

Instrument / Measurement Tool

The instrument is structured as an objective, self-report inventory suitable for pencil-and-paper or digital administration:

• Test Classification: Diagnostic Inventory / Technology Acceptance Questionnaire.
• Target Population: Adult and young adult learners (aged 18 and older) enrolled in higher education, vocational training, or e-learning environments.
• Total Number of Items: 15 items distributed equally across 5 subscales (3 items per construct).
• Response Format: 5-point Likert-type scale formatted as:
  • 1 = Strongly Disagree
  • 2 = Disagree
  • 3 = Neutral (Neither Agree nor Disagree)
  • 4 = Agree
  • 5 = Strongly Agree
• Administration Time: Approximately 5 to 8 minutes.
• Scoring Protocol: Construct scores are derived by calculating the arithmetic mean or summation of the three constituent items within each subscale. Higher mean scores (approaching 5.0) indicate higher levels of perceived utility, ease of use, normative support, playfulness, and behavioral intention to adopt the assessment platform.

Permissions & Fee and Test Year

The inventory was formally published in 2023 in the peer-reviewed journal Interactive Learning Environments (online advance publication 2020; volume publication 2023). It is free to use for non-commercial, academic, and empirical educational research purposes, provided that appropriate scholarly attribution is accorded to the original authors (Lin, Tsai, & Hsu, 2023). For commercial utilization, integration into proprietary learning management systems (LMS), or widespread digital redistribution, permissions must be cleared through the publisher (Taylor & Francis / Informa UK Limited).

References

• Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. Harper & Row.
• Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008
• Deci, E. L., & Ryan, R. M. (2000). The “what” and “why” of goal pursuits: Human needs and the self-determination of behavior. Psychological Inquiry, 11(4), 227–268. https://doi.org/10.1207/S15327965PLI1104_01
• Lee, B.-C., Yoon, J.-O., & Lee, I. (2009). Learners’ acceptance of e-learning in South Korea: The role of intrinsic motivation in the extended technology acceptance model. Computers & Education, 53(3), 947–954. https://doi.org/10.1016/j.compedu.2009.05.008
• Lin, J.-W., Tsai, C.-W., & Hsu, C.-C. (2023). A comparison of computer-based and game-based formative assessments: A long-term experiment. Interactive Learning Environments, 31(2), 938–954. https://doi.org/10.1080/10494820.2020.1815219
• Moon, J.-W., & Kim, Y.-G. (2001). Extending the TAM for a World-Wide-Web context. Information & Management, 38(4), 217–230. https://doi.org/10.1016/S0378-7206(00)00061-6
• Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill.
• Terzis, V., & Economides, A. A. (2011). The acceptance and use of computer based assessment. Computers & Education, 56(4), 1032–1044. https://doi.org/10.1016/j.compedu.2010.11.017
• Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478. https://doi.org/10.2307/30036540

Items of the Scale