Artificial Intelligence Literacy Questionnaire (AILQ)

1. Abstract

The Artificial Intelligence Literacy Questionnaire (AILQ) is a standardized, multidimensional self-report psychometric instrument developed by Ng et al. (2024) to evaluate artificial intelligence (AI) literacy among secondary school students aged 12 to 17 years. Grounded in a comprehensive educational paradigm that synthesizes cognitive, affective, behavioral, and ethical domains, the instrument operationalizes AI literacy beyond mere operational coding skills, measuring the holistic competencies required to navigate, evaluate, and collaborate with emerging algorithmic technologies. The instrument was refined from an initial 60-item pool through expert panel evaluation, cognitive interviews, and exploratory factor analysis, resulting in a finalized 32-item scale administered across a 5-point Likert-type response format ranging from 1 (Strongly Disagree) to 5 (Strongly Agree).

The AILQ is structured around four primary latent dimensions conceptualized under the ABCE framework: (1) Affective Learning, encompassing intrinsic motivation, self-efficacy, confidence, and career interest; (2) Behavioral Learning, capturing sustained behavioral commitment and peer collaboration in AI-driven environments; (3) Cognitive Learning, addressing knowledge application, critical evaluation, and algorithmic creation; and (4) Ethical Learning, examining awareness and appraisal of algorithmic bias, data privacy, transparency, and societal implications. Psychometric validation conducted within secondary school cohorts in Hong Kong established high internal consistency (overall Cronbach’s α = 0.93) and robust structural validity. Confirmatory factor analysis demonstrated the statistical superiority of a second-order model (χ²(452) = 1001.54, p < 0.01; RMSEA = 0.06; CFI = 0.92; TLI = 0.91; SRMR = 0.06) over competing first-order and third-order structures. Convergent and discriminant validity were evidenced via average variance extracted metrics and the Heterotrait–Monotrait ratio of correlations (HTMT < 0.85). The AILQ provides researchers, educators, and curriculum policymakers with a rigorous diagnostic measure for evaluating secondary digital curricula and tracking developmental trajectories in AI competency.

2. Keywords

Artificial Intelligence Literacy, AILQ, Secondary Education, Affective Learning, Behavioral Commitment, Cognitive Competence, AI Ethics, Psychometrics, Confirmatory Factor Analysis, Educational Technology, Algorithmic Literacy, Scale Validation.

3. Authors

The Artificial Intelligence Literacy Questionnaire was conceptualized, developed, and empirically validated by an interdisciplinary team of educational technologists and psychometric researchers across higher education institutions in Hong Kong:

• Davy Tsz Kit Ng (Corresponding Author) — Faculty of Education, The University of Hong Kong, Hong Kong SAR, China. ORCID: 0000-0002-2380-7814. Email: [email protected]
• Wenjie Wu — Faculty of Education, The Chinese University of Hong Kong, Shatin, Hong Kong SAR, China.
• Jac Ka Lok Leung — Division of Integrative Systems and Design, The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong SAR, China. ORCID: 0000-0001-6490-7005.
• Thomas Kin Fung Chiu — Faculty of Education, The Chinese University of Hong Kong, Shatin, Hong Kong SAR, China. ORCID: 0000-0003-2887-5477.
• Samuel Kai Wah Chu — School of Nursing and Health Studies, Hong Kong Metropolitan University, Ho Man Tin, Hong Kong SAR, China. ORCID: 0000-0003-1557-2776.

4. Purpose

The rapid integration of machine learning algorithms, natural language processing models, and generative artificial intelligence into everyday life has introduced profound shifts in educational landscapes. While traditional digital literacy frameworks focus on fundamental computer operation, information retrieval, and basic coding paradigms, they fail to adequately capture the socio-cognitive, affective, and ethical challenges unique to human-AI interaction. The Artificial Intelligence Literacy Questionnaire (AILQ) was developed specifically to address this conceptual and psychometric void in secondary school education.

The primary purpose of the AILQ is to provide a standardized, psychometrically validated diagnostic instrument capable of assessing adolescent competencies across four vital pedagogical domains: affective, behavioral, cognitive, and ethical (the ABCE model). In secondary school contexts (ages 12–17), students increasingly consume and interact with generative systems (e.g., automated recommendation engines, large language models, computer vision systems) often without an explicit understanding of underlying biases, data harvesting mechanics, or cognitive boundaries. The instrument was developed to fulfill several theoretical, empirical, and diagnostic needs:

• Holistic Educational Diagnostic: To serve as a comprehensive screening and benchmark tool for educators to diagnose baseline AI competencies before initiating specialized STEM, computer science, or general digital literacy interventions.
• Curricular Assessment and Evaluation: To quantify the efficacy of newly integrated AI curricula, pedagogical interventions, and public-sector educational policies aimed at equipping young learners for future automated economies.
• Theoretical Demarcation: To separate AI literacy from general technological literacy or programming self-efficacy by measuring AI-specific cognitive creation, collaborative behavioral problem-solving, and normative socio-ethical appraisal.
• Longitudinal and Differential Research: To provide developmental and educational psychologists with a calibrated measurement model to track age-related, gender-differentiated, or socio-economic trajectories in adolescent technological agency and AI career interest.

By capturing students’ feelings, behavioral patterns, deep intellectual processing, and ethical scrutiny, the AILQ transitions educational assessment away from purely technical examinations toward an authentic evaluation of future-ready citizenship.

5. Psychological Construct

The construct of Artificial Intelligence Literacy measured by the AILQ is operationalized as a multidimensional, second-order overarching capacity comprising four interrelated primary dimensions (Affective, Behavioral, Cognitive, and Ethical learning) and eight granular sub-competencies:

1. Affective Learning Dimension

The affective domain encompasses the emotional, motivational, and value-based orientations that dictate how students engage with and perceive artificial intelligence systems. Rather than viewing technology purely as an analytical tool, this dimension recognizes that technological adoption is closely tied to internal affective drives. It consists of four sub-components:

• Intrinsic Motivation: Derived from Ryan and Deci’s Self-Determination Theory, this reflects the internal curiosity, inherent enjoyment, and intellectual satisfaction experienced when exploring AI mechanisms, free from external academic pressures.
• Self-Efficacy: Grounded in Albert Bandura’s social cognitive framework, this reflects students’ perceived capability to master complex AI principles, operate machine learning tools, and troubleshoot algorithmic failures.
• Confidence: The subjective feeling of security, low apprehension, and emotional comfort when interacting with emerging automated platforms.
• Career Interest: The prospective aspirations of learners regarding vocational pathways in computer science, data analytics, algorithmic engineering, or AI-integrated industries.

2. Behavioral Learning Dimension

The behavioral domain assesses the tangible actions, patterns of engagement, and collaborative strategies deployed by students when encountering AI-enabled environments. It shifts focus from passive conceptualization to active agency, operationalized across two sub-components:

• Behavioral Commitment: The sustained effort, perseverance, time investment, and active participation demonstrated by the student when solving complex problems using artificial intelligence tools.
• Collaboration: The capacity to engage in peer-to-peer communicative discourse, co-design technological artifacts, and leverage distributed cognition within group-based AI learning tasks.

3. Cognitive Learning Dimension

The cognitive domain measures intellectual faculties, analytical processing, and creative problem-solving abilities applied to AI systems. Structured in accordance with revised Bloomian taxonomies, it evaluates higher-order thinking skills across three sub-components:

• Knowledge Application: The ability to select, configure, and execute relevant AI tools to resolve real-world computational or academic challenges.
• Evaluation: Critical appraisal of algorithmic outputs, identifying hallucinations, recognizing system limitations, and validating the accuracy of probabilistic predictions.
• Creation: The synthesis of computational thinking, data structuring, and machine learning models to design novel, functioning AI artifacts or solutions.

4. Ethical Learning Dimension

The ethical domain represents a defining pillar of modern technological literacy, recognizing that AI tools inherently involve ethical considerations. It captures students’ critical awareness of algorithmic bias, fairness, transparency, autonomous agency, intellectual property, surveillance, and data privacy rights. It evaluates whether adolescents can ethically appraise the deployment of predictive systems and foresee adverse socio-technical outcomes.

6. Theoretical Framework

The architecture of the AILQ is rooted in a synthesis of educational taxonomy, social cognitive theory, and contemporary digital literacy frameworks. The conceptual foundation integrates three primary theoretical pillars:

1. The Tripartite Model of Attitudes and Educational Learning Domains

The classic educational taxonomy established by Benjamin Bloom and expanded by Anderson and Krathwohl delineates human learning into cognitive, affective, and psychomotor/behavioral domains. Ng et al. adapted this framework into the ABCE Approach (Affective, Behavioral, Cognitive, Ethical), arguing that emerging technologies require an explicit, autonomous ethical domain alongside Bloom’s traditional learning dimensions. Algorithmic autonomous systems do not merely execute deterministic user commands; they generate predictive inferences, introduce systemic biases, and manipulate personal data, necessitating moral agency as an essential learning domain.

2. Bandura’s Social Cognitive Theory and Self-Determination Theory

The integration of affective and behavioral learning draws heavily on Bandura’s Social Cognitive Theory, particularly the mechanism of triadic reciprocal determinism. Environmental interactions (collaborative AI workshops), internal cognitive-affective states (AI self-efficacy, intrinsic motivation), and actual behavioral execution (behavioral commitment) continually influence one another. Concurrently, Self-Determination Theory provides the foundation for the intrinsic motivation sub-construct, postulating that students sustain meaningful technological literacy when their basic psychological needs for autonomy, competence, and relatedness are met within computational spaces.

3. AI Literacy Competency Frameworks (Touretzky and Long & Magerko)

The cognitive and ethical dimensions are aligned with Touretzky et al.’s “Five Big Ideas in AI” (perception, representation and reasoning, learning from data, natural interaction, and societal impact) and the competency framework established by Long and Magerko (2020). These models assert that computational literacy must advance from functional tool operation to deep algorithmic understanding, system evaluation, and sociotechnical critique. The AILQ operationalizes these constructs into an empirically verifiable measurement model tailored to secondary school educational assessment.

7. Validity

The psychometric validation of the AILQ followed a rigorous multi-phase process designed to establish content, construct, convergent, and discriminant validity in accordance with the Standards for Educational and Psychological Testing.

Content and Face Validity

The initial instrument pool of 60 items was drafted following an extensive literature review of AI educational frameworks. Content validity was established via expert review panels comprising university professors specializing in educational technology, secondary school computer science teachers, and psychometricians. Items were evaluated for linguistic appropriateness, clarity for adolescent readers, and construct alignment. Additionally, qualitative cognitive interviews and a pilot study were conducted with secondary school students to refine ambiguities and eliminate redundant items, reducing the instrument pool prior to broad empirical testing.

Construct and Factorial Validity

Factorial validity was established through sequential exploratory and confirmatory factor analyses. The exploratory factor analysis (EFA) demonstrated that the extracted factors accounted for 51.41% of the total cumulative variance, confirming substantial explanatory power across the underlying theoretical facets. Subsequent confirmatory factor analysis (CFA) tested competing architectural models, confirming that a second-order model provided an optimal balance of parsimony and structural fit over alternative first-order or third-order structures.

Convergent Validity

Convergent validity was evaluated using the Average Variance Extracted (AVE) criterion alongside standardized factor loadings. Across the validated latent factors, factor loadings generally exceeded the recommended 0.60 threshold. The AVE exceeded the conventional 0.50 benchmark for the majority of the latent factors. While the Intrinsic Motivation (AVE = 0.44) and AI Ethics (AVE = 0.46) dimensions fell marginally below the strict 0.50 threshold, researchers retained them based on Fornell and Larcker’s criterion: if composite reliability (CR) remains well above 0.70 to 0.80, the convergent validity of the latent construct is considered psychometrically acceptable.

Discriminant Validity

Discriminant validity was rigorously evaluated using the Heterotrait–Monotrait ratio of correlations (HTMT), which is recognized as superior to traditional Fornell–Larcker cross-loadings in detecting multicollinearity among close educational constructs. All calculated HTMT ratios fell below the conservative 0.85 threshold, confirming that while affective, behavioral, cognitive, and ethical competencies are interrelated components of overall AI literacy, they remain distinct empirical constructs.

8. Reliability

The internal consistency and measurement reliability of the AILQ were evaluated using multiple psychometric indices across both the global instrument and its nested subscales:

• Overall Internal Consistency: The global 32-item instrument demonstrated high internal consistency, yielding an overall Cronbach’s α of 0.93. This confirms that the aggregated instrument possesses high reliability suitable for research applications and group-level educational assessments.
• Dimension-Level Reliability: Internal consistency across the individual dimensions and subscales consistently demonstrated acceptable to high coefficients:
• Affective Learning Subscales: Cronbach’s α values ranged between 0.78 and 0.88 across intrinsic motivation, confidence, self-efficacy, and career interest.
• Behavioral Learning Subscales: Behavioral commitment and peer collaboration exhibited Cronbach’s α values between 0.81 and 0.86.
• Cognitive Learning Subscales: Knowledge application, system evaluation, and algorithmic creation exhibited alpha reliabilities between 0.82 and 0.89.
• Ethical Learning Subscale: The AI ethics dimension yielded a Cronbach’s α of 0.80.
• Composite Reliability (CR): To address the limitations of Cronbach’s alpha regarding tau-equivalence assumptions, composite reliability was computed for all latent dimensions, with values ranging from 0.79 to 0.91, well exceeding the established 0.70 benchmark.

9. Factor Analysis

The structural properties and latent architecture of the AILQ were systematically evaluated through exploratory factor analysis (EFA) followed by confirmatory factor analysis (CFA) using structural equation modeling.

Exploratory Factor Analysis (EFA)

During the exploratory phase, principal axis factoring with promax (oblique) rotation was conducted to identify the underlying dimensionality of the initial item pool. Oblique rotation was selected based on the theoretical assumption that educational, affective, and cognitive competencies are inherently correlated. The EFA revealed a stable factor structure capturing the four core dimensions (Affective, Behavioral, Cognitive, Ethical) across eight underlying sub-factors, accounting for 51.41% of the total variance.

Confirmatory Factor Analysis (CFA) & Model Comparison

To confirm the structural validity of the refined 32-item scale, competing models were evaluated using structural equation modeling (SEM) with maximum likelihood estimation:

• First-Order Model: Assumed eight correlated yet distinct latent factors. While fit was acceptable, it failed to account for higher-order theoretical commonalities.
• Third-Order Model: Posited a singular overarching third-order construct (AI Literacy) branching into second-order ABCE domains, which further split into third-order sub-facets. This model exhibited signs of over-parameterization and diminished parsimony.
• Second-Order Model (Favored Structure): Specified the four primary dimensions (Affective, Behavioral, Cognitive, Ethical) as first-order latent constructs loading onto a unified second-order construct of overall AI Literacy, with subscales loading appropriately onto their respective overarching domains.

The second-order model exhibited strong goodness-of-fit indices:

• Chi-Square / Degrees of Freedom: χ²(452) = 1001.54, p < 0.01 (χ²/df = 2.21, well within the conventional ≤ 3.0 threshold).
• Root Mean Square Error of Approximation (RMSEA): 0.06 (indicating a close structural fit).
• Comparative Fit Index (CFI): 0.92 (exceeding the standard 0.90 threshold).
• Tucker–Lewis Index (TLI): 0.91 (demonstrating solid comparative goodness of fit).
• Standardized Root Mean Square Residual (SRMR): 0.06 (below the 0.08 cutoff for acceptable residual variance).

Item loadings across all factors were statistically significant (p < 0.001), confirming that the second-order model accurately represents the multidimensional structure of secondary school AI literacy.

10. Instrument / Measurement Tool

The operational characteristics and administrative guidelines for the Artificial Intelligence Literacy Questionnaire (AILQ) are summarized below:

• Instrument Name: Artificial Intelligence Literacy Questionnaire (AILQ)
• Instrument Type: Standardized self-report multidimensional psychometric scale
• Target Population: Secondary school students (Adolescents, aged 12 to 17 years)
• Number of Items: 32 items (refined from an initial 60-item pool)
• Response Format: 5-point Likert-type scale:
  • 1 = Strongly Disagree
  • 2 = Disagree
  • 3 = Neutral
  • 4 = Agree
  • 5 = Strongly Agree
• Dimensional Structure:
  • Affective Learning: Measured via four sub-facets (Intrinsic Motivation, Self-Efficacy, Confidence, Career Interest)
  • Behavioral Learning: Measured via two sub-facets (Behavioral Commitment, Collaboration)
  • Cognitive Learning: Measured via three sub-facets (Knowledge Application, Evaluation, Creation)
  • Ethical Learning: Measured via the AI Ethics subscale
• Scoring Protocol:
  • Subscale Scores: Computed by calculating the mean or sum of items within each sub-dimension. Higher scores indicate greater competency, interest, commitment, or ethical awareness within that specific domain.
  • Dimensional Scores: Calculated by aggregating subscale scores for Affective, Behavioral, Cognitive, and Ethical learning.
  • Overall AI Literacy Index: Derived from the global average or composite score of all 32 items. Higher aggregate scores indicate advanced holistic artificial intelligence literacy.
• Administration Time: Approximately 10 to 15 minutes.

11. Permissions & Fee and Test Year

The Artificial Intelligence Literacy Questionnaire was published in 2024 in the British Journal of Educational Technology. The instrument was developed by academic researchers at The University of Hong Kong, The Chinese University of Hong Kong, The Hong Kong University of Science and Technology, and Hong Kong Metropolitan University.

The scale was developed for educational and research purposes. In accordance with standard academic publishing conventions, the questionnaire may generally be utilized by educators, psychologists, and academic researchers for non-commercial educational evaluations and scientific studies, provided appropriate academic attribution and citation are given to Ng et al. (2024). Commercial use, automated digital application redistribution, or substantial adaptations typically require formal permission from the copyright holders (the authors and the publisher, John Wiley & Sons Ltd). Researchers wishing to obtain the original complete questionnaire items, localized translations, or administration guidelines are advised to consult the original journal publication or contact the corresponding author, Dr. Davy Tsz Kit Ng, directly via email at [email protected].

12. References

Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. Longman.

Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Prentice-Hall.

Deci, E. L., & Ryan, R. M. (2000). The “what” and “why” of goal pursuits: Human needs and the self-determination of behavior. Psychological Inquiry, 11(4), 227–268. https://doi.org/10.1207/S15327965PLI1104_01

Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39–50. https://doi.org/10.1177/002224378101800104

Henseler, J., Ringle, C. M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43(1), 115–135. https://doi.org/10.1007/s11747-014-0403-8

Long, D., & Magerko, B. (2020). What is AI literacy? Competencies and design considerations. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1–16). Association for Computing Machinery. https://doi.org/10.1145/3313831.3376727

Ng, D. T. K., Wu, W., Leung, J. K. L., Chiu, T. K. F., & Chu, S. K. W. (2024). Design and validation of the AI Literacy Questionnaire: The affective, behavioural, cognitive and ethical approach. British Journal of Educational Technology, 55(3), 1082–1104. https://doi.org/10.1111/bjet.13411

Touretzky, D., Gardner-McCune, C., Martin, F., & Seehorn, D. (2019). Envisioning AI for K-12: What should every child know about AI? Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 33(01), 9795–9799. https://doi.org/10.1609/aaai.v33i01.33019795

13. Items of the Scale