Digital Competence for Learning Assessment Test (Digitest)

1. Abstract

The Digital Competence for Learning Assessment Test (Digitest) is a standardized psychometric instrument designed to evaluate primary and lower secondary school students’ digital competence specifically situated within learning contexts. Developed by Margus Pedaste, Külli Kallas, and Aleksandar Baucal in 2023 at the Centre for Educational Technology, Institute of Education, University of Tartu, the instrument operationalizes modern frameworks of digital competence by synthesizing cognitive-behavioral abilities and socio-cognitive motivational dynamics. Digitest comprises 41 performance-based and self-report items mapped across nine structural subscales: Perceived Control, Behavior-Related Attitudes, Behavioral Intention, Digital Operations, Communication in the Digital World, Digital Content Programming, Creation of Digital Materials, Protection of Oneself and Others in the Digital World, and Legal Behavior in the Digital World. The instrument uses six diverse item delivery formats: single-choice multiple-choice questions, multiple-choice questions with multiple correct options, sequential phase-ordering tasks, matching items, hotspot/picture-marking tasks, and open-ended performance prompts, scored dichotomously as correct or incorrect. Psychometric evaluation in an empirical cohort of Estonian learners across grades 3 through 9 demonstrated robust empirical characteristics. Item Response Theory (IRT) analyses confirmed acceptable infit, outfit, and item discrimination indices, with expected item correlations exceeding 0.30 for nearly all items. Confirmatory factor analyses demonstrated that a two-factor higher-order model—comprising a cognitive-behavioral higher-order factor and a motivational-affective higher-order factor—provides an optimal theoretical and structural fit aligned with Weinert’s foundational competence paradigm. Scale composite reliabilities across the latent dimensions ranged from 0.65 to 0.91, establishing Digitest as a theoretically grounded, ecologically valid diagnostic tool for educational technology researchers, school psychologists, and pedagogical practitioners.

2. Keywords

Digital Competence, Digital Learning, Digitest, Educational Technology, Item Response Theory, Weinert Competence Model, Perceived Control, Behavioral Intention, Primary Education, Secondary Education, Psychometrics, Assessment Literacy

3. Authors

The Digital Competence for Learning Assessment Test (Digitest) was developed and validated through a collaborative scholarly initiative between the University of Tartu in Estonia and the University of Belgrade in Serbia:

• Margus Pedaste, Ph.D. — Professor of Educational Technology, Institute of Education, Faculty of Social Sciences, University of Tartu, Tartu, Estonia. ORCID: 0000-0002-5087-9637. Email: [email protected].
• Külli Kallas, M.A. — Researcher and Doctoral Candidate, Institute of Education, Faculty of Social Sciences, University of Tartu, Jakobi 5, 51005 Tartu, Estonia. Email: [email protected] (Corresponding Author).
• Aleksandar Baucal, Ph.D. — Professor of Developmental Psychology and Educational Psychology, Department of Psychology, Faculty of Philosophy, University of Belgrade, Belgrade, Serbia. ORCID: 0000-0002-7965-7659. Email: [email protected].

4. Purpose

The primary purpose of the Digital Competence for Learning Assessment Test (Digitest) is to provide a comprehensive, multidimensional, and standardized measurement instrument to evaluate the digital competence of primary and lower secondary school students (specifically grades 3 through 9, corresponding roughly to ages 8 to 16) applied directly to educational activities and autonomous learning. While general digital competence frameworks (such as the European Commission’s DigComp framework) assess general digital literacy across adult citizenship contexts, Digitest was explicitly engineered to address the ecological realities of modern instructional environments, where digital tools are not merely social media or leisure mechanisms, but cognitive instruments for knowledge acquisition, problem-solving, collaborative inquiry, and academic material production.

Historically, educational institutions and empirical researchers relied on self-report questionnaires to assess digital literacy. However, educational measurement literature has consistently shown that self-reports of technological fluency suffer from self-enhancement biases, Dunning-Kruger effects, and gendered discrepancies in perceived versus actual competence. Digitest overcomes these limitations by integrating authentic performance-based tasks—measuring applied problem solving, computational logic, data transformation, and safety practices—with socio-cognitive affective dimensions. This dual architecture ensures researchers can assess not only whether a student believes they can manage a digital challenge, but whether they possess the actual operational abilities to execute it effectively.

In clinical, educational, and school psychology contexts, Digitest serves critical diagnostic and formative functions:

• Identification of Learning Inequities: Diagnosing specific digital skill deficits among disadvantaged or neurodivergent student cohorts, thereby supporting tailored pedagogical interventions.
• Curricular Evaluation: Permitting school districts, curriculum designers, and educational policy bodies to monitor systemic longitudinal shifts in digital readiness following educational technology interventions.
• Theoretical Differentiation: Untangling the complex interplay between students’ internal motivational states (perceived control, attitudes, and behavioral intentions) and their objective operational skills (programming, material authoring, legal compliance).

5. Psychological Construct

The psychological construct assessed by Digitest is Digital Competence for Learning. Rather than viewing digital competence as an isolated technical skill, the test models it as a complex, multifaceted latent psychological architecture encompassing cognitive capacities, behavioral performance repertoires, and socio-cognitive motivational dispositions. Through extensive exploratory and confirmatory factor modeling, Digitest operationalizes this construct across nine correlated primary latent dimensions categorized under two overarching higher-order domains:

Motivational-Affective Dimensions

• Perceived Control: Rooted in Bandura’s self-efficacy construct and Ajzen’s Theory of Planned Behavior, this subscale captures students’ subjective evaluations of their capability, autonomy, and external agency when navigating unfamiliar technological systems, software crashes, or complex digital interfaces.
• Behavior-Related Attitudes: Assesses the affective and cognitive appraisals students hold toward utilizing digital devices for academic tasks, reflecting beliefs regarding the perceived utility, efficiency, and intrinsic satisfaction derived from educational digital practices.
• Behavioral Intention: Quantifies students’ conscious plans, readiness, and deliberate commitments to integrate digital tools into future academic problem-solving, collaboration, and learning endeavors.

Cognitive-Behavioral Dimensions

• Digital Operations: Evaluates fundamental operational mastery, including file architecture management, operating system navigation, system configuration, hardware interaction, and basic troubleshooting of technological breakdowns.
• Communication in the Digital World: Measures applied knowledge of netiquette, digital collaboration platforms, audience-appropriate channel selection, information sharing, and constructive participatory practices within academic virtual networks.
• Digital Content Programming: Assesses computational thinking, logical sequence design, algorithmic reasoning, loop construction, and the ability to debug basic code or automation workflows within visual or text-based educational programming environments.
• Creation of Digital Materials: Evaluates students’ capacity to author, synthesize, format, and edit complex digital artifacts, including multimedia presentations, spreadsheets, hyperlinked text documents, and curated digital portfolios.
• Protection of Oneself and Others in the Digital World: Gauges security literacy, password protection, recognition of malware and phishing attempts, data privacy strategies, ergonomics, cyberbullying mitigation, and physiological well-being during digital immersion.
• Legal Behavior in the Digital World: Measures understanding of digital citizenship ethics, creative commons licensing, intellectual property laws, plagiarism deterrence, copyright restrictions, and academic integrity regulations.

6. Theoretical Framework

The theoretical architecture of Digitest is grounded in Franz E. Weinert’s (2001) seminal competence model, augmented by the socio-cognitive dynamics of Icek Ajzen’s Theory of Planned Behavior, and localized within the conceptual educational technology frameworks proposed by Pedaste et al. (2021) and Adov (2022).

Weinert (2001) conceptualized human competence not merely as rote declarative knowledge or static cognitive intelligence, but as an integrated nexus of specialized cognitive capabilities, executable behavioral skills, motivational orientations, volitional readiness, and social-moral responsibility that enable an individual to master complex, situated real-world demands. Applying Weinert’s model to educational technology, Pedaste and colleagues posited that proficient digital learning requires an indivisible synergy between two structural components:

1. Cognitive-Behavioral Component: The objective, operational, and procedural ability to utilize hardware, author multimedia, manipulate symbolic-computational structures, communicate effectively, and maintain ethical-legal digital safety.
2. Motivational-Affective Component: The underlying psychological engine that translates technical capacity into sustained academic effort, mediated by subjective control beliefs, positive utility evaluations, and explicit behavioral intentions.

To articulate the motivational branch, the authors utilized the Theory of Planned Behavior (Ajzen, 1991), which states that actual behavior is immediately preceded by behavioral intention, which in turn is co-determined by attitudes toward the behavior, subjective social norms, and perceived behavioral control. While earlier theoretical iterations included social aspects (such as perceived social norms), empirical analyses indicated that social normative items failed to form a distinct psychometric factor among primary and secondary school learners, instead cross-loading heavily across multiple motivational dimensions. Consequently, the authors refined the theoretical framework to center upon perceived control, attitudes, and intentions as the motivational anchors that activate cognitive-behavioral performance across the six functional operational domains.

7. Validity

The validation of Digitest followed rigorous, multi-method psychometric procedures with an empirical sample of Estonian school students across grades 3 through 9:

Content and Construct Validity

Content validity was established via a collaborative expert panel at the Centre for Educational Technology at the University of Tartu. Panelists cross-referenced each candidate item against the target competencies identified in empirical literature (Adov, 2022; Pedaste et al., 2021) and international benchmark frameworks (such as DigComp 2.2). Construct validity was verified using both exploratory and confirmatory factor analysis, confirming that the items accurately converge onto the hypothesized psychological subscales without excessive construct irrelevance.

Item Response Theory (IRT) Diagnostics

To assess item quality at the latent trait level, Digitest was analyzed using Item Response Theory (IRT) models:

• Item Infit and Outfit: Infit and outfit mean-square statistics were evaluated across all performance items. Although minor fluctuations were observed outside the strictly conventional 0.70–1.30 boundaries for a small subset of tasks, these deviations were determined to be benign given the preservation of high discrimination values and the contextual heterogeneity of performance-based testing in primary school populations.
• Item Discrimination and Point-Measure Correlations: Expected item-total correlations exceeded the accepted threshold of 0.30 for almost all items (with only two items exhibiting correlations between 0.20 and 0.30, an acceptable floor for exploratory developmental cohorts). Estimated item discrimination indices were robust; only two items within the communication domain (COMM1 and COMM2) fell outside recommended optimal boundaries, retaining sufficient informational yield to justify retention in the final 41-item instrument without degrading overall scale validity.

Convergent and Discriminant Validity

The structural separation between motivational-affective subscales and cognitive-behavioral subscales demonstrated robust discriminant validity. Learners exhibiting high perceived control did not automatically achieve flawless operational scores on advanced programming or legal compliance tasks, confirming that Digitest successfully discriminates between subjective technological confidence and objective functional competence.

8. Reliability

The internal consistency of Digitest was evaluated using composite reliability (CR) coefficients across the latent variables defined by the structural factor model, offering a more precise evaluation than traditional Cronbach’s alpha coefficients when working with congeneric, multidimensional scales containing mixed-format items.

Across the nine primary latent dimensions, composite reliabilities ranged from 0.65 to 0.91, reflecting solid to excellent internal consistency:

• Motivational Latent Scales: Demonstrated high composite reliabilities, generally ranging between 0.82 and 0.91, indicating that students hold coherent, well-structured beliefs regarding their perceived technological control, attitudes, and behavioral intentions.
• Cognitive-Behavioral Latent Scales: Demonstrated composite reliabilities ranging from 0.65 (for concise operational dimensions with fewer items, such as specialized digital operations or legal behavior) to 0.86 (for broader computational and content creation subscales). In educational testing involving multidimensional, dichotomously scored performance tasks among young learners, composite reliabilities above 0.65 are recognized as meeting empirical standards for group-level diagnostic assessments.

9. Factor Analysis

The structural architecture of Digitest was established through an integrated psychometric sequence of Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA):

Exploratory Factor Analysis (EFA)

EFA was initially deployed to examine the latent structure of the attitude and motivational items. In early theoretical models, a 10-factor structure was hypothesized, which incorporated an explicit dimension assessing the social aspects of digital tool usage (subjective norms and social peer pressures). However, both three-factor and four-factor exploratory extractions revealed that items measuring social aspects did not load onto an independent, coherent factor. Instead, these items exhibited substantial cross-loadings across general attitudes and perceived control. Consequently, based on initial factor loadings, scree plot inspection, and composite reliability calculations, the social aspects dimension was formally eliminated, refining the primary structural model to nine robust latent factors.

Confirmatory Factor Analysis (CFA)

Subsequent CFA was conducted on the full 41-item instrument to contrast competing structural models:

• A primary 9-factor correlated first-order model.
• A 3-factor higher-order model.
• A 2-factor higher-order model (Cognitive-Behavioral Competence and Motivational-Affective Competence).

The CFA results indicated that the 9-factor first-order model, the 3-factor higher-order model, and the 2-factor higher-order model all demonstrated comparable, acceptable goodness-of-fit indices (RMSEA < 0.06, CFI > 0.90, TLI > 0.90, SRMR < 0.08). However, the researchers selected the 2-factor higher-order model as the definitive structural representation because it provided the most theoretically coherent alignment with Franz Weinert’s (2001) foundational competence paradigm, cleanly partitioning the nine subscales into an overarching cognitive-behavioral factor (Operations, Communication, Programming, Creation, Protection, Legal Behavior) and an overarching motivational-affective factor (Perceived Control, Attitudes, Behavioral Intention).

10. Instrument / Measurement Tool

The Digital Competence for Learning Assessment Test (Digitest) is a performance-based and socio-cognitive testing battery designed for digital or computer-assisted administration in primary and secondary schools.

• Test Type: Standardized diagnostic assessment combining objective performance items and self-report socio-cognitive subscales.
• Item Count: 41 items in the validated operational form.
• Target Population: Primary and lower secondary school students (Grades 3–9; ages 8–16).
• Language Available: Estonian (original validation instrument).
• Item Formats: Six distinct interaction formats are employed across the test to assess applied learning performance:
  1. Multiple-choice questions with only one correct option (single-select).
  2. Multiple-choice questions with more than one correct option (multi-select).
  3. Tasks to form a sequence of phases (chronological/procedural ordering tasks).
  4. Matching items (pairing concepts, tools, or functional actions).
  5. Marking something in a picture (visual hotspot identification).
  6. Open-ended questions (short constructed responses evaluating applied solutions).
• Scoring Rules: All 41 items are marked dichotomously as either Correct (1) or Incorrect (0). For multi-select and matching tasks, full credit is awarded only when the complete criterion matching/selection standard is fulfilled according to the predefined test rubric. Subscale scores are calculated as the arithmetic sum of correct items within each domain, with higher-order cognitive-behavioral and motivational-affective indices derived through structural latent scoring or standardized domain aggregations.

11. Permissions & Fee and Test Year

• Test Year: 2023.
• Copyright & Permissions: The Digital Competence for Learning Assessment Test (Digitest) is copyrighted by the authors (Margus Pedaste, Külli Kallas, and Aleksandar Baucal) and the University of Tartu. The instrument is accessible for non-commercial academic research and institutional educational assessment upon requesting official permission from the corresponding author.
• Corresponding Contact: Külli Kallas, Institute of Education, University of Tartu, Jakobi 5, 51005 Tartu, Estonia. Email: [email protected].
• Commercial Use: No commercial licensing is permitted without formal institutional agreement with the University of Tartu.
• Fee: Free of charge for non-commercial educational and scientific research purposes.

12. References

• Adov, L. (2022). Pedagogical and technological aspects of digital competence: Fostering and assessing students’ and teachers’ digital competence for learning (Doctoral dissertation, University of Tartu). University of Tartu Press.
• Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179–211. https://doi.org/10.1016/0749-5978(91)90020-T
• Pedaste, M., Kallas, K., & Baucal, A. (2023). Digital competence test for learning in schools: Development of items and scales. Computers & Education, 203, Article 104830, 1–19. https://doi.org/10.1016/j.compedu.2023.104830
• Pedaste, M., Leijen, Ä., Pata, K., & San Pedro, M. O. Z. (2021). Developing a digital competence test for primary and lower secondary school students. In Communications in Computer and Information Science (Vol. 1429, pp. 24–36). Springer. https://doi.org/10.1007/978-3-030-77880-4_3
• Weinert, F. E. (2001). Concept of competence: A conceptual clarification. In D. S. Rychen & L. H. Salganik (Eds.), Defining and selecting key competencies (pp. 45–65). Hogrefe & Huber Publishers.

13. Items of the Scale