Dialogue Skills Assessment Scale

Abstract

The Dialogue Skills Assessment Scale (DSAS; Daif-Allah & Al-Sultan, 2023) is an observational behavioral rating instrument designed to evaluate the acquisition, execution, and mastery of interpersonal communication and conversational competencies among adult learners of Arabic as a Second Language (ASL). Developed within the pedagogical context of an experimental investigation into the efficacy of simulation and role-play techniques at Qassim University, the scale operationalizes conversational competence across four interrelated behavioral standards: Self-esteem, Good listening, Expression of opinion, and Respect for others. Each dimension encompasses five criterion-referenced behavioral indicators, yielding a 20-item assessment protocol scored on an objective dichotomous presence-absence format (one mark per observed indicator, maximum score of 20 points, scaled to a 100% mastery distribution). Grounded in the communicative competence frameworks of Canale and Swain and the assessment rubrics of Idham (2022), Al-Harahsheh (2017), and Huba and Freed (2000), the instrument demonstrates acceptable internal consistency reliability (Cronbach’s alpha = 0.73). This article provides a comprehensive psychometric and theoretical review of the DSAS, analyzing its conceptual underpinnings in sociolinguistic theory, pragmatics, and sociocultural learning theory, while detailing its structural composition, scoring protocols, diagnostic utility, empirical boundaries, and methodological avenues for structural equation modeling and construct validation.

Keywords

Dialogue Skills Assessment, Arabic as a Second Language, Communicative Competence, Active Listening, Affective Filter, Interactional Competence, Second Language Acquisition, Pragmatic Competence, Role-Play Pedagogy, Observational Rating Scale, Educational Measurement, Oral Language Assessment

Authors

The Dialogue Skills Assessment Scale was developed by Ayman Sabry Daif-Allah and Muhammad Sultan Al-Sultan.

• Ayman Sabry Daif-Allah: Department of English Language and Translation, College of Arabic Language and Social Studies, Qassim University, Buraydah, Saudi Arabia. ORCID: 0000-0003-4019-3583. Correspondence email: [email protected].
• Muhammad Sultan Al-Sultan: Department of Arabic Language and Arts, College of Arabic Language and Social Studies, Qassim University, Buraydah, Saudi Arabia.

Purpose

The primary purpose of the Dialogue Skills Assessment Scale (DSAS) is to provide an empirically grounded, criterion-referenced observational metric for assessing the multifaceted nature of conversational and dialogic competencies in adult second language learners. Although oral proficiency testing in second language acquisition (SLA) has historically prioritized structural dimensions such as phonological accuracy, lexical diversity, and syntactic complexity, modern linguistic paradigms increasingly emphasize interactional competence—the capacity of speakers to co-construct meaning, navigate discourse turn-taking, and maintain relational harmony during real-time communicative exchanges.

In instructional and research settings, the DSAS fulfills both diagnostic and evaluative functions. Methodologically, it was engineered to evaluate experimental pedagogical interventions, specifically the deployment of structured, semi-structured, and open role-play simulations designed to foster real-world communicative fluency. Traditional pen-and-paper assessments or decontextualized oral interviews often fail to capture the affective, behavioral, and sociolinguistic nuances of face-to-face debates, collaborative negotiations, and spontaneous interpersonal exchanges. The DSAS bridges this psychometric gap by decomposing dialogic performance into observable, discrete behavioral indicators across four foundational communicative domains.

From an applied perspective, the DSAS serves language educators, clinical communication specialists, and educational psychometricians by providing:

• A standardized diagnostic rubric to detect specific interpersonal deficits (e.g., communicative anxiety, interruption habits, or lack of argumentative coherence) in L2 speakers before and after targeted instructional modules.
• A formative assessment tool that language instructors can use during in-class role-play activities to deliver granular, actionable feedback to adult learners regarding their pragmatic and interactional behavior.
• An evaluative research instrument to benchmark the efficacy of communicative language teaching (CLT) methodologies against conventional, teacher-centered instructional approaches.
• A cross-cultural bridge for evaluating Arabic language acquisition, an area that historically lacked standardized, empirically validated instruments for measuring non-grammatical aspects of dialogic competence such as prosocial politeness strategies and affective composure.

Psychological Construct

The DSAS conceptualizes dialogue skills not as a monolithic oral trait, but as an integrated, multidimensional composite of affective, cognitive, pragmatic, and prosocial behaviors. The construct encompasses four distinct standards:

1. Self-Esteem (Affective-Behavioral Poise)

Within communicative interactions, self-esteem represents the interlocutor’s internal subjective appraisal of their communicative competence and personal worth, manifest through observable confidence, decisiveness, and resilience. Rooted in the psychological construct of willingness to communicate (WTC) and the mitigation of foreign language anxiety, this dimension reflects an individual’s capacity to maintain composure during cognitive load. The behavioral manifestations captured by the scale include projecting vocal and conceptual certainty, making rapid conversational decisions without protracted avoidance behaviors, maintaining physical and postural self-presentation, sustaining cognitive clarity under social scrutiny, and persisting through conversational impasses without emotional disengagement or surrender.

2. Good Listening (Receptive and Interactional Attentiveness)

Listening competence in dialogic discourse transcends passive auditory decoding; it constitutes an active, co-constructive communicative endeavor known as active listening. This construct involves continuous verbal and nonverbal signaling that confirms message reception, verifies shared understanding, and honors the interlocutor’s conversational floor. The DSAS assesses five discrete behavioral expressions of good listening: delivering verbal backchanneling tokens (e.g., affirming phrases indicating sustained attention), requesting explicit clarification of ambiguous linguistic or informational content rather than relying on erroneous assumptions, probing for deeper conceptual clarification of the counterpart’s perspective, observing floor-yielding norms by refraining from untimely interruptions, and systematically capturing critical points made by the interlocutor (e.g., through note-taking or focused cognitive retention) to formulate responsive dialogue.

3. Expression of Opinion (Discursive and Argumentative Structuring)

The capacity to articulate personal viewpoints represents the cognitive-linguistic engine of dialogic exchange. This dimension requires interlocutors to externalize complex cognitive schemas into coherent, persuasive, and socially contextualized verbal discourse. The construct captures structural organization, evidential grounding, temporal discipline, cognitive openness, and collaborative problem-solving. Operationally, an individual scoring high on this dimension organizes their arguments into logically sequenced points, presents rational evidence and substantiating claims, respects conversational time boundaries without monopolizing the floor, demonstrates cognitive flexibility by tolerating divergent viewpoints, and engages in dynamic bidirectional exchange aimed at resolving disagreements rather than sustaining rigid polarization.

4. Respect for Others (Prosocial and Pragmatic Politeness)

The relational viability of dialogue depends upon prosocial regulation, interpersonal politeness, and face-management. In second language interaction, where cultural and ideological differences frequently intersect, respecting the interlocutor’s personal and social identity is vital for constructive communication. This construct evaluates the affective tone and ethical orientation brought to interpersonal discourse. It manifests behaviorally through prosocial conversational openings (e.g., initiating dialogue with warmth, greetings, and relational benevolence), validating the partner’s verbal contributions while refraining from mockery or condescension, exhibiting tolerance for individual differences (including personal interests, hobbies, and talents), inhibiting aggressive emotional reactivity and verbal violence, and preserving unconditional positive regard for the counterpart’s perspective even in the presence of fundamental cognitive or ideological disagreement.

Theoretical Framework

The structural architecture of the Dialogue Skills Assessment Scale is grounded in three converging theoretical paradigms: the Communicative Competence Model, Sociocultural Theory, and the Affective Filter Hypothesis.

First and foremost, the instrument operationalizes the seminal communicative competence frameworks developed by Dell Hymes (1972) and later refined for language pedagogy by Canale and Swain (1980) and Bachman (1990). Hymes rejected purely formalist definitions of language competence (such as Noam Chomsky’s grammatical competence), arguing that a competent speaker must know not only what is grammatically correct, but what is socially appropriate, feasible, and contextualized. Canale and Swain expanded this model into four domains: grammatical competence, sociolinguistic competence, discourse competence, and strategic competence. The DSAS directly embodies sociolinguistic and discourse competencies: the Expression of opinion standard captures discourse organization and coherence, while the Respect for others standard operationalizes sociolinguistic rules of appropriateness, face-saving strategies, and politeness phenomena (Brown & Levinson, 1987).

Second, the scale is theoretically anchored in Lev Vygotsky’s Sociocultural Theory of Mind (1978), which posits that higher mental functions, including linguistic cognition, originate through social interaction and dialogic mediation within the Zone of Proximal Development (ZPD). From this perspective, dialogue is not merely an external display of pre-existing internal linguistic knowledge; it is the communicative space where cognition and language are dynamically co-constructed. By using the scale to evaluate role-play interventions, Daif-Allah and Al-Sultan (2023) drew upon the dialogic principles of Mikhail Bakhtin (1981), who emphasized that meaning exists not in an isolated speaker, but in the interactive tension between the speaker and the listener. The inclusion of the Good listening standard reflects this reciprocal, Bakhtinian view of dialogue, acknowledging that active reception and responsive turn-taking are as fundamental to communication as verbal production.

Third, the Self-esteem dimension reflects Stephen Krashen’s Affective Filter Hypothesis (1982) and modern models of foreign language anxiety. Krashen argued that affective variables—namely self-confidence, anxiety, and motivation—act as an emotional barrier that can impede language acquisition and fluent output. When second language learners experience low self-esteem or heightened communicative apprehension, their affective filter rises, leading to cognitive freezing, fragmented articulation, and avoidance. By systematically tracking affective composure, cognitive clarity, and behavioral determination, the DSAS quantifies the external behavioral manifestations of a lowered affective filter, validating the hypothesis that experiential learning practices like role-playing bolster self-efficacy (Bandura, 1997) and conversational fluency.

Validity

The initial development and deployment of the Dialogue Skills Assessment Scale by Daif-Allah and Al-Sultan (2023) established essential forms of validity, while identifying specific psychometric domains that warrant continued empirical testing:

Content and Face Validity

Content validity was established through systematic domain sampling and item synthesis adapted from established communicative rubrics, notably the works of Idham (2022), Al-Harahsheh (2017), and Huba and Freed (2000). The 20 behavioral indicators were developed to span the cognitive, affective, and interactional requirements of collegiate-level discourse. The items underwent qualitative expert panel evaluation by university-level specialists in Arabic language pedagogy, second language acquisition, and applied linguistics at Qassim University. This review ensured that the indicators exhibited face validity, instructional relevance, linguistic clarity, and cultural alignment with the communication norms expected in academic and semi-formal Arabic interaction.

Criterion-Related and Construct Sensitivity

The construct sensitivity of the scale was supported by the experimental findings reported by Daif-Allah and Al-Sultan (2023). In their quasi-experimental investigation comparing an experimental cohort instructed via structured role-play techniques against a control cohort taught via traditional lecture-based methods, the DSAS detected statistically significant performance divergence between groups on the post-test. The experimental group exhibited marked increases across all four standards, providing evidence for the scale’s sensitivity to instructional manipulation and its capacity to measure intervention-driven skill acquisition.

Psychometric Gaps and Future Construct Validation

Because the primary focus of the initial publication was pedagogical efficacy, formal psychometric analyses of construct validity—such as convergent validity against standardized language tests (e.g., ACTFL OPI or CEFR oral scales), discriminant validity against general non-communicative cognitive tests, and Multitrait-Multimethod (MTMM) modeling—remain to be conducted. Future psychometric investigations should examine:

• Convergent Validity: Correlating DSAS aggregate and subscale scores with validated interactional rubrics, such as the Pragmatic Competence Scale or established communicative anxiety inventories (e.g., the FLCAS).
• Discriminant Validity: Demonstrating that DSAS performance dissociates from pure lexical recall or static grammatical recognition tests, confirming that it measures distinct interactive and pragmatic constructs.
• Ecological and Predictive Validity: Determining the degree to which DSAS scores predict success in spontaneous real-world intercultural conversations, academic oral defenses, and professional Arabic language negotiations.

Reliability

The reliability of the Dialogue Skills Assessment Scale has been evaluated primarily through internal consistency estimation, yielding acceptable psychometric indices for an exploratory observational metric:

Internal Consistency

Daif-Allah and Al-Sultan (2023) reported an overall Cronbach’s alpha coefficient of 0.73 for the 20-item scale when administered to advanced-level adult learners in the Arabic Language Teaching Unit for Non-Native Speakers at Qassim University. In psychometric convention (Nunnally & Bernstein, 1994; George & Mallery, 2003), an alpha coefficient between 0.70 and 0.80 denotes acceptable internal consistency reliability, indicating that the 20 behavioral indicators measure a cohesive underlying domain of dialogue competence without excessive item redundancy.

Inter-Rater and Test-Retest Considerations

Because the DSAS is structured as an observational behavioral rating rubric scored by educational evaluators rather than a self-report questionnaire, its overall operational reliability is fundamentally dependent upon inter-rater reliability (scorer agreement). When multiple evaluators assess live or video-recorded role-play interactions, calculating Cohen’s kappa (κ) for individual dichotomous items or the Intraclass Correlation Coefficient (ICC; Shrout & Fleiss, 1979) for overall composite scores is necessary to control for evaluator leniency or severity bias.

Additionally, while test-retest reliability data have not been formally published, the stability of the measure over time should be interpreted in light of ongoing pedagogical intervention: while stable baseline scores would be expected over a short, non-interventional test-retest window (e.g., 7 to 14 days), meaningful shifts in DSAS scores are expected following structured role-play training, reflecting the dynamic nature of communicative skill acquisition.

Factor Analysis

In the original validation study by Daif-Allah and Al-Sultan (2023), formal exploratory factor analysis (EFA) or confirmatory factor analysis (CFA) was not reported due to the bounded sample size of advanced non-native Arabic university students enrolled in the intensive language unit. The scale’s four-dimensional architecture was derived deductively from prior pedagogical and communicative models (Al-Harahsheh, 2017; Huba & Freed, 2000; Idham, 2022) rather than induced through data-driven latent variable extraction.

Recommended Factor Analytic Configurations

To establish structural construct validity, future large-scale psychometric studies should formally test the latent dimensionality of the 20 items using structural equation modeling (SEM) frameworks. The theoretical architecture implies three plausible competing structural models:

• Unidimensional Model: A baseline model specifying all 20 indicators loading directly onto a single, overarching latent factor of General Dialogue Competence. Given the distinct affective, cognitive, and relational requirements of the subscales, this model would likely display suboptimal fit indices.
• Correlated Four-Factor Model: A four-factor specification representing the four theoretical standards: Self-esteem (ξ₁), Good listening (ξ₂), Expression of opinion (ξ₃), and Respect for others (ξ₄), where each latent construct is defined by its five corresponding indicators, with inter-factor correlations freely estimated. This model reflects the conceptual design proposed by the authors.
• Bifactor or Hierarchical Higher-Order Model: A higher-order specification wherein a general dialogic competence factor explains common variance across all 20 indicators, while four orthogonal group factors capture specific domain variance unique to the four subdimensions. This approach allows researchers to determine whether the DSAS is best interpreted as a single global composite or as four separate subscale profiles.

Given the binary nature of the scoring system (0 = absent, 1 = present), subsequent CFA evaluations must utilize appropriate robust estimation techniques designed for categorical variables, such as WLSMV (Diagonally Weighted Least Squares with mean and variance adjustment), rather than standard maximum likelihood (ML) estimation, to avoid biased parameter estimates, distorted standard errors, and inflated fit statistics.

Instrument / Measurement Tool

The operational specifications of the Dialogue Skills Assessment Scale are structured as follows:

• Test Type: Observational Behavioral Rating Scale / Performance Rubric (Evaluator-Administered or Peer-Assessment Protocol).
• Construct Assessed: Interpersonal Dialogue and Conversational Skills in Second Language Contexts.
• Target Population: Adult second language learners (aged 18 years and older), specifically developed and calibrated for advanced-level university students of Arabic as a Second Language.
• Number of Standards / Dimensions: 4 standards (Self-esteem; Good listening; Expression of opinion; Respect for others).
• Number of Items / Indicators: 20 discrete behavioral indicators (5 indicators per standard).
• Response Scale: Dichotomous criterion-referenced performance marking. The scale consists of four standards, with five indicators each, that measure specific dialogue skills. One mark is assigned to each indicator if the target behavior is observed, and zero marks are assigned if the behavior is absent.
• Scoring Formula and Metric:
  • Subscale Scores: Each standard yields a raw score ranging from 0 to 5 points.
  • Aggregate Score: The sum of all four standards yields a total raw score ranging from 0 to 20 marks.
  • Percentage Equivalence: Total Score: 20 (100%). Each individual indicator represents 5% of the total available score.
• Administration Format: Evaluators observe interactive language exercises (e.g., paired role-plays, competitive debates, small-group consensus tasks) lasting between 10 and 20 minutes, systematically checking off observed behavioral criteria in real time or from high-definition audiovisual recordings.

Permissions & Fee and Test Year

The Dialogue Skills Assessment Scale was formally published in 2023 by Ayman Sabry Daif-Allah and Muhammad Sultan Al-Sultan in the academic journal Education Sciences.
In accordance with open-access scientific publishing standards, the scale is distributed under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0). Under this open-access framework:

• Commercial Use: No commercial fee is required; however, commercial users must respect licensing stipulations and provide appropriate attribution.
• Educational and Research Use: Researchers, educators, and psychometricians are permitted to copy, distribute, adapt, and build upon the instrument free of charge for non-commercial and scholarly purposes, provided appropriate scholarly citation is accorded to the original authors.
• Correspondence Contact: For inquiries regarding institutional adaptation or pedagogical implementation, contact Dr. Ayman Sabry Daif-Allah at Qassim University via email at [email protected].

References

• Al-Harahsheh, A. M. (2017). The effect of using dialogue journals on developing writing skills of Jordanian secondary school students. Journal of Education and Practice, 8(15), 148–156.
• Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford University Press.
• Bakhtin, M. M. (1981). The dialogic imagination: Four essays (C. Emerson & M. Holquist, Trans.). University of Texas Press.
• Bandura, A. (1997). Self-efficacy: The exercise of control. W. H. Freeman and Company.
• Brown, P., & Levinson, S. C. (1987). Politeness: Some universals in language usage. Cambridge University Press. https://doi.org/10.1017/CBO9780511813085
• Canale, M., & Swain, M. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 1(1), 1–47. https://doi.org/10.1093/applin/1.1.1
• Daif-Allah, A. S., & Al-Sultan, M. S. (2023). The effect of role-play on the development of dialogue skills among learners of Arabic as a second language. Education Sciences, 13(1), Article 50. https://doi.org/10.3390/educsci13010050
• George, D., & Mallery, P. (2003). SPSS for Windows step by step: A simple guide and reference (4th ed.). Allyn & Bacon.
• Huba, M. E., & Freed, J. E. (2000). Learner-centered assessment on college campuses: Shifting the focus from teaching to learning. Allyn & Bacon.
• Hymes, D. (1972). On communicative competence. In J. B. Pride & J. Holmes (Eds.), Sociolinguistics: Selected readings (pp. 269–293). Penguin Books.
• Idham, A. (2022). Developing a dialogue skills checklist for foreign language students. Journal of Educational and Psychological Studies, 16(2), 210–225.
• Krashen, S. (1982). Principles and practice in second language acquisition. Pergamon Press.
• Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill.
• Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. https://doi.org/10.1037/0033-2909.86.2.420
• Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.

13. Items of the Scale (Questionnaire)