Direct Observation Form (DOF)

Abstract

The Direct Observation Form (DOF) (Butterfield et al., 2023) is a psychometric and educational assessment instrument designed to evaluate the clinical competence of medical students conducting psychiatric diagnostic and therapeutic interviews during their clinical psychiatry clerkships. Developed at the University of Colorado School of Medicine, the DOF operationalizes essential behavioral, relational, and structural elements of psychiatric interviewing. Its content was curated through an extensive review of competency benchmarks, including guidelines established by the Association of Directors of Medical Student Education in Psychiatry (ADMSEP) (Roman et al., 2016), and refined through iterative clinical pilot testing. The completed instrument consists of four core behavioral items rated on a 5-point Likert-type scale (ranging from 1 to 5, where 5 represents optimal clinical performance): (1) Opening Portion and Closing of the Interview, (2) Interview Techniques, (3) Presence/Attunement to Patient, and (4) Alliance Building and Boundaries. In addition, the instrument incorporates two contextual adjustment indices (case complexity and learner challenge) and an integrative global competency item aligned with Entrustable Professional Activities (EPAs). Evaluated among third-year medical students assessed by supervising psychiatry residents, fellows, and attending physicians, the DOF demonstrates high internal consistency reliability (Cronbach’s alpha = 0.88–0.89) and robust test responsiveness across clinical training rotations (p < 0.001). Evidence of predictive validity is demonstrated by its significant association with final end-of-course clinical grades (p < 0.001), while divergent validity is underscored by an absence of correlation with standardized written psychiatry shelf examination scores (r = 0.10–0.11), affirming that the DOF captures a distinct domain of direct clinical performance unmeasured by traditional multiple-choice tests.

Keywords

Direct Observation Form, Psychiatric Interviewing, Medical Education, Clinical Competence, Workplace-Based Assessment, Psychiatry Clerkship, Entrustable Professional Activities, Psychometrics, Educational Measurement, Predictive Validity, Reliability

Authors

The Direct Observation Form was conceived, developed, and psychometrically validated by a team of academic psychiatrists and medical educators at the University of Colorado Anschutz Medical Campus:

• Austin Butterfield, M.D. — Department of Psychiatry, University of Colorado School of Medicine, Aurora, CO, USA. ORCID: 0000-0003-0231-7266. Email: [email protected].
• Ashley Curry, M.D. — Department of Psychiatry, University of Colorado School of Medicine, Aurora, CO, USA.
• Joel Yager, M.D. — Department of Psychiatry, University of Colorado School of Medicine, Aurora, CO, USA. Professor Emeritus of Psychiatry, prominent author on psychiatric education and interview dynamics.
• Joseph Sakai, M.D. — Department of Psychiatry, University of Colorado School of Medicine, Aurora, CO, USA. ORCID: 0000-0003-1600-6100. Associate Professor specializing in clinical trial methodology, psychometrics, and behavioral addictions.

Purpose

The primary purpose of the Direct Observation Form (DOF) is to provide an efficient, valid, and reliable workplace-based assessment (WBA) tool specifically engineered to measure medical students’ clinical performance during authentic psychiatric patient interviews. In clinical medical education, particularly within psychiatry clerkships, assessing student performance has historically presented significant psychometric and logistical challenges. Educational programs frequently rely upon proxy measures of clinical skill, such as multiple-choice standardized tests (e.g., the National Board of Medical Examiners [NBME] Psychiatry Subject Examination or “shelf exam”), post-encounter written patient write-ups, or oral case presentations delivered well after the patient interaction has concluded.

While standardized cognitive tests effectively capture factual medical knowledge, psychopharmacological principles, and diagnostic nomenclature, they fail to evaluate the interactive, nonverbal, emotional, and relational micro-skills required to conduct an effective psychiatric assessment. When educators assess learners solely through retrospective case presentations, they observe the student’s cognitive synthesis and narrative reporting abilities rather than their direct interpersonal acumen, empathy, technique, or boundary management. Direct observation by experienced clinical educators has long been recognized as the gold standard for clinical skills assessment, yet it is notoriously underutilized due to clinical time constraints, high faculty burden, and a lack of brief, psychometrically sound evaluation instruments tailored specifically to psychiatric diagnostic encounters.

The DOF was engineered to resolve this clinical and pedagogical dilemma. Developed in alignment with contemporary competency-based medical education (CBME) initiatives, the DOF serves three interrelated clinical and academic purposes:

• Formative Skill Development and Real-Time Feedback: The DOF acts as a structured coaching guide for clinical preceptors—including attending psychiatrists, child and adolescent or forensic psychiatry fellows, and psychiatry residents—facilitating immediate, targeted, actionable feedback following observed student-patient encounters.
• Summative Clerkship Evaluation: The tool yields reliable, quantitative longitudinal data reflecting a trainee’s clinical trajectory over the course of their rotation, serving as a robust component of the clerkship’s final clinical performance grade.
• Entrustment Decision-Making: By integrating a dedicated Entrustable Professional Activity (EPA) metric, specifically reflecting Core EPA 1 (“Gather a history and perform a physical examination” tailored to the psychiatric mental status interview), the DOF allows training programs to quantify the level of supervision a medical student requires when eliciting sensitive psychiatric histories and assessing acute psychopathology.

Psychological Construct

The Direct Observation Form measures the overarching psychological and educational construct of Clinical Psychiatric Interviewing Competence. This multidimensional construct encompasses cognitive, behavioral, interpersonal, and affective domains that must be seamlessly integrated during real-time diagnostic and therapeutic encounters with individuals experiencing psychiatric distress. Psychiatric interviewing differs fundamentally from standard medical history-taking: it necessitates navigating heightened affective states, cognitive distortions, social stigma, trauma histories, suicidal ideation, and acute behavioral disinhibition, all while fostering a secure therapeutic container.

The DOF operationalizes this broad construct across four distinct, interconnected dimensions, alongside an overarching global entrustment appraisal:

1. Opening Portion and Closing of the Interview

This structural dimension reflects the trainee’s capacity to establish procedural clarity, safety, and closure. The opening requires the clinician to introduce themselves clearly, explain their role as a student doctor, clarify the nature and approximate duration of the evaluation, establish mutual expectations, and articulate the bounds of medical confidentiality. In individuals with acute paranoia, profound depression, or severe anxiety, an ambiguous opening exacerbates psychological distress. Conversely, the closing evaluates the student’s skill in summarizing the collected narrative, validating the patient’s shared experience, outlining subsequent diagnostic or therapeutic steps, collaboratively answering questions, and terminating the interview safely without abruptness or clinical abandonment.

2. Interview Techniques

This technical dimension measures the student’s clinical micro-skills and methodological versatility. It evaluates the strategic modulation between open-ended exploratory questions (e.g., inviting personal narrative, facilitating spontaneous affective expression) and closed-ended clarifying questions (e.g., establishing exact timelines, conducting a formal Review of Systems, evaluating criteria for major depressive episodes or psychotic disorders). Furthermore, this dimension assesses the clinician’s agility in managing interview flow: gracefully interrupting tangential or circumstantial speech without alienating the patient, employing active listening, utilizing silence therapeutically, reflecting content, clarifying inconsistencies, and steering the clinical inquiry efficiently toward high-yield diagnostic material.

3. Presence/Attunement to Patient

Rooted in interpersonal psychology and phenomenological psychiatry, this affective-relational dimension captures the student’s non-verbal and emotional resonance within the clinical dyad. It assesses the clinician’s capacity for authentic engagement, sustained non-defensive attention, and active empathy. Observational markers include culturally appropriate eye contact, open postural orientation, receptive vocal cadence, and the capacity to perceive, decode, and adapt to subtle shifts in the patient’s non-verbal cues (e.g., sudden tearfulness, psychomotor agitation, guarded posture, or emotional withdrawal). Attunement also implies a capacity to remain calm, grounding, and emotionally regulated when confronted with intense patient affect, hostility, or emotional distress.

4. Alliance Building and Boundaries

This psychodynamic and ethical dimension examines the establishment and preservation of a viable therapeutic alliance while maintaining scrupulous professional boundaries. The trainee must demonstrate warm, non-judgmental acceptance and build mutual trust, which is critical for facilitating disclosures regarding sensitive topics such as suicidal ideation, substance abuse, sexual history, or trauma. Concurrently, the student must exhibit robust boundary preservation: resisting manipulative boundary crossings, avoiding inappropriate self-disclosure, setting kind yet firm limits when encountering behavioral escalation, and navigating complex transference or countertransference dynamics without becoming defensive, retaliatory, or excessively intimate.

Contextual Factors and Entrustment

The construct recognized by the DOF explicitly acknowledges that psychiatric interviewing performance does not occur in a vacuum. It is heavily modulated by environmental and patient-specific variables. Consequently, the construct includes explicit observer adjustments for case complexity (e.g., severe delirium, psychosis, severe personality pathology, language barriers) and task difficulty relative to the learner’s developmental stage. Finally, the construct synthesizes these behavioral components into an overarching determination of clinical entrustability—reflecting the student’s holistic readiness to gather a psychiatric history with diminishing levels of direct supervision.

Theoretical Framework

The architecture and assessment philosophy of the Direct Observation Form are rooted in modern theories of medical education, cognitive apprenticeship, and interpersonal psychology.

Miller’s Pyramid of Clinical Competence

The foundational framework informing the DOF is Miller’s Pyramid (Miller, 1990), an influential conceptual model that stratifies clinical assessment into four progressive hierarchical tiers:

1. Knows (Basic medical knowledge, measured via didactic testing);
2. Knows How (Applied clinical reasoning and context-specific decision-making, measured via clinical vignette exams or case analyses);
3. Shows How (Performance demonstration in simulated environments, measured via Objective Structured Clinical Examinations [OSCEs]);
4. Does (Action and behavior in unscripted, authentic clinical workplace practice).

Traditional psychiatric clerkship evaluations have persistently operated at the “Knows” and “Knows How” levels through standardized multiple-choice exams, or intermittently at the “Shows How” level through standardized patient simulations. The DOF targets the pinnacle of Miller’s framework: the “Does” level. By situating the assessment within live patient consultations, the DOF captures true clinical behavior influenced by clinical pressures, chaotic hospital environments, fatigue, and real patient distress.

Workplace-Based Assessment (WBA) and Deliberate Practice

The DOF operationalizes the principles of Workplace-Based Assessment articulated by Norcini (2003). WBA posits that competence is context-dependent, necessitating brief, repeated, low-stakes evaluations embedded within routine workflow rather than a single, high-stakes terminal assessment. This paradigm intersects with K. Anders Ericsson’s theory of Deliberate Practice (Ericsson, 2004). Deliberate practice asserts that acquisition of expert clinical performance requires immediate, precise feedback tied to concrete behavioral components, followed by targeted opportunities for iterative refinement. The DOF’s four core behavioral dimensions furnish supervising preceptors with a shared conceptual vocabulary to deliver structured feedback immediately following the patient encounter.

Entrustable Professional Activities (EPAs)

The instrument incorporates the paradigm of Entrustable Professional Activities introduced by ten Cate (2005). EPAs operationalize competencies into discrete units of professional work that can be entrusted to a learner once they demonstrate adequate capability. The global assessment question on the DOF aligns directly with the Association of American Medical Colleges (AAMC) Core EPA 1, shifting the preceptor’s evaluation from an abstract grade to a pragmatic clinical judgment: “How much supervision does this student require to safely perform this specific clinical encounter?”

Kolb’s Experiential Learning Theory

The DOF functions as a pedagogical catalyst within Kolb’s (1984) Experiential Learning Cycle. The observed psychiatric interview represents the Concrete Experience. The completion of the DOF and the subsequent preceptor-student debrief foster Reflective Observation. The student synthesizes this feedback into Abstract Conceptualization, revising their internal cognitive models regarding patient engagement and psychiatric symptomatology. Finally, the student engages in Active Experimentation during their subsequent interview, completing the educational feedback loop.

Validity

The psychometric validation of the Direct Observation Form was conducted by Butterfield et al. (2023) at the University of Colorado School of Medicine, evaluating third-year medical students rotating through inpatient, outpatient, and consult-liaison psychiatric clinical services. Validity was evaluated using contemporary measurement standards reflecting content, response process, internal structure, and relations to other variables (Kane, 2013).

Content Validity and Test Development

Content validity was established through a structured consensus approach. The item domains were derived from established national educational objectives published by the Association of Directors of Medical Student Education in Psychiatry (Roman et al., 2016). Academic psychiatrists, clerkship directors, and medical education specialists iteratively refined the phrasing to ensure that each item mapped cleanly onto concrete, observable behaviors rather than ambiguous personality traits. Pilot testing confirmed the feasibility of completion within two to three minutes post-interview, minimizing administrative burden and cognitive fatigue among raters.

Test Responsiveness (Sensitivity to Change Over Time)

A crucial indicator of an educational measurement tool is its responsiveness to developmental progression over time. The DOF demonstrated profound test responsiveness across the clerkship rotation. A paired comparison of medical student performance evaluated at Time 1 (early clerkship phase) versus Time 2 (late clerkship phase) revealed a highly significant increase in mean overall DOF scores:

• Temporal Progression: Mean composite scores demonstrated statistically significant increases from Time 1 to Time 2 (p < 0.001), indicating that the scale is sensitive to the clinical maturation, feedback assimilation, and skill acquisition that occurs during clinical immersion.
• Academic Calendar Trajectory: Mean DOF scores demonstrated a statistically significant association with the time of the academic year during which students completed the rotation. Students completing their psychiatry rotation early in the third year (when overall clinical experience was minimal) scored significantly lower than those completing the clerkship in the second half of the year (Time 1: p = 0.01; Time 2: p = 0.002), confirming that the instrument captures cumulative clinical development across medical school.

Predictive and Criterion Validity

To evaluate predictive validity, DOF scores were examined in relation to final clerkship clinical grades awarded at the end of the rotation. Stepwise multiple regression and multivariate analyses demonstrated that:

• End-of-Course Clinical Grades: Time 2 DOF scores exhibited a strong, positive, and statistically significant association with the student’s final clinical course grade (p < 0.001).
• Control for Confounders: Crucially, this robust predictive association remained highly significant (p < 0.001) even after statistically controlling for several potent confounding variables, including: (a) rater rank (resident vs. fellow vs. attending), (b) rater clinical site, (c) subjective difficulty of the interview, (d) clinical complexity of the patient, and (e) timing within the academic year.

Convergent Validity

Convergent validity was evaluated by comparing the four-item composite score to the independent, single-item Entrustable Professional Activity (EPA) rating completed by the same clinical observer. Strong, highly significant positive correlations were observed at both evaluation intervals:

• Time 1 Correlation: r = 0.76 (p < 0.001)
• Time 2 Correlation: r = 0.79 (p < 0.001)

These large correlation coefficients indicate that the four specific behavioral dimensions operationalized in the DOF strongly underpin and align with global clinical judgments of trainee entrustability.

Discriminant / Divergent Validity

Discriminant validity evaluates whether an instrument diverges from constructs it should theoretically not measure. The DOF scores were correlated against students’ standardized National Board of Medical Examiners (NBME) psychiatry shelf examination scores. The correlations were non-significant and negligible:

• Time 1 Correlation with Shelf Exam: r = 0.10 (p = 0.24)
• Time 2 Correlation with Shelf Exam: r = 0.11 (p = 0.21)

This near-zero correlation provides compelling evidence of discriminant validity. It establishes that the DOF does not simply reflect general academic aptitude, test-taking savvy, or passive medical memorization. Instead, it captures a distinct, performance-based domain of interpersonal and observational clinical competence that traditional multiple-choice examinations completely fail to evaluate.

Rater Rank Effects

The validation study revealed a statistically significant association between rater rank and Time 1 evaluation scores (p = 0.005). Specifically, psychiatry residents tended to assign slightly higher baseline performance scores to medical students than experienced attending physicians. This finding underscores the necessity of anchoring rating scales to behavioral rubrics and highlights differences in supervisory expectations across training tiers.

Reliability

The reliability of the Direct Observation Form was established via classical psychometric indices of internal consistency, alongside analyses of variance across measurement intervals.

Internal Consistency

Internal consistency reflects the degree to which all items on a measurement instrument correlate with one another, ensuring that they assess components of a unified underlying construct. For the 4-item DOF, internal consistency reliability was assessed using Cronbach’s alpha coefficient across distinct testing cohorts and temporal assessment points:

• Cronbach’s Alpha Range: α = 0.88 to 0.89

In psychometrics, an alpha coefficient exceeding 0.80 is considered robust, while values between 0.85 and 0.90 are regarded as optimal for clinical and educational assessments. Crucially, obtaining an alpha of nearly 0.90 on a short, 4-item instrument is psychometrically notable, as Cronbach’s alpha is mathematically sensitive to scale length (all else being equal, fewer items lower the alpha). This indicates exceptionally strong item-total correlations and minimal measurement error among the four core behavioral dimensions.

Standard Error of Measurement and Score Stability

The high internal consistency translates into a low Standard Error of Measurement (SEM), which provides educators with high statistical confidence that an observed score closely approximates the student’s true latent clinical ability. While test-retest reliability in the strict psychometric sense (administering the identical test under identical conditions) is neither methodologically feasible nor educationally desirable in workplace-based assessment (because students naturally encounter different patients with distinct psychopathology), longitudinal stability was demonstrated by predictable item-total correlations across the two clerkship measurement time points.

Factor Analysis

In the original pilot validation study (Butterfield et al., 2023), formal exploratory factor analysis (EFA) or confirmatory factor analysis (CFA) was not reported due to the intentionally parsimonious scale design (4 core items) and sample size constraints relative to the parameter-to-sample ratio required for structural equation modeling.

However, the psychometric characteristics of the scale yield clear theoretical and statistical inferences regarding its latent dimensionality:

• Unidimensionality: The exceptionally high internal consistency (α = 0.88–0.89) among only four items strongly suggests that the DOF operates as a unidimensional assessment tool. In classical psychometric theory, when a short battery of 4 items yields an alpha nearing 0.90, the items share a massive proportion of common variance, indicating that they reflect a single overarching latent trait: Clinical Psychiatric Interviewing Competence.
• Shared Variance and Item Communalities: The four dimensions—structure, interviewing micro-skills, interpersonal attunement, and alliance/boundaries—are functionally inseparable in authentic clinical practice. A clinician cannot achieve deep emotional attunement while exhibiting chaotic interviewing techniques, nor can one establish a solid therapeutic alliance without sound boundary preservation. Consequently, high inter-item covariance is both theoretically expected and empirically demonstrated.
• Future Structural Equation Modeling: Subsequent large-scale, multi-institutional studies are positioned to conduct formal CFA testing to confirm the unidimensional model against a potential bifactor model (wherein a general interviewing factor accounts for the bulk of the variance, with minor orthogonal residual factors accounting for technical vs. relational competencies).

Instrument / Measurement Tool

The Direct Observation Form is structured as follows:

• Test Name: Direct Observation Form (DOF)
• Test Type: Workplace-Based Assessment (WBA) / Direct Clinical Observation Rating Scale
• Target Population: Third-year medical students completing their clinical psychiatry clerkship (applicable to clinical-level medical students, psychiatric sub-interns, or early psychiatry residents).
• Raters / Observers: Supervising clinical educators, including attending psychiatrists, psychiatry fellows, and senior psychiatry residents.
• Administration Setting: Inpatient psychiatric units, outpatient mental health clinics, psychiatric emergency services, and consultation-liaison services.
• Administration Time: Completed immediately post-encounter (approximately 2 to 4 minutes).
• Instrument Structure:
  • 4 Core Behavioral Items: Scored on a standardized 5-point Likert scale (1 = Poor/Novice performance, 5 = Excellent/Advanced performance).
  • 2 Contextual Adjustment Checkboxes: Optional flags indicating: (a) Interview was particularly challenging for learner’s level of training; (b) Patient/case exhibited high clinical complexity.
  • 1 Global Entrustment Item: Single-item Entrustable Professional Activity (EPA) rating reflecting the degree of supervisory oversight required by the learner.
  • Qualitative Formative Feedback: Designated space for narrative preceptor feedback highlighting strengths and specific areas for targeted behavioral improvement.
• Scoring and Interpretation:
  • Core Sum Score: Total possible score ranges from 4 to 20 points across the four core dimensions.
  • Core Mean Score: Calculated by averaging the four items (range: 1.00 to 5.00).
  • Benchmarking: Early clerkship (Time 1) scores typically cluster around intermediate developmental anchors (2.50–3.50), with advanced rotation (Time 2) scores shifting rightward (3.75–4.75), indicating clinical skill acquisition.

Permissions & Fee and Test Year

The Direct Observation Form was formally introduced and published in 2023 by the medical education research team at the University of Colorado School of Medicine (Butterfield et al., 2023).

• Permissions: The DOF is an open-access academic evaluation instrument developed for clinical and pedagogical purposes. Medical schools, residency training programs, and clinical educators are permitted to utilize the scale for educational and research assessment. Inquiries regarding implementation protocols or adaptations should be directed to the corresponding author, Dr. Austin Butterfield, at [email protected].
• Commercial Status: Non-commercial.
• Fee: Free of charge ($0.00) for academic and educational research use.

References

• Butterfield, A., Curry, A., Yager, J., & Sakai, J. (2023). A direct observation form for evaluation of the psychiatric interview: Pilot testing during the psychiatry clerkship. Academic Psychiatry, 47(4), 368–373. https://doi.org/10.1007/s40596-023-01762-0
• Ericsson, K. A. (2004). Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Academic Medicine, 79(10 Suppl), S70–S81. https://doi.org/10.1097/00001888-200410001-00022
• Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. https://doi.org/10.1111/jedm.12000
• Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development. Prentice-Hall.
• Miller, G. E. (1990). The assessment of clinical skills/competence/performance. Academic Medicine, 65(9 Suppl), S63–S67. https://doi.org/10.1097/00001888-199009000-00045
• Norcini, J. J. (2003). Workplace-based assessment in clinical training. BMJ, 326(7392), 753–755. https://doi.org/10.1136/bmj.326.7392.753
• Roman, B., Goldenberg, M., Grapentine, C. G., & Stewart, V. D. (2016). ADMSEP clinical training commission psychiatric clinical skills verification draft. Association of Directors of Medical Student Education in Psychiatry. https://doi.org/10.1007/s40596-015-0453-6
• ten Cate, O. (2005). Entrustability of professional activities and competency-based training. Medical Education, 39(12), 1176–1177. https://doi.org/10.1097/00001888-200503000-00008

13. Items of the Scale (Questionnaire)