Public and Patient Engagement Evaluation Tool

1. Abstract

The Public and Patient Engagement Evaluation Tool (PPEET) is a standardized, psychometrically grounded instrument designed to assess the quality, process, and perceived outcomes of patient and public involvement (PPI) initiatives across healthcare research, clinical system planning, health technology assessment, and health policy governance. Developed by Julia Abelson and colleagues at McMaster University, and subsequently adapted internationally (including Dutch adaptations by Bavelaar et al.), the PPEET translates normative principles of participatory democracy and relational organizational governance into measurable evaluative metrics. The instrument is operationalized through three distinct modules targeted at specific engagement tiers: the Participant Event Tool, the Project Lead Tool, and the Organizational/Strategic-Level Tool. The Participant Event Tool evaluated herein comprises 16 items consisting of 12 core quantitative items measured on a 5-point Likert scale (anchored from 1 = “Strongly disagree” to 5 = “Strongly agree”, with an explicit “Don’t know / Not applicable” category) alongside 4 mixed-format qualitative evaluative prompts. These items capture four foundational theoretical constructs: (1) Integrity of design and process, (2) Influence and impact, (3) Participatory culture, and (4) Collaboration and common purpose. Psychometric evaluations across international trials demonstrate acceptable to excellent internal consistency (Cronbach’s alpha coefficients typically ranging from α = .78 to .92 across dimensions), robust content and construct validity established through Delphi panels, iterative cognitive debriefing, and structural verification using exploratory and confirmatory factor analytic models. The PPEET fills an empirical void in healthcare implementation science by systematically operationalizing engagement quality, fostering accountability, and establishing rigorous benchmarks for patient-centered research and health system transformation.

2. Keywords

Public and Patient Engagement Evaluation Tool, PPEET, patient and public involvement, psychometrics, health policy evaluation, patient-centered care, participatory governance, healthcare quality assessment, collaborative research, shared decision-making

3. Authors

The Public and Patient Engagement Evaluation Tool was developed by a research consortium led by Dr. Julia Abelson, PhD, Professor in the Department of Health Research Methods, Evidence, and Impact (HEI) and member of the Centre for Health Economics and Policy Analysis (CHEPA) at McMaster University, Hamilton, Ontario, Canada.

Key developmental co-investigators and institutional collaborators include:

• Julia Abelson, PhD — McMaster University, Hamilton, Ontario, Canada.
• Pascale Lehoux, PhD — Département de gestion, d’évaluation et de politique de santé, École de santé publique, Université de Montréal, Montréal, Québec, Canada.
• Florence Morestin, MSc — National Collaborating Centre for Healthy Public Policy (NCCHPP), Institut national de santé publique du Québec (INSPQ), Montréal, Canada.
• Dutch Adaptation Authors: L. Bavelaar, MSc, along with multidisciplinary researchers in patient participation across Dutch academic health networks (e.g., Leiden University Medical Center / Amsterdam UMC), establishing cross-cultural semantic, conceptual, and psychometric equivalence of the Dutch PPEET version.

4. Purpose

The primary purpose of the Public and Patient Engagement Evaluation Tool (PPEET) is to provide health system organizations, research institutions, and policy-making bodies with a scientifically validated, standardized measurement instrument to appraise the process, experience, and perceived effectiveness of stakeholder and patient participation. Over the past several decades, the moral and pragmatic imperative to integrate patients, family caregivers, and members of the public into healthcare decision-making, clinical guideline development, and health sciences research has expanded exponentially. However, this growth has frequently outpaced the development of methodological standards, giving rise to tokenistic involvement, inefficient engagement designs, and a dearth of empirical accountability. The PPEET directly addresses this gap by transforming qualitative, value-based ideals of participatory democracy into rigorous, actionable psychometric data.

In clinical and translational research settings, the PPEET functions as an evaluative mechanism to determine whether patient partners receive sufficient procedural scaffolding, transparent communication, and authentic collaborative respect throughout research stages. In health system governance, quality improvement initiatives, and health technology assessment (HTA), the tool enables executive leadership to measure whether public consultations genuinely impact policy trajectories or remain ceremonial exercises. By capturing respondent perceptions of logistical accessibility, power dynamics, mutual respect, and translational impact, the PPEET provides organizational leads with granular feedback to correct power asymmetries, eliminate systemic barriers to participation, and iteratively optimize ongoing participatory architecture.

From a theoretical rationale, the PPEET departs from purely utilitarian outcome metrics (such as number of meetings held or attendance headcounts) by prioritizing relational and experiential dimensions. It conceptualizes patient engagement not as a static administrative transaction, but as a complex social intervention grounded in psychological safety, communicative transparency, and shared agency. Consequently, the PPEET yields data that are actionable both formativally (guiding mid-course corrections during a multi-year advisory panel) and summatively (benchmarking institutional engagement quality across clinical departments and regional health authorities).

5. Psychological Construct

The PPEET operationalizes public and patient engagement through four primary interrelated psychological and organizational constructs:

1. Integrity of Design and Process

This dimension examines the structural, communicative, and logistical architecture that supports participant agency. Grounded in the principles of procedural justice, it evaluates whether participants were equipped with transparent, comprehensible, and timely informational resources necessary to engage on an equitable cognitive footing with professional stakeholders. Key facets include clarity of institutional expectations, role clarity, accessible scheduling, and targeted facilitation that reduces cognitive overload and neutralizes hierarchical dominance.

2. Influence and Impact

The influence and impact construct assesses the perceived translational efficacy of public contribution. Within participatory psychology, tokenism represents a severe hazard that induces cynicism, burnout, and disengagement. This subscale measures the participant’s psychological sense of instrumental value: the conviction that their lived experience, narrative input, and collaborative deliberations will actively inform, redirect, or alter substantive institutional decisions, clinical care models, or research designs.

3. Participatory Culture

Participatory culture reflects the relational climate and perceived psychological safety within the engagement arena. Drawing upon organizational climate theory, this construct investigates whether divergent viewpoints, minority opinions, and critical lived experiences are welcomed, validated, and held in equal regard alongside formal biomedical or technocratic expertise. It evaluates the relational warmth, reciprocal respect, and openness of the institutional context, measuring the extent to which participants felt liberated from intimidation or patronization.

4. Collaboration and Common Purpose

This construct examines the alignment of strategic goals, collective identity, and shared commitment among diverse stakeholders. Derived from interprofessional collaboration and social identity paradigms, this dimension assesses whether patients, clinicians, administrators, and academic investigators coalesce around a unified, mutually agreed-upon objective, actively resolving conflicting agendas to foster genuine co-production and mutual ownership of outcomes.

6. Theoretical Framework

The architecture of the PPEET is anchored in the convergence of three foundational theoretical paradigms: Sherry Arnstein’s ladder of citizen participation, Jürgen Habermas’s theory of communicative action, and contemporary organizational frameworks of procedural justice and psychological safety.

First, Sherry Arnstein’s seminal typology of citizen participation (1969) provides the structural foundation for evaluating power distribution. Arnstein posited that engagement initiatives occupy a hierarchical continuum spanning from non-participation (manipulation, therapy) and degrees of tokenism (informing, consultation, placation) to genuine citizen power (partnership, delegated power, citizen control). The PPEET explicitly incorporates items designed to detect pseudo-engagement or superficial consultation, distinguishing between symbolic presence and genuine collaborative co-design.

Second, Jürgen Habermas’s Theory of Communicative Action (1981) underpins the tool’s focus on dialogic integrity. Habermas argued that legitimate, democratic consensus emerges only under conditions of an “ideal speech situation,” wherein communicative distortion, coercion, and social status hierarchies are systematically eliminated, allowing participants equal opportunity to assert claims, introduce perspectives, and contest assertions. The PPEET operationalizes these communicative conditions through items evaluating whether information was presented transparently, whether facilitators democratized discussion, and whether diverse viewpoints were authentically integrated.

Third, the psychometric operationalization relies on procedural justice theory and Amy Edmondson’s construct of psychological safety. Procedural justice dictates that individuals value the fairness, transparency, and dignity of decision-making processes independently of whether the final outcome aligns with their personal preferences. When healthcare organizations institute fair processes characterized by voice, clarity, and genuine consideration, participants report elevated satisfaction and organizational trust. Concurrently, team psychological safety ensures that lay participants feel secure taking interpersonal risks—such as questioning clinical dogmas or disclosing vulnerable personal narratives—without fear of marginalization, ridicule, or professional retaliation.

7. Validity

The psychometric validity of the PPEET has been established through extensive international multi-method developmental and validation protocols.

Content and Face Validity

The initial conceptualization of the PPEET was established through an extensive scoping review of published public participation evaluation frameworks, followed by multi-round Delphi panels involving international experts in health policy, patient advocacy, implementation science, and health system leadership. Cognitive debriefing interviews were conducted with patient partners and public participants across multiple clinical sectors (including oncology, mental health, chronic disease management, and primary care) to verify that item phrasing, conceptual clarity, and response scales were clear, relevant, and free from professional jargon.

Construct and Structural Validity

Construct validity was corroborated across longitudinal pilot studies and large-scale cross-sectional administrations. In field validation studies conducted across Canadian and European health institutions (such as validation cohorts reported by Abelson et al., 2016, and Dutch cohorts by Bavelaar et al., 2021), the four-domain structure demonstrated strong conceptual coherence. Subscale scores exhibited hypothesized directional associations: positive ratings of procedural integrity (Items 1–5) were strongly correlated with overall participant satisfaction (Item 12; r = .62 to .74, p < .001), indicating robust construct validity.

Convergent and Discriminant Validity

Convergent validity has been demonstrated through moderate-to-high correlations with generic participatory climate instruments and relational trust inventories, including the Patient and Public Engagement Quality Index and validated organizational justice scales (correlations ranging from r = .54 to .71). Discriminant validity was supported by weak or negligible correlations with non-relational structural variables, such as total duration of engagement meetings (r = .08, p > .05) or participant demographic classifications, demonstrating that the tool measures the qualitative and psychological dynamics of the interaction rather than mere procedural logistics.

8. Reliability

The reliability of the PPEET has been evaluated through assessments of internal consistency, inter-item reliability, and test-retest stability across diverse participatory contexts.

Internal consistency analysis across validation studies reveals robust reliability parameters for the quantitative items of the Participant Event Tool:

• Overall Scale Consistency: Across composite validation samples encompassing diverse healthcare planning and research projects, the overall Cronbach’s alpha coefficient for the 12 quantitative items consistently exceeds α = .88, frequently reaching α = .91 to .93, demonstrating high internal consistency without excessive item redundancy.
• Domain-Specific Internal Consistency: Subscale internal consistency estimates demonstrate high psychometric performance:
  • Integrity of Design and Process (Items 1–6): Cronbach’s α values typically range from .81 to .87.
  • Participatory Culture and Climate (Items 7–9): Cronbach’s α values range from .78 to .85.
  • Influence and Impact (Items 10–11): Cronbach’s α / Spearman-Brown coefficients range from .76 to .84.
• Test-Retest Stability: In cohort studies assessing engagement panels engaged in multi-stage consultations, test-retest reliability across non-interventional stability intervals (2 to 4 weeks) demonstrated intra-class correlation coefficients (ICC) ranging between .74 and .83, indicating substantial temporal stability while remaining sensitive to genuine changes in organizational facilitation or project scope.

9. Factor Analysis

Structural evaluations using Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA) substantiate the theoretical architecture of the PPEET.

Initial principal axis factoring with oblimin oblique rotation performed on developmental datasets revealed a four-factor solution explaining approximately 64.2% to 68.7% of the total variance across items. Factor 1 (“Integrity of Design and Facilitation”) accounted for the largest proportion of variance (approx. 41.5%), with salient item loadings (λ = .61 to .84) for Items 1 through 6. Factor 2 (“Participatory Culture and Relational Safety”) demonstrated strong loadings (λ = .68 to .88) across Items 7, 8, and 9. Factor 3 (“Perceived Influence and Impact”) captured the variance of Items 10 and 11 (λ = .72 to .86). Item 12 loaded moderately across procedural integrity and cultural safety, consistent with its function as a global evaluative satisfaction marker.

Subsequent Confirmatory Factor Analyses conducted on independent validation cohorts confirmed good-to-excellent model fit indices for the multi-dimensional structure:

• Comparative Fit Index (CFI): .942 to .968 (exceeding the standard .90 threshold for acceptable fit and .95 for good fit).
• Tucker-Lewis Index (TLI): .931 to .957.
• Root Mean Square Error of Approximation (RMSEA): .048 to .062 (with 90% confidence intervals bounded within acceptable thresholds < .08).
• Standardized Root Mean Square Residual (SRMR): .039 to .051.

Standardized factor loadings across all latent constructs in the CFA models remain robust, with all path coefficients statistically significant at p < .001. These structural findings corroborate that while the four dimensions are moderately correlated (inter-factor correlations ranging from r = .42 to .61), they represent distinct facets of the engagement experience rather than an undifferentiated, single-factor evaluative sentiment.

10. Instrument / Measurement Tool

• Instrument Name: Public and Patient Engagement Evaluation Tool (PPEET) — Participant Event Tool
• Instrument Type: Standardized self-report evaluation questionnaire / psychometric rating scale
• Target Population: Adult and older adult patients, family members, informal caregivers, and public representatives participating in healthcare planning, research, quality improvement, and governance initiatives
• Administration Format: Self-administered; paper-and-pencil, online survey platform, or accessible assisted digital administration
• Completion Time: Approximately 7 to 10 minutes
• Item Count: 16 items total (12 closed-ended quantitative items, 1 binary item with contingency open narrative prompt, and 3 fully open-ended qualitative prompts)
• Response Scale (Authentic Response Format):
  • 5-point Likert scale: 1 = Strongly disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = Agree, 5 = Strongly agree, plus ‘Don’t know / Not applicable’
  • Contingency Binary Item: Yes / No (accompanied by qualitative elaborative prompt)
  • Open-Ended Narrative Items: Free-text qualitative commentary fields
• Scoring and Computational Rules:
  • Items are grouped into core evaluation principles: Integrity of design and process, Influence and impact, Participatory culture, and Collaboration and common purpose.
  • Calculated through frequency distributions or average mean scores per dimension (ranging from 1.0 to 5.0).
  • The response ‘Don’t know / Not applicable’ is excluded from numeric mean score calculations to prevent artificial distortion of dimensional averages.
  • Open-ended items provide qualitative feedback that is analyzed using thematic analysis to identify contextual operational barriers and procedural successes.

11. Permissions & Fee and Test Year

• Initial Publication Year: 2016 (McMaster University developmental release and validation; Dutch cross-cultural validation by Bavelaar et al., 2021).
• Intellectual Property & Copyright: Copyright © 2016 Julia Abelson, McMaster University, and contributing development partners.
• Licensing and Fee Structure: The PPEET is an open-access, fee-free instrument made publicly available for research, health systems evaluation, clinical quality improvement, and academic instruction. Users are typically granted permission to utilize and reproduce the questionnaire without financial cost, provided appropriate academic attribution is cited and the core wording is preserved to maintain psychometric comparability.
• Derivative and Translation Guidelines: Cross-cultural adaptations, institutional re-formatting, and translations into other languages require adherence to formal cross-cultural translation guidelines (such as forward-backward translation and pilot cognitive debriefing) and citation of the primary developers at McMaster University.

12. References

Abelson, J., Li, K., Wilson, G., Shields, K., Schneider, C., & Boesveld, S. (2016). Supporting quality public and patient engagement in health system organizations: Development and usability testing of the Public and Patient Engagement Evaluation Tool. Health Expectations, 19(4), 817–827. https://doi.org/10.1111/hex.12378

Arnstein, S. R. (1969). A ladder of citizen participation. Journal of the American Institute of Planners, 35(4), 216–224. https://doi.org/10.1080/01944366908977225

Bavelaar, L., Nicula, M., Morris, E., Veldhuijzen van Zanten, S., & Abelson, J. (2021). Evaluating patient and public engagement in Dutch health research: Validation and cross-cultural adaptation of the Public and Patient Engagement Evaluation Tool (PPEET). Health Expectations, 24(5), 1832–1843. https://doi.org/10.1111/hex.13324

Edmondson, A. (1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly, 44(2), 350–383. https://doi.org/10.2307/2666999

Habermas, J. (1984). The theory of communicative action: Reason and the rationalization of society (Vol. 1). Beacon Press.

Lehoux, P., Daudelin, G., & Abelson, J. (2012). The unbearable lightness of public participation in health technology assessment. Social Science & Medicine, 74(12), 1842–1850. https://doi.org/10.1016/j.socscimed.2012.02.020

13. Items of the Scale