Behavioral Symptoms, Implementation
Conditions
Keywords
LOCI, measurement-based care, evidence-based practice, behavioral health, youth mental health, leadership, implementation science
Brief summary
This study will investigate the effects of an organizational implementation strategy called Leadership and Organizational Change for Implementation (LOCI), relative to training and technical assistance only, on fidelity to, and youth service outcomes of, a well-established digital measurement-based care intervention called the Outcomes Questionnaire-Analyst in outpatient community mental health clinics.
Detailed description
Using a cluster randomized, controlled, hybrid type III effectiveness-implementation design, this trial will investigate the effects of LOCI on the fidelity and and clinical outcomes of a digital measurement-based care (MBC) system called the Outcomes Questionnaire Analyst (OQ-A). The trial will enroll up to 22 outpatient mental health clinics that serve youth and randomly assign them using covariate constrained randomization to either LOCI or training and technical assistance only. Within each clinic, up to 2 first level leaders will be recruited (max N of 40 total) and a minimum of 3 clinicians will be recruited per site (60 total). Data on youth outpatients who receive services will be collected in two phases. In each phase, a unique cohort of 360 caregivers of youth who participate in services will be sampled from the participating clinics. Caregivers will report on the service outcomes and experiences of eligible youth who receive services. The total caregiver enrollment for two phases will be 720 (360\*2).
Interventions
BEHAVIORALLeadership for Organization Change and Implementation (LOCI)
LOCI is a multicomponent implementation strategy that engages organizational executives and first-level leaders (i.e., those who administratively supervise clinicians) to build an organizational climate to support the implementation of a focal evidence-based practice (EBP) with fidelity. In this study, the focal EBP is the OQ-A system. LOCI includes two overarching components: (1) monthly organizational strategy meetings between executives and LOCI consultants/trainers to develop and embed policies, procedures, and practices that support implementation of a focal EBP, and (2) training and coaching for first-level leaders, to develop their skills in leading implementation. The aim of these components is to develop an organizational implementation climate in which clinicians' perceive that use of the OQ-A with high fidelity is expected, supported, and rewarded.
BEHAVIORALTraining and Technical Assistance Only
All leaders and clinicians in participating clinics will receive standardized OQ-A training and technical assistance provided by the OQ-A purveyor organization. This includes an initial, 6-hr, in-person OQ-A training; two, live, virtual, 1-hr booster trainings, offered 3 and 5 months after the initial training; and, year-round technical assistance from the OQ-A purveyor organization. Technical assistance includes virtual training sessions, online library of training videos, and customer care representative for technical support. In addition, to encourage participation in the study, a set of four 1-hr, web-based general leadership seminars will be offered to leaders in the control condition. These will cover topics ranging from effective leadership, to giving effective feedback.
Sponsors
University of California, San Diego
University of Central Florida
University of Pennsylvania
National Institute of Mental Health (NIMH)
Boise State University
Study design
Allocation
RANDOMIZED
Intervention model
PARALLEL
Primary purpose
HEALTH_SERVICES_RESEARCH
Masking
DOUBLE (Subject, Caregiver)
Masking description
Clinicians and caregivers of youth were naïve to condition; however, masking of clinic leaders was not possible due to the nature of the LOCI strategy (which entails leadership training and consultation).
Eligibility
Sex/Gender
ALL
Age
4 Years to No maximum
Healthy volunteers
Yes
Inclusion criteria
Participants for this project will fall into four groups: 1) clinic executives and upper leaders (i.e., Chief Executive Officer, Executive Directors, program administrators, n=20), 2) clinic first-level leaders (i.e. clinical supervisors, n=20), 3) clinicians serving children with emotional and behavioral disorders (n=120), and 4) caregivers of children with emotional and behavioral disorders who receive outpatient mental health services within participating clinics (n=720). Caregivers will be recruited in two distinct cohorts of 360 each. Inclusion criteria for all groups of participants are intentionally broad in line with the pragmatic nature of the trial. Inclusion Criteria for Clinics 1. Provide outpatient psychotherapy services to children ages 4 to 18 who have emotional and/ or behavioral disorders and their families 2. Have at least 3 full time equivalent clinicians on staff 3. Not currently implementing a digital measurement-based care system clinic wide Inclusion Criteria for Executives and Upper Leaders 1\. Identified as Chief Executive Officer, Executive Director, or high-level administrator at an enrolled clinic Inclusion Criteria for first-level leaders 1\. Identified as a clinical supervisor or clinical work-group supervisor/ leader at an enrolled clinic Inclusion Criteria for Clinicians 1. Employed as staff at participating clinic 2. Provides outpatient mental health services (psychotherapy) to youth clients Inclusion Criteria for Caregivers of Youth Who Receive Services at Participating Clinics 1. Child is ages 4 to 18 years at intake, 2. Child has been diagnosed with an emotional or behavioral disorder by clinic staff 3. Clinical staff at the site determined that the youth's treatment needs can be appropriately addressed by the clinic
Design outcomes
Primary
| Measure | Time frame | Description |
|---|---|---|
| Percentage Fidelity to the OQ-A System Experienced by the Youth (0-100%) | 0-6 months after youth's baseline/ entry into treatment | Fidelity to the OQ-A will be measured by using electronic meta-data from the OQ-A system combined with caregiver reported information on the number of sessions the youth attended. For each youth, a fidelity index will be generated that represents the product of two quantities: (a) the youth's completion rate (i.e., number of measures administered relative to the number of sessions attended within the 6-month observation period), and (b) the youth's viewing rate (i.e., the number of feedback reports viewed by the clinician relative to the number of measures administered). Note that this product is equivalent to the ratio of viewed feedback reports to total sessions; it represents an events/trials proportion. MBC fidelity index scores summarize the level of MBC fidelity experienced by each youth (range=0-1). Higher scores indicate the youth experienced greater fidelity to MBC. |
| Change From Baseline to 6-months in Youth Total Problems Score on the Short Form Assessment for Children (SAC) - Phase I Cohort | 0-6 months after youth's baseline/ entry into treatment | The SAC Total Problem Score is a 48-item measure of youth internalizing (e.g., anxious, depressed) and externalizing (e.g., aggressive, noncompliant, overactive) behaviors (range=0-96) completed by caregivers of youth. Total Problem Score was assessed at baseline (youth's entry into treatment) and monthly for 6 months, change from baseline to month 6 is reported. |
| Change From Baseline to 6-months in Youth Total Problems Score on the Short-form Assessment for Children (SAC) - Phase II Cohort | 0-6 months after youth's baseline/ entry into treatment | The SAC Total Problem Score is a 48-item measure of youth internalizing (e.g., anxious, depressed) and externalizing (e.g., aggressive, noncompliant, overactive) behaviors (range=0-96) completed by caregivers of youth. Total Problem Score was assessed at baseline (youth's entry into treatment) and monthly for 6 months, change from baseline to month 6 is reported. |
Countries
United States
Participant flow
Pre-assignment details
Data were only collected from caregivers of youth who received services at participating clinics. Caregivers reported on youth's mental health service experiences and outcomes. Consequently, started and completed numbers shown below only reflect caregivers of eligible youth who participated in services (i.e., number of caregivers enrolled). Youths were not enrolled.
Participants by arm
| Arm | Count |
|---|---|
| LOCI (Intervention) The Leadership for Organization Change and Implementation (LOCI) strategy is a multicomponent implementation strategy that engages organizational executives and first-level leaders (i.e., those who administratively supervise clinicians) to build an organizational climate to support the implementation of a focal evidence-based practice (EBP) with fidelity. In this study, the focal EBP is the OQ-A system. LOCI includes two overarching components: (1) monthly organizational strategy meetings between executives and LOCI consultants/trainers to develop and embed policies, procedures, and practices that support implementation of a focal EBP, and (2) training and coaching for first-level leaders, to develop their skills in leading implementation. The aim of these components is to develop an organizational implementation climate in which clinicians' perceive that use of the OQ-A with high fidelity is expected, supported, and rewarded. | 386 |
| Training and Technical Assistance Only (Control) All leaders and clinicians in participating clinics will receive standardized OQ-A training and technical assistance provided by the OQ-A purveyor organization. This includes an initial, 6-hr, in-person OQ-A training; two, live, virtual, 1-hr booster trainings, offered 3 and 5 months after the initial training; and, year-round technical assistance from the OQ-A purveyor organization. Technical assistance includes virtual training sessions, online library of training videos, and customer care representative for technical support.
In addition, to encourage participation in the study, a set of four 1-hr, web-based general leadership seminars will be offered to leaders in the control condition. These will cover topics ranging from effective leadership, to giving effective feedback. | 300 |
| Total | 686 |
Withdrawals & dropouts
| Period | Reason | FG000 | FG001 |
|---|---|---|---|
| Phase I | Lost to Follow-up | 5 | 5 |
| Phase II | Lost to Follow-up | 11 | 11 |
Baseline characteristics
| Characteristic | Total | Training and Technical Assistance Only (Control) | LOCI (Intervention) |
|---|---|---|---|
| Age, Categorical <=18 years | 0 Participants | 0 Participants | 0 Participants |
| Age, Categorical >=65 years | 0 Participants | 0 Participants | 0 Participants |
| Age, Categorical Between 18 and 65 years | 0 Participants | 0 Participants | 0 Participants |
| Phase I: Family Income $25,751 to $35,535 | 37 Participants | 18 Participants | 19 Participants |
| Phase I: Family Income $35,536 to $51,500 | 39 Participants | 20 Participants | 19 Participants |
| Phase I: Family Income Less than $25,750 | 47 Participants | 20 Participants | 27 Participants |
| Phase I: Family Income More than $51,500 | 109 Participants | 59 Participants | 50 Participants |
| Phase I: Family Income Unknown or not reported | 2 Participants | 0 Participants | 2 Participants |
| Phase II: Family income $25,751 to $35,535 | 70 Participants | 25 Participants | 45 Participants |
| Phase II: Family income $35,536 to $51,500 | 84 Participants | 29 Participants | 55 Participants |
| Phase II: Family income Less than $25,750 | 91 Participants | 34 Participants | 57 Participants |
| Phase II: Family income More than $51,500 | 198 Participants | 91 Participants | 107 Participants |
| Phase II: Family income Unknown | 9 Participants | 4 Participants | 5 Participants |
| Phase II: Parent Highest Education Level Associate's degree or some college | 205 Participants | 76 Participants | 129 Participants |
| Phase II: Parent Highest Education Level Bachelor's degree | 117 Participants | 53 Participants | 64 Participants |
| Phase II: Parent Highest Education Level Graduate degree | 59 Participants | 25 Participants | 34 Participants |
| Phase II: Parent Highest Education Level High school graduate/ GED or less | 66 Participants | 29 Participants | 37 Participants |
| Phase II: Parent Highest Education Level Unknown | 5 Participants | 0 Participants | 5 Participants |
| Phase II: Youth age | 10.9 years STANDARD_DEVIATION 3.7 | 11.6 years STANDARD_DEVIATION 3.8 | 10.5 years STANDARD_DEVIATION 3.6 |
| Phase II: Youth Hispanic/ Latinx Ethnicity | 61 Participants | 21 Participants | 40 Participants |
| Phase II: Youth race American Indian or Alaskan Native | 5 Participants | 0 Participants | 5 Participants |
| Phase II: Youth race Asian | 5 Participants | 2 Participants | 3 Participants |
| Phase II: Youth race Black or African American | 5 Participants | 3 Participants | 2 Participants |
| Phase II: Youth race Multiracial | 33 Participants | 13 Participants | 20 Participants |
| Phase II: Youth race Pacific Islander or Hawaiian Native | 1 Participants | 0 Participants | 1 Participants |
| Phase II: Youth race Prefer not to disclose | 3 Participants | 0 Participants | 3 Participants |
| Phase II: Youth race Prefer to self-describe | 17 Participants | 5 Participants | 12 Participants |
| Phase II: Youth race White | 383 Participants | 160 Participants | 223 Participants |
| Phase II: Youth received prior mental health treatment | 263 Participants | 110 Participants | 153 Participants |
| Phase II: Youth sex Female | 263 Participants | 114 Participants | 149 Participants |
| Phase II: Youth sex Male | 185 Participants | 67 Participants | 118 Participants |
| Phase II: Youth sex Prefer not to disclose | 4 Participants | 2 Participants | 2 Participants |
| Phase II: Youth Short-form Assessment for Children (SAC) Total Problem Score | 34.9 units on a scale STANDARD_DEVIATION 13.4 | 34.7 units on a scale STANDARD_DEVIATION 13.6 | 35.1 units on a scale STANDARD_DEVIATION 13.3 |
| Phase I: Parent Highest Education Level Associate's degree or some college | 90 Participants | 42 Participants | 48 Participants |
| Phase I: Parent Highest Education Level Bachelor's degree | 54 Participants | 28 Participants | 26 Participants |
| Phase I: Parent Highest Education Level Graduate degree | 42 Participants | 24 Participants | 18 Participants |
| Phase I: Parent Highest Education Level High school graduate/GED or less | 48 Participants | 23 Participants | 25 Participants |
| Phase I: Parent Highest Education Level Unknown or not reported | 0 Participants | 0 Participants | 0 Participants |
| Phase I: Youth age | 11.7 years STANDARD_DEVIATION 3.8 | 12.1 years STANDARD_DEVIATION 4 | 11.3 years STANDARD_DEVIATION 3.7 |
| Phase I: Youth Hispanic/ Latinx Ethnicity | 35 Participants | 14 Participants | 21 Participants |
| Phase I: Youth Race American Indian or Alaskan Native | 3 Participants | 2 Participants | 1 Participants |
| Phase I: Youth Race Asian | 2 Participants | 1 Participants | 1 Participants |
| Phase I: Youth Race Black or African American | 3 Participants | 2 Participants | 1 Participants |
| Phase I: Youth Race Multiracial | 16 Participants | 6 Participants | 10 Participants |
| Phase I: Youth Race Pacific Islander or Hawaiian Native | 0 Participants | 0 Participants | 0 Participants |
| Phase I: Youth Race Prefer not to disclose | 4 Participants | 1 Participants | 3 Participants |
| Phase I: Youth Race Prefer to self-describe | 5 Participants | 2 Participants | 3 Participants |
| Phase I: Youth Race White | 201 Participants | 103 Participants | 98 Participants |
| Phase I: Youth received prior mental health treatment | 124 Participants | 58 Participants | 66 Participants |
| Phase I: Youth sex Female | 125 Participants | 63 Participants | 62 Participants |
| Phase I: Youth sex Male | 108 Participants | 54 Participants | 54 Participants |
| Phase I: Youth sex Prefer not to disclose | 1 Participants | 0 Participants | 1 Participants |
| Phase I: Youth Short-form Assessment for Children (SAC) Total Problem Score | 35.6 units on a scale STANDARD_DEVIATION 13.2 | 34.1 units on a scale STANDARD_DEVIATION 12.8 | 37.1 units on a scale STANDARD_DEVIATION 13.6 |
| Race/Ethnicity, Customized American Indian or Alaskan Native | 0 Participants | 0 Participants | 0 Participants |
| Race/Ethnicity, Customized Asian | 0 Participants | 0 Participants | 0 Participants |
| Race/Ethnicity, Customized Black or African American | 0 Participants | 0 Participants | 0 Participants |
| Race/Ethnicity, Customized More than one race (Multiracial) | 0 Participants | 0 Participants | 0 Participants |
| Race/Ethnicity, Customized Pacific Islander or Hawaiian Native | 0 Participants | 0 Participants | 0 Participants |
| Race/Ethnicity, Customized Prefer not to disclose | 0 Participants | 0 Participants | 0 Participants |
| Race/Ethnicity, Customized Prefer to self-describe | 0 Participants | 0 Participants | 0 Participants |
| Race/Ethnicity, Customized Unknown or not reported | 0 Participants | 0 Participants | 0 Participants |
| Race/Ethnicity, Customized White | 0 Participants | 0 Participants | 0 Participants |
| Sex/Gender, Customized Female | 0 Participants | 0 Participants | 0 Participants |
| Sex/Gender, Customized Male | 0 Participants | 0 Participants | 0 Participants |
| Sex/Gender, Customized Prefer not to disclose | 0 Participants | 0 Participants | 0 Participants |
Adverse events
| Event type | EG000 affected / at risk | EG001 affected / at risk |
|---|---|---|
| deaths Total, all-cause mortality | 0 / 0 | 0 / 0 |
| other Total, other adverse events | 0 / 0 | 0 / 0 |
| serious Total, serious adverse events | 0 / 0 | 0 / 0 |
Outcome results
Primary
Change From Baseline to 6-months in Youth Total Problems Score on the Short Form Assessment for Children (SAC) - Phase I Cohort
The SAC Total Problem Score is a 48-item measure of youth internalizing (e.g., anxious, depressed) and externalizing (e.g., aggressive, noncompliant, overactive) behaviors (range=0-96) completed by caregivers of youth. Total Problem Score was assessed at baseline (youth's entry into treatment) and monthly for 6 months, change from baseline to month 6 is reported.
Time frame: 0-6 months after youth's baseline/ entry into treatment
Population: Analysis includes youths from Phase I cohort.
| Arm | Measure | Value (MEAN) | Dispersion |
|---|---|---|---|
| LOCI (Intervention) | Change From Baseline to 6-months in Youth Total Problems Score on the Short Form Assessment for Children (SAC) - Phase I Cohort | -8.17 units on a scale | Standard Error 1.19 |
| Training and Technical Assistance Only (Control) | Change From Baseline to 6-months in Youth Total Problems Score on the Short Form Assessment for Children (SAC) - Phase I Cohort | -4.04 units on a scale | Standard Error 1.15 |
p-value: 0.023Mixed Models Analysis
Primary
Change From Baseline to 6-months in Youth Total Problems Score on the Short-form Assessment for Children (SAC) - Phase II Cohort
The SAC Total Problem Score is a 48-item measure of youth internalizing (e.g., anxious, depressed) and externalizing (e.g., aggressive, noncompliant, overactive) behaviors (range=0-96) completed by caregivers of youth. Total Problem Score was assessed at baseline (youth's entry into treatment) and monthly for 6 months, change from baseline to month 6 is reported.
Time frame: 0-6 months after youth's baseline/ entry into treatment
Population: One clinic was excluded from analyses because it was an extreme outlier.
| Arm | Measure | Value (MEAN) | Dispersion |
|---|---|---|---|
| LOCI (Intervention) | Change From Baseline to 6-months in Youth Total Problems Score on the Short-form Assessment for Children (SAC) - Phase II Cohort | -4.82 units on a scale | Standard Error 0.7 |
| Training and Technical Assistance Only (Control) | Change From Baseline to 6-months in Youth Total Problems Score on the Short-form Assessment for Children (SAC) - Phase II Cohort | -5.19 units on a scale | Standard Error 1.02 |
p-value: 0.782Mixed Models Analysis
Primary
Percentage Fidelity to the OQ-A System Experienced by the Youth (0-100%)
Fidelity to the OQ-A will be measured by using electronic meta-data from the OQ-A system combined with caregiver reported information on the number of sessions the youth attended. For each youth, a fidelity index will be generated that represents the product of two quantities: (a) the youth's completion rate (i.e., number of measures administered relative to the number of sessions attended within the 6-month observation period), and (b) the youth's viewing rate (i.e., the number of feedback reports viewed by the clinician relative to the number of measures administered). Note that this product is equivalent to the ratio of viewed feedback reports to total sessions; it represents an events/trials proportion. MBC fidelity index scores summarize the level of MBC fidelity experienced by each youth (range=0-1). Higher scores indicate the youth experienced greater fidelity to MBC.
Time frame: 0-6 months after youth's baseline/ entry into treatment
Population: Youths from the Phase I Cohort were included in this analysis.
| Arm | Measure | Value (MEAN) | Dispersion |
|---|---|---|---|
| LOCI (Intervention) | Percentage Fidelity to the OQ-A System Experienced by the Youth (0-100%) | 23.1 percentage fidelity achieved | Standard Error 9 |
| Training and Technical Assistance Only (Control) | Percentage Fidelity to the OQ-A System Experienced by the Youth (0-100%) | 3.4 percentage fidelity achieved | Standard Error 2.5 |
p-value: 0.014Mixed Models Analysis
Primary
Percentage Fidelity to the OQ-A System Experienced by the Youth (0-100%)
Fidelity to the OQ-A will be measured by using electronic meta-data from the OQ-A system combined with caregiver reported information on the number of sessions the youth attended. For each youth, a fidelity index will be generated that represents the product of two quantities: (a) the youth's completion rate (i.e., number of measures administered relative to the number of sessions attended within the 6-month observation period), and (b) the youth's viewing rate (i.e., the number of feedback reports viewed by the clinician relative to the number of measures administered). Note that this product is equivalent to the ratio of viewed feedback reports to total sessions; it represents an events/trials proportion. MBC fidelity index scores summarize the level of MBC fidelity experienced by each youth (range=0-1). Higher scores indicate the youth experienced greater fidelity to MBC.
Time frame: 0-6 months after youth's baseline/ entry into treatment
Population: Analyses based on an intent-to-treat approach. Note that one clinic was identified as an extreme outlier and consequently was excluded from analyses.
| Arm | Measure | Value (MEAN) | Dispersion |
|---|---|---|---|
| LOCI (Intervention) | Percentage Fidelity to the OQ-A System Experienced by the Youth (0-100%) | 5.65 percentage fidelity achieved | Standard Error 4.79 |
| Training and Technical Assistance Only (Control) | Percentage Fidelity to the OQ-A System Experienced by the Youth (0-100%) | 0.97 percentage fidelity achieved | Standard Error 1.27 |
p-value: 0.188Mixed Models Analysis