Relationship to CBT outcome and dropout of decision support tools of the written case formulation, list of treatment goals and plot of symptom scores

Behaviour Research and Therapy 142 (2021) 103874

Available online 5 May 2021

Relationship to CBT outcome and dropout of decision support tools of the

written case formulation, list of treatment goals and plot of symptom scores

Vael Gates

, Megan Hsiao

, Garret G. Zieve

, Rebecca Courry

, Jacqueline B. Persons

University of California, Berkeley, USA

Oakland Cognitive Behavior Therapy Center, USA

ARTICLE INFO

Keywords:

Outcome

Dropout

Case formulation

Decision support

Measurement-based care

Private practice

ABSTRACT

Many patients who receive cognitive behavior therapy (CBT) for mood and anxiety disorders fail to respond or

drop out of treatment. We tested the hypotheses that therapist use of each of three decision support tools, a

written case formulation, a list of treatment goals, and a plot of symptom scores, was associated with improved

outcome and reduced dropout in naturalistic CBT provided to 845 patients in a private practice setting. We

conducted regression analyses to test the hypotheses that the presence of each tool in the clinical record was

associated with lower end-of-treatment scores on the Beck Depression Inventory (BDI) and the Burns Anxiety

Inventory (BurnsAI), and lower rates of premature and uncollaborative dropout. We found that the presence of a

written case formulation in the clinical record was associated with lower rates of both types of dropout. A list of

treatment goals was associated with lower end-of-treatment scores on the BDI and the BurnsAI, and a lower rate of

uncollaborative but a higher rate of premature dropout. A plot of symptom scores was associated with lower end-

of-treatment scores on the BDI, and lower rates of both types of dropout. Results suggest that therapist use of a

written case formulation, list of treatment goals, and a plot of symptom scores can contribute to improved

outcome and reduced dropout in CBT.

Despite its strong evidence base, many patients who receive cogni-

tive behavior therapy (CBT) for mood and anxiety and related disorders

fail to respond to it. For example, Cuipers et al. (2014) reported that

response rates of empirically-supported treatments for major depressive

disorder ranged from 44% to 53%. In addition, many patients terminate

treatment prematurely and thus fail to receive a full dose of treatment.

Fernandez et al. (2015), in a meta-analysis, reported that more than 25%

of patients dropped out of cognitive behavioral therapy (CBT).

Clinicians who use the methods of evidence-based practice rely on a

range of strategies and decision support tools to guide decision-making

and solve problems that can otherwise lead to nonresponse and dropout

(American Psychological Association, 2006). First, clinicians of all

theoretical orientations view the case formulation as a key tool (Eells,

2007). The case formulation describes the factors that appear to cause

and maintain a particular patient’s particular symptoms and problems,

and it thus identies the treatment targets and guides the therapist’s

efforts to overcome problems that can arise in treatment. Despite its

importance to practitioners, the case formulation is under-studied, and

we have little information about the relationship of the case formulation

to outcome and dropout (Persons & Hong, 2016). Second, cognitive

behavioral therapists develop with their patients an agreed-upon list of

treatment goals. The list of treatment goals individualizes the treatment

and focuses it on concrete, specic, achievable objectives. A shared list

of goals is considered to be part of the therapeutic alliance (Bordin,

1994), and we know that the alliance is related to both outcome (Hor-

vath et al., 2011) and dropout (Roos, J. & Werbart, A., 2013), but we

don’t know whether the strategy of writing down a list of the patient’s

concrete, specic treatment goals and using them to guide treatment

affects outcome and dropout.

Third, evidence-based practitioners rely on a plot of symptom scores.

The patient completes a symptom scale before the session, and the

therapist and patient together review the plot showing the trajectory of

scores and use it to guide decision-making. A recent review showed that

measurement-based care, dened as “the systematic evaluation of pa-

tient symptoms before or during each clinical encounter to inform

behavioral health treatment” leads to improved psychotherapy outcome

(Lewis et al., 2018). But only one of the samples that Lewis et al.

reviewed were American samples of outpatients with mood and anxiety

disorders (Hawkins et al., 2004). All three of these tools (the case

formulation, the list of treatment goals, and progress monitoring data)

* Corresponding author. Oakland Cognitive Behavior Therapy Center, 5625 College Avenue, Suite 215, Oakland, CA, 94618, USA.

E-mail address: [email protected] (J.B. Persons).

Contents lists available at ScienceDirect

Behaviour Research and Therapy

journal homepage: www.elsevier.com/locate/brat

https://doi.org/10.1016/j.brat.2021.103874

Received 26 April 2020; Received in revised form 30 December 2020; Accepted 19 April 2021

Behaviour Research and Therapy 142 (2021) 103874

represent elements of evidence-based practice as dened by the Amer-

ican Psychological Association (2006), and appear in widely-used clin-

ical manuals (J. S. Beck, 1995; Eells, 2007; Persons, 2008), but little is

known about their contribution to outcome and dropout in routine

clinical practice. More empirical support is needed for these tools that

are widely viewed as core elements of evidence-based practice.

To address this gap in the literature, we tested the hypotheses that

therapist use of each of the decision support tools of the written case

formulation, list of treatment goals, and plot of symptom scores were asso-

ciated with improved outcome and reduced dropout in a large sample of

patients who received naturalistic cognitive behavior therapy (CBT) for

symptoms of depression and anxiety in a private practice setting. We

studied outcome as dened by scores on two symptom measures at the

end of treatment: the Beck Depression Inventory and the Burns Anxiety

Inventory. We studied dropout as dened in two ways. We studied

premature dropout, dened as ending treatment before, in the therapist’s

judgment, the treatment has been tried for long enough to help the

patient accomplish their treatment goals, and uncollaborative dropout,

dened as ending treatment against the therapist’s advice or without

discussing the termination with the therapist. We predicted that the

presence in the clinical record of a written case formulation, a list of

treatment goals, and a plot of scores on a measure of depression or anxiety

would be associated with improved outcome and reduced premature

and uncollaborative dropout.

1. Method

1.1. Participants

Participants were 845 adults who received individual naturalistic

cognitive behavior therapy during the years 1981–2009 from the second

author or one of 20 therapists at the group private practice she estab-

lished in 1995. All participants gave written consent for data from their

clinical record to be used for research purposes. The procedures used to

establish and maintain the Persons Archival Database we studied in this

investigation were reviewed and approved by the Behavioral Health

Research Collective Institutional Review Board. The Persons Archival

Database is a completely de-identied database, and no master code list

links the data in the database to the names of the patients in the

database.

Patients in the sample had an average age of 36.2 years (SD = 12.3)

and had completed an average of 16.6 years (SD = 2.6) of education.

Patients identied as 58.6% female and 41.2% male; 0.2% were of un-

specied gender. 80.9% patients were Caucasian, 2.5% were African

American, 2.6% were Hispanic, 6.3% were Asian, 1.9% were of other

ethnicity, and 5.8% were of unspecied ethnicity. 20.0% of patients

received adjunctive psychosocial treatment (e.g., group therapy or

couple therapy), 63.2% did not, and 16.8% were unspecied. 50.7% of

patients received adjunctive pharmacotherapy, while 38.1% did not,

and 11.1% were unspecied. 86.4% of patients had an anxiety or

depressive disorder or both. Diagnoses were assigned by the treating

therapist based on the most current version of the Diagnostic and Sta-

tistical Manual of Mental Disorders at the time the patient was treated.

Patients were treated by one of twenty therapists.

The 845 patients studied here were drawn from a database of 1469

adult patients. Patients were excluded from the larger sample if they had

incorrect or missing data describing how many total sessions they

completed (n = 14) or sought consultation only, not treatment (n = 24).

Most importantly, patients were excluded if they had fewer than three

sessions in the database (n = 586). We excluded these cases so that the

computations required to carry out the multiple imputation strategy we

used to interpolate missing data could be carried out in a reasonable

length of time, and so we could use the same patient sample for all data

analyses we conducted. If patients completed more than one course of

treatment, only the rst course was analyzed to simplify analysis.

1.2. Treatment

Treatment consisted of individual cognitive behavior therapy (CBT),

typically provided weekly and based on a case formulation-driven mode

of treatment (Persons, 1989, 2008). The case formulation-driven mode

of treatment called for the therapist to develop, with the patient, an

individualized formulation of the case and a written list of treatment

goals, to make a plot in each session of scores on symptoms relevant to

the patient’s treatment goals, and to use these three tools to guide

decision-making in therapy. However, therapists worked in a group

private practice, not a clinic, and there was no policy or requirement in

the practice that clinicians use these decision support tools, and thus

therapists made their own decisions about use of the tools. The most

common symptom measures used by the therapists in the practice were

the Beck Depression Inventory and the Burns Anxiety Inventory, and

therefore we selected those two scales as our outcome measures for this

study. Treatment was naturalistic because it did not depend on a pro-

tocol that described a pre-determined number and frequency of sessions,

and it allowed the therapist to make adjustments in the treatment (e.g.,

increasing session frequency or adding adjunctive psychosocial treat-

ment, such as couple therapy, or adjunctive pharmacotherapy) based on

the results of the progress monitoring data and other factors (e.g., the

patient’s preference). Treatment was open-ended in duration and ended

ideally when patient and therapist agreed that the patient had reached

the patient’s goals but sometimes ended for other reasons (e.g., the

patient moved, the therapist left the practice, the patient felt they had

accomplished all that could be done in the therapy, or the patient ended

treatment against the therapist’s advice). Most therapists were Ph.D.

psychologists; one was an L.C. S.W. The mean (and standard deviation)

of the length of treatment in days was 286.0 (SD = 573.9), and the

number of sessions was 24.0 (SD = 32.7).

1.3. Measures

Symptoms of depression. We assessed symptoms of depression

with the original version of the Beck Depression Inventory (A. T. Beck

et al., 1961). The BDI is a widely-used, 21-item self-report measure of

the severity of depressive symptoms that has been shown to have good

internal consistency (

= 0.86 for psychiatric patients) and good

convergence with other measures of depressive symptoms (A. T. Beck

et al., 1988). Total score could range from 0 to 63.

Symptoms of anxiety. We assessed symptoms of anxiety with the

Burns Anxiety Inventory (BurnsAI), a 33-item self-report inventory

measuring 6 anxious feelings (e.g., anxiety, nervousness, worry or fear),

11 anxious thoughts (e.g., feeling that you’re on the verge of losing

control) and 16 physical symptoms (e.g., a lump in the throat). Each

symptom was rated on a 0 to 3 scale ranging from 0 (not at all) to 3 (a

lot). Total score could range from 0 to 99. Burns and Eidelson (1998)

reported, in a sample of 483 outpatients, that the BurnsAI had a Cron-

bach’s alpha of 0.94, indicating high internal consistency, and it was

correlated 0.86 (p < 0.001) with the Anxiety subscale of the Symptom

Check List-90 (Derogatis et al., 1976). We used the BurnsAI because it

covered the full range of anxious symptoms we observed in our patients,

its classication of anxiety symptoms as feelings, thoughts, or physical

symptoms was clinically helpful, and it was sensitive to change.

Premature dropout. Premature dropout was coded as present (a

score of 1) when, after treatment ended and the database was being

Of the 845 patients, 2 were seen by Therapist 1; 107 by Therapist 2; 23 by

Therapist 3; 20 by Therapist 4; 3 by Therapist 5; 25 by Therapist 6; 2 by

Therapist 7; 1 by Therapist 8; 70 by Therapist 9; 54 by Therapist 10; 16 by

Therapist 11; 16 by Therapist 12; 3 by Therapist 13; 315 by Therapist 14 (last

author); 28 by Therapist 15; 66 by Therapist 16; 3 by Therapist 17; 42 by

Therapist 18; 12 by Therapist 19; 36 by Therapist 20; and 1 by an unspecied

therapist.

V. Gates et al.

Behaviour Research and Therapy 142 (2021) 103874

assembled, the patient’s therapist answered “no” to the question: “Has

the therapy been given a fair shake/tried for long enough to help patient

accomplish their treatment goals?” Reasons for the therapist’s judgment

that termination was premature varied widely from case to case

depending on the patient’s treatment goals.

Uncollaborative dropout. Uncollaborative dropout was coded as

present (a score of 1), when, after treatment ended and the database was

being assembled, the patient’s therapist answered “no” to the question:

“Did the patient and therapist work well together on the termination,

agree on it and discuss it fully?” The termination was coded as uncol-

laborative, for example, if the patient ended treatment by simply

cancelling a session and never rescheduling it.

Premature and uncollaborative dropout were not mutually exclusive,

and each patient received a score of 1 (yes) or 0 (no) for each type of

dropout.

Therapist use of decision support tools: written case formula-

tion, list of treatment goals, and plot. To assess therapist use of the

decision support tools of case formulation-driven CBT, the therapist

reviewed each patient’s clinical record after treatment had ended, and

coded each tool (written case formulation, list of treatment goals, or plot

of behavior or symptoms) 1 if it was present, and 0 if it was not present.

Therapists used a coding manual to make these coding decisions. The

coding manual called for a code of 1 on the case formulation item “if there

was a written case formulation of any quality in the chart, and 0 other-

wise. This should be a written formulation of the case, not just a brief

mini-formulation, e.g., a Thought Record or diagram of a panic cycle.”

The coding manual called for a code of 1 on the list of treatment goals if

“the goals or objectives of treatment are stated in the clinical chart prior

to the termination note. . . . This is not the goals/objectives for what the

patient wants to accomplish between one session and another but must

be the Goals or Objectives of treatment. The word ‘Goals’ or ‘Objectives’

must appear, and there should be a list, except in rare cases where there

is a single goal. It is not sufcient to state, ‘The patient seeks treatment to

work on OCD symptoms,’ or similar.” Plot was coded 1 if a plot of scores

on the Beck Depression Inventory or Burns Anxiety Inventory appeared

in the chart with at least one score entered on the plot, and 0 otherwise.

1.4. Data analysis

We tested the hypotheses that the presence in the clinical record of

the three decision support tools (case formulation, treatment goals, and

plot), was associated with lower end-of-treatment scores on the Beck

Depression Inventory and Burns Anxiety Inventory, and reduced likeli-

hood of premature and uncollaborative dropout. We conducted a linear

regression to predict each outcome variable (end-of-treatment symptom

scores on the Beck Depression Inventory (BDI) and the Burns Anxiety

Inventory (BurnsAI)), and we conducted a logistic regression to predict

each dropout variable (premature and uncollaborative dropout). In each

analysis, the independent variables were the three decision support tools

(coded 0/1 to indicate whether the tool was present in the clinical re-

cord), the number of sessions the patient spent in therapy, and the

identity of the patient’s therapist. In the analyses of end-of-treatment

BDI and BurnsAI scores, we also controlled for the patient’s score on

the measure in the rst session. We included all the decision support

tools in the regressions in order to examine the effect of each tool on

outcome and dropout while controlling for the contributions of the other

tools.

To control for therapist identity, we included therapist as a variable

in the regression analyses. Thus, our analyses examined the relationship

between the decision tools and outcome/dropout within each therapist’s

caseload. To enter therapist identity into the regressions, we created one

dummy-coded categorical variable (coded 0–1) for each of the 20

therapists. Rather than report effects for each individual therapist, we

reported a cumulative result of the effect of therapist identity on the

dependent variable. To do that, for each regression analysis, we

compared two nested models, one including the therapist variable and

one without the therapist variable. We computed the p-value using the F

distribution for each regression to determine whether the larger model

that included the therapist variable was statistically signicantly

different from the smaller model that did not include the therapist

variable.

We handled the therapist variable as a xed effect rather than a

random effect. Our rationale for this decision was two-fold. Random-

effects and xed-effects models usually give similar results, and random-

effects models require making an assumption (Gardiner et al., 2009) that

we did not believe was justied, the assumption that therapist effects are

uncorrelated with therapist propensity to use the decision support tools

we are studying.

As an additional aid to understanding the relationship between the

decision support tools and the dropout variables, we computed a mea-

sure of correlation between each independent variable and each dropout

variable, the Jaccard similarity coefcient (Jaccard index). The Jaccard

index ranges from 0 to 1, with 1 indicating maximal similarity.

Effect size. Because our sample size was so large, very small effects

could be statistically signicant. Therefore, we calculated effect sizes. In

the linear regressions examining the relationship between the tools and

outcome, we used the change in adjusted R

(designated as ΔR

) as an

effect size. Adjusted R

, the coefcient of determination adjusted for the

number of variables, is a statistical measure of how much variance a

regression model explains. We calculated the ΔR

by subtracting the

adjusted R

of a regression model that omitted the variable of interest

(case formulation, treatment goals, or plot) from the adjusted R

of the

model that included the variable of interest.

In the logistic regressions examining the relationship between the

tools and dropout, we used the odds ratio (e

Est.

) as an effect size. The

odds ratio describes the relative odds of the dependent variable (pre-

mature or uncollaborative dropout) occurring given the inclusion of the

specied independent variable. We are most interested in the odds ratio

Est.

) for each decision support tool parameter of the regressions, which

indicates the estimated odds of a premature/uncollaborative dropout by

a patient whose clinical record has the decision support tool (case

formulation, list of treatment goals, or plot) as compared to the odds of

dropout by a patient whose chart does not have the decision support

tool. To increase the interpretability of the odds ratio, we converted the

odds ratio to a relative risk ratio for a baseline risk level, using as the

baseline risk level the percentage of patients in our sample who dropped

out prematurely or uncollaboratively.

We carried out the analyses using R Core Team (2018) and Python

(Python Software Foundation, n.d.), using Jupyter Notebooks (Project

Jupyter, n.d.), and used the packages Amelia II (Honaker et al., 2012)

and Zelig (R Core Team, 2007) to conduct the multiple imputations

analyses.

2. Results

2.1. Preliminary analyses

Missing data. Our dataset contained 845 patients with 15987 ses-

sions of data. 104 patients (12.3%) were missing data about case

formulation, 100 patients (11.8%) were missing data about treatment

goals, and 88 patients (10.4%) were missing data about plot. 70 patients

(8.3%) did not have any BDI scores, and 8427 sessions (52.7%) did not

have a BDI score. 177 patients (20.9%) did not have any BurnsAI scores,

and 10,357 sessions (64.8%) did not have a BurnsAI score. 91 patients

(10.8%) were missing data about premature dropout, and 83 patients

(9.8%) were missing data about uncollaborative dropout. Data about the

decision support tools and dropout were missing when we were unable

to interview the treating therapist to obtain this information. BDI and

Burns AI scores were missing when we could not locate the medical

record or when the therapist did not monitor outcome with a symptom

scale or selected another measure (e.g., the YBOCS or a daily log of skin-

picking behavior) to monitor outcome.

V. Gates et al.

Behaviour Research and Therapy 142 (2021) 103874

Multiple imputation. To handle missing data, we used the statisti-

cal technique of multiple imputation (Enders, 2017). In this approach,

missing data were estimated from existing data multiple times (multiple

“imputations”) with some degree of randomness, and these multiple

estimates were pooled for a nal result. By averaging multiple ran-

domized imputations, we can incorporate the variance of estimated

values, so that estimates that the system has low condence in will have

high variance, and estimates that are highly supported by the existing

data will have low variance. This strategy reduces bias in the estimated

values. We used the package Amelia II (Honaker et al., 2012) to carry out

the imputations. We computed ve imputations. To improve our

imputed values, we incorporated our data’s structure into the imputa-

tion model. Specically, we modeled our data as time series within

patient cross-sections, meaning we assumed that patients’ values would

vary over time, and that each patient could have a different starting

point and rate of change. Amelia II modeled patients’ patterns by

computing rst-order polynomial regressions for each patient, using

session number to represent time. We included the following as vari-

ables for each patient: number of sessions in treatment, number of days

in treatment, age, number of years of education, gender, ethnicity,

whether adjunctive pharmacotherapy was provided, whether adjunctive

psychosocial treatment was provided, whether they had a

depression-related diagnosis, whether they had an anxiety-related

diagnosis, the identity of their therapist (numerically coded), whether

they had a written case formulation in their chart, whether they had a

list of treatment goals in their chart, whether they had a plot of symptom

scores in their chart, whether they were coded as a premature dropout,

whether they were coded as an uncollaborative dropout, and their BDI

and BurnsAI scores for each session. Imputed BDI and BurnsAI scores

were lower-bounded at 0 and upper-bounded at the maximum score for

each measure, and the number of sessions, days in treatment, age, and

years of education were lower-bounded at 0, 0, 18 and 0 respectively. All

missing data for these variables were imputed. Final BDI and BurnsAI

scores were determined after this multiple imputation analysis. To speed

up computation, we used a ridge prior that was set to 10% of the total

number of sessions (for details on Amelia II’s statistical options,

including ridge priors, see Honaker et al. (2012)).

Reliability of coding for therapist decision support tools: writ-

ten case formulation, list of treatment goals, and plot. To evaluate

the inter-rater reliability of coding of the clinical record for presence of a

written case formulation, a list of treatment goals, and a plot of symptom

data, we conducted a small study of clinical records of 20 patients

treated by the last author during the years 2008–2020. These records

were randomly selected from a larger database the last author has

collected. Patients gave consent for use of their records for research

purposes, and this study was reviewed by the IRB of the Behavioral

Health Research Collective. We were unable to conduct this study using

the data from the Persons Archival Database used in the present study

because no master code list exists to link the patient clinical record to the

data in the Persons Archival Database. Two authors (R.C. and J.B.P.)

rated each of the 20 clinical records using the same coding manual

(described on page 9) that therapists used to rate these variables in the

Persons Archival Database. The two therapists showed nearly perfect

agreement on ratings of the presence in the clinical record of all three

decision support tools: a written case formulation, a list of treatment

goals, and a plot of symptom data. The clinicians agreed on 19 of 20

ratings of a written case formulation, 19 of 20 ratings of a list of treat-

ment goals, and 20 of 20 ratings of a plot of progress monitoring data,

indicating that the presence in the clinical record of the decision support

tools in the clinical record can be reliably rated.

Therapist use of the decision support tools. We had data about the

presence of a written case formulation (present or absent) for 741 par-

ticipants; of those 741 cases, 581 (78%) had a written case formulation

in the chart. We had data for 745 participants about the presence of a list

of treatment goals; of those 745 cases, 505 (68%) had a list of treatment

goals in the chart. We had data for 757 participants about the presence

of a plot of BDI or BurnsAI data; of those 757 cases, 424 (56%) had a plot

in the chart.

Patient outcome. Table 1 presents imputed scores on the Beck

Depression Inventory and Burns Anxiety Inventory for all patients, and

for patients whose charts did and did not include a case formulation, list

of treatment goals, and plot. (The equivalent information for the original

data, before the multiple imputation procedure, is presented in Sup-

plementary Table 1.)

Patient dropout. We had data on premature dropout for 754 pa-

tients; of those, 394 (52%) were premature dropouts. We had data on

uncollaborative dropout for 762 patients; of those, 231 (30%) were

uncollaborative dropouts. These percentages were used as “baseline

risk” estimates for the relative risk ratio calculations for each type of

dropout.

2.2. The effect of decision support tools on outcome and dropout

We hypothesized that the presence in the clinical record of decision

support tools of case formulation, treatment goals, and plot would be

associated with reduced end-of-treatment BDI and BurnsAI scores, and

reduced premature and uncollaborative dropout. To test our hypotheses

about end-of-treatment BDI and BurnsAI scores, we conducted a linear

regression for each symptom score, where the dependent variable was

the end-of-treatment score on the symptom measure, and the indepen-

dent variables were the three decision support tools (coded as present or

absent), the initial score on the symptom measure, the identity of the

therapist, and the total number of sessions of treatment. To test our

hypotheses about dropout, we conducted a logistic regression for each

type of dropout, where the dependent variable was premature or

uncollaborative dropout, and the independent variables were the deci-

sion support tools (coded as present or absent), the identity of the

therapist, and the total number of sessions of treatment. We multiplied

all p values by four to Bonferroni-correct for the number of regression

analyses. Table 2 reports the results of these regressions. We examine

results for outcome and dropout in turn.

Outcome. First we examine the degree to which the presence of the

decision support tools in the clinical record was related to improved

patient outcome on the Beck Depression Inventory (Table 2). We found

that Treatment Goals (β = − 1.4, p = 0.007) and Plot (β = − 1.6, p = 0.04)

but not Case Formulation (β = − 0.6, p = 0.6) were statistically signi-

cant predictors of end-of-treatment Beck Depression Inventory (all p

Table 1

Estimated means and standard deviations (S.D.) of initial and nal outcome

scores of subcategories of patients, after multiple imputation replaced missing

data. Lower scores indicate fewer symptoms.

n = 845 Beck Depression Inventory Burns Anxiety Inventory

Initial Score Final Score Initial Score Final Score

Mean ± S.D. Mean ± S.D. Mean ± S.D. Mean ± S.D.

All Patients 17.45 ± 8.89 12.03 ± 9.02 28.79 ±

15.61

19.98 ±

14.62

Sorted by Tool

Case Formulation

Present 17.16 ± 8.41 11.66 ± 8.68 28.55 ±

15.58

19.73 ±

14.52

Absent 18.53 ±

10.41

13.35 ±

10.08

29.65 ±

15.73

20.89 ±

14.98

Treatment Goals

Present 17.22 ± 8.64 11.48 ± 8.72 28.36 ±

15.84

18.62 ±

14.42

Absent 17.93 ± 9.37 13.16 ± 9.54 29.65 ±

15.12

22.77 ±

14.64

Plot

Present 18.21 ± 8.87 12.01 ± 9.00 29.96 ±

15.52

19.64 ±

14.48

Absent 16.48 ± 8.83 12.04 ± 9.07 27.29 ±

15.62

20.41 ±

14.80

V. Gates et al.

Behaviour Research and Therapy 142 (2021) 103874

values are Bonferroni-corrected). These beta coefcients indicate that

we would expect that patients whose clinical record included a list of

treatment goals would have an end-of-treatment BDI score that was 1.4

points lower on average compared to patients whose clinical record did

not include a list of treatment goals, and that patients whose clinical

record included a plot of symptoms would have an end-of-treatment BDI

score that was 1.6 points lower. So although statistically signicant, the

effects of the Treatment Goals and Plot variables on end-of-treatment

BDI score were small, about 1.5 points on the Beck Depression In-

ventory. The measure of effect size, ΔR

, reported in Table 2 also shows

that the effects of Treatment Goals and Plot on end-of-treatment BDI

scores were small.

For the Burns Anxiety Inventory, our linear regression analysis

showed that Treatment Goals was a statistically signicant predictor of

end-of-treatment score on the Burns Anxiety Inventory (β = − 4.2, p =

0.04), but not Plot (β = − 3.4, p = 0.09) or Case Formulation (β = 0.09, p

= 3.8) (all p values were multiplied by 4 to Bonferroni-correct for the

total number of regression analyses). These beta coefcients indicate

that we would expect that patients whose clinical record included a list

of treatment goals would have an end-of-treatment BurnsAI score that

was 4.2 points lower on average (fewer symptoms) than patients whose

clinical record did not include a list of treatment goals. As the size of the

beta coefcient indicates, the effect of the Treatment Goals variable on

end-of-treatment Burns Anxiety Inventory score was small (the score on

the measure ranges from 0 to 99). The measure of effect size, ΔR

, re-

ported in Table 2 also shows that the effect of Treatment Goals on end-

of-treatment BurnsAI score was small.

In both the BDI and BurnsAI analyses, the initial score on the mea-

sure and therapist identity were statistically signicant predictors of the

end-of-treatment score on the measure (p < 0.0001), but total number of

therapy sessions and the intercept parameters were not.

In sum, two decision support tools, Treatment Goals and Plot, were

statistically signicantly related to end-of-treatment scores on one or

two of the outcome measures. Treatment Goals was a statistically sig-

nicant predictor of both end-of-treatment BDI and Burns AI scores, and

Plot was a statistically signicant predictor of end-of-treatment BDI

score. All effects were small in size.

Dropout. First we examine the degree to which the presence of the

decision support tools in the clinical record was related to Premature

dropout. As predicted and shown in Table 2, Case Formulation (β =

− 0.5, p < 0.0001) and Plot (β = − 0.4, p < 0.0001) were statistically

signicantly related to reduced Premature dropout compared to when

these tools were not present in the clinical record. However, contrary to

prediction, Treatment Goals was positively related to Premature dropout

(β = 0.3, p < 0.0001), indicating that a list of Treatment Goals was in the

chart was statistically signicantly related to increased premature

dropout.

The effect sizes (estimated odds ratios) of the decision support tools

on Premature dropout were 0.6 for Case Formulation, 1.3 for Treatment

Goals, and 0.7 for Plot. The fact that the odds ratio for Case Formulation

was less than one indicates that if patients had a case formulation in

their clinical record, they were less likely to drop out prematurely than if

they did not have a case formulation in their clinical record; the same is

true for Plot. The fact that the odds ratio for Treatment Goals was greater

than one indicates that if patients had a list of treatment goals in their

clinical record, they were more likely to drop out prematurely than if

they did not have a list of treatment goals in the clinical record. Another

way to consider the odds ratio is to convert it to a risk ratio for a given

baseline risk level. Using a baseline risk level of premature dropout of

52% and the odds ratios generated from our model t, we calculated

that a patient whose chart had a case formulation was 0.74 times as

likely to drop out prematurely as a patient without a case formulation, a

patient whose chart had a list of treatment goals was 1.13 times as likely

to drop out prematurely as a patient without a list of treatment goals,

and a patient whose chart had a plot was 0.80 times as likely to drop out

prematurely as a patient without a plot.

Next we examine the degree to which the presence of the decision

support tools in the clinical record was related to Uncollaborative

dropout. As Table 2 shows, Case Formulation (β = − 0.3, p = 0.0004),

Treatment Goals (β = − 0.4, p < 0.0001), and Plot (β = − 0.6, p < 0.0001)

were each statistically signicantly related to Uncollaborative dropout

in the predicted direction, indicating that the presence of a case

formulation, a list of treatment goals, or a plot in the clinical record each

predicted reduced uncollaborative dropout compared to when these

tools were not present in the clinical record.

As reported in Table 2, the estimated odds ratios for Case Formula-

tion, Treatment Goals, and Plot were 0.8, 0.7, and 0.5, respectively. The

fact that all odds ratios were less than one indicates that patients with

each of these tools in their clinical record were less likely to drop out

uncollaboratively than patients without each of these tools. Using a

baseline risk level of uncollaborative dropout of 30%, and the model-

derived odds ratios, we calculated that patients who had a case formu-

lation were 0.82 times as likely to drop out uncollaboratively as patients

without a case formulation, 0.77 times as likely to drop out

Table 2

Parameter estimates for linear and logistic regression models predicting outcome and dropout from the decision support tools of a written case formulation, list of treatment goals,

and plot of symptom scores.

n = 845 Beck Depression

Inventory Est. ± S.E.

ΔR

p Burns Anxiety

Inventory Est. ± S.E.

ΔR

p Premature Est.

± S.E.

Est.

p Uncollaborative Est.

± S.E.

Est.

Intercept − 2.5 ± 2.7 1.4 5.3 ± 7.3 1.9 2.8 ± 0.6 0 1.3 ± 0.3 4e-

Case

Formulation

− 0.6 ± 0.4 4e-4 0.6 0.09 ± 1.8 9e-4 3.8 − 0.5 ± 0.08 0.6 0 − 0.3 ± 0.07 0.8 4e-

Treatment

Goals

− 1.4 ± 0.4 0.005 7e-3 − 4.2 ± 1.6 0.02 0.04 0.3 ± 0.05 1.3 0 − 0.4 ± 0.05 0.7 0

Plot − 1.6 ± 0.6 0.007 0.04 − 3.4 ± 1.5 0.01 0.09 − 0.4 ± 0.05 0.7 0 − 0.6 ± 0.05 0.5 0

Total # Sessions 4e-4 ± 0.007 3.8 − 0.001 ± 0.01 3.8 − 0.01 ± 6e-4 0 − 0.002 ± 3e-4 0

Initial Score 0.3 ± 0.04 0 0.3 ± 0.02 0

Therapist F(19,76.3) = 8.3 0 F(19,76.1) = 3.9 0 F(19,10901.4) = 85.8 0 F(19,463.4) = 20.5 0

Note. All p-values were Bonferroni-corrected for the number of regression analyses. A p-value of 0 represents signicance at <0.0001. ΔR

signies the difference

between adjusted R

for the full regression model, and adjusted R

for the regression model without the decision support tool. e

Est.

is the odds ratio. Both ΔR

and e

Est.

are effect size measures. Reported ΔR

are means over the ve imputations; means ± standard errors across imputations are as follows: Case formulation/BDI: 4e-4 ±

1e-4; Treatment Goals/BDI: 0.0047 ± 0.0013; Plot/BDI: 0.0068 ± 0.0019; Case formulation/BurnsAI: 9e-4 ± 5e-4; Treatment Goals/BurnsAI: 0.017 ± 0.0053; Plot/

BurnsAI: 0.0113 ± 0.0038. Condence intervals around odds ratios are as follows: Case formulation/Premature Dropout: 0.58, CI = [0.50,0.67]; Treatment Goals/

Premature Dropout: 1.31, CI = [1.19,1.44]; Plot/Premature Dropout: 0.66, CI = [0.60,0.72]; Case formulation/Uncollaborative Dropout: 0.76, CI = [0.67,0.87];

Treatment Goals/Uncollaborative Dropout: 0.70, CI = [0.63,0.77]; Plot/Uncollaborative Dropout: 0.53, CI = [0.48,0.59].

V. Gates et al.

Behaviour Research and Therapy 142 (2021) 103874

uncollaboratively as patients without a list of treatment goals, and 0.62

times as likely to drop out uncollaboratively as patients without a plot.

The total number of sessions a patient completed (p < 0.0001), the

therapist identity (p < 0.0001), and the intercept parameter (p

premature

< 0.0001, p

uncollaborative

= 0.0004) were all statistically signicant pre-

dictors of both types of dropout.

The mean Jaccard index across imputations for the relationship be-

tween the decision support tools and Premature dropout was 0.31, 0.36,

and 0.27, for Case Formulation, Treatment Goals, and Plot, respectively.

The mean Jaccard index across imputations for the relationship between

the decision support tools and Uncollaborative dropout was 0.22, 0.17,

and 0.18, respectively, for Case Formulation, Treatment Goals, and Plot,

respectively. These indices describe the relationship between each de-

cision support tool and type of dropout, separate from the presence of

the other tools, therapist, or number of sessions.

In sum, Case Formulation, Treatment Goals, and Plot each predicted

statistically signicant reductions in both Premature and Uncollabor-

ative dropout, with the exception that, contrary to our prediction,

Treatment Goals predicted an increased rate of Premature dropout.

3. Discussion

We found that all three of the decision support tools we studied, a

written case formulation, a list of treatment goals, and a plot of symptom

scores, were associated with improved outcome and/or reduced dropout

in our sample of outpatients treated with naturalistic CBT in a private

practice setting. Tools had differing effects on dropout and outcome. We

found that two of the three decision support tools we studied, a list of

treatment goals, and a plot of symptom scores, were associated with

improved outcome. Patients whose medical record included a written

list of treatment goals had lower end-of-treatment scores on the Beck

Depression Inventory and Burns Anxiety Inventory than patients whose

medical record did not include a list of treatment goals. Patients whose

medical record included a plot of symptom scores had lower end-of-

treatment scores on the Beck Depression Inventory than patients

whose medical record did not include a plot. Effects of the decision

support tools on outcome were statistically signicant but small. The

tools with statistically signicant effects were associated with expected

average reductions in the end-of-treatment BDI score of 1.4 points (a

written list of treatment goals) and 1.6 points (a plot of symptom scores),

and an expected average reduction in the end-of-treatment BurnsAI

score of 4.2 points (a written list of treatment goals). These small effects

are similar to the small effects of decision support tools seen in some

other studies (e. g., Delgadillo et al., 2018; Kendrick et al., 2016), and

several of the studies reviewed by Lewis et al. (2018). We might have

obtained larger effects if we had been able to measure the degree to

which therapists actually used each tool; instead, we had only an indi-

rect measure of the therapist’s use of the tool, namely whether the tool

was present in the clinical record. Therapists may have had a written

formulation in the chart, for example, but not relied on it in their work,

or they may have relied on a case formulation without writing it down.

We might have also obtained larger effects if we had assessed the quality

of the case formulation; Abel et al. (2016) showed that therapist

competence in case conceptualization was related to sudden gains in

patients with treatment-resistant depression. And the list of treatment

goals might have been more convincingly related to outcome if we had

assessed outcome by assessing progress toward the goals on the list.

Unfortunately, our eld has not yet developed a strong measure for

assessing progress toward idiographic goals. Also, there is some evi-

dence that decision support tools contribute more to outcome when the

patient begins treatment with a poor outcome than when early progress

is good (cf. Lambert et al., 2005; Vittengl et al., 2019) and that the

currently-available tools are most effective at predicting deterioration

than outcome (e.g., Delgadillo et al., 2018; Lambert & Shimokawa,

2011).

The decision support tools we studied were more predictive of

dropout than of outcome in our sample. All three tools were statistically

signicantly related to a lower rate of both types of outcome we studied,

with the exception that a list of treatment goals was a statistically sig-

nicant predictor of a higher rate of premature termination. Based on our

model-derived odds ratios and estimated baseline sample rates, we

calculated that patients were 0.74 or 0.80 times as likely to drop out

prematurely when the clinical record had a written case formulation or a

plot of symptom scores, respectively. We calculated that patients were

0.82, 0.77, or 0.62 times as likely to drop out uncollaboratively when the

clinical record had a written case formulation, list of treatment goals, or

plot of symptom scores, respectively.

Contrary to our prediction, we calculated that patients whose chart

included a list of treatment goals were more likely (1.13 times more

likely) to drop out prematurely than patients whose chart did not

include a list of treatment goals. Perhaps this nding resulted from the

fact that the rating of whether the dropout was premature was made by

the therapist, and perhaps the therapist who had a written list of the

patient’s treatment goals in the chart was more likely to be aware of the

patient’s unmet goals and to rate the patient who wanted to end treat-

ment before accomplishing all of their goals as ending treatment pre-

maturely. Relatedly, therapists seem often to have more goals for

patients than do the patients themselves. This is reected by the fact that

in our sample (in which patients were only included if they had had at

least three therapy sessions), a very large proportion of patients were

judged by the therapist to have terminated prematurely (52% of the

patients for whom we had premature dropout data) yet had completed a

substantial number of therapy sessions (mean ± std.err of 16.3 ± 0.9) at

the time they terminated treatment. (For all patients in the sample, the

mean ± std.err was 24.0 ± 1.1 sessions.)

We speculate that the effects of the tools on outcome and dropout

was mediated at least in part by the benecial effects of the use of the

tools on the therapeutic alliance. Notice that all of the decision support

tools we studied entailed a written document that therapist and patient

could review together and use to guide treatment. Thus, the sharing of

the case formulation (Kuyken et al., 2009) and the list of treatment

goals, and the collaborative discussion of progress and decision-making,

may all contribute to the development of a strong therapeutic alliance.

(For example, one of the last author’s patients, a business executive,

volunteered that the author’s practice of collecting and reviewing

outcome data conveyed a willingness to be held accountable that

increased his respect for her.) This notion is consistent with the facts that

agreement on the tasks and goals of therapy is widely viewed as an

element of the therapeutic alliance (Bordin, 1994), and uncollaborative

dropout would seem to be the quintessential example of an alliance

rupture (Safran et al., 2011). A strong alliance has been shown to be

related to both improved outcome (Horvath et al., 2011) and reduced

dropout (Roos & Werbart, 2013).

Of the three tools we studied in our sample, the one that most aligned

with our predictions was the plot of symptom scores, which predicted

reduced end-of-treatment Beck Depression Inventory score, marginally

predicted reduced end-of-treatment Burns Anxiety Inventory score (p =

0.09), and predicted lower rates of both types of dropout. This obser-

vation is consistent with a growing body of studies showing that

measurement-based care and routine outcome monitoring are associ-

ated with improved outcome (see reviews by Carlier et al., 2012;

Goodman et al., 2013; Lewis et al., 2018) and reduced dropout (Janse

et al., 2020; meta-analysis by; de Jong et al., 2012).

The data analyzed in the present study were collected over a long

time period that ended more than 10 years ago (1981–2009). Many

changes occurred in our eld over this time period, including changes in

therapists’ training, the development of new interventions and new

diagnostic systems, and the advent of online tools for the medical record

and progress monitoring. The question of whether these developments

have implications for our results is worth considering. However, our

hypotheses and data analyses were constructed in such a way that in-

sulates our ndings from these changes. We tested the hypothesis that

V. Gates et al.

Behaviour Research and Therapy 142 (2021) 103874

therapist use of decision support tools of a written case formulation, a

list of treatment goals, and a plot of symptom scores affected outcome

and dropout. These questions remain of interest today. In fact, our study

provides some empirical support for the treatment utility of online tools

that make it easier for clinicians to maintain a useful clinical record that

includes a written case formulation and a list of treatment goals, and to

collect and plot progress monitoring data.

Our study has several limitations. Use of the decision support tools

was not randomly assigned, and as a result we cannot conclude that the

therapist’s use of a formulation, plot, and treatment goals caused the

effects on outcome and dropout that we observed. However, because we

controlled for the main effect of therapist in our analyses, we can rule

out the competing account that the decision support tools were related

to outcome and dropout because the use of the tools reects the

conscientiousness or skill level of the therapist rather than the use of the

tools.

Several limitations affect our independent variables, the decision

support tools. We do not have any information about the content or

adequacy of the case formulations or lists of treatment goals; we have

only a rating of whether the tool was present in the clinical record.

Therapists may have used a case formulation to guide the treatment

even if a written formulation was not present in the clinical record. And

the fact that the clinical record included a plot does not indicate that the

therapist reviewed the plot with the patient. We also have no informa-

tion about when in the course of treatment the tools were developed and

the degree to which the therapist used the tool to guide the treatment.

Another limitation is that results cannot be assumed to generalize to

therapists other than the ones studied here, who are not typical of

therapists in the community. All of these therapists collected and plotted

symptom data for at least some of their patients; in contrast, less than

20% of providers use measurement-based care (Lewis et al., 2018). This

fact limits the generalizability of our ndings, as does the fact that pa-

tients were a homogeneous group of highly educated predominantly

White adults who paid high fees for their treatment.

Strengths of our investigation include our study of a sample of pa-

tients with multiple comorbidities who received treatment in a clinical

rather than a research setting, and the fact that most of the data we

studied here were collected in the course of routine clinical practice,

both of which are research strategies that increase the external validity

of our ndings (Weisz et al., 2014). Additional strengths of our study

include our focus on key elements of evidence-based practice (American

Psychological Association, 2006), and our examination of a large sample

of private practice patients that are infrequently represented in the

research literature. A nal strength of our study is that the decision

support tools we studied (the written case formulation, the list of

treatment goals, and the plot of symptom scores) can be used by any

psychotherapist of any discipline or psychotherapy orientation to treat

patients who seek treatment for any disorder or presenting problem.

Author declaration template

There are no nancial conicts of interest.

CRediT authorship contribution statement

Vael Gates: devised and conducted the data analysis and wrote most

of the Results section and some of the Discussion. Megan Hsiao: made

major contributions to organizing and cleaning the database. Garret G.

Zieve: assisted with the database organization, cleaning, and coding.

Rebecca Courry: assisted with coding for the inter-rater reliability

study. Jacqueline B. Persons: collected the data and drafted the

manuscript.

Acknowledgments

The rst author received funding from Defense Advanced Research

Projects Agency grant D17AC0004. We thank Thomas L. Grifths for his

support as the rst author’s thesis advisor, M. D. Edge for statistical

consultation, Cannon Thomas for helpful comments, and Connie Fee for

help cleaning and organizing the database. An earlier version of this

paper was presented at the Association for Behavioral and Cognitive

Therapies in Atlanta on November 22, 2019.

Appendix A. Supplementary data

Supplementary data to this article can be found online at https://doi.

org/10.1016/j.brat.2021.103874.

References

Abel, A., Hayes, A. M., Henley, W., & Kuyken, W. (2016). Sudden gains in

cognitive–behavior therapy for treatment-resistant depression: Processes of change.

Journal of Consulting and Clinical Psychology, 84(8), 726–737. https://doi.org/

10.1037/ccp0000101. PsycINFO.

American Psychological Association. (2006). Evidence-based practice in psychology.

American Psychologist, 61(4), 271–285.

Beck, J. S. (1995). Cognitive therapy: Basics and beyond. Guilford press.

Beck, A. T., Steer, R. A., & Garbin, M. G. (1988). Psychometric properties of the Beck

depression inventory: Twenty-ve years of evaluation. Clinical Psychology Review, 8,

77–100. https://doi.org/10.1016/0272-7358(88)90050-5.

Beck, A. T., Ward, C. H., Mendelsohn, M., Mock, J., & Erbaugh, J. (1961). An inventory

for measuring depression. Archives of General Psychiatry, 4, 561–571. https://doi.

org/10.1001/archpsyc.1961.01710120031004.

Bordin, E. S. (1994). Theory and research on the therapeutic working alliance: New

directions. In A. O. Horvath, & L. S. Greenberg (Eds.), The working alliance: Theory

research and practice. Wiley.

Burns, D. D., & Eidelson, R. J. (1998). Why are depression and anxiety correlated? A test

of the tripartite model. Journal of Consulting and Clinical Psychology, 66(3), 461.

Carlier, I. V. E., Meuldijk, D., Van Vliet, I. M., Van Fenema, E., Van der Wee, N. J. A., &

Zitman, F. G. (2012). Routine outcome monitoring and feedback on physical or

mental health status: Evidence and theory. Journal of Evaluation in Clinical Practice,

18(1), 104–110. https://doi.org/10.1111/j.1365-2753.2010.01543.x.

Cuijpers, P., Karyotaki, E., Weitz, E., Andersson, G., Hollon, S. D., & van Straten, A.

(2014). The effects of psychotherapies for major depression in adults on remission,

recovery and improvement: A meta-analysis. Journal of Affective Disorders, 159,

118–126. https://doi.org/10.1016/j.jad.2014.02.026.

Delgadillo, J., de Jong, K., Lucock, M., Lutz, W., Rubel, J., Gilbody, S., Ali, S., Aguirre, E.,

Appleton, M., Nevin, J., O’Hayon, H., Patel, U., Sainty, A., Spencer, P., &

McMillan, D. (2018). Feedback-informed treatment versus usual psychological

treatment for depression and anxiety: A multisite, open-label, cluster randomised

controlled trial. The Lancet Psychiatry, 5, 564–572. https://doi.org/10.1016/S2215-

0366(18)30162-7.

Derogatis, L. R., Rickels, K., & Rock, A. F. (1976). The SCL-90 and the MMPI: A step in the

validation of a new self-report scale. The British Journal of Psychiatry, 128(3),

280–289.

Eells, T. D. (Ed.). (2007). Handbook of psychotherapy case formulation (2nd ed.). Guilford.

Enders, C. K. (2017). Multiple imputation as a exible tool for missing data handling in

clinical research. Behaviour Research and Therapy, 98, 4–18. https://doi.org/

10.1016/j.brat.2016.11.008.

Fernandez, E., Salem, D., Swift, J. K., & Ramtahal, N. (2015). Meta-analysis of dropout

from cognitive behavioral therapy: Magnitude, timing, and moderators. Journal of

Consulting and Clinical Psychology, 83(6), 1108–1122. https://doi.org/10.1037/

ccp0000044.

Gardiner, J. C., Luo, Z., & Roman, L. A. (2009). Fixed effects, random effects and GEE:

What are the differences? Statistics in Medicine, 28(2), 221–239. https://doi.org/

10.1002/sim.3478.

Goodman, J. D., McKay, J. R., & DePhilippis, D. (2013). Progress monitoring in mental

health and addiction treatment: A means of improving care. Professional Psychology:

Research and Practice, 44(4), 231–246. https://doi.org/10.1037/a0032605.

Hawkins, E. J., Lambert, M. J., Vermeersch, D. A., Slade, K. L., & Tuttle, K. C. (2004). The

therapeutic effects of providing patient progress information to therapists and

patients. Psychotherapy Research, 14(3), 308–327. https://doi.org/10.1093/ptr/

kph027.

Honaker, J., King, G., & Blackwell, M. (2012). Amelia II: A Program for Missing Data

(1.6.2) [Computer Software]. https://gking.harvard.edu/amelia.

Horvath, A. O., Del Re, A. C., Flückiger, C., & Symonds, D. (2011). Alliance in individual

psychotherapy. Psychotherapy, 48(1), 9.

Janse, P. D., de Jong, K., Veerkamp, C., van Dijk, M. K., Hutschemaekers, G. J. M., &

Verbraak, M. J. P. M. (2020). The effect of feedback-informed cognitive behavioral

therapy on treatment outcome: A randomized controlled trial. Journal of Consulting

and Clinical Psychology, 88(9), 818–828. https://doi.org/10.1037/ccp0000549.

de Jong, K., van Sluis, P., Nugter, M. A., Heiser, W. J., & Spinhoven, P. (2012).

Understanding the differential impact of outcome monitoring: Therapist variables

that moderate feedback effects in a randomized clinical trial. Psychotherapy Research,

22(4), 464–474. https://doi.org/10.1080/10503307.2012.673023. PsycINFO.

Kuyken, W., Padesky, C. A., & Dudley, R. (2009). Collaborative case conceptualization.

Guilford.

V. Gates et al.

Behaviour Research and Therapy 142 (2021) 103874

Lambert, M. J., Harmon, C., Slade, K., Whipple, J. L., & Hawkins, E. J. (2005). Providing

feedback to psychotherapists on their patients’ progress: Clinical results and practice

suggestions. Journal of Clinical Psychology, 61(2), 165–174. https://doi.org/10.1002/

jclp.20113.

Lambert, M. J., & Shimokawa, K. (2011). Collecting client feedback. Psychotherapy, 48

(1), 72–79. https://doi.org/10.1037/a0022238.

Lewis, C. C., Boyd, M., Puspitasari, A., Navarro, E., Howard, J., Kassab, H., … Douglas, S.

(2018). Implementing measurement-based care in behavioral health: A review.

JAMA Psychiatry, 76(3), 324–335. https://doi.org/10.1001/jamapsychiatry.2018.

3329.

Persons, J. B. (1989). Cognitive therapy in practice: A case formulation approach. Norton &

company.

Persons, J. B. (2008). The case formulation approach to cognitive-behavior therapy. Guilford.

Persons, J. B., & Hong, J. J. (2016). Case formulation and the outcome of cognitive

behavior therapy. In N. Tarrier, & J. Johnson (Eds.), Case formulation in cognitive

behaviour therapy (2nd ed., pp. 14–37) Routledge.

Project Jupyter. (n.d.). Jupyter Notebook. https://jupyter.org.

Python Software Foundation. (n.d.).Python Language Reference (3.6.5) [Computer

software]. http://python.org.

R Core Team. (2007). ls: Least squares regression for continuous dependent variables.

http://zeligproject.org.

R Core Team. (2018). R: A language and environment for statistical computing. R

foundation for statistical computing. https://www.r-project.org/.

Roos, J., & Werbart, A. (2013). Therapist and relationship factors inuencing dropout

from individual psychotherapy: A literature review. Psychotherapy Research, 23,

394–418. https://doi.org/10.1080/10503307.2013.776628.

Safran, J. D., Muran, J. C., & Eubanks-Carter, C. (2011). Repairing alliance ruptures.

Psychotherapy, 48(1), 80–87. https://doi.org/10.1037/a0022140. PsycINFO.

Vittengl, J. R., Clark, L. A., Thase, M. E., & Jarrett, R. B. (2019). Estimating outcome

probabilities from early symptom changes in cognitive therapy for recurrent

depression. Journal of Consulting and Clinical Psychology, 87(6), 510.

Weisz, J. R., Ng, M. Y., & Bearman, S. K. (2014). Odd couple? Re-envisioning the relation

between science and practice in the dissemination-implementation era. Clinical

Psychological Science, 2(1), 58–74. https://doi.org/10.1177/2167702613501307.

V. Gates et al.