|
CHAPTER 3.
METHODOLOGICAL ISSUES IN ECONOMIC EVALUATIONS
3.1 Introduction
Conceptually, it is difficult to
identify, measure and value the costs and consequences of health
care, or alcohol and drug abuse programs. Many choices have to be
made and no accepted methodology exists for conducting economic
evaluations. Drummond et al. (1997) set forth 28 methodological
questions that need to be answered in economic evaluations. In the
absence of an accepted methodology, it is not surprising that the
estimated economic evaluations of alcohol and drug abuse treatment
programs are not consistent.
The literature on health care
economic evaluations in general and on alcoholism and drug abuse
treatments in particular is fairly recent and much of it is not of
very good methodological quality (see Drummond et al. 1997 Chapter
3; Gold et al., 1996; Saxe, 1983; and Holder, Lennox, and Blose,
1992). To make good decisions, government officials and program
managers need reliable and valid information that is as free from
bias as possible. Biases refer to factors other than those studied
in an evaluation that affect the outcomes considered in
evaluations. Sound methodological studies control for extraneous
factors that may influence evaluation results. Ideally, we want to
eliminate competing explanations of our findings to have confidence
that we have properly measured the relationships specified in the
evaluation.
This chapter focuses on the major
elements of economic evaluations and the potential biases associated
with each of them. First, we consider the conceptual framework and
the analyst’s point of view. Next, we focus on the treatment design
that addresses the questions of what treatments were provided to
which patients for how long. Third, we discuss the problems in
identifying, measuring, and valuing the costs of treatments. The
next section examines the same problems with respect to outcomes or
consequences of treatments. Section five discusses the issue of
research design with special attention given to the choices of the
control group and the alternative program considered. The following
section focuses on sampling questions. Section seven discusses the
issue of the timing of the costs and benefits of programs and the
need for discounting. The next section stresses the importance of
incremental analysis of costs and consequences of alternatives. The
issues of uncertainty and statistical analysis are considered in the
following section. The conclusions on methodological issues are
presented in the final section.
3.2
The Conceptual Framework
As noted in chapter 2, economic
evaluation is the comparative analysis of alternative courses of
action in terms of both their costs and consequences (Drummond, et
al., 1997, p. 8). The basic tasks of any economic evaluation are to
identify, measure, value, and compare the costs and consequences of
the alternatives being examined. The focus of economic evaluations
differs and the perspective of the analyst determines the relevant
costs and consequences that should be included in the evaluation.
A very simple conceptual
framework for analyzing the categories of costs and consequences
that are relevant to an economic evaluation of health services and
programs is given in Figure 3.1. There are three general categories
of costs associated with health care programs. Health care sector
costs (C1) consist of the costs of organizing and
operating the program, including dealing with the adverse events
caused by the program.
Source: Drummond, et al. 1997, p. 19.
Both variable and fixed or overhead costs
must be considered. Patient and family costs (C2)
include any out-of-pocket expenses incurred by patients and/or
family members as well as the value of resources and time that they
contribute to the treatment process. Other sector costs (C3)
refer to resources consumed in other sectors such as those provided
by other public agencies and volunteers.
The simple conceptual framework
assumes there are three general categories of outcomes or
consequences of the program, as indicated in Figure 3.1. Health
state changes relate to changes in the physical, social or emotional
functioning of patients. The values of the health state effects (E)
can be obtained either by health state preference scores (U) or
willingness-to-pay (W). Other value (V) may be created by the
program (e.g. reduction in anxiety to the patient and family
members). Finally, resources may be freed and cost-savings
generated in the healthcare sector (S1), to patients and
their family (S2), and to other sectors (S3).
As we shall see later, this last outcome is particularly important
in the case of alcohol and drug abuse treatment programs.
In reality, conceptualization is
far more complex and complicated than that shown in Figure 3.1. The
researcher needs to provide a theoretical explanation of the
services provided to patients and explain why they are likely to
have the effects hypothesized. He needs to identify all of the
important consequences of the treatments for all the affected
parties. The researcher must specify the time frame of the
analysis. The time frame needs to be long enough to encompass all
of the major consequences that flow from the program.
If the least important, less
relevant, or wrong variables and relationships are selected for
study, the usefulness of the evaluation may be limited and the
results biased. If the post-intervention period is not long enough
to capture all of the future costs and consequences of the program,
the results can be misleading. In health care evaluations, the
event pathway for the analysis usually extends over the course of
the episode of illness, which frequently is the patient’s expected
lifetime. The possibility of future health care interventions and
their costs are incorporated in the analysis (Drummond et al. 1997,
p. 127). In contrast, evaluations of alcohol and drug abuse
treatment programs focus on the short-term effects of single
episodes of treatment.
Because economic evaluations
invoke comparisons of treatment programs, the selection of the
appropriate comparator is crucial in a cost-effectiveness study. In
general, studies should compare a new intervention or program to
existing programs. This is particularly true if one is considering
replacing an existing program with an alternative (Gold et al.
1996). It can be expensive and time consuming to gather information
on existing programs, so researchers often choose a “do-nothing”
option, which assumes that no other treatment programs for the same
illness are in operation. The “do-nothing” option program has no
costs and consequences. Therefore, the incremental
cost-effectiveness algorithm
 is based on the program under evaluation.
3.3
The Analyst’s Point of View
A number of possible economic
evaluations can be made depending on the perspective adopted by the
analysts and the estimation methods they use. Let us consider four
different analytical perspectives. Analyst A is a welfare economist
whose primary interest is on the efficient allocation of scarce
resources within the economy. He places considerable emphasis on
the values individuals place on outcomes, since individuals are
considered to be the best judges of their own welfare. Analyst A is
particularly interested in conducting cost-benefit analysis to
assess whether the program is worthwhile.
Analyst B is a budget director
who sponsors economic evaluations to help her allocate the health
care budget. She is only interested in the costs and consequences
in the health care sector. Analyst B believes that health outcomes
should be measured in natural units or health state preference
scores, but not in dollars through willingness-to-pay valuations.
She is interested in cost-minimization analysis (CMA),
cost-effectiveness analysis (CEA), or cost-utility analysis (CUA),
but not cost-benefit analysis (CBA).
Analyst C is a health care
researcher who has a broad societal perspective and therefore
believes that both health sector and non-health sector costs and
consequences should be included in the analysis. He or she
recognizes that some of the costs and consequences are easier to
express than others. Analyst C rejects the use of
willingness-to-pay evaluations and therefore does not conduct
cost-benefit analysis.
Analyst D represents the
taxpayer’s perspective. He is only interested in the question of
whether investing taxpayer dollars in the program is a good
investment in the sense that the cost-savings to other public
programs in the healthcare sector and non-healthcare sector will
more than cover the initial outlay of funds. Analyst D is not
concerned about patient and family costs (C1) or
consequences (S2) or other non-public sector costs (i.e.
cost of volunteers) or cost-savings.
Table 3.1 presents a number of
possible formulations of economic evaluation based on different
analysts’ perspectives and the costs and consequences shown in
Figure 3.1. Analyst A, the welfare economist, favors the cost
benefit analysis indicated by evaluation 4.a, where W represents
the total value of all consequences. In our literature
Table 3.1 Possible Formulations of Economic Evaluation in Health
Care
(1)
Cost-minimization analysis
1.a (C1 - S1)
1.b (C1 + C2 + C3) - (S1
+ S2 + S3)
1.c (C1 + C ) - (S1 + S)
(2)
Cost-effectiveness analysis
2.a (C1 - S1)/E
2.b [(C1 + C2 + C3) – (S1
+ S2 + S3)]/E
2.c [(C1 + C) – (S1 + S)]/E
(3)
Cost-utility analysis
3.a (C1 - S1)/U
3.b [(C1 + C2 + C3) - (S1
+ S2 + S3)]/U
(4)
Cost-benefit analysis
4.a (W’) –
(C1 + C2 + C3)
search, we could find no full-blown CBA of
alcohol and drug abuse treatment programs to determine if they are
worthwhile.
Analyst B, the health sector
budget director, whose focus is limited to the health sector
exclusively, would be interested in the cost-minimization analysis
1.a, the cost-effectiveness analysis 2.a, or the cost-utility
analysis 3.a.
Analyst C, the health care
researcher with the broader perspective would be interested in cost
minimization analysis 1.b, the cost-effectiveness analysis 2.b, and
the cost-utility analysis 3.b.
Analyst D, representing the
taxpayers’ perspective, would be interested in cost-minimization
analysis 1.c where  represents other public non-health care costs and
 represents other public non-healthcare resource
savings. He or she also would be interested in cost-effectiveness
analysis 2.c. Since analyst D does not care about patient and
family costs and consequences, he would have no interest in
conducting CUA or CBA studies. The vast majority of alcohol and
drug abuse treatment economic evaluations have been based on Analyst
D perspective; perhaps because they have been conducted or sponsored
by parties interested in increasing public spending on these
programs.
3.4
Treatment Design
Alcohol and drug abuse programs
vary in terms of patient characteristics, treatment settings,
services offered, and practitioner characteristics. Because these
features interact to affect treatment outcome, it is difficult to
assess the relative effectiveness of individual programs. For
meaningful economic evaluations to be made, both the primary program
and the alternative must be fully described in terms of who does
what to whom, where, and how often and the consequences for each
group of patients (Drummond et al. 1997). The components of each
program should be well-enough specified so that readers can compare
the subject of the CEA to other programs and know whether their
cost-effectiveness is likely to be similar or very different (Gold
et al. 1996, p. 42). By tracking the types and amount of different
treatments provided to individuals in different patient groups and
the consequences, program managers and others can learn what types
of treatments work best for which patients. Treatment subgroup
analysis can help managers to redesign their program or create new
programs that are more efficient. Such analysis can also help
decision markers to target patient populations for new programs on
the basis of their likely efficiency (Gold, et al. 1996, p. 42).
Of course, the costs of
treatments to the individuals in each patient group also would have
to be tracked. Some subgroups can be expected to consume more
treatment resources than others. Many alcohol and drug abuse
program patients relapse and return for more treatments (Richman,
1983; Holder, Lennox, and Blose, 1992; French, 1995; and Cartwright,
1998). Economic evaluations of alcohol and drug abuse programs are
based on individual episodes at specific programs. They usually
make no distinction between first-time patients and those returning
for more treatment. Readmitted patients and new patients are likely
to have different expected costs and outcomes, and evaluations that
fail to account for the mix of new and readmitted patients may
produce misleading results (Richman, 1983). At the very least,
economic evaluations should track the costs and consequences for
these two groups separately.
The alcohol and drug abuse
evaluation literature frequently makes a distinction between the
relative effectiveness of “inpatient” versus “outpatient” treatment
programs. Frequently, however, these studies fail to account for
differences in the specific services provided or for differences in
the patient populations. Thus, we cannot be sure if the reported
differences in cost-effectiveness by type of setting are real or
illusionary. We need to know what treatments are provided for whom
under what conditions.
One last potential problem needs
to be mentioned. Sometimes particular types of patients are
assigned to receive certain treatments and not others. The
assignment of patients to treatments makes it impossible to isolate
the effects of treatments or types of patients on outcomes. If
patients were randomly assigned to different types of treatments, we
could clearly estimate the effects of different treatments on
outcomes. If patient assignment is present, the estimated effects
of treatments on outcomes will be biased.
3.5
Identification, Measurement, and Valuation of Costs
The cost of alcohol and drug
abuse treatment depends on the amount of resources used to treat
patients. As noted in Figure 3.1, there are three main categories
of costs of health care programs: healthcare sector costs (C1),
patients and family costs (C2), and other sector costs (C3).
The relevant costs to be included in any given evaluation depend on
the viewpoint of the analyst (see Table 3.1). For example,
patient’s travel costs are a cost from the patient’s point of view
and from the view of society (i.e., welfare economist), but not a
cost from the State Director of Health Services, program manager, or
taxpayer (Drummon, et al. 1997, p. 52).
In a recent study, Cartwright
(1998) identified eight cost categories for drug abuse treatment.
(1)
Operating expenses
This
category includes salaries of treatment personnel, and expenditures
on methadone and legal drugs, pharmacy services, drug testing, data
collection, ancillary services, supplies, and room and board.
(2)
Overhead expenses for administration, space and insurance
(3)
Screening and assessment expenses
(4)
Ancillary services
This
category includes: childcare, housing, social work services,
transportation assistance, and welfare. He argues these are not
strictly drug abuse treatment resources and are not counted in all
evaluations.
(5)
Medical care
This
category includes examinations, screenings, and treatments.
Cartwright says these are not strictly treatment expenditures and
some evaluations exclude them from consideration.
(6)
Dropouts
Resources
expended by the program on dropouts should be included as a cost.
(7)
Criminal justice supervision
This
category includes additional resources incurred for special programs
for addicts such as drug courts, parole and probation special
programs, and diversion to treatment programs. Cartwright says
these are not strictly treatment expenses and so they could be
ignored in the evaluation.
(8)
Lost earnings
This refers
to foregone employment income while in treatment.
Once program costs have been
identified, they still must be measured and evaluated. Ideally, the
researcher could determine from accounting records the amount of
resources consumed by each patient during the treatment episode.
Market prices when available can be used to determine the dollar
cost of resources used by each patient. Volunteer and
patient/family time inputs are more difficult to value. Researchers
can use the patient’s market wage and the market wage for workers
similar to volunteers as the unit price for these non-market
resources (Drummond, et al. 1997, p. 55).
Alcohol and drug abuse program
evaluations rarely attempt to estimate individual patient costs of
treatment. Rather, program accounting data on income and
expenditures are usually obtained for the period of record, and the
total treated patients are divided into total program costs to
estimate an average cost per patient (Cartwright, 1998). There are
several problems with this approach (NIDA 1999). First, not all
patients use the same amount of resources. Some patients show up
for all appointments; others frequently do not show up. Some
patients leave treatment early while others stay the course. In
addition, patients’ outcomes and behavior change in association with
the amount of resources consumed. To analyze cost-effectiveness and
cost-benefit accurately, cost as well as effectiveness and benefit
must be measured separately for each patient.
A trade off is involved in making
cost calculations. Average patient costs are less precise but
easier to make. Micro-costing is more accurate, but more difficult
to calculate. In alcohol and drug abuse evaluations, researchers
have primarily chosen to estimate patient average costs. The
National Institute of Drug Abuse (NIDA, 1999) is encouraging drug
abuse programs to begin micro-costing, that is, to calculate the
cost per procedure for each patient so that the program can
determine which procedures work for which patients so that they can
become more cost-effective.
All relevant costs should be
included in the analysis. Capital costs are the costs to purchase
the major capital assets required by the program; generally
equipment, building, and land. Frequently, the capital costs are
not listed in the accounts or budgets because they have been funded
earlier. Some annual budgets include an item called depreciation
that relates to capital cost, but does not fully represent it.
There are several methods of measuring and valuing capital costs in
an economic evaluation. According to Drummond et al. (1997, p. 60),
the best method is to annuitize the initial capital outlay over the
useful life of the asset. This method automatically incorporates
both the depreciation aspect and the opportunity cost of the capital
cost. Other methods can be used to estimate capital costs, but
these costs should never be ignored in economic evaluations.
Shared or overhead costs also
should not be ignored. Alcohol and drug abuse treatment services
are often offered in an inpatient setting. The alcohol and drug
abuse treatment programs must be assessed for their share of the
overhead costs. There are a number of methods for allocating these
costs to the programs (see Drummond, et al. (1997, pp. 62-66). In
economic evaluations some attempt must be made to include such costs
in the analysis.
In general, economic evaluations
of health care intervention focus on the individual patient. They
analyze the costs and consequences of the intervention over the
individual’s expected lifetime. Such evaluations tend to assign all
the credit for life extension to the intervention. It would make
sense to consider the costs of the provision of additional care in
added years of life since they are a consequence of the program if
data are available (Drummond, et al., 1997, pp. 57-58). This means
that increases in costs associated with the health care intervention
may occur many years in the future.
Evaluations of alcohol and drug
treatment programs focus on the changes in costs and outcomes for
single episodes over fairly short periods of time. They tend to
ignore long-term benefits such as life-extensions and the long-term
additional costs associated with the added years of life.
Because economic evaluations make
comparisons of programs and services at one point in time, the
timing of program costs and consequences that do not occur entirely
in the present must be taken into account. Individuals and society
have a time preference. They prefer to receive dollars or resources
now as opposed to later so they can benefit from them in the
interim. In economic evaluations, “future dollar cost and benefit
streams are reduced or ‘discounted’ to reflect the fact that dollars
spent or saved in the future should not weigh as heavily in
programme decisions as dollars spent or saved today” (Drummond, et
al. 1997, p. 39). While there is a debate among economists about
the choice of discount rate, there is little debate over whether
costs should be discounted in economic evaluations (Drummond, et al.
1997, pp. 72-74). Because the focus in alcohol and drug abuse
program evaluations is on short-run effects, it is rare for such
evaluations to discount future costs and consequences.
3.6
Identification, Measurement, and Valuation of Program
Consequences or Outcomes
In framing a cost-effectiveness
analysis, the researcher should attempt to identify all of the
possible consequences or outcomes of the program. As shown earlier
in Figure 3.1, there are three general types of consequences
associated with health care interventions.
(1)
Patient health effect changes
A health care intervention may lead to
a gain in life years and/or an improvement in the quality of life.
Health-related quality of life itself can incorporate many domains
of health. The measurement of health outcomes is relatively
straightforward. Effects relating to mortality can be measured in
life-years gained or deaths averted. Effects relating to morbidity
might be measured by reductions in disability days or improvement on
some index of health status measuring physical, social, or emotional
functioning (Drummond, et al. 1997, Chp. 4).
Often in health cost-effectiveness
analysis, intermediate rather than final outcome measures are
calculated. For example, in the treatment of hypertension
effectiveness has been measured in mmHg blood pressure reduction.
In the treatment of hypercholesterolaemia, effectiveness has been
measured in % serum cholesterol reduction. Finally, asthma
treatment effectiveness has been measured in episode-free drugs.
Drummond, et al. (1977, pp. 102) argue that if a CEA uses
effectiveness data on intermediate outcomes, the analysis should
either make a case for the intermediate endpoint having value or
clinical relevance in its own right or provide evidence of a link
between the intermediate outcome and a final outcome.
Improved health could lead to greater
periods of employment and higher earnings for the individual and
more tax revenue for government. These indirect health productivity
effects are usually measured by the change in the gross earnings of
the individual. There are some concerns that this method of
valuation raises equity considerations in an evaluation but it is
commonly used (Drummond et al. 1997, p. 106).
(2)
Other value
While participation in a program may
cause a reduction in anxiety to the patient and family quality of
life, no serious attempts have been made to measure and value this
outcome.
(3)
Resource savings
Effective health care interventions can
generate cost-savings in the healthcare sector. The value of these
savings is measured by the expected decrease in health care
resources consumed. Improved health can reduce the patient and
family resource cost, although such savings are frequently ignored
in health care cost-effectiveness analysis. Finally, improved
health status and indirect health productivity effects may lead to
resource savings in the non-health care sector. For example, the
amount of government income and services provided to the patient and
his family may be reduced.
The choice of which of these
types of outcomes is relevant to economic evaluations depends on the
view of the analysts. If the study is conducted by Analyst A who
takes a society view in the allocation of scarce resources, then it
will use cost-benefit analysis. All outcomes except those involving
transfer payments will be included in the analysis. Under
cost-benefit analysis, health care outcomes will be measured in
dollars according to willingness to pay (W) and other outcomes also
will be measured in dollars. Outcomes involving resources saved in
other sectors, which are simply transfer payments, will be ignored
by Analyst A.
From a health budget director’s
point of view (i.e., Analyst B), only health outcomes, measured in
natural units or health state preference scores, and resources saved
in the healthcare sector (S1) should be included in the
analysis. The budget director wants to know if the health care
intervention is likely to increase or decrease his budget. Analyst
B does not worry about health productivity changes or changes in
other non-health care costs such as patient’s time, volunteer time
and costs falling on other agencies.
Analyst C health care researchers
take a broad societal view so they would include both health status
changes and changes in resource use in both the health and
non-health care sectors, but they would ignore changes in transfer
payments. Health outcome changes would be measured in natural units
under cost-effectiveness analysis and in health state preferences
under cost-utility analysis. Resources saved in the healthcare
sector and by the patient and family would be measured in dollars.
Analyst D, who takes the
taxpayer’s perspective, will like Analyst A ignore the health
productivity effects accruing to the patient, but will include any
increase in tax revenue flowing from them. He will ignore changes
in non-health care use of resources by patient/family and
volunteers. Analyst D will be quite interested in resources saved
by other public agencies because these could be used to provide
other public services or to provide some taxpayer relief.
As just seen, a number of types
of economic evaluations of health care interventions can be
undertaken, and the relevant outcomes that are included depends on
the analyst’s viewpoint. Economists tend to stress the society
point of view and the need to consider all costs and consequences in
the analysis (Drummond et al. 1997, p. 106). They recognize
analytical difficulties and data limitations will often preclude the
full measurement and valuation of all costs and consequences in
monetary terms. In general, economic evaluations of health care
interventions focus on the life time changes in the health status of
the individual and the changes in resource costs in the health care
sector, because these are quantitatively the most important and the
major goal of such interventions is to improve health.
The focus of economic evaluations
of alcohol and drug abuse treatment programs is generally not on
changes in the patient’s health status per se. Rather, they focus
on changes in the patient’s behavior and the associated changes in
resource use in the healthcare and non-health care sectors, but not
by the patient and their families. As Saxe (1983) notes, the
controversy over what should be the measure of successful outcomes
in treating alcohol problems is still ongoing. Traditionally,
abstinence from alcohol use has been the preferred measure of
treatment effectiveness. More recently, other measures have been
used related to drinking behavior such as frequency of drinking,
number of ounces of alcohol ingested, number of binges, days of
abstinence, number of relapses, and percentage of days without
alcohol related problems. Some studies have considered other
outcome measures, including work adjustments, family adjustments,
problems with the law, psychological well-being, and continuation of
treatment. A few researchers have used physical health in the
context of cost-benefit analysis of treatment. Virtually all of
these studies focus on the short-term effects of each episode of
treatment. They make no attempt to calculate the long-term or
lifetime effects of alcohol treatments.
According to the National
Institute on Drug Abuse (NIDA, 1999), the primary outcome desired by
all substance abuse treatment programs is total permanent abstinence
from illicit drugs. To achieve this goal, patients have to make
many major changes in lifestyle, attitudes, friends, skills and so
forth. Some programs consider these personal changes as interim
outcomes; others regard them as final outcomes. The NIDA identifies
seven general types of interim outcomes: (1) Relations with peers,
children, spouse/mate, relatives, employer, and others; (2)
Employment; (3) Independent living; (4) Cessation of substance
abuse; (5) HIV transmission behaviors; (6) Physical health; and (7)
Mental health.
Drug abusers use large amounts of
health care resources, and because of their poor health status, they
receive large amounts of non-health care resources through
government transfer programs. Finally, drug abusers inflict high
costs on other parts of the economy through their criminal
activities. For this reason, successful drug abuse treatments,
which eliminate or reduce drug abuse problems and improve health
status can produce a large amount of resource savings to society.
Cartwright (1998) provides the following list of potential
cost-savings from drug abuse treatment programs.
(1)
Individuals will spend less to protect themselves from
predatory criminal acts.
(2)
Fewer victims will be injured during criminal acts,
which will free health care resources
(3)
Less property will be damaged or destroyed because of
the reduction in criminal activity.
(4)
There will be a decrease in victim loss of work income.
(5)
Resources will be saved in the criminal justice system
by law enforcement, courts, corrections, parole, and probation, and
reduced work loss due to incarceration due to the decline in
drug-abusers’ criminal activity.
(6)
There will be a reduction in addicts’ resources used to
commit crimes and an increase in legal employment.
(7)
There may be a reduction in the production and
distribution of illegal drugs and resources can be shifted from the
illegal drug trade into legal activities.
(8)
Health care resource savings would arise from the
patient’s improved health state and reductions in communicable
diseases (e.g. AIDS).
(9)
There would be a decline in drug-related deaths.
(10)
We can expect a reduction in welfare administration costs.
In the next chapter, we shall review the
alcohol and drug abuse cost of illness studies, which estimate the
potential benefits or cost-savings from the social viewpoint. We
shall also review the studies undertaken for governments, which
examine the taxpayer cost of welfare transfer payments related to
drug problems in the welfare population.
Earlier, we discussed the need
for discounting costs if the timing of costs varies between two
programs being evaluated and this practice has become well
established. The issue of discounting the outcomes or consequences
of health care programs is more controversial. There are good pro
and con arguments for discounting the effects of health care
programs, but most economists believe that effects should be treated
in the same way as costs, and discounted at the same rate (Gold et
al. 1996 and Drummond et al. 1997, pp. 107-109). In most economic
evaluations of alcohol and drug abuse treatment programs the issue
is relatively unimportant because of the short period of time
considered.
If the major program outcomes are
not properly identified and measured accurately, the results will
lack validity and lose their usefulness to government decision
makers and program managers.
3.7
Research Design
Economic evaluations should be
based on the best designed and least biased data sources available.
In our analysis, we shall give greater weight to evaluation results
derived from research designs that are less prone to bias.
According to Gold et al. (1996, p. 97) , the hierarchy of research
designs that can be used in a cost-effectiveness analysis is in
descending order: random clinical trials (RCTs); observational
data, including cohort, case-control, and cross-sectional studies;
uncontrolled experiments; and descriptive series
(1)
Randomized clinical trials (RCTs)
Under an RCT patients are
randomly assigned to an “experimental group” which receives
treatment services and to a “control group” which does not receive
treatment services. The advantage of this research design, in
comparison to a nonrandom design, is that it allows differences in
outcomes to be attributed more confidently to the treatment, and not
to preexisting differences in the samples tested.
RCTs have their strengths and
weaknesses when applied to economic evaluations of health care or
alcohol and drug abuse programs. An RCT has strong internal
validity. It eliminates the problem of assignment bias created when
the staff assigns patients to treatment services on the basis of
their perceived needs. RCTs, however, often include
protocol-induced resource consumption, in order to institute
control, which would not be done in everyday practice. This raises
the question of external validity. If RCTs operate under
restrictive conditions, it is possible the evaluation results will
not generalize to larger population groups. Second, RCTs are
expensive and they may not continue long enough to capture the full
economic consequences of the intervention. In this case, models are
required to estimate the likelihood of utilization and health events
occurring in the future (Gold et al. 1996, p. 131). Finally,
because of their high cost, RCTs frequently are limited to
relatively small samples, which may not be representative of the
larger population group (e.g., alcoholics) of interest, which raises
further questions about the external validity of the results. In
conclusion, RCTs are stronger with respect to internal validity, but
they tend to be weaker on realism, and on external validity.
(2)
Randomized Cost-effectiveness trials.
A randomized cost-effectiveness trial
is one in which the trial itself is specifically designed to study
cost-effectiveness, as opposed to RCTs which are designed to study
efficacy. Under this research design, patients are randomly
assigned to a study group, often comparing one course of therapy to
usual care or another active control. Few additional constraints
are imposed. Because the randomized cost-effectiveness trial
examines an intervention in a real-world health care context, it has
greater external validity than an RCT. Unfortunately, randomized
cost-effectiveness trials have lower internal validity than RCTs
“because the relaxed protocol constraints permit the introduction of
potential confounding variables such as patient cross-over, bias due
to lack of binding, and variations in practice patterns” (Gold et
al. 1996, p. 50).
Under both RCTs and randomized
cost-effectiveness trials, primary data are collected at the time
the treatments are administered and the outcomes or consequences of
the treatments are measured through follow-up studies. Frequently,
however, economic evaluations are based on secondary data, which
were collected for purposes other than economic evaluations.
(3)
Retrospective cohort design
This design examines health care
utilization or costs based on secondary data for a patient cohort
that has experienced a given intervention and a cohort that has not,
comparing the two. The problem is that one cannot be sure that the
treatment group does not vary from the non-treatment group in some
important way related to outcome measures because the patients were
not randomly assigned to treatment groups. The potential selection
or assignment bias affecting who did and did not receive an
intervention may limit the value and generalizability of the results
(Gold et al. 1996, p. 50). If the retrospective cohort data set is
large and comprehensive, it may be possible to control for some
confounding variables such as age, sex, severity of disease and
other potential risk factors. In some instances, it may not be
possible to eliminate or reduce the selection bias. For example,
patients may enter treatment at the bottom of a cycle of social
disability productivity when they are “ready to improve.”
Statistically significant comparisons of pre- and post-treatment
functioning therefore can be misleading measures of treatment
effectiveness (Richman, 1983).
A second problem with the retrospective
cohort design is that since the data were collected for purposes
other than the study of cost-effectiveness some important cost data
such as out-of-pocket expenditures, time expended for treatment, and
informal caregiver time are not available retrospectively. The
omission of these costs will bias the estimates of
cost-effectiveness upward.
There are advantages in using secondary
data such as claims databases compared to RCTs. First, the
secondary data are drawn from actual “real world” experience with
the use of an intervention, so economic evaluations based on
secondary data may have greater external validity. Second,
secondary data are less expensive to collect and they may cover an
extended period of time sufficient to calculate the long-term
effects of the intervention. Finally, the secondary data sets tend
to be much larger than RCT or randomized cost-effectiveness data
bases so the effects of the intervention can be estimated more
precisely. In summary, the retrospective cohort design is generally
weaker in controlling bias than the RCT or randomized
cost-effectiveness designs, but it tends to be stronger with respect
to realism and generalizability.
(4)
Descriptive Design
The description gross outcome design
merely tracks patient’s behavior prior, during, and after
treatments. Comparison data for non-treated individuals (i.e., a
control group) are typically not collected. Statistical tests are
conducted to determine if there is a significant difference in
prior, during, and post treatment behavior. The absence of an
external control group raises serious questions with respect to
potential bias and the internal validity of the study. Without a
control group, it is impossible to determine if a significant change
in behavior is due to the treatment or some other factors, which
changed during the observation period. It is possible that early
dropouts could be used as a control group if adequate data were
available (Gerstein 1991), but this has rarely been done in the
alcohol and drug abuse literature.
Another problem with the descriptive
gross outcome design is that results are often presented in
aggregate form, i.e., average per patient measures. Data on
resource use and outcomes are often not differentiated by severity
of initial symptoms or other patient characteristics. As noted
above, average results can be misleading because patients consume
different amounts of resources and they do not respond to treatments
uniformly.
Despite these limitations, until fairly
recently the descriptive gross outcome design was the most commonly
used design in evaluations of alcohol and drug abuse treatment
programs (Saxe 1983; Holden, Lennox, and Blose 1992; French 1995;
and Cartwright 1999). The use of this design is particularly
problematic in the area of alcohol and drug abuse given the evidence
that alcoholics and drug abusers frequently “ramp up” their
substance abuse and consumption of medical services just prior to
entering treatments (Richman 1983). In recent years, a number of
well-controlled retrospective cohort and a few randomized
cost-effectiveness studies of alcohol and drug abuse treatment
programs have been conducted. We shall base our estimates of the
cost-effectiveness of such programs primarily on the basis of these
studies.
(5)
Modeling designs
There are two main groups of models
used in health care economic evaluations, clinical decision-analytic
models and epidemiologically based models (Gold et al. 1996, p.
51). Clinical decision-analytic models analyze clinical decisions
pertaining to an intervention. They trace out the implications of
the clinical decisions over the life of the patient.
Epidemiologically-based models track risk factors and the course of
disease over the patient’s lifetime. Interventions are represented
as changes in probabilities of movement through the event pathway.
Modeling designs have not been used in the alcohol and drug abuse
literature because of the focus on the short-term effects of single
episodes of treatment. If there is a shift toward the long-term
health effects of such programs, modeling designs will become
important.
Most evaluations of alcoholism and drug
abuse programs usually ignore the issue of readmission despite the
fact that relapse rates often exceed fifty percent (Richman 1983 and
Holder, Lennox, and Blose 1992). Without tracking patients who
return for more treatment, cost and effectiveness measures could be
seriously biased. As noted above, treatment cost and outcomes are
often reported on an aggregate per patient basis. If first-time
patients and readmitted patients tend to utilize different amounts
of resources and have different expected outcomes, then programs’
evaluations could be a function of their mix of patients. Ideally,
evaluations should consider the care received by an individual over
a period of time, not just a single treatment episode (French
1995). At the very least, evaluations should track the costs and
outcomes for newly treated patients and readmitted patients
separately to eliminate the influence of patient mix on the
program’s cost-effectiveness ratio. Finally, extrapolating
long-term outcomes from short-term results based on brief follow-up
periods to a single episode could be hazardous. Many patients
experiencing post-treatment declines in drug abuse or abstinence are
likely to suffer relapses in the future. In the absence of tracking
individual patients, we need to model the relapse phenomenon so that
we can make more accurate estimates of the long-term effects of
alcohol and drug abuse treatments. If this is not done, the follow
up periods must be substantially lengthened from one or two years to
perhaps a decade.
3.8
Sampling Design
Sampling refers to decisions
concerning the subjects selected for research. Issues of sampling
concern: (1) eligibility for treatment, (2) selection for
participation in research, and (3) availability for follow-up
research (Saxe 1983, p. 37). If the sample of individuals selected
to be studied in the evaluation is not representative of those about
whom information is desired, the results of the evaluation will lack
generalizability; the results will only apply to a subset of those
participating in the program. For example, if the results are based
on Medicaid claims data, we cannot be sure they apply to the
non-poor population.
In sampling, we are particularly
concerned with the potential problem of self-selection bias. It is
quite possible that individuals who receive alcohol and drug abuse
treatments are not representative of problem drinkers and drug
abusers in the general population. Programs may have an incentive
to exclude patients that are not likely to respond to treatments if
their funding is based on outcomes (Saxe 1983). Currently, however,
private and public facilities in Louisiana have no incentive to
exclude patients that are not likely to respond to treatments. If
programs start focusing more on outcomes, this might create the
aforementioned incentive.
Program participants may be more
disparate or more motivated than non-participants. Often in alcohol
(and drug abuse) evaluations the control group simply consists of
alcoholics (drug abusers) that did not receive treatments. We
cannot be sure how much of the decline in alcohol and drug use in
the pre and post-period is due to treatment as opposed to the
motivation factor. Ideally, we need a control group of equally
motivated non-participants in order to isolate the “pure” treatment
effect on alcohol and drug use. As Cartwright (1998) explains,
there is a source of ineffectiveness that should result in some
outcomes not being counted as a success. Some drug abusers would
have recovered spontaneously from their addiction without treatment,
and therefore their benefit should not be counted. Unfortunately,
the absence of data on alcohol (and drug) abusers who do not seek
treatment limits the ability to generalize results or to establish
realistic spontaneous remission rates (Saxe 1983).
Sampling biases are likely to be
present in follow-up studies used to measure program outcomes.
There are likely to be systematic differences between program
dropouts and those participants who remain in the program and who
are available for follow-up interviews. There is some evidence that
those alcoholics who are difficult to follow up have the poorest
treatment outcomes, although contrary evidence is also available (Saxe
1983). Ideally, all of the participants should be included in the
follow-up analysis and their outcomes recorded, but in practice this
is never done. Consequently, the estimated cost-effectiveness of
alcohol and drug abuse programs is upwardly biased.
3.9
Incremental Analysis of Costs and Consequences of
Alternatives Performed
As noted above, economic
evaluations are comparative in nature. Such evaluations examine the
additional costs that one service or program imposes over another,
compared with the additional effects, benefits, or utilities it
delivers (Drummond et al. 1997, p. 40). Such comparisons are
relatively common in the evaluation of health care interventions but
quite rare in the evaluation of alcohol and drug abuse programs. In
the latter evaluations each program typically is compared with the
“do nothing” alternative, which assumes that non-program
participants incur no costs and receive no treatments. This is
generally not true. Alcohol and drug abusers generally consume more
medical resources than non-abusers and they frequently receive
counseling and advice from medical practitioners.
Economic evaluations of alcohol
and drug abuse programs should consider the marginal change in costs
compared to the marginal change in consequences when the program is
compared to the status quo, which is usually not doing nothing. If
we are interested in the relative cost-effectiveness of two programs
or the cost-effectiveness of adding additional treatments to an
existing program, we need to consider the change in costs relative
to the change in outcomes between the two programs or between the
newly expanded program and the original program. We do not want to
compare each of these four programs with the “do nothing”
alternative. We are interested in the efficient allocation of
scarce resources and the change in outcomes relative to the change
in costs when resources are shifted from one use to another.
3.10 Potential Bias in the Presentation and Interpretation of
Results and Recommendations
Every evaluation will contain
some degree of uncertainty, imprecision, or methodological
controversy (Drummond et al. 1997, p. 109). Some data thought to be
important may not be available and some of the estimates may be
known to be imprecise. As noted earlier, the choice of the
appropriate discount rate is still controversial. As a consequence,
the researcher may use sensitivity analysis to explore the
generalizability of study results to other settings. Sensitivity
analysis generally involves the following three steps: (1) identify
the uncertain parameters for which sensitivity analysis is required;
(2) specify the plausible range over which uncertain factors are
thought to vary; and (3) calculate study results based on
combinations of the best guess, most conservative, and least
conservative estimates. If the results do not appear to be
sensitive to these alterations, then one can have more confidence in
the generalization of the original results.
In presenting and interpreting
the results, the research should always recognize the limitations of
the study. In drawing conclusions from evaluation findings, the
source, nature and extent of potential bias must be considered.
Some of the biases may have been eliminated or reduced through the
use of statistical techniques. No evaluation is perfect. In
drawing inferences and making policy recommendations, the analyst
should be cautious in making judgments about the reliability,
validity, and generalizability of the findings to other programs and
settings.
3.11 Conclusions on Methodological Issues
Conducting economic evaluations
on alcohol and drug abuse treatment programs is difficult.
Evaluation results are highly sensitive to methodological decisions
concerning the conceptual framework; the analyst’s point of view;
the treatment, research, and sampling designs; and the
identification, measurement, and valuation of costs and outcomes.
CEA and CBA studies of alcohol and drug abuse treatment programs
have been fairly rare and often methodologically flawed so it is
difficult to draw any firm conclusions about the economic merits of
such programs (Saxe 1983, Apsler 1991; Apsler and Harding 1991;
French 1995; and Cartwright 1998). In recent years, the quality of
such evaluations has improved. By carefully considering the
methodological soundness and results of individual studies,
conclusions can be drawn concerning the relative cost-effectiveness
of alcohol and drug abuse treatment programs.
|
