Many investigators have proposed the use of systematic reviews to try to aggregate information from the medical literature.[1-3] Systematic reviews of clinical studies have been used widely and have also formed the basis of clinical inputs to many decision analysis models.[4,5] However, systematic reviews have not been applied to the economic literature due, in large part, to the lack of repeated assessments that can be collected in a systematic review, the incomparability of methods for analyzing economic data across clinical studies, and specific methodologic inconsistencies that may be found across studies (eg, different discount rates, different years of cost data, use of charges rather than costs, different types of costs analyzed, and different types of outcome measures).
The paper by Mather and colleagues is an attempt to aggregate data that would be the first step in a systematic review of economic data for a new clinical therapy. It is an ambitious undertaking on the part of the authors but is also limited by many of the methodologic challenges that may prevent widespread use of systematic reviews of economic studies. The authors comment on differences among the studies that they reviewed and many of the limitations that they found in the literature. This comprehensiveness is one of the most important elements of their paper. Interestingly, the majority of studies reviewed have the perspective of a non-United States government health authority.
Of the 15 studies evaluating the costs associated with chemotherapy for the treatment of inoperable, non-small-lung cancer, only 11 met the authors’ criteria for inclusion in this review. The authors point out that all of these analyses were developed from the perspective of the payor, not that of the patient.
Given the limited amount of primary clinical trial data available to address this issue, almost all of these studies were based on secondary review of the literature with data from a variety of sources. In practice, this meant that many of the different analyses used the same clinical trials to estimate the clinical effectiveness of therapy. Although this result may actually improve the comparability of findings across studies, it also suggests that the literature and clinical recommendations may be extremely sensitive to clinical trials with a small number of patients.
Furthermore, the authors note that, of the 11 studies in the review, only 2 assess the quality of life associated with the survival gains produced by treatment. One of these studies reports physician preferences rather than patient preferences for outcomes—a method generally disfavored in the economic literature.
How Could the Presentation of Data Have Been Improved?
What could the authors have done to make the presentation of the clinical and economic data in the articles reviewed easier to understand? First, readers would like to compare the survival rates across the economic studies reviewed. The authors’ Table 2 includes estimates of survival rates for 3 of the 11 protocols included in the review, while Table 1 contains some structural data for all of the studies. Thus, it would have been useful, either in the text or in the tables, to include estimates of survival rates for each of the different treatment arms for all 11 studies.
Second, the authors include a variety of costs from different perspectives and different years. They do not seem to have tried to translate these costs into a single currency or a single year. Just as clinical results from trials of different combinations of chemotherapy are unique and would not be combined, economic results conducted from the perspective of different countries may not be directly translatable. At a minimum, this incompatibility may arise from several factors: (1) Each country has its own health-care system with its own reimbursement structure. (2) Practice patterns may differ across countries, based, in part, on the structure of the health-care system (eg, German statutory health insurance is included as a perspective for one of the trials reviewed; in Germany, patients have a length of stay almost twice that of other European countries). (3) The relative prices for specific services may differ across countries.
Although the lack of an attempt to translate the economic results into a single currency is a flaw of the Mather paper, it is unclear how one would interpret data that are aggregated across such disparate perspectives as a Canadian provincial health plan, the Canadian government, German statutory health insurance, a private payor in the United States, a public health-care payor in Spain, the Italian government, and a hospital without extensive analysis by Mather and colleagues. The incompatibilities among these perspectives again point to the difficulty of conducting systematic reviews of economic analyses.
How Should the Results Be Interpreted?
How should we interpret the results of the Mather et al paper? The authors begin the paper by suggesting that evidence concerning the appropriate role of chemotherapy in this setting is controversial, with meta-analyses pointing to a modest benefit for chemotherapy. According to Mather et al, no definitive studies of sufficient sample size and quality have been reported. Thus, the article focuses on the economic impact of a therapy for which the clinical evidence supporting its use is inconclusive. Economic evaluation cannot be used to answer clinical questions that are unresolved by clinical investigators.
Economic analyses reviewed in this paper tend to focus on the economic evaluation of specific trials. However, until a definitive clinical result has been realized, the economic evaluation may be of limited use in determining clinical policy for the treatment of patients with non-small-cell lung cancer. Economic data based on these studies suggest that if there are clinical benefits of the magnitude reported in some of these small studies, this treatment may potentially be economically attractive.
The authors conclude their paper by suggesting that transparency (clarity in presentation) of economic evaluations may improve our ability to better understand differences in economic analyses before beginning a systematic review. However, given the complexity of economic analyses and the variety of perspectives involved, I am pessimistic about the possibility of standardizing the economic literature at its current state of development. Given that pessimism, critical reviews of the literature, such as that developed by Mather et al, are needed to help us better understand the economic literature and to assess differences between findings of economic studies reported in the literature.
A 52-year-old man presented with an erythematous lesion in the axilla of unknown duration. Surgical excision was performed. What is your diagnosis?
