Levels of evidence in endoscopic research: a longitudinal survey

Abstract

In clinical studies in surgical journals, the proportion of randomized trials is below 10%, and this has remained unchanged over the years. The problems of performing randomized studies in surgery are discussed, and we classify the publications in the only Medline journal aiming at gynecological endoscopy, The Journal of the American Association of Gynecological Endoscopy (JAAGL), according to the publication type and level of evidence over a recent time period, comparing the results with other surgical and general medical journals. The 1994–2002 volumes of the JAAGL are evaluated, as well as a limited number of issues of other leading journals on surgery, obstetrics-gynecology and general medicine. In the JAAGL issues from 1994 to 2002, most articles were based on therapeutic trials (549/862=64%). No significant changes occurred in the distribution of the different categories from 1994–2002. The number of randomized trials in this period is 48 (range: one to nine per year). Of 549 therapeutic clinical studies, the proportion of randomized trials is 8.8% (48/549). Of the 549 therapeutic studies, 432 case series without control groups (79%) were reported. In the endoscopic journal of the general surgeons and the general surgical journal, a similar distribution of publication types and levels of evidence can be seen. In the obstetrical-gynecological journal, 12 out of 22 therapeutic studies (44%) are based on randomized trials. In the journals of general medicine, randomized studies account for 74% of all the published studies. Case series without controls predominate in surgical journals, and the proportion of randomized trials has remained small over the years. Particularly in the fields of endoscopy and minimally invasive surgery with many emerging new techniques, the importance of randomized studies is stressed.

Introduction

By tradition, authority plays an important role in the surgical field. In 1996, Horton published a study that evaluated the number of clinical studies in nine general surgery journals, classified according to study design [1]. Only 12/175 papers (7%) reported original research data derived from randomized trials. This prompted him to recommend collaboration among surgeons and epidemiologists to plan randomized trials. As early as 1980, the term “evidence-based medicine” was introduced [2]. Nowadays, evidence-based medicine has grown to be the modern paradigm by which to practice clinical medicine, and it is defined as the conscientious, explicit and judicious use of the best available evidence to treat individual patients. Also, endoscopic journals draw attention to evidence-based medicine as the appropriate way to develop surgical management [3, 4, 5, 6, 7]. Especially in this era in which many new surgical techniques—minimally invasive or otherwise—are being developed, the profession needs tools to select an evidence-based treatment for their patients. The Journal of the American Association of Gynecologic Laparoscopists (JAAGL) is the only journal that addresses mainly gynecological endoscopy and is in the Medline database. Consequently, publications in this journal will most likely have a high exposure to the practicing gynecological endoscopist.

The objective of this study was to classify the articles in the Journal of the American Association of Gynecological Laparoscopists from 1994 until 2002 according to the level of scientific evidence and to compare the results with other surgical and general medical journals.

Materials and methods

The 1994–2002 volumes of JAAGL were evaluated. As comparative journals, we chose—in a way arbitrarily—journals such as those leading in the fields of endoscopy (Surgical Endoscopy), general surgery (Surgery), obstetrics and gynecology (American Journal of Obstetrics and Gynecology) and general medicine (British Medical Journal and the New England Journal of Medicine). While preparing the manuscript, not all articles in Surgical Endoscopy and Surgery published in 2002 had been processed in Medline yet, and we therefore evaluated the articles published in 2001.

To enable a global impression of distribution of levels of evidence in the other journals, a limited number of issues were studied. Only the articles concerning therapeutic clinical studies were evaluated in the American Journal of Obstetrics and Gynecology (August–December 2001), the British Medical Journal (November–December 2001) and the New England Journal of Medicine (November–December 2001).

With Pubmed (http://www.ncbi.nlm.nih.gov/entrez/query.fcgi), all titles and available abstracts were evaluated according to the publication type: clinical trial (e.g., diagnostic or therapeutic), survey, essay (guideline, review, commentary, correspondence and editorial), fundamental study (pathology, chemistry) and animal study. Abstracts of congress presentations were excluded.

All therapeutic trials were classified according to the level of evidence [8]: systematic review of randomized clinical trials, randomized clinical trial (RCT), cohort study with control group, case-control study, case series and case report.

Results

In the JAAGL issues from 1994 to 2002, most articles were based on therapeutic trials. (549/862=64%, Table 1) The second most frequent publication type is the essay that is not based on a clinical trial (24%). This category consists of guidelines, non-systematic reviews, editorials, book reports and letters to the editor. No significant changes occurred in the distribution of the different categories over time. The number of randomized trials in this period is 48 (range: one to nine per year). This is 6.1% of all articles (n=790). Of 549 therapeutic clinical studies, the proportion of randomized trials is 8.8% (48/549) (Table 2). In another 65 articles (12%), two or more treatment modalities are compared in a non-randomized way. Of the 549 therapeutic studies, 432 case series without a control group (79%) were reported. In 109/432 (25%) of these case series, the study was based on fewer than 25 patients.

Table 1 Publications in the Journal of the AAGL (1994–2002) according to type. *Rounded figures

Table 2 Number of articles according to design in different medical journals. Percentages in rounded figures. ^§1994–2002, ^¢2001, ^¥August–December 2001, *November–December 2001

In the endoscopic journal of the general surgeons (Surgical Endoscopy), a similar distribution of publication types can be seen. Although in Surgery the proportion of randomized trials equals that of the JAAGL, the proportion of non-randomized trials comparing two or more methods is considerably higher, and this also applies to animal studies. In the journal Surgery, mainly non-endoscopic studies are reported. In the four studies regarding endoscopy, two are based on a randomized study. In the American Journal of Obstetrics and Gynecology, 12 out of 22 therapeutic studies (44%) are based on randomized trials. If confined to surgical subjects, this proportion decreases to 4/17 (22%). In the British Medical Journal, ten therapeutic studies consisted of five randomized trials, four controlled trials, one systematic review and no case series. In the New England Journal of Medicine (17 therapeutic studies), these numbers were 14, 1, 0 and 3, respectively. Consequently, in both journals, 20/27 studies (74%) were based on randomization.

Discussion

In evaluating medical interventions, the randomized clinical trial (RCT), if conducted correctly, is the standard as it reduces the risk of bias. In the general medical journals with a high impact factor, more than half of the therapeutic clinical trials are randomized. However, in these journals surgical studies are scarcely published. In a general journal of obstetrics and gynecology (American Journal of Obstetrics and Gynecology), still a substantial proportion (22%) of the surgical studies are based on randomization. In two endoscopic journals in gynecology and in general surgery, this proportion is about 6–10% and does not change considerably over time. This shows that the best (randomized) studies are likely to be sent to the higher impact journals.

According to a literature review [9], the proportion of randomized trials between 1980 and 1990 in three major international surgical journals was the same (9%). In less than a quarter of these surgical randomized studies, a surgical technique was addressed, while in most articles, medical therapies were studied in surgical patients. If surgical randomized trials were assessed in terms of quality [9], basic issues such as sample size calculation were ignored in more than 90% of the trials reviewed. In an analysis of 40 randomized clinical trials in laparoscopic surgery [10], six trials were well conducted, while 22 (55%) had a poor score.

The difficulties of randomized trials in surgery are well established. Contrary to medical treatment, surgery is considered irreversible, which generates stronger treatment preferences by the patients and their surgeons. This makes the recruitment of patients difficult. Moreover, having to confess uncertainty by wishing to undertake a trial may undermine the patient’s confidence in the surgeon. Methodological problems are common in surgical trials: differences in surgical skills regarding the innovative procedure compared to the conventional procedure may cause a bias of the results. Blinding of the surgeon is obviously impossible, and in most cases neither the patient nor the investigator can be blinded for the allocated treatment. As surgical studies are often multicentric in order to recruit a sufficient number of patients, problems arise about agreement on the protocol and the surgical technique. Most surgical trials require long-term follow-up to evaluate the results, demanding a well-structured trial organization. The difficulty in raising financial support is considered to be a major impediment to initiating surgical trials.

It is strongly advocated by some [11] to consider the observational study as an alternative to the randomized clinical trial, although incorrect conclusions inferred from historical control groups of up to 40–60% of the cases are reported [12]. Because well-designed, randomized trials have been performed to assess new surgical treatments in the past, the randomized clinical trial appears to be a realistic requirement for surgical evidence and, therefore, remains the standard [13].

It has been argued that surgical procedures and devices should be evaluated in the same way as medical therapy [14]. Recently, the U.S. Food and Drug Administration (FDA) released a “submission guidance” document to guide industry in the process of obtaining marketing approval for thermal endometrial ablation devices [15]. According to the phasing of medical trials, a similar structure was proposed: studies concerning feasibility and safety (phase I, 5–10 patients), feasibility and effectiveness (phase II, at least 20 patients) and safety and effectiveness (phase III). The phase-III study is designed as a multicenter, randomized clinical trial and compares new with conventional techniques (e.g., rollerball ablation). It is the final step in evaluating a new technique or instrument before marketing. It may well be that some regulations—by the government or by professional boards—for the introduction of new surgical techniques, devices and graft materials create an incentive for involved companies to invest their money in randomized trials rather than in “planting trials” with uncontrolled case series.

The British epidemiologist Archie Cochrane wanted to “prevent the introduction of new therapeutic procedures until randomized trials had shown them to be more effective than existing treatments” [16]. Particularly in the field of endoscopy and minimally invasive surgery with many emerging new techniques, we should make more effort to do so.