To the editor of JMPT .. 4/02 Dr. Cates and colleagues1 are to be commended for encouraging critical assessment of clinical practice guidelines. Unfortunately, limitations in their methodology, and the composition of their expert panel, raise serious questions regarding their recommendations. The authors relied on the Cluzeau2 instrument, and ten volunteer appraisers. While the Cluzeau instrument addresses several characteristics of clinical practice guidelines, it fails to adequately address important issues that significantly affect the validity and utility of a clinical practice guideline. These include, but are not limited to, the following: Internal consistency. Guideline developers are free to establish their own criteria for ratings. However, once the criteria are selected, they must be consistently applied. In the Guidelines for Chiropractic Quality Assurance and Practice Parameters 3 (also known as the Mercy Guidelines), under the heading "Procedure Ratings," it is stated, "There must be one or more controlled trials (Class I evidence) for a type A rating of established." This rule applied to Rating System 1. This was not followed in Mercy Chapters 2, 3, 4, 7, 8, and 13, all of which purported to use Rating System 1. Specifically, in Chapter 2, recommendations 2.1.1, 2.3.1, and 2.4.1 received "established" ratings in the absence of Class I evidence. In Chapter 3, recommendations 3.2.1, 3.3.3, 3.4.3, 3.9.1, and 3.9.2 received "established" ratings in the absence of Class I evidence. No Class I evidence is cited to support recommendations 4.1.1, 4.1.2, 4.1.3, 4.1.4, 4.1.5, 4.1.9, 4.1.12, 4.1.13, 4.1.14, 4.2.7, 4.2.8, 4.2.9, and 4.2.25. In Chapter 7, recommendations 7.1.2, 7.3.3, 7.5.4, and 7.5.5 received "established" ratings in the absence of Class I evidence. Recommendation 8.2.1 confers an "established" rating without Class I evidence. Finally, Chapter 13 provides an "established" rating for recommendation 13.1.1 with only Class III evidence cited. In contrast, Council on Chiropractic Practice Clinical Practice Guideline No. 1, Vertebral Subluxation in Chiropractic Practice4 (also known as the CCP Guideline) consistently applies stated criteria. Open process. Interestingly, the Mercy3 guidelines include a section, "The Agency for Health Care Policy and Research and the Development of Clinical Practice Guidelines." It states that an important aspect of guideline development by the Institute of Medicine of the National Academy of Science is an open forum. In such a forum, ".every individual interested in providing oral or written testimony relevant to the guideline is invited to do so." This is in sharp contrast to the rules of quasi-secrecy enforced at the Mercy conference. Committee meetings were closed to official observers, and only commissioners, observers, and support staff was permitted to attend plenary sessions. Only commissioners were permitted to speak. The Council on Chiropractic Practice held an open forum where any interested person could address the panel and present evidence. The panel also accepted written submissions from individuals who were unable to attend the open forum. Advertisements were published soliciting peer-reviewers from the field. All persons who volunteered to serve as peer reviewers were given the opportunity to do so. Although Cates et al claim that the CCP Guidelines do not include representation from several major interest groups within the chiropractic profession; they fail to identify these groups. Broad spectrums of individuals were on the panel, including individuals from the professions of chiropractic, medicine, law, and basic science. A consumer representative was included. Any person who wished to present, or serve as a peer reviewer, was permitted to do so. The finished product reflects the work of over 200 professionals in 12 countries-a far broader cultural representation than Mercy. The recommendations of the CCP guideline were published in the peer-reviewed Journal of Vertebral Subluxation Research5. Curiously, we have no record of Cates or his co-authors participating in the open forum, submitting written evidence, or serving as a peer reviewer. Furthermore, it must be emphasized that the target audience for the CCP Guidelines is the subluxation-based community. The Cates group consisted of 3 persons practicing "chiropractic orthopedics," a practitioner of "chiropractic medicine," a practitioner of "chiropractic orthopedics and sports," a medical physician specializing in occupational medicine, a radiologist, a person who practices "chiropractic and quality assurance and utilization review," and one doctor identified as engaged in the "private practice of chiropractic." Where is the representation from the subluxation-based community: the individuals for whom the CCP guideline was produced? As Powers6 observed, "As the strength of the evidence declines, the composition of the panel and the process it follows become increasingly important determinants of the recommendations." Literature review. CCP employed a comprehensive information gathering process. The process included: 1.. A search of the MEDLINE, MANTIS, and CINAHL databases. 2.. A technique symposium, where representatives of a broad spectrum of techniques for the assessment, correction, and management of vertebral subluxation presented evidence to the panel, and responded to questions by the panel and the other participants. 3.. A leadership forum, where participants discussed issues related to practitioner acceptance of clinical practice guidelines. 4.. An open forum, previously described, where any interested person could present oral and/or written evidence to the panel. Sackett7 defines evidence-based clinical practice as; "The conscientious, explicit, and judicious use of the current best evidence in making decisions about the care of individual patients.(it) is not restricted to randomized trials and meta- analyses. It involves tracking down the best external evidence with which to answer our clinical questions." This concept was embraced by the Association of Chiropractic Colleges in Position Paper No. 18, which states, inter alia, "A subluxation is evaluated, diagnosed, and managed through the use of chiropractic procedures based on the best available rational and empirical evidence." The process followed by CCP to ferret out the best available evidence is in sharp contrast to the biased and incomplete reviews of literature included in the Mercy guidelines. The CCP guideline had the intellectual honesty to include evidence that disagreed with some of the final recommendations. A preponderance of evidence standard was employed when making final recommendations. Practitioner acceptance. The rubber meets the road with practitioner acceptance. A guideline not accepted by those affected by it has little value. Kent et al9 conducted a survey addressing field acceptance of the Mercy guidelines by 454 members of the International Chiropractors Association. While it is acknowledged that this may not be a representative sampling of American chiropractors, the results are striking. When asked, "How would you describe the effects of the Mercy guidelines on your practice?" only 1% responded that they had a favorable effect. 26% replied that they and no effect, and 70% reported that the Mercy guidelines had an adverse effect on their practices. The remaining 3% were non-responsive to the question. In the same study, 63% of respondents experienced instances where the Mercy guidelines were used to cut insurance claims or otherwise deny reimbursement for chiropractic services. According to the World Chiropractic Alliance10, the Mercy guidelines have been rejected by many national, state, and local chiropractic organizations. The National Guideline Clearinghouse lists only the FCLB (Federation of Chiropractic Licensing Boards) as an endorser of the Mercy guidelines. The CCP guideline has been incorporated as part of the International Chiropractors Association's Recommended Clinical Protocols and Guidelines for the Practice of Chiropractic11. Additionally, the World Chiropractic Alliance has endorsed the CCP guideline. In the State of Washington, the CCP guideline has been accepted as defining an acceptable standard of care. Acceptance by subluxation-based chiropractic practitioners throughout the United States and Canada has been excellent. Until the publication of the Cates et al paper, no national association, peer-reviewed publication, or other authoritative body has criticized the CCP guidelines. Timeliness. Although the authors acknowledge that a major problem with Mercy lies in the age of the document. It is based upon literature published prior to 1993, and has not been updated. The authors note that according to the National Guidelines Clearinghouse, the Mercy guidelines were "reaffirmed" in 1999. According to its web site, the National Guideline Clearinghouse (NGC)12 is "a public resource for evidence-based clinical practice guidelines. NGC is sponsored by the Agency for Healthcare Research and Quality (formerly the Agency for Health Care Policy and Research) in partnership with the American Medical Association and the American Association of Health Plans." To be included in the NGC, the guideline developer must state, "Documented evidence can be produced or verified that the guideline was developed, reviewed, or revised within the last five years." The first chiropractic guideline to qualify for inclusion in the NGC was Council on Chiropractic Practice Clinical Practice Guideline No. 1, Vertebral Subluxation in Chiropractic Practice. Members of the CCP panel were surprised and concerned when the Mercy guidelines were added to the NGC. There were two concerns: 1) The Mercy guidelines are consensus guidelines, not evidence- based guidelines. 2) The Mercy guidelines were over 5 years old when they were admitted to the NGC. In an effort to determine the process used by the Mercy developers to transmute the dross of consensus into the gold of evidence, and circumvent the timeliness rule, letters of inquiry concerning the "reaffirmation" process were sent to Dr. Leonard Suiter, of the Council on Chiropractic Guidelines and Practice Parameters, and Dr. Scott Haldeman, who chaired the Mercy conference. To date, neither party has responded to our inquiry. If a process was used to "reaffirm" Mercy, the process was seriously flawed, and conducted in secret. As an example, consider the Mercy ratings for physical therapy modalities. Thermal modalities, ultraviolet, and ultrasound received the "established" rating, and electrical modalities were rated "promising to established." These ratings are not supported by contemporary research. In 1997, Feine and Lund13 of McGill University investigated therapies including exercise, ultrasound, thermal agents, acupuncture, low-intensity laser therapy, electrical stimulation, and combination therapies for a variety of musculoskeletal conditions including chronic back pain. These authors concluded, "our results suggest that none of the therapies under review cause improvements in symptoms of chronic musculoskeletal pain or in quality of life that outlast the therapy.including placebo." Van den Hoogen et al14 reported that receiving physical therapy was associated with a longer duration of low back pain. The authors wrote, "at every moment in time, patients receiving physical therapy had a 61% less chance to recover in the following week than patients not receiving physical therapy." A more comprehensive review is beyond the scope of this letter. Interested parties are referred to other studies15,16. Given the widespread utilization of physical therapy modalities by chiropractors, relying upon the clearly obsolete Mercy recommendations could result in adverse clinical outcomes, and astronomical costs. Yet this is but one issue where new information would likely alter the recommendations. Recent publications addressing orthopedic testing, patient safety, instrumentation, and biomechanical x-ray analysis would likely change the ratings given by the Mercy panel. Interested readers may consult Council on Chiropractic Practice Clinical Practice Guideline No. 1, Vertebral Subluxation in Chiropractic Practice4 and Recommended Clinical Protocols and Guidelines for the Practice of Chiropractic11 for reference material. Given the silence of those involved in the "reaffirmation" process of the Mercy guidelines, one cannot exclude the possibility that no review process, including an appropriate literature review, was conducted. The "reaffirmation" was inadequate at best, fraudulent at worst. Ratings. Cates et al feel that the CCP guideline lacks an acceptable rating scheme. At issue is the applicability of specific evidence to a given clinical encounter. Some would argue that the "gold standard" for clinical evidence is the randomized clinical trial, with the lowly case report near the bottom of the hierarchy. The CCP panel questioned the validity of such a hierarchy. Our decision was to classify the nature of the evidence, and permit the attending chiropractor to determine how applicable it was to an individual patient. According to Coulter17, the randomized clinical trial (RCT) was first proposed by the British statistician Austin Bradford Hill in the 1930s. Since then, the RCT has received a plethora of praise and a paucity of criticism. Coulter notes that according to The Office of Technology Assessment, "Objections are rarely if ever raised to the principles of controlled experimentation on which RCTs are based." Despite such widespread enthusiasm, A.B. Hill recognized that clinical research must answer the following question: "Can we identify the individual patient for whom one or the other of the treatments is the right answer? Clearly, this is what we want to do.There are very few signs that they (investigators) are doing so18." Inability of the RCT to deal with patient heterogeneity, and the confounding factors which characterize individual clinical encounters, severely limits its utility in guideline development. As Coulter noted, "The clinical trial is an experiment performed on an unreal, unknown, mysterious entity-an assembly of sick people who have some features in common.the results of the trial cannot even be extrapolated to the individual patient, who (not some faceless member of a 'homogeneous group') is still the object of medical ministration17." The CCP panel determined that clinical decision-making should rely on objective indicators of a favorable outcome. In the case of analysis of vertebral subluxation, this entails evaluation of the biomechanical and neurophysiological manifestations of vertebral subluxation, and health-related quality-of-life. Use of objective anatomical and functional outcome assessments, performed throughout the course of professional care, enable the clinician to develop an evolving, appropriate clinical strategy. Over thirty years ago, Platt19 wrote, "there are clear clinical or laboratory tests which will rapidly tell whether the treatment is effective.a controlled clinical trial is quite often the wrong way to assess the value of a.treatment." Implicit in chiropractic practice is an emphasis on the unique constitution of the patient. Recognition of the individual needs, preferences, skills, and objectives of the patient and doctor will enable clinicians to develop meaningful and effective clinical strategies. To operationalize these concepts, the CCP guideline emphasizes the use of reliable and valid indicators of vertebral subluxation to implement a course of chiropractic care. Practitioner discretion. The CCP guideline has been criticized for permitting practitioner discretion in determining frequency and duration of care. The Mercy3 guidelines state that an adequate trial of treatment/care is as follows: "A course of two weeks each of two different types of manual procedures (four weeks total) after which, in the absence of documented improvement, manual procedures are no longer indicated." A nearly identical recommendation may be found in a 1991 RAND Corporation publication20. This is an example of an opinion masquerading as an evidence- based recommendation. Shekelle20 acknowledged that "There exists almost no data to support or refute these values for treatment frequency and duration, and they should be regarded as reflecting the personal opinions of these nine particular panelists." Unfortunately, such candor was not evident in the Mercy guidelines. The CCP panel carefully considered input from individuals who wished to include specific time and frequency recommendations. Since our objective was to produce evidence-based recommendations, we could not rely on speculation or wishful thinking. No one was able to produce evidence supporting the validity of any method to prospectively determine the frequency and duration of care necessary to correct or stabilize vertebral subluxations. The panel did, however, find reliable and valid technologies to measure anatomical and functional manifestations of vertebral subluxation. It is this evidence that formed the basis for our recommendation that frequency and duration of care should be determined on an individual basis, using appropriate assessments. Existence of vertebral subluxations. My initial reaction to Cates et al questioning "the significance and existence of the vertebral subluxation" was that they were attempting to introduce humor into their paper. After re-reading the section several times, I suspect this is not the case. As McCoy21 observed, this is "akin to saying that dental caries do not exist, and left unanswered poses a threat to public health, since a person with a subluxation may not get needed care." I feel Dr. McCoy was too kind. Any chiropractor that seriously questions the existence of vertebral subluxation in the 21st century needs remedial education. A good place to start is the Journal of Vertebral Subluxation Research, a peer-reviewed journal indexed by MANTIS and CINAHL. Such "scientists" who deny the existence of vertebral subluxation may wish to consider membership in the Flat Earth Society, whose members refuse to consider evidence that is at variance with their dogma. It may be instructive to note that the WCA, ACA, ICA, FSCO acknowledge vertebral subluxation. Association of Chiropractic Colleges (ACC) Position Paper No. 18 states, inter alia, that the chiropractic profession "focuses particular attention on the subluxation." This Position Paper was signed by the president of every chiropractic college in North America, and adopted by every national and international chiropractic organization of consequence. There may be debate concerning an operational definition of vertebral subluxation, a matter complicated by a diverse array of alternative terms. Rome22 listed 296 variations and synonyms used by medical, chiropractic, and other professions. Rome concluded, "It is suggested that, with so many attempts to establish a term for such a clinical and biological finding, an entity of some significance must exist." Yet debate concerning an operational definition is a far cry from questioning the very existence of vertebral subluxation. As Gentempo23 wrote, "Only a fool with a dishonest political agenda could ignore this." Conclusion. Clinical Practice Guideline development is a continuing process. Although the Mercy group has apparently disbanded, the Council on Chiropractic Practice is still actively involved in reviewing literature, which will form the basis for guideline revision. While Cates et al have raised some interesting issues, their analysis is flawed, and their conclusion is misleading. Clearly, a meaningful assessment of clinical practice guidelines requires a more comprehensive approach. As Sackett7 wrote, "External clinical evidence can inform, but can never replace, individual clinical expertise, and it is this expertise that decides whether the external evidence applies to the individual patient at all and, if so, how it should be integrated into a clinical decision. Similarly, any external guideline must be integrated with individual clinical expertise in deciding whether and how it matches the patient's clinical state, predicament, and preferences, and thereby whether it should be applied." Sincerely, Christopher Kent, D.C. President Council on Chiropractic Practice Mailing Address: 195 N. Franklin Turnpike Ramsey, NJ 07446 e-mail cbkent@ix.netcom.com References 1.. Cates JR, Young DN, Guerriero DJ, Jahn WT, Armine JP, Korbett AB, et al: Evaluating the quality of clinical practice guidelines. J Manipulative Physiol Ther 2001;24(3):170-176. 1.. Cluzeau F, Littlejohns P, Grimshaw J, Feder G, Morgan S: Development and application of a generic methodology to assess the quality of clinical guidelines. Int J Qual Health Care 1999;11:21-8. 1.. Haldeman S, Chapman-Smith D, Petersen DM. Guidelines for chiropractic quality assurance and practice parameters: proceedings of the Mercy Center Consensus Conference. Gaithersburg (MD): Aspen Publishers; 1993. 1.. Council on Chiropractic Practice. Vertebral subluxation in chiropractic practice. (Clinical practice guideline no. 1.) Chandler (AZ): The Council; 1998. 1.. Council on Chiropractic Practice. Clinical practice guideline (Number 1) Vertebral subluxation in chiropractic practice: abbreviated version. Journal of Vertebral Subluxation Research 1998;2(3):1-18. 1.. Power EJ: From the Congressional Office of Technology Assessment. JAMA 1995;274(3):205. 1.. Sackett DL: Evidence-based medicine (editorial). Spine 1998;23(10):1085-6. 1.. Association of Chiropractic Colleges. Position paper no. 1. July, 1996. http://www.chirocolleges.org/paradimt.html 1.. Kent C, Rondberg T, Dobson M: A survey response regarding the appropriateness of professional practice guidelines to subluxation- based chiropractic. Journal of Vertebral Subluxation Research 1996;1(2):1-6. 1.. Position paper. The Guidelines for Chiropractic Quality Assurance and Practice Parameters (Mercy Guidelines). World Chiropractic Alliance. Chandler (AZ). http://www.worldchiropracticalliance.org/positions/mercy.htm. 1.. Recommended clinical protocols and guidelines for the practice of chiropractic. International Chiropractors Association. Arlington (VA): ICA; 2000. 1.. National Guideline Clearinghouse. http://www.ngc.gov. 1.. Feine JS, Lund JP: An assessment of the efficacy of physical therapy and physical modalities for the control of chronic musculoskeletal pain. Pain 1997;71:5. 1.. van den Hoogen HJM, Koes BW, Deville W, van Eijk JT, Bouter LM: The prognosis of low back pain in general practice. Spine 1997;22(13):1515-21. 1.. Gam AN, Johannsen F: Ultrasound therapy in musculoskeletal disorders: a meta-analysis. Pain 1995;63(1):85-91. 1.. Beckerman H, Boulter LM, van der Heijden GJ, de Bie RA, Koes BW: Efficacy of physiotherapy for musculoskeletal disorders: what can we learn from the research? Br J Gen Pract 1993;43(367):73-7. 1.. Coulter H: The controlled clinical trial. An analysis. Washington (DC): Center for Empirical Medicine. Project Cure; 1991. 1.. Hill AB: Reflections on controlled trial. Ann Rheum Dis 1966;25(2):107-13. 1.. Platt R: Doctor and patient. Lancet 1963;41:1156-8. 1.. Shekelle PG, Adams AH, Chassin MR, Hurwitz EL, Park RE, Phillips RB, Brook RH: The appropriateness of spinal manipulation for low back pain. Santa Monica (CA): RAND Corporation;1991. 1.. McCoy M: The man who loves to bust quacks. JVSR Newsletter. May, 2001. http://www.jvsr.com/researchupdate/20010502/index.htm. 1.. Rome P: Usages of chiropractic terminology in the literature: 296 ways to say "subluxation": complex issues for the vertebral subluxation. Chiropractic Technique 1996;8(2):49-60. 1.. Gentempo P: Reason vs. mysticism in chiropractic. American Chiropractor 2001;23(3):41.