Toward ecologically valid studies of innovative therapies

A key concern in the development and deployment of innovative therapeutic technologies is the availability of validation studies that support clinical practice. Specifically, the fields of biofeedback and neuromodulation have undergone significant evolution in the past 20 years, with emerging methods including advanced analysis of signals from the heart and brain, and increasingly elaborate feedback schema. These are often based on clinical experience, professional judgment, intuition, and repeated experimentation and variations in techniques.

Controlled studies have been put forth as a “gold standard” in the evaluation of emerging therapeutic methods. It has been put forth that a randomized, blind, sham controlled, crossover design is “the only way to go.” History, however, has shown a very different picture. The rigid RCT approach suffers from a number of critical problems that have, overall, served more to confound and obfuscate the benefits of these methods. Among the flaws in using RCT’s are a lack of ecological validity, problems with subject expectations and the experimental environment, unfamiliarity with the workings of practical therapeutic work, and at times, sheer incompetence on the part of the practitioners.

Controlled studies typically cost 100’s of thousands, even millions of dollars, and require significant planning and resources. They may lack ecological validity, and are often conducted by non-practitioners, and may be conducted in other than an active clinic. If methods are to innovate, the control group should be the existing practice standard, including medications. Studies that have taken this approach have been successful (Meisel, Duric). Hammer (2012), for example, demonstrated similar results for an emerging method, compared to a “traditional” neurofeedback treatment, for insomnia.

Another problem with RCT’s is the cost of absorbing all the treatment costs for all participants, for ethical and scientific reasons. This is then confounded with the fact that all participants are getting free treatment, which affects the level of investment and expectation, putting it outside normal clinical parameters. Studies that fit into routine clinical work can be much more flexible, in that the “control” group, which would be treatment as normal, could be conducted and paid for as usual, and compared with an experimental treatment. This would improve the ecological validity, in that clients receiving actual treatments have the same expectations and costs as encountered in actual clinical work.