Excerpt from: A COMPARISON OF OUTCOME VARIABLES IN PSYCHOTHERAPY

Chapter Two: Literature Review

Telehealth

The problems of distance and the delivery of mental health services have been studied for over 50 years (Baer, et al., 1997; Shannon, Bashshur, Lovett, 1986). telehealth, one of the fields that arose to address this problem, made its first significant stride five decades past. An early experiment with video links resulted in a live one-way and later two-way connection between a psychiatric unit in Nebraska and a state mental hospital. The connection allowed university psychiatrists to perform neurological assessments, provide consultation, and education (Wittson & Benschoter, 1972).

Since this time telehealth has made significant strides into virtually all areas of health services (Fields, 1996; Baer et al., 1997). However, only in the current decade have health services via distance technologies become both prevalent and financially feasible (Baer, et al., 1997; Dakins & Jones, 1996). Recently the field has experienced a rapid explosion. It was estimated that by the end of 1996 almost 30% of rural hospitals would be using some form of telehealth (Nickelson, 1996). Given the increased number of studies and links to information about various programs on the Internet, it would appear that this estimation was fairly accurate.

Part of this explosion has been due to recent legislation. From 1994 to 1995, the state and federal budget allocations exceeded 100 million for telehealth programs. There were 13 different federal government agencies overseeing this area. In the 104th congress there were 15 pieces of legislation introduced about telehealth. The recent 1996 Telecommunication Act was the culminating act of many of these efforts. However, legislative movements have not stopped there. Additional bills such as the Comprehensive telehealth Act of 1996 have been reintroduced (Nickelson, 1996). There have also been recent movements to authorize and require that services provided through telehealth technologies be reimbursed. California was recently the first to see a statewide mandate that services provided through telehealth technologies be reimbursed ("State law update," 1997). The impetus behind many of these movements lies in the three main problems to which telehealth hopes to provide solutions.

Bashshur (1997) notes that, despite our best efforts, the health care system continues to face three particularly tenacious problems:

1. uneven geographic distribution of health care resources throughout the country, including health care facilities and health manpower;
2. inadequate access to health care on the part of certain segments of the population, including the under-privileged, isolated, and confined;
3. unabating rise in the cost of care, including the costs borne by both public and private payers. (p. 5)

It for these reasons that the potentials of telehealth are appealing to all types of health care providers.

Telehealth: Who and How Will It Benefit Us?

Our nation has developed in a fashion that has facilitated the centralization of most health services. This development has left a large portion of the population without access to many of the health services that are readily available to others. Although many programs have been attempted (e.g., training programs, encouragement to relocate) all have failed to close this gap (Murray & Keller, 1991). telehealth is seen as the answer that will bridge the gap between the unevenly distributed providers and the underserved recipients (Bashshur, 1997).

One of the most commonly cited underserved populations is those living in rural areas. Though many urban dwellers often think of people living in rural areas as small in number, there are in fact, more than 60 million Americans, that is about ¼ of the population, living in rural areas (Office of Technology Assessment [OTA], 1990). People living in rural areas are not found solely in Alaska and Montana, but throughout the country. The extent of this problem can be illustrated through the examination of the coverage of a community mental health center.

Community mental health centers were developed after the deinstitutionalization of state mental health hospitals. Based on an urban model, they seek to serve populations of 70,000 to 250,000. For many urban areas, this results in a short trip to the local community mental health center should a problem arise. However, in rural areas, mental health centers serve areas ranging from 5,000 to 60,0000 square miles (Murray & Keller, 1991).

Yet another population that telehealth can help build a connection to are Native Americans. Like other members of rural America, Native Americans often live in areas geographically isolated from health services. For many Native Americans, health services are simply too far away to be utilized (Bhatara, Fuller, & William, 1995). telehealth can be their link to better health care.

Distance can also be a temporary situation. Military personnel typically have ample access to services when they reside on base. This situation changes when they move abroad or are otherwise in the field. A conventional solution would be to provide every company with health personnel and equipment. However, not every situation will require every specialized type of health professional. Following the conventional model would lead to the third problem cited by Bashshur (1997), exorbitant costs. telehealth helps to side step the issue of cost by enabling people to call upon health professionals virtually, as needed.

Another isolated population is prisoners. In 1994 there were over 1,500,000 men and women in various prisons in the United States. This number has continued to increase over the last few years, and shows no sign of abatement. Among the many health problems are the numerous cases of infectious diseases and numerous prisoners with mental health problems (Hipkens, 1997). Surveys indicate that anywhere from 7-15% of inmates have mental disorders, with about a third of these disorders being depression, severe forms of schizophrenia, or manic depression (Foderaro, 1994; National Alliance for the Mentally Ill and Public Citizens Health Research Group, 1992). telehealth services provide a way to efficiently, safely, and more cheaply bridge the gap between health professionals and prisoners (Hipkens, 1997).

For groups that are home bound, telehealth is an ideal solution to a complex problem. The elderly are one group that fits this category. The geriatric population is on the rise, and they typically have a higher need for all types of health services (Siu, Ke, & Becket, 1989). Although many elderly might live in an urban area, their infirmities often make it difficult to travel even short distances and many are home bound. Distance technologies can help to address this problem in a way no other solution can.

Elderly are not the only population who are isolated because of medical reasons. Several specific mental disorders, such as agoraphobia and obsessive-compulsive disorder, keep people isolated and bound to their home or a relatively small geographical area. As is the case with the elderly, telehealth provides a way to intervene in a situation that normally would probably go untreated. In fact, this technology might actually provide a better way to facilitate the treatment of these disorders than a typical face-to-face delivery mode. One of the more empirically validated types of treatments for phobias is systematic desensitization (Spiegler, 1983). telehealth technologies would enable professionals to intervene at a less intrusive level and systematically desensitize the client through not only mediating less intrusive communication technologies, but computer modeling and virtual reality technologies as well. In this case telehealth may not only provide a more effective way of reaching the client, but also of treating the client.

telehealth helps to redistribute services to a wider variety of people and reach populations that have formerly been underserved. Tied into these two objectives is a decrease in costs. telehealth increases costs in terms of equipment and connection fees, but decreases costs in other less obvious ways. Opportunity costs, such as travel time and lost work due to travel, are decreased for both consumers and providers. Costs of specialized services are reduced due to increased access and a lesser need to retain a specialist in an area where they might not have been fully utilized. telehealth, because of its increased ease of access and decreased opportunity costs, increases the likelihood that a consumer will seek treatment earlier. Earlier treatment often leads to greater prevention and reduced long term care costs (Bashshur, 1997). Some estimates indicate that assessment conducted via telehealth could save 36 billion in national health care spending (Nickelson, 1996). All of these points suggest that telehealth might provide a viable solution to the problems of distribution of resources, unequal access, and escalating costs.

Telehealth: Technologies in Mental Health

telehealth is broad in its application and possibilities. In this study I have not sought answers to all areas of telehealth, but instead have chosen to focus on a particular service, psychotherapy, and a particular form of treatment delivery, two-way audio and two-way video. For this reason I will focus my review of the use of telehealth technologies specifically to studies that relate to these technologies and psychotherapy.

Two-Way Audio

The telephone has been an important tool for the mental health care professional for many years. It is used for scheduling, emergency sessions, follow ups, alternate sessions, and intake assessment. Furthermore, it has become the mode of preference for crisis work and emergency sessions (Liebson, 1997; Haas et al., 1996). Despite all this, it has made relatively little headway into the realm of traditional psychotherapy. People seem to have a variety of opinions as to why it should not be used. Haas et al. (1996) suggests that where the therapy takes place can have a significant impact. If the client calls from home, the therapeutic relationship might be compromised. Nagy (1987) argues that clinicians may have trouble maintaining attention throughout a telephone session. Other arguments against its use tend to follow along these lines, but the main argument is that the telephone deprives you of essential non-verbal cues, thus lessening its effectiveness (Haas et al., 1996).

The other camp argues that, although the telephone eliminates many non-verbal cues, many emotional factors can still be heard over the telephone (Tausig & Freeman, 1988). Woodson Reynolds, a practitioner who provides traditional therapy over the phone, claims he is more able to pay attention to vocal signals than he would in person ("Practice Issues," 1997). Last, as the literature, though limited, shows, there is no research to support that telephone counseling does not work ("Practice Issues," 1997; Haas et al., 1996).

Telephone therapy has garnered very little empirical research. However, there have been a few case studies and subjective reports of studies that examined its use in traditional therapy situations. One of the first such examinations of the efficacy of telephone therapy was a short report on three cases in which clients were given psychotherapy over the phone (MacKinnon & Michels, 1970). The therapy was started because extenuating circumstances prevented the clients from meeting with the psychiatrists face-to-face. The first client was a depressed woman who had gone to Nevada to obtain a divorce. She had previously been in therapy for several years and then continued therapy for several months via the telephone. The authors report that the therapy was helpful. The second case was a depressed woman with anxiety and hypochondriacal trends. Due to a pregnancy she had to remain in bed. Therapy was continued with her over the phone for several months. Again, the authors report that this case had a positive outcome. The third case was a phobic housewife. She was treated for only a few sessions over the phone, which initially seemed helpful. However, it was later realized she was using the phone to avoid face-to-face encounters and treatment over the phone was terminated. As might be expected from case vignettes, the outcome assessment in this paper was subjective and inconclusive.

In another case study, Grumet (1979) reported on a long-term therapy relationship he had with one client. Unlike the previous study, Grumet went into much greater detail about his case. Initially he met with the client face-to-face. The client was a single 26-year-old woman who had been diagnosed with chronic schizophrenia and "deeply-entrenched character pathology with prominent features of hysteria and impulsivity" (p. 579). After receiving repeated phone calls and having unproductive office visits the two of them established a structure and fees for both phone and office visits. The office visits continued to dwindle, but the phone calls went unabated, occurring daily at times, and going on for many years. He reports that over this time the client was able to live independently, successfully hold a clerical job for eight years, renew a relationship with her mother, and learn how to drive (she had a driving phobia). Her diagnosis, over this time, was downgraded to personality disorder of mixed type. It was the opinion of the author that for certain clients, like the one in the case study, telephone therapy is not only effective, but also better than face-to-face.

In a slightly different realm, Tolmach (1985) talks about some of the advantages that telephone therapy provided over face-to-face in an intervention program. The program she worked with, City Lights , attempted to provide comprehensive schooling, therapeutic intervention and assistance to adolescents who have had multiple placements and have been written off as "unteachable, intractable, and untreatable" (p. 214). In her program they placed periodic phone calls to the teenagers in order to communicate further reinforcement and eventually establish a trusting therapeutic relationship. Though no quantitative data is offered, she says that the telephone interaction gave the teenagers a greater sense of control and helped to establish trust. She claims that establishing trust with these children in face-to-face meetings would have been more difficult and that the telephone was probably superior in this instance.

Another area in which telephone therapy has been used is family therapy. Conducting family therapy can be inherently more difficult because of the need to bring multiple people together at one time. This becomes particularly problematic when one or more parents live or work in a different location or the child resides in a far away treatment facility. Hines (1994) has used the telephone to treat families in several of these types of situations. From his perspective, the use of the telephone, while not ideal, has proved to be invaluable in several cases and has helped to lead to successful resolutions. He also offers many helpful suggestions for providing therapy via telephone therapy, but, like the others, his reports are subjective and reveal little about the actual process and outcome in telephone therapy.

One of the few examinations of telephone therapy to use a controlled study design was one that examined the use of the telephone to provide group therapy. Evans, Smith, Werkhoven, Fox, and Pritzl (1986) reported on a study of cognitive group therapy for elderly (55 and over) outpatients who had recently left an inpatient rehab unit. The therapy was conducted via a telephone conferencing system. All of the participants were former residents of the rehab inpatient unit. From this sample, participants were randomly selected. From the selected population they obtained 43 participants who completed the study.

In order to assess the pre and post state of these clients, each participant was given the Wakefield self-rating depression scale, the UCLA Loneliness Scale, and the Life Satisfaction Index. A goal attainment scale was used to determine if the clients met their desired goals. Unfortunately their report is void of any specific data, but they do summarize their results. The treatment group had only three people who reported minimal goal achievement and two who reported partial goal achievement. The other 16 members reported adequate goal achievement. They also reported that change in loneliness was positively correlated with goal attainment and that the treated group showed a significant decrease in reported loneliness both immediately following treatment and at a three month follow-up. They did not find a significant correlation between depression or life satisfaction and goal attainment. They state that overall goal attainment was high, cautiously suggesting that this group was equivalent to other successful groups that had used goal attainment as their measure of outcome.

In a parallel but not identical realm, there have been some empirical examinations of behavioral therapy delivered through the telephone. Swinson, Fergus, Cox, and Wickwire (1995) assigned a group of 42 individuals living in rural Ontario who had been diagnosed with panic disorder with agoraphobia to two groups, treatment through telephone and wait list control. The study exhibited a relatively high level of control, accounting for factors such as medication and illness severity (e.g., no comorbidity with depression or substance abuse).

Outcome assessment was conducted using a variety of measures assessed at pre, post, and a three-month follow-up period. The measures used were: Fear Questionnaire (FQ), State-Trait Anxiety Inventory¾ trait version (STAI-T), Anxiety Sensitivity Index (ASI), Beck Depression Inventory (BDI), and Symptom Checklist 90 (SCL-90). In addition, a treatment satisfaction questionnaire was administered to the client and the closest indicated relative at post and follow-up.

The treatment itself consisted of eight manualized behavioral therapy sessions delivered by a bachelor's level clinician. The results of the study indicate that there were significant treatment C time interactions on the Fear Questionnaire Agoraphobia subscale of avoidance and fear ratings as well as the Anxiety Sensitivity Index. Within group effects were significant on the BDI (p < 0.001), the STAI-T (p < 0.005) and the SCL-90 (p < 0.001). Their post-hoc analysis indicated that while both groups improved significantly on the BDI, only the treatment group improved significantly on the STAI-T. The treatment group also exhibited significant improvement on the fear target ratings (p < 0.001) and the avoidance target ratings (p < 0.001). The control group was not administered these ratings.

The wait list group was administered the treatment after their wait period and follow-up analysis indicated that there were no significant differences in gain on any of the measures between the two treatment groups. Furthermore the gain for both groups was maintained on all measures and further significant gains were made on the target fear ratings. Finally, the patients reported a high level of satisfaction with the treatment method.

The improvements found in this study are similar to those that were found in face-to-face treatments using behavior therapy. The findings suggest that, at least for behavior therapy, the use of the telephone is equivalent in effectiveness to face-to-face methods. Though there is some question as to the actual improvement that the treatment itself provided, this study provides strong support for the probable equal effectiveness of face-to-face and two-way audio psychotherapy.

As can be seen in this review, there has been very little research done in this area and while there are some exceptions, most of it consists of subjective reports (Haas et al., 1996). Although these types of reports are useful in establishing a direction to follow when examining an issue, they do not provide any conclusive evidence. However, this type of technology is not completely devoid of empirical examinations. One important area of psychotherapy, assessment, has received some empirical attention.

Aneshensel, Frerichs, Clark and Yokopenic (1982), in an effort to find different and more efficient methods of obtaining quality community data, examined interviews conducted face-to-face and through a telephone. The purpose of the interviews was to assess, on a community level, depression. They randomly solicited 832 homes for participation in a study. The respondents were then randomly assigned to one of two conditions, face-to-face (n = 238) and phone (n = 308). The interviews were conducted using the Center for Epidemiologic Studies-Depression Scale (CES-D). In assessing the results they found no significant differences between the two methods in nonresponse to symptom items, preference for specific response categories, reliability, mean level of depression, or proportion classified as depressed. In addition, there were no significant interactions of mode of interview with sociodemographic characteristics. The findings suggest that assessment using a set protocol can be performed with the same accuracy over a telephone as would occur face-to-face.

Baer, Brown-Beasley, Sorce, and Henriques (1992) also performed an examination of this same issue. In this study they compared administrations of the Yale-Brown Obsessive Compulsive Scale and the Clinical Global Improvement Scale over three different assessment delivery modes, telephone administration by a person, telephone administration by a computer, and paper and pencil self-administration. The participants were 18 patients who met the DSM-III-R criteria for obsessive-compulsive disorder. All of the clients were administered these scales using each of the three delivery modes. The three administrations for each client occurred on the same day and were each conducted one hour apart. Their findings demonstrate equivalence between the delivery methods. Correlations between the methods ranged from r = 0.97 to r = 1.00. ANOVA analysis revealed no differences between the methods. Although the results are not terribly surprising, it does lend further support to the ability to conduct assessments over the phone.

In a study that combined design elements of the previous two studies, Fenig, Levav, Kohn, and Yelin (1993) compared assessments using a psychiatric screening scale, the Demoralization Scale of Psychiatric Epidemiology Research Interview, between two groups which contained both face-to-face and telephone interviews. The two groups consisted of women born in Poland between 1919 and 1924, who had experienced the Holocaust (n = 49) and who had lived in prestate Israel (n = 47) during the same time period. The second group was chosen to match the sociodemographic characteristics of the first.

Within the two groups they were further broken down into people who would be administered the interview face-to-face and through telephone. Correlations for the combined groups between the two interviewing modes was high (r = .78). Item analysis revealed that in the Holocaust group there was a significant difference between interviewing modes on six of the 27 scales. This group reported a higher severity of problems and depression when interviewed over the telephone than in person. In the second group, there were no differences across modes. As the authors point out, the data partially lends itself towards the support of equivalency of the two modes. They surmise that the differences that were found could be due to regression to the mean or a facilitation of symptom expression by the telephone. They further emphasize that because of the high correlations between modes and the absence on any differences in the second group, the delivery modes are probably equally effective.

Although the examinations of assessment are not conclusive, they do lend support to the use of the telephone in one aspect of psychotherapy. Unfortunately, overall the studies lack any conclusive empirical support for or against the use of two-way audio. With the advent of pay per call therapy, 1-900 therapy (Haas et al., 1996), and cheap two-way audio connections over the Internet, can mental health providers afford not to examine empirically the effectiveness of this technology? In this study I sought to address empirically this question and determine if psychotherapy can be effectively delivered using a two-way audio system.

Two-Way Video

In 1955 the Nebraska Psychiatric Institute (NPI), under the direction of Wittson, installed a one way closed circuitry microwave television system. This system was expanded a few years later to become an interactive two-way video system. The connection enabled psychiatrists from NPI in Omaha to communicate with the staff at the state mental hospital in Norfolk, 112 miles away. The services provided consisted of speech therapy, neurological examinations, diagnosis of difficult psychiatric cases, case consultations, research seminars, education, and training (Benschoter, Wittson, & Ingham, 1965; Wittson & Benschoter, 1972).

In their assessment of the effectiveness of this technology, they report that some people, clients and mental health workers, were initially reluctant to use the technology, but that the people who worked with it were extremely positive about their experience. The problems that arose were hesitancy on the part of the staff to use the technology, technical problems, and high costs (Wittson & Benschoter, 1972). However, the report is entirely subjective and lacked any empirical evaluations, and unfortunately, this dearth of empirical examination appears to have perpetuated itself through the years (Sleek, 1997).

Wittson, Afflect, and Johnson (1961) also used the Nebraska video link to provide group psychotherapy. The link enabled a psychiatrist at the Institute to run a general therapy group with clients at the state mental hospital. In their evaluation of the effectiveness of this treatment, they indicated that overall it was effective when compared to face-to-face treatment. Furthermore, they believed that the differences in outcome had more to do with the therapist and the clients than the delivery method used. They also noted that, after an initial adjustment, the clients were comfortable with the equipment. The only real problem with the treatment arose when some of the clients in the group with anti-social tendencies decided to whisper so the therapist could not hear them. Again, the reports given were short on data and high on subjectivity.

Thomas Dwyer initiated the second major excursion into two-way video use in 1968 at Boston Massachusetts General Hospital. In this instance they established a two-way microwave video link between a hospital in Boston and its satellite station at Logan International airport. While the link only bridged a distance of three miles, these were miles that, due to heavy traffic, were difficult for many to traverse. The project was a success in many ways. In a little over two and half years over 150 consultations took place. The number of contacts that each patient had ranged from two to 20 (Dwyer, 1973). Due to the positive response by the staff and clients, they established a second link to Bedford Veterans Administration hospital, 20 miles away.

Dwyer states that many people, after using the technology, were positive about its potential for one-on-one and group therapy situations. He also claims that the technology appeared to make it easier for some patients to communicate with the psychiatrist. On the other end of the spectrum, some patients were initially anxious when using the technology, but after a short while, this anxiousness dissipated. In the end, although there was a positive response from both the clinicians and staff, the project was terminated due to lack of funding.

The last major use of two-way video prior to the late 80's was a project that connected Dartmouth's medical school and a rural health center in Claremont, New Hampshire, a town that was 20 miles away. This link was used primarily for assessment of clients at the rural mental health center and for education. An evaluation of the program indicated that the patients had little trouble interacting with the system and that even the paranoid patients accepted it. There were a total of 199 patients seen, with 73% of these being new patients and 27% of them follow-ups. This project, like the others, was terminated. In this case the project ended due to both a decrease in use, because a psychiatrist moved to the remote area, and a lack of funding (Solow, Weiss, Bergen, & Sanborn 1971).

The 70's and early 80's were hard on telehealth and of those programs that were began before 1986, only one survived (Perednia & Allen, 1995). However, the dramatic increase in technology and the advent of the modern computer helped to bring a fresh surge of growth in the number of telehealth programs. The resurgence of interest also brought additional studies that, while still lacking, were an improvement over previous evaluations. These studies provided better assessments of satisfaction with two-way video treatment and assessments of its substitutability for face-to-face assessments.

Dongier, Tempier, Lalinec-Michaud, and Meunier (1986) made one of the first controlled studies of satisfaction in this area. Using a two-way closed circuit television system (CCTV) the researchers connected two floors of a hospital. The system was then used to compare psychiatric interviews conducted via two-way video to those conducted face-to-face. Patients were selected from in-patient and outpatient departments. Three psychiatrists (consultants) were used to conduct the study.

In order to obtain a representative sample of the clients seen by these psychiatrists, their last 200 consecutive cases were examined. A sample of 50 patients was then obtained that matched the type and range of clients these psychiatrists normally saw. These patients were then seen through the two-way video system. A control group of 35 patients, matched for diagnosis, was seen face-to-face. Following each interview the psychiatrists, as per hospital protocol, consulted with the rest of the staff.

In order to assess the CCTV, measures were given after the interviews. Clients were asked to rate various aspects of the interview, as compared to previous experiences with psychiatric interviews. Each of the aspects were rated on a five point scale ranging from much below average to much better than average. The aspects rated were these:

• feeling at ease during the interview
• ability to express oneself
• feelings after termination of the interview
• did they perceive the psychiatrist as feeling at ease or not
• quality of the interpersonal relationship
• will the interview help in their treatment (Dongier et al., 1986, p. 33).

The people involved in the consultation (the psychiatrists and other team members) were asked to rate, using the same scale, the following aspects:

• the preliminary discussion between consultant and consultee
• the patient / consultee or patient consultant relationship
• the post-interview discussion
• the written conclusions for diagnosis, management and treatment
• the global evaluation of the usefulness of the interview (Dongier et al., 1986, p. 33).

The majority of clients rated the CCTV system "above average" in comparison to the previous interviews. However, there were no significant mean differences (p > .10) between the control and experimental groups. The consultees, like the clients, rated the CCTV interviews better than past interviews. However, they rated the CCTV inferior to the control group for global assessment (p < .05), diagnosis (p < .01), and written consultation (p < .10). The consultants rated the CCTV inferior on all aspects (p < .05), a large statistical difference. However, practically this was quite low, with an overall global mean difference score below 0.5 on a five-point scale. Given the small practical difference, it is questionable how relevant this would be when taking into account the possible benefits of reaching a population who normally would go untreated.

One of the only other studies that attempted to establish control when evaluating this area was conducted by Ball, McLaren, Summerfield, Lipsedge, and Watson (1995). This study examined four modes of interaction: face to face, telephone, hands free telephone, and a desktop computer based videoconference system. The sample was quite small (n = 6) with most of the patients having severe pathology (three schizophrenics, one paranoid, one depressed person, and one mixed neurotic disorder). Ball et al. report that for the seriously ill, the video system produced the highest level of frustration. The authors state that the clients felt least understood when interacting in that way. In contrast, there were no differences in the psychiatrists' evaluations of the different modes. Unfortunately the design of this study and the assessment methods used are not well explained. The frustration of the clients could easily have been due to poor video or audio quality and the satisfaction of the clinicians due to familiarity and a bent towards the use of technology. Unfortunately, these types of problems are evident in other studies as well.

The University of Kansas Medical center is the site of another look at satisfaction. In this instance the center is connected to a rural outpost in Hays. In an effort to ease transportation problems yet increase access to health services they established a program in which a psychometrician obtains the client's medical information and sends it to the neuropsychologist who reviews it before the client's appointment. The client then meets virtually with the neuropsychologist for 30 to 60 minutes. The neuropsychologist then sends orders to the psychometrician to administer psychological tests and then send back the results. In a follow-up interview the neuropsychologist meets virtually with the client and family to report the results. Exit interview data were obtained from patients with Parkinson's disease. Overall these patients found the mode of service favorable. Furthermore, all of them viewed the mode as providing an acceptable means to improved health care access (Troster, Paolo, Glatt, Hubblem & Koller, 1995). However, again there is a lack of any control in this study and very little data is reported.

Oregon's RodeoNet telepsychiatry program provides additional evidence for satisfaction with two-way video technology. The program uses over 45 different interactive video sites to conduct consultations between psychiatrists, clients, and mental health providers. Initially satisfaction evaluations were given to the community mental health workers after every consult. They claim that 70% of the workers found the system satisfactory (Dakins & Jones, 1997). Unfortunately, no additional information is provided that would enable a better evaluation of these findings.

A similar program is APPAL-LINK. This program was created to provide additional psychiatric services to most of the rural sections of the Appalachian southwestern Virginia (Graham, 1996). Using compressed two-way video they conduct assessments and follow-ups giving each patient a complete DSM-IV diagnosis, a rating on the Psychiatric Symptom Assessment Scale, and a mental status examination. Following the completion of the assessment the local nurse administers a client satisfaction questionnaire. After six months of operation they logged 126.5 hours of service. Over six months 39 patients were followed. Their diagnosis had a wide range, but a majority had severe psychiatric disorders. Patient acceptance of the program was almost universally high, with 90% giving positive ratings on the satisfaction questionnaire. Furthermore, there have been no significant adverse effects seen with any of the patients. Nor is there much evidence of any phobic symptoms or anxiety in regard to being on camera. continued follow-up reports are scheduled, and it will be interesting to see if these positive findings are retained.

Comprehensive telehealth programs can also provide additional side benefits to clients that increase their satisfaction. In one telehealth program in Germany elderly clients are given a video system for their home. This system is used to provide consultations, education, and facilitate social interaction. In interviews the clients repeatedly gave high praise to the system. They found it extremely helpful in providing treatment, education and enabling them to talk to their friends and family. The most common benefit cited was an increased sense of freedom and control (Gott, 1994). For elderly, who are often home bound, this type of intervention appears to have great potential.

Satisfaction is an important aspect when comparing delivery methods, but substitutability is paramount. Two different studies have examined this area. In each of the studies face-to-face assessment was compared to two-way video and in each of the cases equivalence of the methods was found.

The first of these studies examined the equivalency of the Folstein Mini-Mental State Examination (MMSE) when conducted through two-way video and face-to-face. The patients were from an acute psychiatric ward. Each patient took the MMSE using both delivery modes. The order of assessment was randomly allocated and the second test was administered within 48 hours of the first. Eleven of the original twelve patients completed both assessments. The scores between the two delivery modes were highly correlated (r = .89). Scores on the video mode were then re-scored to correct illegible material. The newly scored protocols had even higher correlations with the face-to-face mode (r= .92). Furthermore, there were no significant differences between the original video and the corrected video scores. Last, when comparing the two modes, none of the protocols contradicted each other in their suggestion of client cognitive impairment (Ball, Scott, McLaren, & Watson, 1993).

A second examination in this area was performed by Baer, et al. (1995). In this study they had raters present both locally and remotely. In one condition the patients were asked questions by the local rater and in the second condition they were asked questions by the remote rater, but in both conditions the remote and local rater rated the answers to the questions. Each of the clients assessed met the DSM-III-R criteria for obsessive-compulsive disorder. The measurements administered to the clients were the Yale-Brown Obsessive Compulsive Scale, Hamilton Depression Rating Scale, and Hamilton Anxiety Rating Scale. In analyzing the results they found near perfect inter-rater reliability on all three scales (r = .99, r = .98, and r = .97). In addition to the ratings, they asked each client to rate their comfort level during the interview, their ability to express themselves, the quality of the interpersonal relationship, and the helpfulness of the interview. The mean scores for each of these categories fell into the "average" to "better than average" range.

The raters themselves were asked to compare this experience to a live one in terms of comfort, ease of expressing one's self, and usefulness of the same scale. In each of these cases the mean scores were also either "average" or "above average." Although interviews in which a client is rated on a straight-forward instrument are not the same as building a successful therapeutic relationship, this study does provide further evidence for the substitutability of different modes as well as the apparent comfort people have with this type of technology.

Although the number of studies exploring the use of this technology have been small and their applicability to outcome in psychotherapy at times indirect, there are works in progress that seem to confirm the equality between the two modes of delivery that the previous studies only hinted at. The first report is preliminary and therefore no data is available. However, in an interview of Paul Zinnik, a psychologist with the Army's Medical Advanced Technology Management Office, Mr. Zinnick stated that their initial feasibility studies of individual and group psychotherapy via telehealth technologies found that it had enormous potential for most clients, but that it does not work well with patients that have severe mental disorders (Cooper, 1997). Of course, like so many of the reports before this, a reasonable critique of these findings is impossible.

A more enlightening study is one being conducted by Glueckauf and his team of researchers (Glueckauf et al., 1999; Glueckauf et al., 1998). They are currently conducting a study examining the effectiveness of the telephone and video teleconference in providing psychotherapy for epileptic teenagers and their families who live in rural areas. Families in this study were provided with six sessions and given the option of continuing for an additional four sessions. The study used a wide variety of measures to assess different variables.

Thus far they have had 39 families participate in the research. Based on their preliminary findings they found that attrition was substantially lower in home-based video and audio counseling as opposed to office-based face to face conditions. In terms of outcome they found substantial improvement across all three conditions. Similar results were found in respect to therapeutic alliance.

A Closing Review

telehealth has been lauded as an answer to the problems of underserved populations, a lack of evenly distributed resources, and burgeoning health care costs (Bashshur, 1997). Current actions in Congress and by private companies have helped to turn telehealth into a reality. Unfortunately this reality has not been developed out of empirically validated studies of its effectiveness in comparison to traditional treatment methods. This study seeks to address one particular area of this gap, psychotherapy.

In this study I have assessed psychotherapy outcome across three different treatment delivery modes, face-to-face, two-way audio, and two-way video. Based on the review of the literature I developed four hypotheses:

• All three treatment modes will show significant improvement when compared to the control group.
• There will be no significant difference in outcome among the three treatment modes.
• Within the audio and video treatment groups comfort with audio and video communication will increase over time in their respective treatment groups.
• Within each of the three treatment groups client aptitude variables, comfort with communication and communication technologies will play a small, but significant part in explaining the level of outcome reached.

The type of therapy provided will not play a significant role in treatment outcome across modes.

The first two hypotheses are tied together. The truth of the first is suggested by the literature examining the effectiveness of psychotherapy which indicates that treatment, regardless of form, is effective (e.g., Smith & Glass, 1977; Smith, Glass, & Miller, 1980). The second hypothesis, each treatment will be equally effective, is more controversial. It has been argued that for psychotherapy to occur, both people need to be able to see each other. Furthermore, conventional wisdom argues that the therapist and the client need to be in the same room. In essence, this wisdom concerns social presence.

Social presence can be defined as "the degree to which a medium is perceived as conveying the presence of the communicated participants" (Short, Williams & Christie, 1976). Media richness depends on words combined with nonverbal cues and communication context (Daft & Lengel, 1984, 1986; Trevinos 1987). This would seem to lend support to the difference in treatment effects across the different modes. Rutter (1987) found that when nonverbal cues were not available, less spontaneous and relaxed conversation occurred. Rice (1993) later found that people consistently ranked face-to-face highest, followed by video, audio and written memo. In another study subjects reported greater social presence after completing a task in an audio and visual situation than in an audio only situation (Short et al., 1976). However, there is no evidence of a direct link between presence and therapy effectiveness. Furthermore, presence is a complex concept that is not entirely media driven.

Heeter (1992) notes that placing a person in the presence of another, whether it be by telephone or in person, will evoke a level of presence. Once together, the task itself can lead to greater or lesser presence (Heeter, 1992). On the surface level, it might seem that a less rich form of media would lead to a poorer therapeutic relationship and thus a significantly lower outcome level. However, I would argue that the therapy relationship itself is a more significant factor in the development of presence and therefore outcome would not be affected by the minor differences caused by different forms of media. In a study comparing various video conferencing systems, audio conferencing systems, and face to face, Sellen (1995) found that people tended to use more verbal cues to mediate the conversation when no picture was available. In a therapy relationship the therapist, a skilled relationship builder, is likely to be able to find different ways to establish the level of presence necessary for a successful therapeutic relationship.

Perhaps the greatest indirect support for the equivalence and effectiveness of these different modes of delivery comes from the education realm. Educators have been using various forms of distance education since the earlier part of this century (Russell, 1997). As one might expect, the use of these different modes of delivery, such as two-way audio, two-way, video, and print based, has resulted in a variety of different empirical studies. Russell (1997) provides a summary of over 200 studies that examined the effectiveness of education delivered through video or audio as compared to face-to-face. The unrelenting finding was no significant difference. Although studies examining education and psychotherapy are not completely transferable this, coupled with the subjective support for equality found in the limited studies comparing various modes of psychotherapy delivery, lends strong support for the proposed equivalence in effectiveness across these three modes of delivery.

In psychotherapy similar average outcomes can be obtained even though different mechanisms in each instance are at work on different client characteristics (Stiles, Shapiro, & Elliot, 1986). In essence my third hypothesis seeks to address a part of Paul's (1967) question, "What kind of therapy, or elements thereof, benefits what kind of client?" Although similar average outcomes are expected across treatments, I expect that there are different client characteristics or aptitudes interacting with the three different treatment modes. Essentially this question is one of aptitude by treatment interaction (ATI). ATI methods have been used for a number of years in an attempt to discover which client variables interact with which treatments (Shoham-Salomon, 1991). To illustrate how this type of analysis is relevant to my study, a definition of ATI terms and their relations to my study is warranted.

Aptitude refers to any measurable person characteristic that is identified before and/or after the treatment that accounts for the person's end state after the given treatment. In my study aptitude will refer to comfort with communication variables. Specifically, I will assess three different comfort with communication variables; comfort with face-to-face communication, comfort with two-way audio communication, and comfort with two-way video communication. Treatment refers to any manipulable situation including such things as environmental characteristics and therapist characteristics. Treatment in this instance is the three different delivery methods that will be used. The last term, interaction, is the degree to which two or more treatments differ from persons who also differ on one or two aptitude measures (Snow, 1991). In this case, the interaction will refer to the degree to which outcome for each of the three delivery modes differs across each of the three comfort variables.

ATI research is difficult at best and to find significant effects there are several steps that should be taken. One of the most important is in the selection of aptitudes to be examined. Shoham-Salomon and Hannah (1991) indicate that it is imperative that the choice of aptitudes be grounded in theory. Following this guideline a few factors become apparent. In the two-way audio literature there were several case reports indicating that some clients were more comfortable with phone therapy and that in some instances probably did much better than they might have had face-to-face treatment been used. This would indicate that comfort with two-way audio communication might be indicative of different outcomes in two-way audio and face-to-face situations.

The two-way video literature reveals a similar, though less striking phenomenon. In two-way video situations there seemed to be initial trepidation in the use of the technology followed by a gradually increasing level of comfort and use. The presence literature offers some explanation for this phenomenon, noting that new technology may initially be detrimental to presence, but that as the user becomes more familiar with the medium, this effect will disappear (Held & Durlach, 1992). Given these factors, in a situation that would permit a more lengthy treatment, I would predict that a comfort with two-way video communication would not relate to treatment outcome across mediums. However, the therapy used in this study is very brief and therefore a significant number of the clients might not have the necessary amount of time to become completely comfortable with the medium. Therefore it seems quite possible that comfort with two-way video communication will affect outcome level in two-way video therapy situations.

Last, conventional wisdom would indicate that people who are comfortable with communicating are more likely to benefit from talk therapy than those who are not comfortable. This would probably be especially true in a brief therapy situation. However, the presence literature also suggests that continued experience with the medium may actually increase presence (Heeter, 1992). For this reason I except there to be differences in level of comfort with particular treatment modes, but that this will change over time, thus resulting in a negligible effect on treatment outcome.

Given three different aptitudes I hypothesize the following: Comfort with communication technologies, specifically audio and video, will increase significantly over time with exposure to the given communication mode. In each group the level of comfort with the mode in which therapy is delivered will be greater than with the comfort with the other two methods of delivery. Furthermore, its contribution to outcome will be significant, but small.

In the past therapy has been shown to be effective regardless of the type of treatment provided (Lambert & Bergin, 1994). However it is important to re-establish this finding when examining novel methods of providing service. I believe that the type of treatment provided will not play a significant role in outcome level in any of the treatment modes.

The field of telehealth and psychotherapy in particular is approaching a turning point in history. Technology is advancing at a rate that will enable us to do what some had only dreamed of doing before. This study seeks to explore the effectiveness of this technology and thus establish some empirical guidelines in an area where there are next to none.

REFERENCES

NOTE: Several of these references are for parts of the original thesis which were not included in this excerpt.

Allen, D. & Allen, A. (1994). Teleradiology 1994. Telemedicine Today, 2, 21-23.

Aneshensel, C. S., Frerichs, R. R., Clark, V. A., & Yokopenic, P. A. (1982). Measuring depression in the community: A comparison of telephone and personal interviews. Public Opinion Quarterly, 46, 110-121.

Baer, L., Brown-Beasley, M. W., Sorce, J., Henriques, A. I., (1992). Computer-assisted telephone administration of a structured interview for obsessive-compulsive disorder. American Journal of Psychiatry, 150, 1737-1738.

Baer, L. Cukor, P., Coyle, J. T. (1997). Telepsychiatry: Application of telemedicine. In R. L. Bashshur, J. H. Sanders, & G. W. Shannon (Eds.), Telemedicine: Theory and practice (pp. 265-290). Springfield, IL: Charles C Thomas.

Baer, L., Cukor, P., Jenike, M. A., Leahy, L., O'Laughlen, J., & Coyle, J. T. (1995). Pilot studies of telemedicine for patients with obsessive-compulsive disorder. American Journal of Psychiatry, 152, 1383-1385.

Ball, C. J., McLaren, P. M., Summerfield, A. B., Lipsedge, M. S., & Watson, J. P. (1995). A comparison of communication modes in adult psychiatry. Journal of Telemedicine and Telecare, 1, 22-26.

Ball, C. J., Scott, N., McLaren, P. M., & Watson, J. P. (1993). Preliminary evaluation of a Low-Cost VideoConferencing (LVCV) system for remote cognitive testing of adult psychiatric patients. British Journal of Clinical Psychology, 32, 303-307.

Bashshur, R. L. (1997). Telemedicine and the health care system. In R. L. Bashshur, J. H. Sanders, & G. W. Shannon (Eds.), Telemedicine: Theory and practice (pp. 265-290). Springfield, IL: Charles C Thomas.

Battle, C. C., Imber, S. D., Hoehn-Saric, R., Stone, A. R., Nash, E. R., & Frank, J. D. (1966). Target complaints as criteria for improvement. American Journal of Psychotherapy, 20, 184-192.

Beck, A. T., Rush, A. J., Shaw, B. F., & Emery, G. (1979). Cognitive therapy of depression. New York: Guilford.

Benschoter, R. A., Wittson, C. L., Ingham, C. G. (1965). Teaching and consultation by television. Hospital and Community Psychiatry, 16, 99-100.

Berger, M., & Callister, S. (1981). A review of comments from consumer satisfaction instruments. Ogden, UT: Weber Mental Health Center.

Bergin, A. E., & Garfield, S. L. (Eds.) (1971). Handbook of psychotherapy and behavior change. New York, NY: Wiley.

Bergin, A. E., & Garfield, S. L. (Eds.) (1994). Handbook of psychotherapy and behavior change (4th ed.). New York, NY: Wiley.

Beutler, L. E. & Crago, M. (1983). Self-report measures of psychotherapy outcome. In M. J. Lambert, E. R. Christensen, & S. S. DeJulio (Eds.) The Assessment of psychotherapy outcome. New York, NY: John Wiley & Sons.

Bhatara, B. S., Fuller, V. S., & William, C. (1995). An educational perspective of reservation mental health service deficit: The South Dakota experience. American Indian and Alaska Native Mental Health Research, 6, 56-68.

Brown, J., Dreis, S., & Nace, D. K. (1999). What really makes a difference in psychotherapy outcome. In M. A. Hubble, B. L. Duncan, & S. D. Miller (Eds.), The Heart and Soul of Change: What Works in Therapy (pp. 389-406). Washington, DC: American Psychological Association.

Budman, S. H., & Stone, J. (1983). Advances in brief psychotherapy: A review of recent literature. Hospital and Community Psychiatry, 34, 939-946.

Burton, C. A., & Nichols, M. P. (1978). Behavioral target complaints form. Psychological Reports, 42, 219-226.

Campbell, D. T. & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychology Bulletin, 56, 81-105.

Cartwright, D. S. (1975). Patient self report measures. In I. E. Waskow & M. B. Parloff (Eds.), Psychotherapy change measures. Rockville, MD: National Institute of Mental Health.

Castonguay, L. G., Goldfried, M. R., Wiser, S. L., Raue, P. J., & Hayes, A. M. (1996). Predicting the effect of cognitive therapy for depression: A study of unique and common factors. Journal of Consulting and Clinical Psychology, 64, 497-504.

Cohen, E. (1997, January 17). Shrinks aplenty online, but are they credible? The New York Times: CyberTimes [Newspaper, selected stories on line]. Retrieved November 3, 1997 from the World Wide Web: http://www.nytimes.com/library/cyber/week/011797shrinks.html

Cooper, G. (March/April, 1997). Therapy.com. Networker, p. 17.

Costa, P. T., Jr. & McCrae, R. R. (1992). NEO five factor inventory. Odessa, FL: Psychological Association Research Inc.

Daft, R. L., & Lengel, R. H. (1984). Information richness: A new approach to managerial behavior and organization design. Research in Organizational Behavior, 6, 191-233.

Daft, R. L., & Lengel, R. H. (1986). Organizational information requirements, media richness and structural design. Management Science, 32, 554-571.

Dakins, D. R., & Jones, E. (1996). Cream of the crop: 10 outstanding telemedicine programs. Telemedicine and telehealth Networks. Miller-Freeman. Retrieved November 3, 1997 from the World Wide Web.

Derogatis, L. R. (1993). Brief symptom inventory: Administration, scoring, and procedures manual¾ II. Minneapolis, MN: National Computer Systems.

Derogatis, L. R. (1994). Symptom checklist-90-r: Administration, scoring, and procedures manual (3rd ed.). Minneapolis, MN: National Computer Systems.

Derogatis, L. R. & Melisaratos, N. (1983). The brief symptom inventory: An introductory report. Psychological Medicine, 13, 595-605.

Derogatis, L. R. & Spencer, P. M. (1982). The brief symptom inventory manual. Baltimore: John Hopkins School of Medicine.

De Jong, P. & Hopwood, L. E. (1994). Outcome Research on Treatment Conducted at the Brief-Family Therapy Center, 1992-1993. In S. D. Miller, M. A. Hubble, & B. L. Duncan (Eds.), Handbook of Solution-Focused Brief Therapy (pp. 272-298). San Francisco, CA: Jossey-Bass Publishers.

de Shazer, S. (1988). Clues: Investigating solutions in brief therapy. New York: Norton.

Diagnostic and Statistical Manual of Disorders (fourth edition). (1994). Washington, DC: American Psychiatric Association.

Dongier, M., Tempier, R., Lalinec-Michaud, M., & Meunier, D. (1986). Telepsychiatry: Psychiatric consultation through two-way television: A controlled study. Canadian Journal of Psychiatry, 31, 32-34.

Dwyer, T. F. (1973). Telepsychiatry: Psychiatric consultation by interactive television. American Journal of Psychiatry, 130, 865.

Endicott, J., Spitzer, R. L., Fleiss, J. L., & Cohen, J. (1976). The global assessment scale. Archives of General Psychiatry, 33, 766-771.

Evans, R. L., Smith, K. M., Werkhoven, W. S., Fox, H. R., & Pritzl, D. O. (1986). Cognitive telephone group therapy with physically disabled elderly persons. The Geronotologist, 26, 8-10.

Eysenck, H. J. (1952). The effects of psychotherapy: An evaluation. Journal of Consulting Psychology, 16, 319-324.

Fenig, S., Levav, I., Kohn, R., & Yelin, N. (1993). Telephone vs face-to-face interviewing in a community psychiatric survey. American Journal of Public Health, 83, 896-898.

Field, M J. (1996). Telemedicine: A guide to assessing telecommunications in health care. Washington D. C.: National Academy Press.

Fiske, D., Hunt, H., Lubrosky, L., Orne, M., Parloff, M., Reiser, M., & Tuma, A. (1970). The planning of research on effectiveness of psychotherapy. American Psychologist, 25, 727-737.

Foderaro, L. (1994). For some mental ill inmates in New York state, punishment is the treatment. The New York Times, 144, 17.

Garfield, S. L. (1994). Research on client variables in psychotherapy. In A. E. Bergin & S. L. Garfield (Eds.), Handbook of psychotherapy and behavior change (4th ed.) (pp. 190-228). New York: Wiley.

Gelso, C. J. (1979). Research in counseling: Methodological and professional issues. Counseling Psychologist, 8, 7-30.

Gelso, C. J. & Johnson, D. H. (1983). Explorations of time-limited counseling and psychotherapy. New York: Teacher's College Press.

Glueckauf, R. L., Hufford, B., Whitton, J., Baxter, J., Schneider, P., Kain, J., & Vogelgesang, S. (1999). telehealth: Emerging technology in rehabilitation and health care. In M. G. Eisenberg, R. L. Glueckauf, & H. H. Zaretsky (Eds.), Medical aspects of disability: A handbook for the rehabilitation professional (2nd ed.) p. 625-639). NY: Springer.

Glueckauf, R. L., Whitton, J., Baxter, J., Kain, J., Vogelgesang, S., Hudson, M., & Wright, D. (1998). Videocounseling for families of rural teens with epilepsy: Project update. telehealthNews [On-Line], 2(2), 1-3. Retrieved November 15, 1997 from the World Wide Web: http://www.telehealth.net/subscribe/newslettr_4a.html#1

Gott, M. (1994). telehealth and telemedicine: Executive summary of a European foundation research project. Dublin, Ireland: Loughlinstown House Shankill, Co.

Graham, M. A. (1996). Telepsychiatry in Appalachia. American Behavioral Scientist, 39, 602-615.

Grigsby, J. (1997). Telemedicine in the United States. In R. L. Bashshur, J. H. Sanders, & G. W. Shannon (Eds.), Telemedicine: Theory and practice (pp. 265-290). Springfield, IL: Charles C Thomas.

Grumet, G. W. (1979). Telephone therapy: A review and case report. American Journal of Orthopsychiatry, 49, 574-584.

Haas, L. J., Benedict, J. G., & Kobos, J. C. (1996). Psychotherapy by telephone: Risks and benefits for psychologists and consumers. Professional Psychology: Research and Process, 27, 154-160.

Heeter, C. (1992). Being there: The subjective experience of presence. Presence, 1, 262-271.

Held, R. M., & Durlach, N. I. (1992). Telepresence. Presence, 1, 109-112.

Henry, W. P., Strupp, H. H., Butler, S. F., Schacht, T. E., & Binder, J. L. (1993). The effects of training in time-limited dynamic psychotherapy: Mediators of therapists' response to training. Journal of Consulting and Clinical Psychology, 61, 434-440.

Hines, M. H. (1994). Using the telephone in family therapy. Journal of Martial and Family Therapy, 20, 175-184.

Hipkens, J. H. (1997). Telemedicine in correctional systems. In R. L. Bashshur, J. H. Sanders, & G. W. Shannon (Eds.), Telemedicine: Theory and practice (pp. 265-290). Springfield , IL: Charles C Thomas.

Hoehn-Saric, R., Frank, J., Imber, S., Nash, E., Stone, A., & Battle, C. (1964). Systematic preparation of patients for psychotherapy - I. Effects on therapy behavior and outcome. Journal of Psychiatric Research, 2, 267-281.

Huberty, C. J., & Morris, J. D. (1989). Multivariate analysis versus multiple univariate analyses. Psychological Bulletin, 105, 302-308.

Hufford, B. J., Glueckauf, R. L., & Webb, P. M. (1999). Home-based, interactive videoconferencing for adolescents with epilepsy and their families. Rehabilitation Psychology, 44, 1-18.

Johnson, P. R., & Thorn, B. E. (1989). Cognitive behavioral treatment of chronic headache: Group versus individual treatment format. Headache, 29, 358-365.

Key, P. (1997, May 5). Counselor nets new clients with on-line service. Indianapolis Star/News [Newspaper, selected stories on line]. Retrieved November 15, 1997 from the World Wide Web: http://www.metanoia.org/imhs/indiana.htm

Kokotovic, A. M., & Tracey, T. J. (1990). Working alliance in the early phase of counseling. Journal of Counseling Psychology, 37, 16-21.

Koss, M. P., & Shiang, J. (1994). Research on brief psychotherapy. In A. E. Bergin & S. L. Garfield (Eds.), Handbook of psychotherapy and behavior change (4th ed., pp. 664-700). New York, NY: Wiley.

Lambert, M. J., & Bergin, A. E. (1994). The effectiveness of psychotherapy. In A. E. Bergin & S. L. Garfield (Eds.), Handbook of psychotherapy and behavior change (4th ed., pp. 143-189). New York, NY: Wiley.

Larsen, D. L., Attkison, O., Hargreaves, W. A., & Nguyen, T. D. (1980). Assessment of client/patient satisfaction: Development of a general scale. Evaluation Program Planning, 2, 197-207.

Lauter, J. (1978). Sample size requirements for the T2 test of MANOVA. Biometrical Journal, 20, 389-406.

Lebow, J. L. (1982). Consumer satisfaction with mental health treatment. Psychological Bulletin, 91, 244-259.

Liebson, E. (1997). Telepsychiatry: Thirty-five years experience. Medscape Mental Health, 2 (7), 1. Retrieved November 15, 1997 from the World Wide Web: http://www.medscape.com/Medscape/MentalHe...02.n07/mh3112.liebson/mh3112.liebson.html

Locke, E. A. The nature and causes of job satisfaction. The Handbook of Industrial and Organizational Psychology. New York, NY: Rand McNally College Publications, 1976, 1297-1350.

MacKinnon, R. A. & Michels, R. (1970). The role of the telephone in the psychiatric interview. Psychiatry, 33, 82-93.

Maheu, M. & Gordon, B. (1997). Preparing for the 21st century: Psychological telehealth today. Symposium at the meeting of the American Psychological Association, Chicago, IL.

Manning, W. & DuBois, P. (1962). Correlational methods in research on human learning. Perceptual and Motor Skills, 15, 288-321.

May, R. J. (1990). Are wait lists really a problem? A follow-up survey of wait list dropouts. Journal of College Student Development, 31, 564-566.

McKeon-Charkalis, D. (1997). The web can be a link to your shrink. USA Today: Health [Newspaper, selected stories on line]. Retrieved November 15, 1997 from the World Wide Web: http://www.metanoia.org/imhs/usatoday.htm

Miller, R. C., & Berman, J. S. (1981). The efficacy of cognitive-behavior therapy: A quantitative review of the research evidence. Paper presented at the annual meeting of the American Psychological Association, Los Angeles, CA.

Mintz, J. Kiesler, D. J. (1982). Individualized measures of psychotherapy outcome. In P. C. Kendall & J. N. Butcher, (Eds.), Handbook of research methods in clinical psychology (pp. 429-460). New York, NY: John Wiley & Sons.

Murray, J. & Keller, P. (1991). Psychology and rural America: Current status and future directions. American Psychologist, 46, 220-231.

Nagy (1987, November). Electronic ethics. APA Monitor, p. 3.

National Alliance for the Mentally Ill and Public Citizens Health Research Group (1992). Joint report with the public citizen's health research group, Criminalizing the seriously mentally ill: The abuse of jails as mental hospitals. National Alliance: Arlington, V. A.

Nichols, M. P. (1974). Outcome of brief cathartic psychotherapy. Journal of Consulting and Clinical Psychology, 42, 403-410.

Nickelson, D. W. (1996). Behavioral telehealth: Emerging practice, research, and policy opportunities. Behavioral Sciences and the Law, 14, 443-457.

Norville, R., Sampson, H., & Weiss, J. (1996). Accurate interpretations and brief psychotherapy outcome. Psychotherapy Research, 6, 16-29.

Office of Technological Assessment, U.S. Congress. (1990). Health care in rural America, OTA-H-434. Government Printing Office: Washington, D. C.

Paul, G. L. (1967). Strategy of outcome research in psychotherapy. Journal of Consulting Psychology, 31, 109-118.

Pedhazur, E. J. (1982). Multiple regression in behavioral research: Explanation and prediction (2nd ed.). New York, NY: Harcourt Brace College Publishers.

Peterson, R. L., Hodge, E. A., Kafer, L. G., Matthews, S. R., Pfeifle, H. H., & Van Heuse, C. L. (1981). A group counseling emphasis at a university counseling service. Psychotherapy Theory and Practice, 18, 526-536.

Practice issues: Can you build a practice on telephone services? (1997, April). Psychotherapy Finances, 23, 1-2.

Rice, R. E. (1993). Media appropriateness: Using social presence theory to compare traditional and new organizational media. Human Communication Research, 19, 451-484.

Russell, T. L. (1997). The "No Significant Difference Phenomenon." Retrieved November, 15 1997 from the World Wide Web: http://teleeducation.nb.ca/nosignificantdifference/

Rutter, R. D., (1987). Communicating by telephone. Elmsford, NY: Pergamon.

Schneider, P. L. & Reinhart, J. (1998, August). The Relationship Between Environmental Perception, Satisfaction, Self-efficacy, and Anxiety in a Two-way Audio/Video Classroom. Poster presentation at APA, San Francisco, CA.

Selgiman, M. E. P. (1995). The effectiveness of psychotherapy: The Consumer Reports study. American Psychologist, 50, 965-974.

Sellen, A. J. (1995). Remote conversations: The effects of mediating talk with technology. Human-Computer Interaction, 10, 401-444.

Shannon, G., Bashshur, R., & Lovett, J. (1986). Distance and the use of mental health services. The Milbank Memorial Quarterly, 64, 302-330.

Shoham-Salomon, V. (1991). Introduction to special section on client therapy interaction research. Journal of Consulting and Clinical Psychology, 59, 203-204.

Shoham-Salomon, V. & Hannah, M. T. (1991). Client therapy interaction in the study of differential change processes. Journal of Consulting and Clinical Psychology, 59, 217-225.

Short, J. Williams, E., & Christie, B. (1976). The social psychology of telecommunications. London: Wiley.

Siu, A., Ke., G., & Beck, J. (1989). Geriatric medicine in the United States: The current activities of former trainees. Journal of the American Geriatric Society, 37, 272-276.

Sleek, S. (1997). Providing therapy from a distance: Video conferencing helps practitioners maintain strong relationships with patients they can't see regularly. APA Monitor, 28(8), 1+.

Smith, M. & Glass, G. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32, 752-760.

Smith, M., Glass, G., & Miller, T. (1980). The benefits of psychotherapy. Baltimore, MD: John Hopkins University Press.

Snow, R. E. (1991). Aptitude treatment interaction as a framework for research on individual differences in psychotherapy. Journal of Consulting and Clinical Psychology, 59, 205-216.

Solow, C., Weiss, R. J., Bergen, B. J., & Sanborn, C. J. (1971). 24 hour psychiatric consultation via TV. American Journal of Psychiatry, 127, 120-23.

Spiegler, M. D. (1983). Contemporary Behavioral Therapy. Mayfield Mountain View, California: Publishing Company.

"State law update: State legislative update." (1997). Center for Telemedicine Law. Retrieved November 20, 1997 from the World Wide Web: http://www.ctl.org/news/jun97/statelawupdate.html

Stevens, J. (1996). Applied multivariate statistics for the social sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.

Stiles, W. B., Shapiro, D. A., & Elliott, R. (1986). Are all psychotherapies equivalent? American Psychologist, 41, 165-180.

Strupp, H. H., & Hadley, S. W. (1977). A tripartite model of mental health and therapeutic outcomes. Archives of General Psychiatry, 36, 1125-1136.

Swinson, R. P., Fergus, K. D., Cox, B. J., & Wickwire, K. (1995). Efficacy of telephone-administered behavioral therapy for panic disorder with agoraphobia. Behavior Research and Therapy, 33, 465-469.

Tausig, J. E., & Freeman, E. W. (1988). The next best thing to being there: Conducting the clinical research interview by telephone. American Journal of Orthopsychiatry, 58, 418-427.

Thompson, L. W., Gallagher, D., & Breckenridge, J. (1987). Comparative effectiveness of psychotherapy for depressed elders. Journal of Consulting and Clinical Psychology, 55, 385-390.

Tolmach, J. (1985). "There ain't nobody on my side": A new day treatment program for Black urban youth. Journal of Clinical Child Psychology, 14, 214-219.

Tracey, T. J. (1988). Relationship of responsibility attribution congruence to psychotherapy outcome. Journal of Social and Clinical Psychology, 7, 131-146.

Tracey, T. J. (1989). Client and therapist session satisfaction over the course of psychotherapy. Psychotherapy, 26, 177-182.

Tracey, T. J. & Dundon, M. (1988). Role anticipations and preferences over the course of counseling: An interactional examination. Journal of Counseling Psychology, 31, 13-27.

Tracey, T. J. & Ray, P. B. (1984). The stages of successful time-limiting counseling: An interactional examination. Journal of Counseling Psychology, 31, 13-27.

Trevino, L. K., Lengel, R. H., Gerloff, E. A., & Muir, N. K. (1990). The richness and media choice in organizations: A symbolic interactionist perspective. Communication Research, 14, 553-575.

Troster, A. I., Paolo, A. M., Glatt, S. L., Hubble, J. P., & Koller, W. C. (1995). "Interactive video conferencing" in the provision of neuropsychological services to rural areas. Journal of Community Psychology, 23, 85-88.

Volsky, T., Jr., Magoon, T. J., Norman, W. T., & Hoyt, D. P. (1965). The outcomes of counseling and psychotherapy. Minneapolis, MN: University of Minnesota Press.

Ware, J. E. (1978). Effects of acquiescent response set on patient satisfaction ratings. Medical Care, 16, 327-336.

Williams, R. H. & Zimmerman, D. W. (1996). Are simple gain scores obsolete? Applied Psychological Measurement, 20, 59-69.

Wittson, C. L., Afflect, D. C., & Johnson, V. (1961). Two-way television in group therapy. Mental Hospitals, 12, 22-23.

Wittson, C. L. & Benschoter, R. A. (1972).Two-way television: Helping medical center reach-out. American Journal of Psychiatry, 129, 624-27.

Wittson, C. L. & Dutton, R. (1956). A new tool in psychiatric education. Mental Hospitals, 7, 11-14.

.