Table: Categories of Evaluation Questions for Comparing Telemedicine to Alternative Health Services

1. What are the effects of the application on the clinical process of care compared to the alternative(s)?
2. What were the effects of the application on patient status or health outcomes compared to the alternative(s)?
3. What are the effects of the application on access compared to the alternative(s)?
4. What are the costs of the application for patients, private or public payers, providers and other affected parties compared to the alternative(s)?
5. How did patients, clinicians and other relevant parties view the application and were they satisfied with the application compared to the alternative(s)

Note: Each question assumes that an analysis of results will control for or take into account severity of illness, comorbidities, demographic characteristics and other relevant factors

Table: Evaluating Quality of Care and Health Outcomes

What are the effects of the Telemedicine application on the clinical process of care compared to the alternative(s)

1. Was the application associated with differences in the use of the health services (e.g. office visits, emergency transfers, diagnostic tests, length of hospital stay)?
2. Was the application associated with differences in appropriateness of services (e.g. underuse of clearly beneficial care)?
3. Was the application associated with differences in the quality, amount or type of information available to clinicians or patients?
4. Was the application associated with differences in patients’ knowledge of their health status, their understanding of the care options or their compliance with care regiments?
5. Was the application associated with differences in diagnostic accuracy or timeliness, patient management decisions or technical performance?
6. Was the application associated with differences in the interpersonal aspects of care?

What were the effects of the telemedicine application on immediate, intermediate or long term health outcomes compared to the alternative(s)?

1. Was the application associated with differences in physical signs or symptoms?
2. Was the application associated with differences in morbidity or mortality?
3. Was the application associated with differences in physical, mental or social and role functioning?
4. Was the application associated with differences in health-related behaviours (e.g. substance abuse)?
5. Was the application associated with differences in patient satisfaction with their care or patient perceptions about the quality or acceptability of the care they received?

Note: Each question assumes that an analysis of results will control for or take into account severity of illness, comorbidities, demographic characteristics and other relevant factors

Table: Evaluating Access to Care

Did Telemedicine affect the use of services or the level or appropriateness of care to the alternative(s)

1. What was the utilisation of telemedicine services before, during and after the study period for target population and clinical problem(s)?

2. When offered the opinion of a telemedicine service, how often did patients

• accept or refuse an initial service or fail to keep an appointment
• accept or refuse a subsequent service or fail to keep an appointment?

3. What was the utilisation of specified alternative services before, during and after the study period for the target population and clinical problem(s)?

• consultants travelling to distant sites
• patients travelling to distant consultants
• consultation by mail or courier
• transfer to other facilities
• self care

4. Was the telemedicine application associated with a difference in overall utilisation (e.g. number of services or rate) or indicators of appropriateness of care for

• speciality care
• primary care
• transport services
• services associated with lack of timely care?

Did the application affect the timeliness of care or the burden of obtaining care compared to the alternative(s)?

1. Was the difference in the

• timing of care
• appointment waiting times for referrals

2. What were patient attitudes about the

• timeliness of care
• burden of obtaining care
• appropriateness of care

3. What were the attitudes of attending and consulting physicians and other personnel about the

• timeliness of care
• burden of providing care
• appropriateness of care?

Note: Each question assumes that an analysis of results will control for or take into account severity of illness, comorbidities, demographic characteristics and other relevant factors

Table: Evaluating Health Care Costs and Cost-Effectiveness

What were the costs of the telemedicine application for participating health care providers or health plans compared to the alternative(s)

1. Was an application associated with differences in attending clinicians’ costs for personnel, equipment, supplies, administrative services, travel or other items? Was an application associated with differences in revenues or productivity? What was the net effect?
2. Was an application associated with differences in consulting clinicians’ or consulting organisations’ costs for personnel, equipment, supplies, space, administrative services, travel or other items? Was an application associated with differences in revenues or productivity? What was the net effect?
3. Was an application associated with differences in the cost per service, per episode of illness or per member (health plan enrolee, capitated lives) per month?

What were the costs of the telemedicine application for patient and families compared to the alternative(s)?

1. What was the application associated with differences in direct medical costs for patients of families?
2. What was the application associated with differences for patients or families in other direct costs (e.g. travel, child care) or indirect costs (e.g. lost work days)?

What were the costs for the society overall compared to the alternative(s)?

1. Was an application associated with differences in total health care cost, the cost per service, per episode of illness or per capita?

How did the costs of the application relate to the benefits of the telemedicine application compared to the alternative(s)?

Note: Each question assumes that an analysis of results will control for or take into account severity of illness, comorbidities, demographic characteristics and other relevant factors

Table: Evaluating Patient Perceptions

Were patients satisfied with the telemedicine service compared to the alternative(s)

1. How did patients rate their physical and psychological comfort with the application?
2. How did patients rate the convenience of the encounter, its duration, its timeliness and its cost?
3. How did patients (and family members) rate the skills and personnel manner of the consultant and the attending personnel (e.g. primary care physician, nurse practitioner)?
4. Was the lack of direct physical contact with the distant clinician acceptable?
5. How did patients rate the explanations provided to them of what their problem was and what was being recommended?
6. Did patients have concerns about whether the privacy of personal medical information was protected?
7. Would patients be willing to use the telemedicine service again?
8. Overall, how satisfied were patients with the telemedicine service they received?

Note: Each question assumes that analysis of results will control for or take into account prior experiences with the health care system, severity of illness, comorbidities, demographic characteristics and other relevant factors

Table: Desirable Attributes of Evaluation Criteria

Reliability / Reproducibility An evaluation instrument or criterion is reliable if repeated use under identical circumstances by the same or different users produces the same results

Validity An evaluation instrument or criterion is valid if it measures the properties, qualities or characteristics it is intended to measure

Responsiveness An evaluation instrument or criterion is responsive if it can detect important differences in outcomes across evaluation groups or time periods

Interpretability An evaluation instrument or criterion is interpretable if users find the results of its application understandable

Feasibility An evaluation instrument or criterion is feasible if users can accomplish the required activities, collect the necessary information and analyse the resulting data within available evaluation resources and without imposing excessive burdens on those whose co-operation is required for the evaluation

Flexibility An evaluation instrument or criterion is flexible if it is adaptable to a variety of evaluation problems and circumstances

Documentation An evaluation instrument or criterion is documented if the protocols for applying and interpreting it are specified and if evidence of its successful use is summarised or cited

Elements of an Evaluation Plan

Project description and research question(s): the application or programme to be evaluated and the basic questions to be answered by the evaluation

Strategic objectives: how the project is intended to serve the sponsor or parent organisation’s purposes

Clinical objectives: how the telemedicine project is intended to affect individual or population health by changing the quality, accessibility or cost of care

Business plan or project management plan: a formal statement of how the evaluation will help decision makers judge whether and when the application will be a financially and otherwise sustainable enterprise or less formally, what the project’s management, work plan, schedule and budget will be

Level and perspective of evaluation: whether the focus of the research question(s) and objectives id clinical, institutional, societal or some combination

Research design and analysis plan: the strategy and steps for developing valid comparative information and analysing it

• Experimental and comparison groups: characteristics of (a) the group or groups that will be involved in testing the target telemedicine application and (b) the group or group that will receive alternative services for purposes of comparison
• Technical, clinical and administrative process: as planned and actually implemented, the communications and information systems, the methods for providing medical care and the supportive organisational processes
• Measurable outcomes: the variables and the data to be collected to determine whether the project is meeting its clinical and strategic objectives
• Sensitivity analysis: the inclusion of techniques to assess to what extent conclusions may change if assumptions or values of key variables changed

Documentation: the explicit reporting of the methods employed in the evaluation and the findings so that others can determine how the results were established

Categories of Evaluation Questions for Comparing Telemedicine to Alternative Health Services

1. What were the effects of the application on the clinical process of care compared to the alternative(s)?
2. What were the effects of the application on patient status or health outcomes compared to the alternative(s)?
3. What were the effects of the application on access compared to the alternative(s)?
4. What were the costs of the application for patients, private or public payers, providers and other affected parities compared to the alternative(s)?
5. How did patients, clinicians and other relevant parties view the application and were they satisfied with the application compared to the alternative(s)

Note: Each question assumes an analysis of results will control for severity of illness, comorbidities, demographic characteristics and other relevant factors