Research Update

Issue Number 4, March 2003

Monitoring and Evaluating Physical Activity Programs for Older Adults

By: Anita M. Myers PhD, Professor, Department of Health Studies and Gerontology, University of Waterloo; Adjunct, Canadian Centre for Activity and Aging, University of Western Ontario

Program evaluation involves collecting information on service delivery and clients to help you make informed decisions when planning new initiatives or examining existing services. Evaluation provides the information you need to support your case for new, continued or increased program funding. It also demonstrates your commitment to delivering the highest quality service possible with available resources. While evaluation may be required for funding or reporting, first and foremost it will enable you to be accountable to your primary stakeholders—the clients or participants in your program.

Older adults who take part in active living programs have the right to expect a safe and beneficial program, tailored as much as possible to their needs, abilities and expectations. Conducting evaluation shows potential and current clients that you value their input and feedback.

Benefits of Program Evaluation

Accountability to sponsors and participants
Informed planning and decision-making

Directions for improving service delivery:
• Recruitment and retention
• Meeting client needs and expectations
• Maximizing client benefits
• Minimizing adverse effects
• Maximizing efficiencies

Popular Misconceptions about Evaluation

While program evaluation sounds good in principle, you may wonder if you have the time, money or expertise to conduct it. First, it is important to understand that evaluation differs from “traditional” research. Research examines physical activity interventions under highly controlled conditions, while evaluation examines programs under real-world conditions. You do not have to be a researcher to evaluate your programs!

While evaluation does require time and effort, the alternative (not doing evaluation) can be much more costly. The programs at greatest risk of losing funding are those with no supporting evidence. For example, back in the 1980’s when the Canadian Red Cross Society underwent major restructuring, the Fun & Fitness Program for Seniors survived because of evaluation data. Not only did the findings convince the Board to continue the program, the program coordinators also used the evaluation to secure external funding to expand it across Canada.

In-house resources will affect the scope of evaluation projects you can tackle. However, even grassroots programs with limited resources can undertake simple evaluation activities. As you will see, simple adjustments to record-keeping practices can produce valuable information for decision-making.

Another popular myth is that evaluation may produce negative findings, leading to funding cuts and staff layoffs. This apprehension stems from a misunderstanding that evaluation only looks at whether the program can demonstrate “success” in terms of client outcomes or benefits. Outcome evaluation is only one type of evaluation. And most show that programs tend to be relatively beneficial (for some clients more than for others). It is not productive to think in terms of success or failure. By setting realistic performance objectives and examining various aspects of service promotion and delivery, evaluation will identify which aspects are working well and which may need improvement.

Let’s look at the different types or approaches to program evaluation, as well as the distinction between periodic evaluation studies and ongoing data collection or monitoring. Then we will look at ways to improve client information gathering practices.

Types of Program Evaluation

Program Evaluation Approaches
Needs assessment (planning; market analysis)
Formative evaluation (piloting; market testing)
Process evaluation (routine operation)
Outcome evaluation (effectiveness)
Cost evaluation (efficiency)

The particular stage of program development determines which approach to evaluation is most appropriate. For instance, when planning new programs, it is important to conduct a needs assessment (or market analysis) to examine what services already exist in your area to avoid duplication and to see whether your target audience (potential users) is interested in such a program, given possible fees, location, scheduling, and so on. Once you have made the decision to proceed and begin developing materials (such as brochures, videos, posters), it is important to test the reactions of potential clients through formative evaluation (market or pilot testing). Are materials understandable? Appealing? Motivating? Evaluation at the draft stage can save time and money. Imagine producing 1,000 copies of an exercise video only to discover that older adults did not identify with the actors or found it boring or difficult to follow.

Once your program is up and running, it is important to conduct process evaluation. Are you reaching your intended target audience (recruitment)? Who is using your program or services, how often? Are your classes or groups filled to capacity? Can you meet the demand (is there a wait list, how long)? To what extent are you meeting client expectations? Do participants tend to continue participating or complete the program (retention)? Do some areas of program delivery need improvement?

After you have modified your program, to optimize recruitment and/or delivery, you may want to consider outcome and cost evaluations. Outcome evaluation examines the extent of client improvement, maintenance or prevention (depending on the program’s objectives). It assesses clients before they begin the program and after some period of participation using measures or indicators such as level of physical activity, physical or psychological functioning. Cost analysis, on the other hand, can be fairly simple (cost per case) or highly complex (relating program inputs to program results).

Evaluation and Monitoring

The good news is that most evaluation studies need to be conducted only periodically. Once you have supporting information from your needs assessment, you can proceed with developing your new program. You need to conduct further periodic needs assessment only when if you want to expand your services to new areas or target groups. For example, if you wanted to attract more ethnically diverse older adults in your region you would conduct another needs assessment. You would conduct another process evaluation if you made substantial changes to your promotional or delivery strategies. You may not have to do an outcome evaluation (pre- and post- client comparisons) if studies documenting client benefits of similar programs are available.

Conversely, certain information should be collected on an ongoing basis. Every program should collect background information from each and every new participant and record individual attendance (for class or group-based programs.) Such information is essential for developing a profile of your clientele, including dropout rates. Comparing the same information over time (monitoring or tracking) allows you to examine patterns or trends. For instance, you may see that enrolment patterns are higher at certain times of the year. Or, you may find that your client profile has changed in the last few years (such as more early retirees or different cultural groups or persons with more severe arthritis). Usage patterns may also change (such as greater demand for morning classes, more women using the weight rooms, more men in dance classes). Today’s clients may have different needs and expectations, requiring you to modify promotional and delivery strategies. For instance, if your clientele now includes individuals with more severe arthritis or osteoporosis, you may need to help them to safely enter and exit the pool. Likewise, it may take more time to explain home exercises to clients with cognitive impairments. Either of these scenarios would justify additional program resources.

As well as monitoring possible changes in your client profile and usage patterns, you need to determine how well you are meeting client needs and expectations. This can be done either periodically or on an on-going basis.

Obtaining Participant Feedback

Many programs use client satisfaction questionnaires (also referred to as “evaluation” or “feedback” forms) because they are easy to develop and score. However, you should also understand their limitations.

Satisfaction Questionnaires

Pros
Easy to develop
Easy to adapt
Anonymous
Quickly summarized

Cons
Response bias
Representation bias
Hard to interpret
Limited utility

• Did you know that 80 to 85% of consumers say they are moderately to highly satisfied?
  This response bias has been found across all types of services and programs. Why? Most people don’t like to complain (at least formally) for fear of losing the service or not wanting to ruin it for others. Others simply can’t be bothered filling out the form. Since such surveys tend to be distributed at the end of the session or year-end, less frequent attendees may not be there the day of the survey. And dropouts, those for whom the program has not met their needs and expectations, are already long gone.
• Did you know that two-thirds of consumers switch services due to staff indifference?
  This holds true whether examining health care services, fast foods or active living programs. It is debatable whether participants can judge the technical skills or competency of service providers, but they do know whether they were treated in a friendly, courteous manner. If people like the service provider, they tend to give high overall ratings to the program itself.
• What can you learn from client satisfaction surveys?
  Such surveys typically entail checking or circling a response ranging from highly satisfied to highly dissatisfied (or excellent to poor) for various program features (such as instructor, location, scheduling, music). Even if some people check “dissatisfied,” for such things as music or scheduling, what do these findings tell you? Since such surveys are anonymous, there is no way of determining which clients are dissatisfied, and why. Although surveys may include space for open-ended comments, most people do not bother. Even if you receive some specific comments, you would not want to act without first knowing the extent to which such concerns represent the views of your clientele as a whole.

How to improve client satisfaction surveys:

• Choose more than one class or time period for distribution (not close to holidays).
• Have someone other than the group instructor or leader distribute these surveys.
• Include a confidential envelope and a drop-off box for returning surveys.
• Mail the survey (with an addressed, postage paid envelope) to all registrants. This will help reach less frequent users and those who have left the program.
• Always report the response rate (percent completing the survey divided by the total number of registrants) when presenting the findings.

You should also consider other feedback strategies. Suggestion boxes and comment cards should be readily available and promoted. A phone call to absent members (shown to reduce attrition in commercial fitness clubs) can be used to both offer support and gather information. You should also periodically conduct focus groups.

Focus groups are an excellent way to obtain in-depth client feedback. Focus or discussion groups allow several participants (usually 8 to 12 per group) to exchange ideas, elaborate on impressions and explore issues in more detail. This method is highly recommended for needs assessments and formative evaluation (to solicit the input of potential clients) and for process evaluation (to solicit feedback from current clients).

Where to start?

Start by examining your current record-keeping practices concerning client information. Is the information easy to access? Is it in usable form? When working with community recreation and seniors’ centres, we found that most collected enrolment or registration data. Many instructors also recorded class attendance using registration lists, although some did total head counts only. While instructors knew a lot about individual participants, typically only client age and gender were available from registration forms (usually stored in file folders).

A simple background questionnaire given to all new registrants, together with individual attendance/usage data, is all that you need to develop a more detailed profile of your clientele. As well as age and gender, you may want some other demographic information (such as marital status, education), some basic health information (such as mobility, vision or hearing problems that may affect participation), as well as reasons for joining, how they heard about the program, how they normally get to the program, other physical activities, possible reservations, and so on. Simple databases can help you manage and link different sources of information (to compare characteristics of frequent versus infrequent users, adherers versus dropouts), and monitor patterns over time.

(Simple Ways to Get Started)

• Record individual attendance or usage
• Gather background on new clients
• Set up a simple database to manage, link and track client information
• Set up a procedure for contacting absent clients
• Conduct periodic focus groups to solicit in-depth client feedback

There are simple ways to improve your current record-keeping systems to begin profiling your clients and monitoring your programs. Depending on your program’s stage of development, you should also consider periodic evaluation studies. Most participants will be happy to provide information for evaluation purposes once you explain why it is important and how it will be used (refer to Tips for Older Adults). Invite some of your program’s participants to sit on your evaluation planning committee to ensure appropriateness of tools and relevance of activities. Involving them in the evaluation process shows you value their ideas. Undertaking evaluation shows that you are committed to improving service delivery and supplying evidence for continued or increased program support and funding.

Publications

The Canadian Centre for Activity and Aging (CCAA) systematically evaluates all its programs, covering both programs that are conducted at the Centre and the community outreach programs, and also spanning the continuum from healthy to frail older adults. The book Program Evaluation for Exercise Leaders uses examples from the CCAA, as well as examples from other sectors (such as public health, community recreation, rehabilitation) to help program managers and deliverers, particularly those working with older adults, evaluate their own programs. The book contains prototypes for developing performance objectives, client consent forms and background questionnaires and protocols for contacting absent clients. It also provides a 12-step guide for conducting focus groups, shows you how to protect client information (by assigning unique codes based on the date and order people register), how to select tools for different types of evaluation, and how to manage, analyze and interpret the data.

If you are trying to promote active living in your community, you can learn a great deal from the evaluations of the Community Health Activities Model Program for Seniors (CHAMPS) in California, and the Aging Well and Healthy initiative in the Netherlands. If you work with frail older adults, look at the evaluation articles on the CCAA Home Support Exercise and Functional Fitness Long-term Care Programs. Articles tracking the participation and adherence patterns of older adults in various classes at the CCAA, as well as the one by T.S.Gavin (profiling older adults who join Tai Chi and line dancing programs offered at recreation and seniors’ centres), provide important benchmarks or comparisons for other program deliverers. But remember, you still need to evaluate your own programs.

References and Resources: For the list of references in this article, please contact the ALCOA office or visit our web site. For more information on the book Program Evaluation for Exercise Leaders (ISBN 0-88011-709-5), go to www.HumanKinetics.com or call toll free in Canada 1-800-465-7301.

We would like to thank Liz Cyarto, instructors and participants at the CCAA, for the valuable input on this article. The views expressed are those of the author and not necessarily those of Health Canada.

Back to Research Update