The author is a Senior Research Associate at the Institute for Learning Innovation, Annapolis, Maryland, USA.

Citation: This paper may be cited as: [Author.] 2005. [Article title.] In Ted Trzyna, ed., The Urban Imperative. California Institute of Public Affairs, Sacramento, California.

1.  INTRODUCTION

Museums in many parts of the world are government-funded and hardly need to worry about audiences, both in terms of number of visitors as well as quality of the museum experience museums provide their visitors. However, museums in some countries, notably the United States, are operated by non-profit organizations and rely on fees and, more and more, outside sources of funding. A typical museum in the United States receives its operating income from ticket sales, corporate donations, endowments, special events, and – increasingly – grants, either from governmental agencies or private foundations (Falk and Dierking 1995; 2000; 2002).

Museum visitation is comparatively high in the United States. According to the U.S. National Science Board (2002), in 2001 the average American adult visited a science-related museum, zoo, aquarium, or other science-related informal learning environment approximately three times, and 66 percent of the adult population stated they had visited such an institution during the preceding year. In comparison, according to the European Commission (2001), only 37 percent of Europeans in 2001 stated that they had visited similar informal science institutions during the previous year. While these are impressive numbers, visitorship has declined after the terrorist attacks of September 11, 2001. The concurrent decline in world stock markets and the associated shrinkage in corporate donations led to serious monetary shortfalls in many U.S. museums. The questions since have become: How can we attract more visitors? How can we attract repeat visitors? And, most urgently, museums have been asking themselves: How can we attract those who currently opt not to visit museums?

Attracting current non-visitors has become a problem for museums on another front as well. Many funders of museum programs or exhibitions require a high level of accountability for the grants they give, be they the National Science Foundation (the premiere U.S. federal government agency for science), the Howard Hughes Medical Institute (a major private source for science education initiatives), or the National Institutes of Health (a U.S. federal agency that funds health communication initiatives).  Almost all major sources of grant funding for museums require that an “evaluation” be conducted. The major purpose of evaluations is not whether funds received were spent for the purpose they were granted. Evaluations also don’t assess whether a museum that promised to build 20 exhibits ultimately built them or whether a science program at a zoo actually attracted the promised visitorship. These input/output measures are only part of evaluations. In addition, funders like to know whether the initiatives they fund achieve the broader learning objectives that formed the basis for the grant proposal. Funders of urban outreach initiatives, for instance, might not only be interested in output-related measures such as the number of people reached by such an initiative, but might also like to know how the initiative impacted the thinking and behavior of the people who were reached by the urban outreach. 

As part of almost any proposal, however, museums must now state to what degree their programs or exhibits address the need to attract what is commonly referred to in the United States as “underserved communities.” This term is a euphemism for “poor people” and immigrants. In urban centers of the U.S., these are mostly low-income people of color, and new immigrants with limited command of English.

Not only do many funders require that grantees address how they would serve these “underserved communities” when receiving grant monies; individuals who comprise the vaguely defined target group of “underserved visitors” are also the typical “non-museum visitor” (and, when they do come through the door, are referred to as “non-traditional visitors”).  Museums in the U.S. are thus forced by their need to attract new audiences and the need to satisfy funding requirements to service groups which, for whatever reason, have traditionally not visited museums.

Initially, the need for evaluation was seen by the museum community as a burdensome requirement, but museums have increasingly come to understand evaluation not only as a useful tool for accountability, but also as a tool for planning and management. 

This article will explore three aspects of museum evaluation that might be of relevance to the protected areas community. First, it discusses general ideas behind program evaluation, ideas that will make it clear why evaluation is as much a planning tool as it is an instrument to demonstrate outcomes to external parties. Second, the article will touch briefly on how evaluation can be used to prove program effectiveness: to satisfy funders, but even more so to ensure that programs are successful. Third, the article will discuss lessons learned from evaluation studies of museum programs that addressed issues similar to those likely to confront the protected areas community when it engages in urban outreach.

2.  WHAT IS EVALUATION, AND HOW IS IT RELEVANT TO URBAN OUTREACH?

“Evaluation” is done to ensure that programs are on track and successful. Good evaluations do that by measuring achievements or outcomes against goals and objectives during crucial stages of program development. Evaluation is thus not simply a tool to assess program outcomes and document success or failure; it can also be used to ensure program success throughout the process of program development. In order to serve such a broad purpose, evaluation research should be embedded into program development and defining such important program milestones as:

• Identifying target audiences (e.g., the entire urban population, schoolchildren, residents near certain urban protected areas, decision-makers in the community);

• Defining realistic goals and objectives for each specific target audience;

• Identifying all important stakeholders in the project and involving them in the process by asking about their needs and perspectives;

• Creating programs and program elements that address each objective for each of the target audiences (if there are more than one);

• Avoiding programs and program elements that cannot be mapped to an objective or goal;

• Testing all intermediary steps and products to ensure their efficacy;

• Documenting the degree to which the program has achieved its goals and objectives for each of the previously defined target audiences.

Evaluation should be seen as an integral part of any project, no matter whether an outside funding agency requires it or not, since it allows program staff to stay goal-oriented and keep the audience in mind.

In the museum field it is customary to identify four stages of evaluation. Only the last of these stages, the “summative” evaluation, actually documents the outcomes of the entire project and is normally conducted by an outside, neutral evaluator. The three prior stages simply guarantee program success. The four stages are: 

(a)  Front-end evaluations. These provide input to decisions about how to develop a program in advance of the planning stage.

(b)  Formative evaluations. These provide information to improve the program during the design and development stage.

(c)  Remedial evaluations. These provide information to improve the program during implementation and allow for corrections once programs are underway.

(d)  Summative evaluations. These assess outcomes or impacts of a settled program.

In fact, the first three stages of an evaluation can be, and often are, done by program staff, usually aided by outside consultants or evaluators. Evaluation as described so far is often associated with written surveys or interviewers armed with clipboards. However, good evaluation uses the entire spectrum of educational and social science research methodology, from quantitative surveys to face-to-face interviews, from observations to focus groups, and ideally triangulates research methods to produce robust results.

Front-end evaluation

A number of fundamental questions arise at the beginning of a program or exhibition design, which can then become the focus of a front-end evaluation. The results from the front-end evaluation will lead to more informed decisions about the general direction and objectives of a project and its overall implementation. Front-end studies are generally surveys or involve focus groups and answer such questions as:

• Is there a need for this program?

• What do we know about the needs that the program will address?

• What is recognized as best practice in this area?

• What does the relevant research or conventional wisdom tell us about this need?

• What do likely audiences already know, and what do they not know?

• More specifically in the area of urban outreach, questions might include:

• Who are visitors and non-visitors to urban parks and protected areas?

• What are people’s understandings and misconceptions regarding conservation, biodiversity, and habitats?

• What attitudes and beliefs do urban populations hold about urban parks, protected areas, and nature conservation?

• What understandings, attitudes, and beliefs do members of the protected area community have about possible visitors or supporters of protected areas?

• Who has conducted successful urban outreach initiatives and what can be learned from them?

Formative and remedial evaluation

The purpose of a formative evaluation is to assess ongoing project activities at several points in time to provide information for program improvement. It asks:

• How is the program going?

• Is the delivery working?

• Is it consistent with the program plan?

• How could the delivery be changed to make it more effective?

This kind of research can take place during the early conceptual design phase all the way through the early days of program implementation, when it is commonly referred to as “remedial” evaluation. Formative studies can also seek information on whether or not the program, or more often specific program elements, are meeting the stated goals or objectives. It is a “rehearsal” of sorts for the summative (outcome) evaluation.

Early stages of urban outreach program development might raise the following questions (although more specific questions for program delivery depend on the precise nature of the outreach activities that are part of the program):

• Do target audiences understand the specific outreach messages, and are these messages relevant?

• Do outreach activities reach the target audiences?

• What specific outreach approaches or techniques do audiences respond to most positively? What are their preferences or levels of satisfaction?

• Do audiences enjoy the outreach activities, programs, or products?

• What benefits do visitors perceive that they derive from experience with the project or subject matter?

• To what degree is the project meeting established goals and objectives? To what degree do program elements and individual activities and products achieve their intended goals and objectives?

Summative evaluation

The purpose of a summative evaluation is to assess the effectiveness of the program as it relates to the initial goals and objectives. In short, did the program do what it was intended to do? This type of study is conducted after or very near the end of an initiative and answers such questions as:

• To what degree does the program achieve goals and objectives?

• What direct or indirect impact, short-term and long-term, does the program have on the target group?

• To what degree can changes in the target groups’ understanding, attitudes, emotions, or behaviors be attributed to their experience with the program?

• Have the needs of those served by the program been fulfilled?

• What are unintended outcomes (positive as well as negative) of the program?

The results of a summative evaluation depend only in part on the quality of the program. Other factors that influence an outcome evaluation are:

• Have short- and long-term goals and objectives been well defined and serviced by program elements? Were those goals and objectives realistic?

• Has the target group been well defined, and has program design specifically addressed the target group?

• Are evaluation measures valid and credible, and do they actually measure what they purport to measure?

• Do evaluators, program designers, and funders agree on outcome measures?

Summative or outcome evaluations can help urban outreach efforts in many ways. Among others, summative evaluations:

• Document the outcomes of urban outreach programs and thereby provide crucial information on tasks accomplished and challenges still to be faced;

• Identify what works and what doesn’t work, and what allows the community to learn and grow;

• Ensure that outreach efforts are done systematically and are outcome-oriented;

• Provide the target groups’ perspective and thus ensure that outreach efforts actually address audiences’ perspectives;

• Provide a level of accountability to funders and the community that creates trust and should ultimately lead to sustained support. 

Understand what urban residents associate with protected areas

Definitions of “parks” and “protected areas” used by the protected area community might not be aligned with expectations of urban residents. Expectations stem from experience, and people’s responses need to be assessed based on the context of their life experience. Urbanites, for instance, might associate “parks” with recreational activities such as sports or barbecuing, rather than such functions as protection of biodiversity. Protected areas that do not fulfill urbanites’ expectations for well-managed recreational parks might even be considered dangerous, unaesthetic, or wasteful and underutilized. It is thus important to first elucidate what urban residents understand by “protected area” or “urban park.”

Understand who the “true” audience is

One major difference between “museums,” broadly defined, and protected areas, be they in urban settings or elsewhere, is that the protected area community is seeking support from people who may not personally experience the protected areas. In contrast, museums are striving to gain more, or more diverse, visitors, and use their visitor statistics to gain funding and support from donors, foundations, and/or governmental agencies. However, museums and protected area agencies share a desire to have citizens understand the value of what they are trying to accomplish. Within the museum profession there is a new movement to think about the role museums play within the larger communities of which they are part, both as a moral imperative as well as a long-term strategy for sustained support through relevance. Clearly, museums, parks, protected areas, and other community resources create value within the larger context of the community, although the total benefit within that community context might not be well known  (especially in the case of museums), or might not have been communicated well (especially in the case of protected areas).

Protected areas, for instance, might not seek to have more visitors, and certainly do not seek support only from visitors. Instead, appreciation and support for protected areas is sought from a range of stakeholders for benefits that may not be immediately personal.  Thus, those responsible for protected areas should ideally determine at the local level the answers to these questions:

• Do people know what protected areas are? Are they aware of the difference between “parks” and “protected areas,” and between “protected” and “wilderness” areas?

• Who benefits from protected area management, and are benefits broadly disseminated and communicated within the urban community?

• Are beneficiaries of protected areas aware of the benefits?

3.  HOW TO USE EVALUATION TO PROVE PROGRAM SUCCESS AND PROGRAM EFFECTIVENESS

Setting realistic goals and objectives

As described above, any assessment of success needs to start with objectives. Success should ideally be measured against goals and objectives, and funders increasingly want measurable objectives against which results can be evaluated. Hence, the first step of any project planning is defining realistic goals and objectives. Ideally, it is these goals and objectives that determine program development, implementation, and evaluation. Urban outreach projects could embrace a range of objectives and goals, some of which are stated below in very generic terms (and should never remain stated in such generality for any specific outreach program):

• Increased knowledge of urban populations about the benefits of parks and protected areas;

• Eradicated misconceptions about parks and protected areas;

• Increased awareness of parks, protected areas, and their benefits among crucial stakeholders;

• Increased appreciation for parks and protected areas;

• Increased willingness to support parks and protected areas;

• Increased financial and non-monetary support for parks and protected areas.

Revising goals and objectives

Funders commonly write lofty goals into their funding requirements. Organizations seeking funds often include these goals in their grant proposals and later realize that the goals were actually unattainable. Science museums and science centers have often claimed that their efforts increase their audiences’ understanding and appreciation of science. And environmental and conservation education often claims to influence audiences’ environmental awareness or attitudes. But attitude and value development is a slow process, and when measured, depend on the particular state of mind of the person whose attitude change is to be measured (Ajzen and Fishbein 1977; 1980; Diekmann and Preisendörfer 1992; Fliegenschnee and Schelakovsky 1998; Hines, Hungerford and Tomera 1986-87; Kollmuss and Agyeman 2002; Zelezny 1999; Zimmermann 1996). Claims of changing deeply ingrained aspects of the human mind and soul need to be revisited and, if necessary, restated or changed.

Museums and nature centers in the United States have learned through numerous evaluation studies and much research that their impact on audiences depends on how much, relative to other factors, they actually interacted with their audiences. For example, objectives need to be calibrated depending on whether audiences experience an outreach effort for 10 minutes, 30 minutes, 2 hours, a day, or weeks, whether they have a one-time experience or repeated experiences, whether the experience is personal, such as a guided tour through a protected area, or a simply a leaflet or public-service announcement on radio or TV (Falk and Dierking 2000).

Create programming that addresses the objectives

Funders prefer a sequence of logical program steps that start with clearly defined objectives for clearly defined audiences, proceed to activities that address each and every objective for every possible target group, define measurable outcomes for these objectives, and then set out to measure them once the program is completed or nears completion. Thus, the crucial next step in program planning is mapping objectives to activities. In brief: What is actually being done to achieve the objectives? Are all objectives addressed in some way by the intended programming?  (Objectives that cannot be linked to programming should be dropped!)  The easiest way to ensure that each program objective is linked to an activity – and, almost as important, that each activity actually addresses a program objective – is to create a matrix of goals and objectives on one axis, and programming and activities on the other. Are all of the cells filled? Where are most of the programming efforts placed?  Ideally, this exercise, commonly referred to as Program Modeling, will also map each goal or objective to at least one measurable outcome, and ultimately specific measures for each outcome.

Define the target group(s)

Objectives need to be defined for various target groups, and target groups can be rather different when it comes to promoting protected areas to urban populations. Possible target groups include, but are not limited to:

• The urban population in general;

• Decision-makers;

• Stakeholders;

• Multipliers (for instance, teachers, community activists, etc.);

• Media and other opinion-makers.

People experience an urban outreach program through their own personal lens, which is shaped by their prior knowledge and beliefs, their attitudes and awareness, their receptivity, and other personal factors. Hence, it is often useful to define a range of different sub-groups within a target group, both to focus program delivery, as well as to make evaluation efforts more effective, since no evaluation will show significant results when programming hasn’t actually served the needs of the audiences. It is thus necessary to distinguish among:

• Those who already know a lot from those who don't;

• Those who are generally supportive from those who are not;

• Those who are already aware from those who are not; and

• Those who already provide support from those who don’t.

Program outcomes should be defined for such sub-groups, and program success should not be assessed by changes in the “average audience,” because an “average audience” does not exist (Prochaska, DiClemente, and Norcross 1992).

Conduct the final summative evaluation

Once goals are clear and are based on front-end research, target groups have been defined, programming has been linked to goals, and sufficient testing has been done to ensure that programming is of high quality, and once the program has been running long enough or comes to its conclusion or outcome, summative evaluations can be conducted that document program success (or failure, as the case may be).

At this juncture, it is important that program staff and the evaluator agree on measures that will validly and reliably capture the outcomes. “Increased knowledge about protected areas’ role in preserving biodiversity,” for instance, should only be a goal if it was directly addressed by programming the issue. And if it was, it is important to realize that there are many different ways to capture knowledge change, not all of which might be appropriate for measuring urban outreach efforts. It is also important to capture unintended outcomes, since the audience and the community is not necessarily restricted to the original objectives, and might take away other, surprising, yet useful aspects from the program (Storksdieck, Ellenbogen, and Heimlich, 2005). In general, research design and research methods should be chosen carefully and from the full spectrum of the social science research repertoire. The old adage holds that if the only tool available is a hammer, all of one’s problems will look like a nail!

Evaluation done in the context of protected areas will mostly be applied research, and resources to conduct such research will be extremely limited. Thus, it is important to conduct the studies efficiently and effectively, which means that:

• No more data than can be analyzed should be collected.

• Data should not be collected because it “would be nice to know.” Only data that will help answer a research question should be collected.

• Making sense of data takes time and experience. It is paramount that enough time be scheduled for this most crucial stage of research. Researchers and program staff often underestimate the time it takes to make sense of data and prepare reports.

• Reports are read by people who have agendas. It is thus important to consider the addressee when developing conclusions and writing the report. In the museum field, addressees range from funders to museum directors, from program developers to marketing specialists.

In the museum field, evaluations are used as a tool to keep projects on track and report outcomes to funders (in fact, evaluation results from one project are often used to gain support for another project).  Evaluation is particularly successful when a few deceptively simple-sounding rules are followed:

• Know your audience and slice your target audience well.

• Develop realistic expectations in funders, and define achievable goals, objectives, and outcomes.

• Create programs that actually provide for experiences that will address the potential changes you envision in the audiences.  Always keep your eyes on your goals.

• Develop a research design and methods that actually measure valid outcomes.

• Choose methods that answer your questions.

• Remember why you do evaluation.

Learning from museum evaluation

Two major areas of museum evaluation might be of relevance to the protected area community. The first one involves the perceived need of museums to reach out to new audiences. Museums in the U.S. (and elsewhere) attract mostly middle- to upper-class, well-educated audiences. A variety of museum initiatives have been conducted, mostly in partnership with community organizations, to reach new, non-traditional audiences and increase appreciation for museums as urban educational resources within those audiences. The second one involves efforts of nature centers, science centers, zoos, and aquariums to convey increased appreciation for urban green space, urban habitats, and nature in the city to urban populations.

The primary factors that connect these projects (or, in fact, any initiative that intends to create an appreciation for something that was not appreciated or even known) were:

• A need to know the audience, their priorities, their concerns, their interests, and their knowledge prior to embarking on educational and outreach initiatives;

• A realization that target audiences are best reached when programming is flexible and allowed to change course where and when needed;

• A need to base decisions on solid data (rather than ideology or gut-feeling), and to always align goals with audiences and program activities.

More specifically, lessons learned through program evaluation in museums and similar institutions that could potentially be transferred to protected areas include the following:

• Solid research into people’s motivations, needs, and constraints, from their perspective, is essential.

• Barriers for support and appreciation might not be obvious. For example, in many cases underserved audiences would not visit museums even when visits were free of charge and free food was served. It became apparent that such audiences were non-visitors for many, other than financial, reasons. Support for protected areas from audiences not traditionally inclined to support them might be forthcoming only if the benefits are personally relevant to their everyday lives, and communicated in such ways as to ensure that they understand the personal benefits derived from protected areas.

• It must be realized that cultural barriers exist for many urban populations. Visiting parks or supporting protected areas are lifestyle issues in the same way that going to a museum and becoming a museum member are lifestyle issues. Knowing about the benefits and even importance of parks and protected areas does not make them any more of a priority for those who struggle with other problems.

• For non-traditional audiences, museum visits often mean leaving their neighborhoods, something they are not accustomed to doing and prefer not to do. Another way of putting this is that some populations are not as readily willing as others to leave a zone of physical or mental comfort. Translated to urban outreach efforts, this finding might signify the need to design program locations around audiences’ comfort zones and primary means of receiving information.

• Urban outreach programs need to communicate forcefully that local communities are welcome to join the protected area movement. Outreach cannot be an afterthought or add-on if it wants to be successful.

• If people don’t go to parks and protected areas, and thus are less likely to appreciate them, bring the parks and protected areas and their benefits to the people by conducting urban outreach programs jointly with schools, churches, and other community organizations.

• Create tangible and personal benefits, even if they are relatively small. Providing neighborhood amenities concurrently with urban protected areas helps residents see that such areas benefit them in their everyday life.

• Conduct field trips for urban people to parks and protected areas.

• Forge strong coalitions with urban community groups that are rooted in the city. Make sure you align your goals with theirs. Ensure that they have the needed resources to participate in your urban outreach efforts. Use community partners to gain insights into community members’ perspectives and opinions.

• Nature, habitats, and green space might hold different meanings for urban populations and might even be seen as dangerous and unattractive. Be aware that people who grew up in the built environment might prefer concrete over grass. Don’t take nature appreciation for granted.

• People who are aware of a protected area or park, and who might normally be willing to visit such areas, might not visit because they take the resource for granted. In a non-visitor study for Fort Snelling (a historic site in Minnesota, USA), respondents were asked whether they knew about the fort and how they knew about it (Lanning and Elton 2001). It turned out that many locals took the existence of the fort for granted and did not visit since they felt no urgency to visit the venue: it would still be there next year.

Museums in the United States now embrace evaluation as a valid and useful tool for better programming. Evaluation is no longer done simply because a funder demands it. Instead, it is seen as an opportunity for institutional growth. Missing so far is broad dissemination of evaluation results. Inasmuch as evaluation becomes second nature in the development of urban outreach programming, the protected area community should be willing to share the lessons learned. This should include failures as well as success stories, since it is from the failures that we learn the most.

4.  REFERENCES

Ajzen, I., and M. Fishbein. 1977. Attitude behavior relations: A theoretical analysis and review of empirical research. Psychological Bulletin 84: 888-918.

Ajzen, I., and M. Fishbein. 1980. Understanding Attitudes and Predicting Social Behavior. Englewood Cliffs, New Jersey: Prentice Hall.

Diekmann, A., and P. Preisendörfer. 1992. Persönliches Umwelthandeln. Diskrepanz zwischen Anspruch und Wirklichkeit. Kölner Zeitschrift für Soziologie und Sozialpsychologie 44(2): 226-251.

European Commission. 2001. Europeans, science and technology. Eurobarometer 55.2.  Luxembourg: Office for Official Publications of the European Communities.

Falk, J.H., and L.D. Dierking. 2000. Learning from Museums. Walnut Creek, California: AltaMira Press.

Falk, J.H., and L.D. Dierking. 2002. Lessons without Limit: How free-choice learning is transforming education. Walnut Creek, California: AltaMira Press.

Falk, J.H., and L.D. Dierking (eds.). 1995. Public Institutions for Personal Learning: Establishing a research agenda. Washington, D.C.: American Association of Museums.

Fliegenschnee, M., and A. Schelakovsky. 1998. Umweltpsychologie und Umweltbildung: eine Einführung aus humanökologischer Sicht. Vienna, Austria: Facultas-Universitäts-Verlag.

Hines, J.M., H.R. Hungerford, and A.N. Tomera. 1986-87. Analysis and synthesis of research on responsible environmental behavior: A meta-analysis. The Journal of Environmental Education 18(2): 1-8.

Kollmuss, A., and J. Agyeman. 2002. Mind the gap: Why do people act environmentally and what are the barriers to pro-environmental behavior? Environmental Education Research 8(3): 239-260.

Lanning, J., and B. Elton. 2001. Historic Fort Snelling School Group Survey. (Unpublished technical report.) Minneapolis, Minnesota: Minnesota Historical Society.

National Science Board. 2002. Science &Engineering Indicators – 2002. (NSB-02-1.) Arlington, Virginia: National Science Foundation.

Prochaska, J.O., C.C. DiClemente, and J.C. Norcross. 1992. In search of how people change: Applications to addictive behavior. American Psychologist 47(9): 1102-1114.

Storksdieck, M., K. Ellenbogen, and J.E. Heimlich. 2005. Changing minds? Reassessing outcomes in free-choice environmental education. Environmental Education Research 11(3): 93–109.

Zelezny, L.C. 1999. Educational interventions that improve environmental behaviors: A meta-analysis. The Journal of Environmental Education 31(1): 5-14.

Zimmermann, L.K. 1996. Knowledge, affect, and the environment: 15 years of research (1979-1993). The Journal of Environmental Education 27(3): 41-44.

This paper copyright © 2004 International Union for

Conservation of Nature and Natural Resources.

Reproduction of this paper for educational or other noncommercial purposes is authorized without prior written permission from the copyright holder provided the source is fully acknowledged. Reproduction of this paper for resale or other commercial purposes is prohibited without prior written permission from the copyright holder.

Web site hosted by the California Institute of Public Affairs,

an IUCN member since 1980

Home page  /  Site map  /  Copyright & caveats 

The copyright notice applies to the entire contents of this Web site.