By the Institute for Methods Innovation
When conducting impact evaluation research, you might get results that are the opposite of what you were expecting or hoping for. This is called negative impact. Although this can be discouraging, a negative impact result is often exactly why an impact evaluation was needed in the first place – to see if you are actually achieving your objectives. The London Zoo case study serves as a good example to see why negative impact can yield a positive outcome.
Evaluation research aims to provide helpful knowledge about how and why particular interventions or processes are working or failing in order to inform future decision-making. In other words, it offers a better understanding of your audience, their motivations and expectations. This enables you to decide what aspects of your engagement strategies are working or not, assess whether you are achieving your objectives, and re-design your approach as needed.
In 2013, Prof. Eric Jensen conducted an independent impact evaluation at the London Zoo to determine if the zoo’s engagement activities with school children were having the desired effect, namely, teaching them about the importance of conservation biology and the delicate relationship between animals and their wildlife habitats.
Prior to the impact evaluation, general feedback from the teachers who took the children on class trips to the zoo, was overwhelmingly positive. Photos showed happy, smiling children engaging with the animals, including a ferret, a snake, and a trained Macaw parrot (Figure 1).
After direct observations, quantitative surveys and qualitative interviews with the children themselves, it soon became apparent that the children often paid most attention to the artificial elements of the engagement activity, such as the behaviour of the trained parrot and the ferret. This, in turn, is what they often remembered and recalled after the engagement activities and not the environmental messaging that was intended. See example excerpts of some of the interview conversations below.
Interviewer: Did you learn anything else about the parrot [regarding its habitat and ecology]? Child 1: Yes, well I liked the parrot. I didn’t know that parrots can actually talk. Interviewer: What did you think of that lesson? Did you like it? Child 2: Yes. Interviewer: What did you like about it? Child 2: I liked all of the animals. I liked the snake, the parrot. I liked all of the animals. Interviewer: And what did you... did you feel that you learned anything? Child 2: Yes, I wish I had a snake and a parrot and what are these things called again? Interviewer: A ferret. Interviewer: What was special about the ferret? What do you remember about it? Child 3: I remember about the [carrying] case [for the ferret]. They have a special case. I like how [the ferret] pulls in the biscuits. I liked it how he was pulling in the biscuits and how he found them all. I really liked it. |
From these and other interviews, it became clear that the children were often far more concerned with the particular animals they were allowed to interact with—how they looked and behaved or how they were handled—than with any environmental lesson about the importance of wildlife habitats and ecology.
The children were also asked to make a drawing of their favourite wildlife habitat, both before and after the engagement activity at the zoo. The drawings provided a similar conclusion to the interviews. One child, prior to the visit, drew a forest with tropical birds and monkeys (Figure 2), but after the visit simply drew a ferret eating biscuits (Figure 3).
However, the results were not all bad. The impact evaluation showed that 41% of educator-guided visits and 34% of unguided visits did produce positive, conservation biology-related learning. Nevertheless, 16% of children on unguided visits and 11% on guided visits showed negative learning. These are significant proportions given the millions of children who visit zoos every year making it important to increase the level of positive learning impacts (while reducing negative impacts).
Overall, the results confirmed the potential educational value of zoo visits for children, but also suggest that zoos’ standard unguided interpretive materials were insufficient for achieving the best outcomes for visiting children. These results frame conservation educators as crucial toolmakers who should develop conceptual resources to enhance children’s understanding of science.
The negative impact demonstrated through the evaluation showed that the London Zoo’s engagement activities were too often not having the desired effect and were not adequately achieving the objective that prompted them in the first place.
What do you do if your impact evaluation produces a negative impact result?
The important thing to remember is that negative impact will always be a possible outcome of doing an impact evaluation. Such a result provides the best opportunity for an impactful intervention. In other words, there can always be negative outcomes to an impact evaluation, and when it occurs, it should be targeted for positive, meaningful change.
The impact evaluation done at the London Zoo was invaluable in many ways. It clearly demonstrated that the engagement activities for school children were not ideal for properly educating them and provided the opportunity for zoo management to rethink their school outreach activities to better align with the objective of teaching kids about conservation biology.
If you want to learn practical tips about conducting a thorough impact evaluation, including how to interpret and respond to a negative impact result, book a place in Methods for Change’s self-paced “Introduction to Evaluation Design” course here.
This article is written by members of the Institute for Methods Innovation team, a registered non-profit devoted to improving evidence-based practice.
Article citation Jensen, E. (2014). Evaluating children's conservation biology learning at the zoo. Conservation Biology, 28 (4). pp. 1004-1011. Read or download the research article here. |
Comentários