For more than 15 years, Christoph Spurk, lecturer and researcher at the Institute of Applied Media Studies (Zurich University of Applied Sciences), has been evaluating media projects in developing or crisis countries. He tells us about challenges and examples of measuring the impact of information :

How can the impact of a media outlet be measured, especially the quality and usefulness of news and information for the population concerned ?
Within a media project, the first thing we recommend is to measure the quality of the reporting. Then, if that information is good enough, you have a good chance of being able to assess its effect on the audience. But how do you measure the quality of news and information? At the Institute of Applied Media Studies, we have developed a tool to identify within a piece of journalistic output (e.g. article, radio programme) criteria related to quality, such as diversity of sources, diversity of perspectives, existence of opposing viewpoints, and provision of context. The piece is evaluated on each criterion and the scores put into a spreadsheet, which gives an overall evaluation. Studies on Tanzanian media show for example that in many newspapers more than 40% of articles had only one source (or none) and only 9% of the articles mentioned opposing viewpoints. Evaluation of the quality of reporting is based on those kinds of statistics.
To measure the impact of the media information produced, we then conduct surveys with target groups. Ideally we ask about their perceptions and behaviour before and after listening to the programme that is being evaluated. This demands time: the effect of media on perception or behaviour is long-term. It takes years to materialize, if at all. These effects should be measured in comparison with control groups that do not listen to the media in question but have similar social, cultural and economic characteristics to its listeners. Then we can measure the specific impact of that programme.

What are the specificities of measuring media impact in crisis contexts ?
In crisis contexts, we face two major challenges. First, reality changes often and rapidly. People suffer traumas. Their perception of reality is volatile. Second, there are practical difficulties to access people and areas we need to reach. Indeed, media surveys demand time and some stability which is lacking during a conflict.
In 2016, for instance, we conducted a survey on Studio Tamani, Fondation Hirondelle’s radio project in Mali. We achieved quite specific results: in its news bulletins, Studio Tamani succeeds in reporting comprehensively about the conflict (much more than other radio stations) and including diverse sources and viewpoints, but it rarely puts its information in context. On the other hand, its “Grand Dialogue” programme gives guests – who often disagree with each other – a simple but crucial opportunity to clearly explain their views. That somehow compensates for the missing background in news.
In addition, the study shows that listening to Studio Tamani makes its audience more likely to favour dialogue as a solution to conflicts.

Has the development of social networks changed the criteria used to measure the impact of the media ?
Not really. Of course we consider social networks but a “like” on a social network is not a “like” in real life, it is not sufficient as appreciation of a media content to be taken into account in our quite detailed media measurements. I think social networks are themselves a new media genre and need to be analyzed as such: what kind of content do they produce, what kind of conversation is conducted there, what effect do social media have on people’s perception and behaviour? But with regard to Africa, we have to acknowledge that social media remain marginal for large portions of the population in this continent, except the young, urban, formally high-educated: a 2016 survey on media daily behaviour in rural areas in four African countries showed that an average 80-90% of people were listening to radio, 20-40% watching TV, less than 10% reading newspapers, less than 3% using Internet or social media

For more than 15 years, you've been evaluating media projects implemented in developing or crisis countries. What lessons do you draw from your research for Western media and journalists?
We’ve been observing in the West the same trend as in Africa: a journalistic treatment of events more than issues. As media financing is decreasing, a succession of fact-telling has gained ground compared to long-term investigation: we have an explosion of information, 24h, but to say the least it often does not increase our understanding.
Regarding media effects, we need to consider what we know from years of media research. Up to the 1930s, with the view to war propaganda, media were considered as very powerful, like a “magic bullet”. That view changed during the next decades, when media researchers analysed media effects as almost to the contrary, namely as small and weak, because humans tend to block any content contrary to one’s mindset. Today, the “contingent theory” is dominant, saying that media effects are specific. They depend on the characteristics of the message, the characteristics of the (active) recipient and the (changing) context. Violent movies may not trigger violent behaviour on everybody but only on the consumers that are more receptive, for example on young, male, emotionally and socially fragile parts of the population. So, today we are almost sure that media effects can be medium up to strong, but they depend on the audience and they have different time frames. Media can influence knowledge in the short term, but changing attitude or behaviour needs much more time, at least 3-5 years.


Interview realised by La Fabrique Documentaire for Fondation Hirondelle’s quarterly newsletter, n°54, “Measuring the utility of information”. The complete issue can be downloaded here : https://hirondelle.org/en/newsletters/download/75_bb6d405d2bfdff1697c1eb24443101e7