HPSR Teaching Blogs

If you have ever taught health policy and systems research (HPSR) at post-graduate level, you will know both the frustrations and intellectual “a-ha” moments that can result from having a classroom of students who have been trained in very different ways.

These differences can manifest in questions about course content (“Why are there so few objective facts and so many competing perspectives?”), difficulties with assignments (“I am not used to writing essays”), as well as bafflement (hopefully followed by clarity) around concepts that are taken for granted by some, but quite foreign to others.

This blog reflects on the concept of analytical generalisation – an idea that often comes up in HPSR methods courses and that can seem unusual to those most familiar with statistical generalisation from samples to populations. In this video, Dr Thameshree Naidu, a participant in CHEPSAA’s course Introduction to Health Policy and Systems Research, describes what an eye-opener the concept was to her.

Why is it relevant?

In general, the value of research may be judged not only by what it says about its particular question, population or setting, but also by its broader implications for other studies, populations or settings⁽¹⁾. Specifically, applied disciplines such as HPSR are often interested in evidence-based practice, which involves applying findings and lessons to people and situations in different times and contexts⁽²⁾. These concerns mean that we have to think about the generalisability of our work.

In thinking about generalisability, one question among many is how we can generalise from qualitative work and associated research designs such as case studies – both of which are key to HPSR’s repertoire. Analytic generalisation is an important, although not the only, answer to this question.

How does it work?

Analytic generalisation involves two key steps. First, the researcher shows how the particular findings from a case study or qualitative investigation relate to a theory or theoretical construct. Second, this same theory or theoretical construct is applied to other people and situations^(1,2). Therefore, generalisation does not occur because one study is a smaller sub-set of the other or because the people in the one study have the same demographic characteristics as in the other, but because the same analytical or theoretical insights and relationships hold in different times and places.

Analytic generalisation happens in many research fields. In international relations, one example is the case study of the 1962 Cuban missile crisis that involved conflict between the United States and the ex-Soviet Union; the facts of which were related to broader theory about decisions in confrontations between superpowers that could then be applied to other international confrontations between other states⁽¹⁾.

Another example concerns work on labour unions, where there was a well-established generalisation that they tended to be run by small groups with little power being shared with ordinary members. Researchers then identified a “deviant” case of a union that was managed much more democratically than the theory predicted, thus using the facts of the case to outline the mechanisms that enabled it to be run differently and using the “deviant” case to refine and limit the scope of the original theoretical generalisation⁽³⁾.

An HPSR example is the authors who have related the actions of different types of frontline implementers (e.g. community-based contraception distributors, nurses, environmental health officials, district managers), implementing different policies (e.g. family planning, user fee removal, community-based health insurance) in a variety of countries (Kenya, South Africa, Ghana, Tanzania) to the theory of street-level bureaucracy^(4,5,6,7). Street-level bureaucracy was first developed in the context of the United States and encompasses a wide range of frontline government workers way beyond the health system, including teachers, police officers, welfare officials and judicial officers. In part through analytical generalisation, it was possible for this theory to be developed and to travel to a different time and a host of other settings.

Improving our research, improving generalisability and building HPSR

Authors have written about research practices that undermine the credibility of analyses and therefore the strength of analytical generalisability. Chief among these are generating artificial coherence by connecting superficial similarities in data, having a flash of insight and then mistakenly assuming that was all there was to discover about a topic, and stopping analysis when it is convenient rather than when full saturation has really been achieved^(2,8).

They have also made many suggestions about how to strengthen analytical generalisation in particular and generalisation more broadly, such as stating the relevant theory at the outset of the research, making sincere efforts to collect data that might refute the theory, working beyond your single case study or project to look for similar results elsewhere, and seeking to ensure that descriptions of research processes and findings are “thick” enough to enable transfer and broader application^(1,2).

Many of these suggestions are completely in line with identified weaknesses in the field of HPSR and calls for improved practice, including the need for more theory-driven work, providing better contextualisation for case studies and richer analyses of health systems and policy processes, and making better use of all the existing case studies by synthesising across them^(9,10,11).

Building the field of HPSR undoubtedly requires us to think about how we can improve our research practice and strengthen our claims about the broader relevance of our work, both to increase its value generally and to increase the chances of our “evidence” being of interest and use to broader audiences. It also requires us to continue our dialogues and negotiations, whether in the classroom or beyond, about those concepts we might not always understand at first or agree with, but that nevertheless occupy central places in some of the nooks and crannies that make up the broad community that is HPSR.

Ermin Erasmus, CHEPSAA coordinator

(1) Yin, R.K. (2010). Analytic Generalization. In Mills, A.J., Durepos, G. & Wiebe, E. (eds.), Encyclopedia of Case Study Research. Thousand Oaks, CA: SAGE Publications, Inc.
(2) Polit, D.F. & Beck, C.T. (2010). Generalization in quantitative and qualitative research: myths and strategies. International Journal of Nursing Studies, Vol. 47: 1451-1458.
(3) Firestone, W.A. (1993). Alternative arguments for generalizing from data as applied to qualitative research. Educational Researcher, Vol. 22(4): 16-23.
(4) Walker, L. & Gilson, L. (2004.) ‘‘We are bitter but we are satisfied’’: nurses as street-level bureaucrats in South Africa. Social Science and Medicine, Vol. 59: 1251–61.
(5) Kamuzora, P. & Gilson, L. (2007). Factors influencing implementation of the community health fund in Tanzania. Health Policy and Planning, Vol. 22: 95–102.
(6) Kaler, A. & Watkins, S.C. (2001). Disobedient distributors: street-level bureaucrats and would-be patrons in community-based family planning programs in rural Kenya. Studies in Family Planning, Vol. 32: 254–69.
(7) Crook, R. & Ayee, J. (2006). Urban service partnerships, ‘street-level bureaucrats’ and environmental sanitation in Kumasi and Accra, Ghana: coping with organisational change in the public bureaucracy. Development Policy Review, Vol. 24: 51–73.
(8) Thorne, S. & Darbyshire, P. (2005). Land mines in the field: a modest proposal for improving the craft of qualitative health research. Qualitative Health Research, Vol. 15: 1105-1113.
(9) Gilson, L. & Raphaely, N. (2008). The terrain of health policy analysis in low and middle income countries: a review of published literature 1994–2007. Health Policy and Planning, Vol. 23(5): 294-307.
(10) Erasmus, E., Orgill, M., Schneider, H. & Gilson, L. (2014). Mapping the existing body of health policy implementation research in lower income settings: what is covered and what are the gaps? Health Policy and Planning, Vol. 29 (Supplement 3): iii35–iii50.
(11) Gilson, L. (2014). Qualitative research synthesis for health policy analysis: what does it entail and what does it offer? Health Policy and Planning, Vol. 29 (Supplement 3): iii1-iii5.