Ethics statement

All interviews, observations and surveys were conducted after obtaining written informed consent. Ethical approval was received from University College London Research Ethics Committee (ref: 3433/003); BADAS Ethical Review Committee (ref: BADAS-ERC/EC/19/00274), Ethiopian Public Health Institute Institutional Review Board (ref: EPHI-IRB-240-2020), National Health Sciences Research Committee in Malawi (ref: 19/03/2264) and Makerere University Institutional Review Board (ref: Protocol 869).

To evaluate the work of the QCN, we conducted multiple embedded case studies [12] in four countries–Bangladesh, Ethiopia, Malawi and Uganda–chosen for the spectrum of contexts they represented. This research design allowed us to investigate in more depth factors that might influence the effectiveness of the network by conducting a cross-country analysis of our case studies [12,13]. Each case study comprised embedded units of analysis: the national, sub-national and local levels, as well as the country’s interactions with the global level of the QCN. Within each unit of analysis, we further delved into and triangulated a multitude of perspectives (such as governments, implementing partners, health facility workers and other partners of the network) and data sources (detailed below) since the inception of the network in 2016 until March 2022. We describe the case study settings in more detail in S2 Text.

To evaluate how the network operated, we conducted in multiple rounds: interviews with key QCN actors across different levels of the health systems; observations of QCN meetings and learning health facilities; a quantitative survey to understand QCN actors’ awareness and perceptions of the network in each country; and analysis of documents related to QCN operationalisation and implementation. Semi-structured interviews were conducted with QCN actors at the global, national and local levels and included participants from diverse backgrounds and affiliations: members of the Ministry of Health (MoH), implementing partners (e.g. UNICEF, USAID), health facility workers (managerial and clinical staff), technical partners, academic partners, WHO and other global partners. We sought to pay particular attention to the perspectives and goals of those carrying out the work of the network [14,15]. In total, 248 interviews were conducted–including follow-up interviews of key stakeholders–over several rounds of data collection between March 2019 and March 2022, across all levels of governance in our four case study countries, and globally (see S2 Text for breakdown of interviews per case study and round of data collection). To accommodate local restrictions related to the global pandemic, some of the interviews at global and national levels were conducted online or by telephone [16]. Initial topic guides were developed for global, national and local stakeholders that were then adapted to each country’s context and translated in local languages as appropriate. Topic guides were further refined for each round of data collection to validate emerging findings; follow-up on QCN progress and emerging lines of inquiry; and to be more specific for the category of stakeholders. Interviews were conducted by several members of the QCN Evaluation Group trained in qualitative methods and familiar with the local contexts and languages. Interviews were then transcribed verbatim and translated into English, if not conducted in English. Interviews were complemented by non-participant observations [17] of international QCN meetings as well as national and sub-national QCN meetings in case-study countries. QCN activities were further observed in each country via visits to two best and two least performing QCN health facilities that were selected based on maternal and newborn health outcomes and other quality of care data used in national schemes relevant for each country. Non-participant observations served several purposes, such as: anchoring the case studies in the real-world setting of the cases [12]; exploring why QCN activities are done (or not) and how those changed over time; probing for any effects and the veracity of QCN monitoring data; and triangulating the data [17]. Observations were completed over several rounds of data collection between March 2019 and March 2022 (see S2 Text for breakdown of observations per case study) by members of the QCN Evaluation Group trained in qualitative methods and familiar with the local setting (culture, language, context). We used templates (see S2 Text) to capture key processes relevant to the focus of the network in each country during observations, as well as unstructured field notes. The focus of observation was sharpened over time enabled by iterative rounds of data analysis and reflection.

Additionally, we adapted a psychometrically validated tool (5 domains, 40 indicators) developed for evaluating clinical networks [18] to evaluate the network at national and local levels in each case study. Over several rounds, we surveyed a variety of network members (e.g. clinicians, managers, advisors) that also included QCN actors beyond our observation sites, totalling 1525 responses across the countries and rounds of data collection (see S2 Text for a breakdown per case study). Respondents had an option to fill in the survey online, via the Opinio platform, or on paper. Finally, we triangulated the data collected with a document review that included all relevant published and unpublished documents and communications relating to QCN at the global, national and district levels in the case study countries. These included strategy and management documents, operational plans, directives, formal minutes, and reports (see S2 Text). We were able to access unpublished documents via WHO and Ministry of Health QCN contacts.

All qualitative data was analysed using a common coding framework developed from several underlying theories that framed the overall QCN evaluation (see S2 Text). In this paper, our analysis was guided by the QCN Theory of Change and monitoring framework [8]–the Leadership, Action, Learning and Accountability (LALA) strategic objectives of QCN–and the environment, structure, process and outcomes of the QCN [19]. All data was coded in NVivo 12, drawing on initial theory in both an inductive and deductive way [17]. Our codebook contained ‘theory’ codes related to underlying theories; each theory was outlined using codes and sub-codes that broke down the different components of the theory. The codebook was further supplemented by ‘case study’ codes to distinguish data specifically relevant to each case study. We describe in more detail how the codebook was developed, piloted and tested by researchers from the QCN Evaluation Group in S2 Text. We worked to ensure consistency in our qualitative data collection and analysis via holding regular interactive group training sessions with all researchers in the team, led by senior members of the team. Over two years, many of our research team, including nine co-authors for this paper, were involved in coding data in the different case studies. Six of the co-authors were also actively involved in local data collection and familiar with the local context. Regular team meetings and rechecks took place during each round of coding in order to ensure inter-coder consistency and that coders remained close to the theories underpinning the codebook. Team meetings were also an opportunity for coders to put forward new codes they deemed relevant to refine the ‘theory’ codes or to address gaps in ‘case study’ codes. The quantitative data was analysed independently in the first place, by conducting descriptive statistics of respondents and the percentages of respondents giving responses to each question. In S2 Text, we provide a more detailed explanation of the quantitative data analysis process. After analysis, the results of the survey were triangulated with the qualitative data.

QCN Evaluation group members who were from, and based in, the countries being researched, conducted the interviews in each country. They were experienced qualitative researchers familiar with the culture, language, context and health system in the country, though not working in the health system or the particular health facilities involved. As such they were non-participant observers and interviewers. We aimed to limit Hawthorne effects–behaviour change in response to being observed–by conducting observations over a number of consecutive days in each selected facility and in a non-obtrusive, non-judgemental way, with our researchers making it clear to those being observed that all data collected would remain anonymous and that there would be no consequences of the observations for those being observed.

QCN effectiveness at the global level

Network activities at the global level, through the QCN secretariat led by WHO, focused on supporting countries with the LALA framework, developing standards and indicators for quality of care, providing technical assistance to MoHs and facilitating learning between countries. The QCN secretariat steered the formulation and distribution of standards, guidelines, and technical documents for quality of care (QoC). In particular, the WHO co-developed with global QCN partners (such as UNICEF, USAID, IHI, Jhpiego, URC ASSIST) and countries: the QoC standards for maternal and newborn care in 2016 –prior to the official launch of the QCN [22], the QoC standards for children and young adolescents in 2018 [23] and the QoC standards for small and sick newborns in 2020 [24]. As part of the network’s monitoring strategy, the WHO co-developed 15 core quality indicators [25] in 2018, aligned with other global MNH initiatives [26–28], that were to be tracked in all QCN facilities and integrated in the countries’ routine information systems to facilitate learning and accountability within and between QCN countries. Support and technical assistance were provided by the QCN secretariat to countries through regular calls, periodic implementation and technical briefs, field visits, and meetings with Ministries of Health (MoHs) to share implementation progress and challenges.

To further facilitate learning between countries, the QCN secretariat organised international meetings where all global partners and network countries sent delegations of eight to ten people, creating opportunities to evaluate progress of the network, share lessons learned and network between countries. These meetings were seen as facilitating both the Learning and Accountability facets of the LALA framework. Global and national participants noted that those meetings evolved in time as the network matured, from the global leadership and technical experts sharing information unidirectionally to national stakeholders, to countries sharing their progress and learning with one another whilst global actors took a background role. Participants also saw the international meetings as providing a necessary chance to “take stock”, evaluate, and challenge the progress of the network. Initially meant to occur annually, in practice only two international meetings have taken place: in February 2017 (Lilongwe), and then March 2019 (Addis Ababa); with smaller meetings focused on technical assistance occurring in between. Because of the COVID pandemic, no further international meetings took place since 2019, except a last meeting in March 2023 (Accra) to share final lessons from countries. In addition, the QCN secretariat launched a network website, shortly after the official launch of the network in 2017, which was used as a global learning platform to share resources on QoC best practices for professionals within and outside QCN [6]. The website consists of: a knowledge library with guidance documents for QoC implementation; a podcast to share health practitioners QoC experiences; a global Community of Practice that any health practitioner can join to discuss ideas, practices and issues related to QoC for MNH; and monthly newsletters providing updates on the network’s activities. Finally, the QCN secretariat facilitated regular webinars, open to anyone, on different topics and issues related to QoC where global partners present or where partner countries share updates, learning, and ideas (see [29] for more on the topic). Interviewees at the global level found this activity important to support and maintain momentum within the network. Overall, many global participants spoke of the impact of the learning emerging from the QCN, steered by the WHO secretariat, and how it should continue beyond the end of QCN as it requires little funding to maintain.

As demonstrated in the first paper of our collection [7], the QCN network at the global level, led by the efforts of the QCN secretariat, was effective in bringing together network countries’ governments and global actors to raise the profile of QoC in MNH on the global agenda. The network at the global level was a catalyser for aligning goals between global actors and network countries, and supporting countries, leading to the emergence of the QCN network in-country.

Leadership

As described in other papers from this collection [7,10], QCN was designed to give governments leadership of in-country implementation. The implementation approach therefore relied on MoHs to align the goals of in-country MNCH partners as well as resources to develop or strengthen national policies, strategies and structures for quality of care in [MNH] health services [8] in order to pursue the network’s four strategic objectives. Thus, in-country, implementation was steered by the MoH through a technical working group (TWG) comprised of a variety of partners with the aim of developing or updating a national MNCH QoC roadmap and operational plan for the implementation of the QoC standards, as well as provide technical and financial support for QoC implementation. Funding for QoC implementation was also to be provided by MoH, according to countries’ roadmaps [30–32], however government funding proved to be limited and unpredictable. In Ethiopia, thanks to their earlier work on QoC [7], by 2017 their TWG was formed and the national roadmap for 2017–2020 was developed and due to be evaluated in 2022. National key informants felt that the roadmap has clearly put goals and strategies for implementation including procedures, indicators and monitoring and evaluation. There was also regular meetings and communication among them to help cohesion. Yet, the roadmap was not well known amongst our survey participants working in QCN health facilities, as only 34% confirmed they were aware of it in Nov-Dec 2021. In Uganda, although the TWG was established in 2017, its work was relatively slow until 2019 when a champion was assigned, according to national interviewees. The roadmap was developed in 2018, then revised in 2020, with an operational plan devised end of 2019. Interviewees in our first round of data collection linked the lack of roadmap and direction early in the network’s implementation with a lack of clarity and coordination, resulting in many actors undertaking various activities across the country without standardisation. Once implemented, our interview and survey results indicate a lack of awareness of the roadmap and operational plan among sub-national and local participants particularly, suggesting those might not be widely utilised. One reason could also be that at the frontline, these documents are presented in the form of tools or guidelines and not necessarily referred by the macro policy labels such as ‘roadmap’. Additionally, interviewees have indicated that adhering to the roadmap and plan was challenging because of different factors but most especially the diversity and approaches of network partners who sometimes used additional tools, local targets or had different schedules. In Malawi, the TWG was established in 2017 and the roadmap made available in 2018. As the roadmap and other strategic documents were not ready at the launch of the network, national interviewees in earlier rounds of data collection described a lack of strategic direction leading to delays in implementation and unorganised activities. In later rounds, national interviewees found the roadmap helpful by providing direction and outlining roles and responsibilities of partners. Some national implementation partners further believed the roadmap helped harmonise the efforts of various partners and ensured resources were used more effectively. In contrast however, awareness of the roadmap was still low (24%) among our 135 survey respondents in our last round in Malawi–predominantly from the sub-national and local levels–whilst awareness of other strategic QCN documents varied between 17% and 48%. Bangladesh developed a broad operational plan for health, nutrition and population for 2017–2022 which helped, along with additional national plans and guidance on quality of care (2015), patient safety (2018), and quality improvement (2019) to establish structures for quality improvement including those used in QCN. By md-2021, 58% of survey respondents at sub-national and national-level in Bangladesh were aware of the operational plan and 82% were aware of the quality improvement standard operating procedure.

The other component of the Leadership QCN strategic objective was the establishment of supportive governance structures in all countries to lead the QCN work within the health system. At the national level, all countries established a directorate within the MoH to oversee QI efforts, including for MNH, and hence QCN activities. Directorates were established before the inception of QCN, in Bangladesh, Ethiopia and Uganda, whilst Malawi set up theirs in 2016, ahead of the official launch of the QCN. The main functions of the national directorates are to oversee the coordination of QI activities and training in the country, establish quality standards, guidelines and standard operating procedures, and to monitor and evaluate them. Additionally, various national QI committees were established to support implementation in Bangladesh, although it has been difficult to assess their impact as few committee meetings took place over the years, according to our participants and our observations.

QCN governance was more developed at the national level of case study countries and lack of governance structures at sub-national and local levels impaired effectiveness of QCN activities. Where governance structures existed at sub-national or local levels their effectiveness was often impeded by a lack of capacity, lack of human resources or lack of ownership of responsibilities, and lack of support from national level.

“Regional quality structure is very weak and it is understaffed. Even those assigned people have lots of responsibilities. They are assigned for every quality-related activity, not just for MNH—not only that—but also do other things and it is difficult” (Implementing partner-National-Ethiopia Round 2)

Corruption was also mentioned as an issue, particularly in Uganda and Ethiopia. For example, in Uganda some key informants mentioned both cases of individuals diverting funds and systemic issues related to procurement of supplies and favouritism in relation to travel, training and mentorship opportunities.

Leadership at national level has been important to coordinate and monitor QI efforts in the countries. Funding for those functions however has been dependent on international donors, which impacted the QCN implementation in-country and raises issues for the sustainability of leadership efforts. Even in countries where network emergence was stronger [7] due to existing policies and initiatives and where leadership and QoC were more institutionalised, effectiveness of the network was still dependent on financial resources that are mostly external and on the commitment of governments to dedicate resources to QoC. In Ethiopia, the government has a budget allocated for quality improvement and QCN was part of it. This budget was small, especially after funds were diverted due to the COVID-19 pandemic and to rehabilitate conflict-affected areas. Implementing partners supported the 48 facilities by supporting coaching and learning activities and supporting the MoH to prepare TWG and annual QCN meetings. In Bangladesh, efforts have been made to secure their own independent budget but that has not been achieved yet according to government participants.

Action

The 8 WHO QoC domains and linked standards for maternal and newborn care [22] were adapted in 2017 in Ethiopia and Uganda, in 2018 in Bangladesh, and in 2019 in Malawi, with technical assistance from other QCN partners. Before this, there were no national QoC standards for MNH in Malawi, Uganda and Ethiopia, though quality of care was incorporated in national plans, while some facilities in Bangladesh had been using the Every Mother Every Newborn standards [33] since 2016. In all countries, the standards were used as a QI reference tool and to assess or audit health facilities, and in some countries, e.g. Uganda, different partners were focused on different standards. Many respondents thought the standards were useful to identify where improvements were needed, for example in Malawi:

“the assessment and the standards are highlighting gaps in the system” (Government-National-Malawi Round2)

Others thought more detail was required to improve quality of care, for example, in Bangladesh:

“the QED [QCN] indicators are very generic indicators to me. It never reflects the quality improvement of a health facility. It should have more sub indicators.” (Implementing Partner-National-Bangladesh Round 4)

Adaptation in Malawi and Uganda further included the addition of a ninth standard focusing on community engagement and social accountability in order to enhance community participation in improving QoC [30,31]. Interviewees at the national level pointed out that this additional standard was a necessary change for the success of the QCN. Regarding the other WHO standards developed by the QCN at the global level–those for children and young adolescents [23], and for small and sick newborns [24]–our data indicate that none of the countries have started to adopt them.

Under QCN’s Action strategic objective, each country was to choose learning districts and health facilities, representatives of others in the country “to serve as laboratories for learning” for implementing QoC activities [9]. By keeping track of those activities and documenting the learning, it was expected that successful interventions would be scaled-up to other districts and facilities. Learning sites were selected in each country (Table 1) by the government, together with implementing partners, some of whom had pre-existing work in the selected districts. Selection of learning sites was based on maternal and neonatal death burden, geography, and pre-existing and planned work. In Ethiopia, the government allocated specific funding for QoC in those QCN facilities in addition to funding provided by implementing partners. Several regions were excluded due to security concerns given the political instability in Ethiopia at the time of selection. Scale-up was not achieved in Ethiopia or Malawi by 2022, whereas in Uganda, and especially Bangladesh some scale-up was achieved (Table 1). Implementing partners operated with a high level of autonomy and independence in terms of what activities to implement and how. Whilst the MoH tried to coordinate and track efforts, participants reported it not to be effective beyond “big” partners such as USAID. In Uganda, in 2019, the MoH worked to improve coordination and strategic implementation by moving the ownership to a different department. Interviewees reported that, beyond the learning sites, implementing partners have been carrying out QCN activities in other districts they operate in. They further noted that this initial siloed and disjointed implementation attempt slowed down the success of the network in Uganda and that it was only three years into the five-year effort that a true network was beginning to form in the country. Although frontline health workers in Uganda were more aware of quality improvement efforts in general, led by the MoH and implementing partners, rather than QCN specifically. In Bangladesh, selection of sites differed as two initial pilot districts were selected in 2017 and implementation responsibilities were split between two partners: UNICEF and Save the Children (funded by USAID), who chose their own pilot learning facilities, either based on high mortality rate and remoteness or on lack of previous QI initiatives.

Whilst capitalising on existing work from partners to implement QoC interventions in QCN sites, this approach resulted in siloed working at sub-national and local levels. QCN activities in learning sites often differed in their content and scope between learning sites–depending on the implementing partners’ presence in the district–and their implementation was dependent on the available partners’ resources and technical support, as financial support to learning sites from the government was limited or non-existent.

“But I find the issue to do with financing more of a cause for us to fail. This is because I look at all the components of the health system and I find… well… I was trying at this particular time to think about the investments that have happened for example in [Case 1-low performing], as a learning district. How much did government commit to the goal that we reduce the maternal mortality rate by 50% in the implementing [of QCN] in the nation and districts by 2022? If we are to be honest, success of every implementing district was dependant on the kind of and the flexibility of partners that are in the district” (Implementing partner-National-Malawi Round 3)

This was further noticed in our health facility observations, where these fluctuations presented a challenge to track what QoC activities on site were considered QCN outputs. In case study QCN sites in Bangladesh, QI activities were more easily identifiable as they were part of the determined bundle of interventions implemented by either Save the Children’s or UNICEF’s projects (depending on the case study). In most case study QCN sites in Ethiopia, Malawi and Uganda, it proved more difficult as many health facility workers were not aware of QCN (Fig 1) or were not able to attribute if a QoC activity was a QCN output. Generally, the type of QCN activities observed in case study sites included: review of routinely collected data, assessment of facilities against QoC standards, Maternal and Perinatal Death Surveillance and Response, community engagement, point of care QI across all of MNH continuum of care. Whilst many respondents at local levels were not aware of QCN, they were aware of quality improvement efforts more broadly, including those pre-dating QCN.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Network awareness (survey data).

https://doi.org/10.1371/journal.pgph.0001751.g001

Overall engagement with QCN was perceived to be higher (and rising) in Uganda, and Bangladesh (though falling) than in Malawi or Ethiopia (Fig 2). In Bangladesh a median of 10–20 hours over 6 months was reported to have been spent directly on network activities, in Malawi 5–10 hours, Ethiopia 5–10 hours, and in Uganda 5–10 hours rising to 20–30 hours per 6 months (Fig 2). A greater proportion of respondents in Bangladesh and Uganda also reported that their views and ideas contributed to the network and that they were able to drive the network agenda (Fig 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Network engagement (survey data).

https://doi.org/10.1371/journal.pgph.0001751.g002

As per the LALA framework, network outputs at sub-national and local levels further involved a capacity building element for QI interventions and monitoring QoC such as QoC training, on-site QI coaches and QI committees within the health facilities. Capacity building and coaching visits covered a wide range of topics, such as intervention packages, data collection and monitoring, as well as training QI committees in developing, implementing and monitoring QoC activities. QI committees were the QCN output most discussed in interviews and observed in case study sites, corroborated by our survey respondents at health facility level whose majority indicated they were part of a QCN/QI committee. Indeed, the QCN led to the establishment or development of various QI committees in health facilities in Bangladesh, Ethiopia, Uganda and Malawi. Including mostly frontline healthcare workers, those committees met regularly (except during COVID-19 disruptions) to identify and prioritise QoC issues in the facility. The resulting local improvement projects were partially successful in some locations (e.g. reducing the incidence of birth asphyxia in a facility in Malawi), sometimes via logistical and funding support of implementing partners (this was mentioned in all countries), and unsuccessful in other locations (e.g. not reducing neonatal deaths in a hospital in Malawi).

In Bangladesh, Malawi and Uganda, capacity building outputs were mainly conducted by implementing partners (with collaboration on some occasions with local academic partners in Bangladesh). In Uganda some respondents thought not optimizing having Makerere University School of Public Health as the research and learning partner was a missed opportunity. In Ethiopia, despite being government-led, capacity building was supported financially and technically by implementing and technical partners. Therefore, consistency of capacity building activities throughout the implementation period was dependent on partners’ presence and resources for learning sites, leading to variable levels of activity between sites.

“It’s much more related to financial muscles and if the committee had a proper funding to run its activities, like its plans to do its supervision at whatever time they should be able do their activity. Unlike now where they have to wait for partners to come in and help. […] So, if there was a proper financial support it would be good to the network” (Government-Local Case 3-Malawi Round 1)

Learning sites activities, and their scale-up, were dependent on the presence and resources of one or several implementing partners, meaning that the amount and duration of resources provided by the implementing partner(s) dictated the level of effectiveness of the network at sub-national and local levels.

“When you bring a five-year learning project, it should have its own resources for implementation and to scale it up with the lessons learnt. This is the reason why the footsteps of the partners were followed. It doesn’t have its own resource. Partners have their own interests; they all have different approaches that they follow. For example, we say learning collaborative should be prepared in three months. Some do it within six months. Some conduct coaching every month, the others do it quarterly. Therefore, it lacks uniformity. This is still because there is no resource. So, partners plan with their own resources. Due to these problems, I can’t say that things have gone the way we want them to.” (Government-National-Ethiopia-Round 1)

For instance, in Bangladesh, committed partners for the duration of QCN implementation, like Save the Children and UNICEF, meant that a constant level of activity was maintained in the QCN facilities they operated in with some change to show for. They were, however, themselves funded by global partners and therefore also dependent on volatile external funding. For example, when UNICEF funding was reduced in some districts this meant that QCN activities carried out by UNICEF staff had to be taken over by health facility workers leading to a drop in QCN activities. On the other hand, some learning sites (e.g. in Malawi and Uganda) had no or little partners involved, leading to a dearth of QCN activities in those sites. For example, as reported by a local stakeholder in Malawi:

"…we have so many projects that we will put on paper and the issues of training QIST members so that they should function properly, we should know their roles and responsibilities. Now, we would have loved if we had a potential funder or sponsor of this program so that at least we know that when we have activities, these are the ones that are going to support us. But as it is now, there is no donor who has come openly to say that we are going to support this program. So, it’s a challenge to us" (Government-Local Case 3-Malawi Round1).

In Uganda, there was some attempt to pool resources from partners, but this effort failed because of lack of trust, and a lack of harmonisation of different tools and methods used by different partners–both exacerbated by the leadership vacuum at the start of QCN in Uganda which later improved. Additionally, as observed in several facilities, this has meant that when partners’ projects came to an end in a given area due to financial constraints or end of funding, partners ceased to support the learning sites–making it difficult for sites to continue QCN activities given their lack of human, technical and material resources, as well as undermining M&E efforts.

“When IHI, WHO, CHAI pulled-out from the network, it was very difficult to cover the sites which were under their organizations. Taking lessons from this, we are working to make the government takeover the activities. There were gaps at collection of common core indicators when the partners pull-out. Partners always depend on the donors as a result there are challenges.” (Government-National-Ethiopia Round 2).

The COVID-19 pandemic further contributed to weakening partners’ support to activities in learning sites [21]. Issues around resources dedicated to QCN are taking place in a context of weak health system capacity, in terms of material and human resources, undermining network effectiveness in all countries. This was the challenge to QCN activities most reported by participants in all countries. The lack of human (including support staff) and material resources was also observed in our case studies facilities as a hindering factor of implementation.

“Looking at the 8 standards for MNCH that we adopted, we realize that implementing those standards isn’t just about quality improvement. We need real inputs and we have seen that without improving infrastructure, without having some of the services or the equipment, for example the newborn care to improve it, we need to have actual equipment, actual services, people need to have skills for managing those newborns. Some of the inputs were not well catered for even in the 6 learning districts and probably that’s why we did not see any reduction in the newborn mortality. So, we have seen that the input level as we implement it, the inputs that we plug into probably were not well addressed. Even just the technical skills not just the quality improvement skills but the technical skills for providing services needed to be strengthened from time to time and needed to be in contact with the people implementing them to be supported at least on a quarterly basis. This did not happen for most of the districts, and so we know why we did not perform well.” (Government-National-Uganda Round 2)

Learning & accountability

Learning and Accountability are two important strategic objectives of the QCN. Here we discuss the infrastructures put in place in the four countries to support learning and accountability. We analysed elsewhere the diffusion of information, innovation and learning processes within the network [29]. Part of this strategy, as mentioned earlier, was the development in 2018 by WHO, global partners and network countries of 15 core quality indicators for MNH–an immense effort when considering the situation before QCN. Among global interviewees, there was general agreement that the dedicated working group had done good work and the indicators were well-received, yet by 2019 participants still described a lack of clarity on how and when data should be collected. The aim was to integrate those core quality indicators to the countries’ routine Health Information Systems (HIS), which required according to some global and national interviewees, a lot of technical support from the network secretariat (with limited manpower) and partners. Despite the complexity of the process, each country has had some success in integrating the 15 core quality indicators into their national health information systems (Table 1), though experience of care indicators have been a challenge, with independent regular systems of data collection from patients proving difficult. Data in learning sites were often collected by implementing partners. In Bangladesh, both the government and the major implementing partners aimed to place quality improvement indicators on the web-based national dashboard, which can be used to download custom reports on various indicators. National interviewees indicated that this dashboard was largely functional, and stakeholders expressed pride in their ability to roll out this new initiative while simultaneously responding to COVID-19. In Ethiopia on the other hand, the network relied on a parallel system to their HIS to collect data on the QCN quality indicators (data from facilities reported up in the system electronically via spreadsheets provided by MoH) until a planned integration to DHIS2 at the next HIS revision. This proved to be a challenge according to our interviewees who reported that the QCN indicators did not align with the previous reporting system and questioned the robustness of the experience of care indicators since the data was not collected by independent actors not involved in the care provided. However, all countries faced structural issues and a lack of capacity in collecting good quality data. For instance, countries still relied on separate surveys to collect data on experience of care indicators, unlike indicators on provision of care that were captured by DHIS2 or a parallel system (for Ethiopia). Additionally, in all countries, capacity building on data collection, analysis, quality and management was crucial due to the lack of capacity at sub-national and local levels but depended on the efforts of the partners supporting the learning facility. Supported by our facility observations, several participants in all countries further brought up concerns over the reliability of the data reported, particularly around patient experiences and mortality figures. The work of the QCN on indicators and monitoring did bring attention to the importance of health data for QoC improvements. In Ethiopia for example, national and local interviewees considered that the network had improved health data documentation, management, use and reporting in health facilities, even if gaps in capacity remain. In our survey, most respondents in Bangladesh (83% rising to 88%) and Uganda (74% rising to 80%) indicated that there were quality improvement indicator dashboards or visualisations at their facility (Fig 1). This was 63% in Ethiopia and just over 50% in Malawi. Locally, in our observations in health facilities, behaviour change around data monitoring seemed to vary. Some of the sites had a system of bulletin boards which visually displayed QI metrics for the preceding several months, including maternal and neonatal mortality, as a means of making the information more accessible and of motivating staff. Some facilities, for example those observed in Bangladesh, also regularly completed partographs to monitor the progress of labour as part of efforts to reduce mortality. In some facilities, data was also discussed in the facility QI committee to improve accuracy and enhance accountability. A few facilities were observed to not be using dashboards, visualising data or using monitoring indicators in their practice. At the end of our data collection, there was still a lot of work to achieve in all countries to collect (reliable) data, integrate to DHIS2, increase the quality of the data collected and improve analysis at the national level. However, from our interviews and observations, the lack of human resources and capacity around M&E makes this difficult to achieve.

“Yes… I think we have not done well in the M&E. I think there is a lot that we would have done. Maybe the problem is that some of the problems are beyond us, because we need full time M&E, looking at the data and ensuring that reports are being done. And when reports are put in the system, analysing the information or extracting the information from whatever platform is there and pushing it back to policy makers and the facilities, so that they can use that information to further improve the network. So, there is a lot that needs to be done in the M&E side” (Government-National-Malawi Round 2)

As a result, the data received from countries was often incomplete or of poor quality leading to a lack of quantitative analysis on how well the network has done on key process and outcome indicators and whether the overall goal of the network of reducing case fatalities by 50% has been achieved. Therefore, the impact of the network on reducing mortality remains unknown, though may be reported in 2023 (WHO-Global Interviews-Round 2). For global interviewees, this goal was always ‘ambitious’ or ‘aspirational’ and they were cautious about the network’s ability to achieve it within the five-year timeframe. Among global actors there was an overall sense that the network needed these bold, ambitious targets in order to gain momentum and attract engagement, funding, and global attention. Most global actors believed these goals were an essential catalyst at the start of the network and continued to serve as important motivation. Some national and local participants in Bangladesh, Ethiopia, Malawi and Uganda believed that some progress had been achieved towards reducing maternal and/or neonatal deaths in some of the learning sites. Our survey results indicated that most respondents in each country perceived the network to be valuable, though more so in Uganda and Bangladesh than in Ethiopia or Malawi (Fig 3). Fig 3 also shows a breakdown of indicators within this domain, with the highest scores being for “would recommend joining the network”, and the network “helps me professionally”, and lower scores for perceiving QCN to have resulted in healthcare improvements.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Network Perceived as Valuable (survey data).

https://doi.org/10.1371/journal.pgph.0001751.g003

As part of the Learning and Accountability strategy, countries were also to involve a research institution to document the lessons learned from the network in order to facilitate the scale-up of the learning to non-learning sites. In Bangladesh and Uganda, the MoH built upon existing partnerships with academic partners such as NIPSOM and Makerere University School of Public Health respectively to achieve this milestone. In Uganda, Makerere was not optimally engaged as the learning partner, working instead in more of a consulting and advisory role for the MoH and WHO, with limited funding, and did not get to interact widely and in-depth with network members. At the end of our data collection, this milestone had not been achieved in Malawi and Ethiopia. From our qualitative data, despite the learning achieved in-country across different platforms [29], the network was not mature enough in any country to facilitate the scale-up of the learning from laboratories to non-QCN facilities through the national research institution in a systematic way. This was due to a combination of factors including lack of funding, an over-reliance on implementing partners and delays due to the COVID-19 pandemic. In Bangladesh, at the time of our last data collection in 2022, NIPSOM had advanced plans to scale-up QoC training in all districts, via allocated funding from the government budget.

Lastly, the Learning and Accountability strategy involved the establishment of a mechanism for community engagement in QoC in the learning sites, in line with the second global goal of the QCN to improve experiences of care for women and their families. Each country took a different approach to include community voices in the learning sites to improve QoC and accountability (Table 1). Some of these built on earlier work with communities pre-dating QCN, though were not always fully integrated with QCN [34]. Some community discussions and engagement mechanisms were developed into action plans with learning facilities, although in practice our data indicate that those are not fully optimised. All in all, the global QCN goal to improve experiences of care and “enable measurable improvements in user satisfaction with the care received” [8] seem to have been relatively neglected in national QCN implementation in all countries. Mirroring this at the global level, documents collected show an emphasis on community engagement in the early phases of QCN development (especially pre-launch) but overtime, little or no mention of this component is made. Malawi achieved the most regarding this goal. Indeed, Malawi was the only country that integrated one indicator (percentage of facilities with functional ombudsman and suggestion box or other feedback mechanism) to DHIS2 linked to patient experience of care. Malawi’s establishment of an ombudsman structure was criticised though for not being independent and consequently rarely used.

Effect of COVID-19, and political instability

The COVID-19 pandemic impacted network effectiveness related to the Action, Learning and Accountability strategic objectives as those were not as advanced as the Leadership one by the time the pandemic hit. Indeed, many participants at all levels of governance mentioned that QCN resources were diverted towards Covid-19; some learning facilities were repurposed to only accommodate Covid-19 patients; focus remained on Infection Prevention Control over all other aspects of QoC; and local restrictions kept patients away. Consequently, participants believed this led to important delays in implementation of interventions and scale-up as well as learning and accountability outputs. Local respondents in all four of our case study countries further indicated that all QCN activities in their facility ceased during periods of heightened restrictions. Additionally, all QCN meetings, trainings and interactions at different levels of governance reduced in frequency, even when an online option was available. Some of the frontline workers interviewed further believed that progress towards reducing case fatalities rates was stalled as a result of the pandemic. Additionally, some of the countries faced periods of instability around elections (e.g. Malawi and Uganda) and conflicts (e.g. Ethiopia) that impacted the work of the network [21].