Skip to main content
Download PDF

Effectiveness of in-service training plus the collaborative improvement strategy on the quality of routine malaria surveillance data: results of a pilot study in Kayunga District, Uganda

Malaria Journal volume 20, Article number: 290 (2021) Cite this article

Abstract

Background

Surveillance data are essential for malaria control, but quality is often poor. The aim of the study was to evaluate the effectiveness of the novel combination of training plus an innovative quality improvement method—collaborative improvement (CI)—on the quality of malaria surveillance data in Uganda.

Methods

The intervention (training plus CI, or TCI), including brief in-service training and CI, was delivered in 5 health facilities (HFs) in Kayunga District from November 2015 to August 2016. HF teams monitored data quality, conducted plan-do-study-act cycles to test changes, attended periodic learning sessions, and received CI coaching. An independent evaluation was conducted to assess data completeness, accuracy, and timeliness. Using an interrupted time series design without a separate control group, data were abstracted from 156,707 outpatient department (OPD) records, laboratory registers, and aggregated monthly reports (MR) for 4 time periods: baseline—12 months, TCI scale-up—5 months; CI implementation—9 months; post-intervention—4 months. Monthly OPD register completeness was measured as the proportion of patient records with a malaria diagnosis with: (1) all data fields completed, and (2) all clinically-relevant fields completed. Accuracy was the relative difference between: (1) number of monthly malaria patients reported in OPD register versus MR, and (2) proportion of positive malaria tests reported in the laboratory register versus MR. Data were analysed with segmented linear regression modelling.

Results

Data completeness increased substantially following TCI. Compared to baseline, all-field completeness increased by 60.1%-points (95% confidence interval [CI]: 46.9–73.2%) at mid-point, and clinically-relevant completeness increased by 61.6%-points (95% CI: 56.6–66.7%). A relative − 57.4%-point (95% confidence interval: − 105.5, − 9.3%) change, indicating an improvement in accuracy of malaria test positivity reporting, but no effect on data accuracy for monthly malaria patients, were observed. Cost per additional malaria patient, for whom complete clinically-relevant data were recorded in the OPD register, was $3.53 (95% confidence interval: $3.03, $4.15).

Conclusions

TCI improved malaria surveillance completeness considerably, with limited impact on accuracy. Although these results are promising, the intervention’s effectiveness should be evaluated in more HFs, with longer follow-up, ideally in a randomized trial, before recommending CI for wide-scale use.

Background

In most African countries, health facility (HF)-based surveillance data on malaria reported through routine health information systems are a critical source of information for disease surveillance and decision-making by national malaria control programmes. Unfortunately, challenges in HF data completeness, accuracy, and timeliness [1,2,3,4,5,6,7,8,9,10,11,12] limit the utility of routinely collected HF data for programmatic monitoring and evaluation [4, 13, 14]. As a result, modelling has been used to estimate malaria morbidity and mortality trends in many countries [15, 16].

Given these limitations in HF surveillance data, periodic cross-sectional population-based surveys have been considered a superior source of information for planning and policy needs [17]. However, such surveys are complex and costly, are not useful for assessing malaria prevalence trends in low-burden settings, and lack the geographic granularity of HF surveillance data. Due, in part, to the global scale-up of malaria control and elimination efforts, there is an increased demand for near real-time information to track malaria burden and guide implementation of interventions [18, 19]. The widespread adoption of District Health Information System 2 (DHIS2) in African countries has improved the availability of HF data by integrating disease-specific surveillance systems into a single digital platform, rendering the timely use of HF surveillance data more feasible.

The DHIS2 platform was established in Uganda in 2012, and the system has been credited with increasing the quality, availability, and use of national HF data substantially [20, 21]. The malaria-specific goals of DHIS2 in Uganda include collating and presenting data needed to calculate indicators on adherence to test-and-treat guidelines [22], malaria burden, and the targeting and evaluation of interventions. While DHIS2 is envisioned as a digital disease surveillance platform, actual data capture remains a manual process involving thousands of health workers from individual HFs. For malaria, data collection requires paper-based data extraction from paper registers sourced from outpatient departments (OPD), inpatient units, laboratories, and pharmacies. Collected data are aggregated monthly at HFs and submitted, as paper reports, to the district level where information is entered into the web-based DHIS2 [20].

Until recently, HF registers lacked several key data fields required to calculate the indicators outlined in Uganda’s malaria strategic goals. To address this deficiency, the Uganda Ministry of Health (MOH) introduced revised HF registers and reporting forms (heretofore referred to as “revised forms”) in July 2015. In addition to capturing the required malaria-related data, the revised forms include new fields to collect data on tuberculosis burden and care, health behaviour, nutritional status, referral, and selected clinical practices.

These ambitious new data collection plans increased the amount and complexity of data that Ugandan health workers were required to record substantially, doubling the number of data entry fields, including sensitive health behaviour questions, and requiring additional equipment for specific measurements (e.g., for blood pressure and blood sugar levels). While the revised forms may improve the quality and utility of collected data, there also is a substantial risk that data quality could be undermined by the increased complexity and volume of work. Recognizing this risk and seeking to improve the overall quality of DHIS2 data from HFs, the Ugandan MOH and malaria stakeholders sought to identify innovative, feasible, context-appropriate, healthcare worker-centred methods—one of which was the collaborative improvement (CI) approach.

CI was developed by the Institute for Healthcare Improvement in 1995 as a team-based approach to improve healthcare worker performance. Over the last 20 + years, the CI approach has been applied across a wide array of healthcare worker performance areas in high-resource [23,24,25,26] and low-resource [26,27,28,29,30,31,32,33,34,35] settings. As an adjunct to traditional training modalities, strategies including CI components have been shown to improve results substantially [32, 34]. The CI approach involves a network of teams (usually in HFs) that identify problems, develop solutions, measure change using agreed-upon indicators, and share results and best practices through periodic learning sessions [28, 36, 37]. Specific CI components include the initial design of a change package by national and international technical experts, networking of facility-based CI teams (proof of concept CI networks can include a few HFs, while CIs with an established intervention may involve 20–100 HFs), training or sensitization towards standards, application of continuous quality improvement techniques known as plan-do-study-act cycles to test changes [38, 39], ongoing coaching and mentorship on the quality assurance methodology, and integration of “shared learning” to drive change and promote innovation.

Although the evidence for CI effectiveness is encouraging, much of it was produced by evaluations with important methodological limitations [40, 41], including inadequate control arm, reliance on self-evaluation and self-reported internal monitoring data as sole outcome measures [27, 32, 42,43,44,45,46], and short baseline periods, which limit the ability to fully account for natural variability in performance [27, 32]. Moreover, because the CI process encourages improved data collection, a positive bias can result when under-reporting at baseline is corrected and inaccurately interpreted as improvement [23, 26]. Lastly, the majority of published CI studies are limited to large tertiary and academic centres with limited evaluation of effectiveness in primary (lower) level HFs [33, 47].

To evaluate the feasibility, cost-effectiveness, and impact of a combined in-service training and CI initiative on the quality of malaria surveillance data, a pilot study of five rural Ugandan HFs was conducted using an interrupted time series design, independent evaluation, and objective data sources. The small scale of the pilot allowed a unique in-depth qualitative exploration of the mechanisms of action of CI components [48].

Methods

The objectives of this study were to estimate the effectiveness and costs of an intervention that combined in-service healthcare worker training plus CI (referred to as TCI), which was designed to improve the quality of routine malaria surveillance data collected at HFs in Uganda.

Study design

This was a one-arm interrupted time series study, which used baseline trends as a counterfactual, to determine the effectiveness of TCI [49]. A cost-effectiveness evaluation was completed using activity-based costing to calculate the total incremental intervention cost divided by the number of improved records observed during the study period. The study had a 12-month baseline period, 5 months of TCI intervention scale-up, 9 months of active TCI implementation, and 4 months of post-intervention evaluation.

Study sites

The study was conducted in five hierarchically-connected public HFs (level II, III, and IV) in the Kayunga District, Central Uganda—a moderate-to-high malaria endemic area. Level II health centres (HC IIs) serve ~ 5000 residents and generally have no laboratory facilities (malaria diagnosis by RDT only); level III health centres (HC IIIs) usually serve 20,000 residents and are expected to have a functioning laboratory (malaria diagnosis by RDT and microscopy) and maternity services; and level IV health centres (HC IV) serve a health sub-district (100,000 residents) and are designed to function as a small hospital serving outpatients and inpatients (malaria diagnosis by RDT and microscopy).

Of the 18 HFs in Kayunga District (8 level II HCs, 8 level III HCs, and 2 level IV HCs), two were excluded due to prior experience with CI or current engagement in externally-funded malaria research or data improvement interventions. All other HFs were eligible (Box 1). The final selection was made from HFs within the same hierarchical unit, balanced by HC level representation: Bbaale HC IV, Lugaasa HC III, Wabwo Oko HC III, Nakyesa HC II, and Kakiika HC II. The detailed characteristics of selected facilities are reported separately, in the paper reporting on the results of the qualitative evaluation [48].

Study outcomes

Study outcomes included data completeness, accuracy, and timeliness, defined as follows (Table 1): completeness: the proportion of OPD register malaria records with all fields complete, per month; accuracy: the relative difference in patient counts or malaria tests between the OPD registers, laboratory registers, and submitted monthly reports, per month; timeliness: the proportion of monthly summary reports submitted within 15 days of the end of the month, as requested by the MOH.

Table 1 Formulae and data sources for completeness and accuracy outcomes
Full size table

Study intervention

Prior to implementation, a full-day expert meeting was held to review relevant evidence and define the scope of the collaborative (Fig. 1). Each HF formed a CI team comprised of representatives from relevant departments (e.g., clinicians, lab, pharmacy, records) in preparation for the study activities. The intervention started in November 2015 with an on-site in-service training (3–4 h at each HF) for HF staff on good malaria data recording and reporting practices. Also in November 2015, a half-day workshop was held to orient district and HF administrators to the CI methodology and the study, followed by the first learning session. CI teams attended a series of 1–2 day learning sessions held approximately every 3 months. Learning sessions provided didactic instruction on CI methodology and allowed teams to share their experiences and discuss challenges. CI teams were responsible for recording study outcomes each month, or more frequently if new changes were implemented, and producing graphs of the outcomes over time (run charts) in quality improvement journals supplied by the CI implementers. Summaries of the implemented changes (see Additional file 1: Annex 1 for examples) and the outcomes documented in the journals were presented and discussed during learning sessions, so CI teams could learn from each other which changes were more or less effective. At the end of each learning session, CI teams developed action plan and identified outcomes to focus on. Each learning session was followed by an “action period” during which plan-do-study-act cycles were implemented to test change ideas. Ongoing CI coaching and mentorship was provided to each team throughout the intervention period. The CI mentor was a trained and experienced professional who provided onsite coaching (1–2 h) on the CI methodology rather than the technical aspects, several times per action period.

Fig. 1
figure 1

Timing of study procedures. TCI in-service training and collaborative improvement, PDSA Plan-do-study-act

Full size image

Each learning session initiated a new intervention phase – for three total phases:

  • Phase I (November 2015–February 2016), improving completeness of OPD and lab registers.

  • Phase II (March–June 2016), improving concordance between the OPD registers, lab registers, and monthly summary reports.

  • Phase III (June–August 2016), harmonizing OPD and lab registers for calculation of standard malaria outcomes.

While the improvements were intended to be cumulative in nature, the CI teams primarily tracked the outcomes reflecting the specific focus of each phase.

In addition to the described outcomes, CI teams also monitored intermediate indicators to assess specific aspects targeted by implemented changes, such as number of malaria patients that can be traced in both OPD and laboratory registers and number of patients with a positive malaria test result in both OPD and laboratory registers. Each team performed independent root cause analyses and identified changes to implement at their health facilities. Aside from the costs associated with training, learning sessions, and coaching, external resources were not provided for changes instituted by the HF CI teams.

A “harvest meeting” was held over two days in September 2016 to complete the TCI intervention. CI teams documented and ranked the most effective changes made by the CI teams, which were compiled into a “change package” (see Additional file 1: Annex 1). This “change package” could be used as a catalogue of changes that improved data quality, which could be adopted by other health centres or by the MOH on a larger scale, considering the local context.

Evaluation

Independent quantitative and qualitative evaluations were carried out alongside the study intervention (Fig. 1, lower section). Quantitative results are presented in this paper, and qualitative results are published separately [48, 50]. Data were collected from the OPD and lab registers and the monthly summary reports by a team of seven trained surveyors during HF visits between October 2015 and January 2017, for the period from July 2014 to December 2016. Individual level data were double-entered for 20% of the records, and all data were compared to the aggregate data collected separately.

Data analyses

Monthly data were analysed using segmented linear regression modelling, based on the following timelines:

Segment 1. Baseline: June 2014 to June 2015 (12 months), retrospective data collected from the registers and monthly reports.

Intervention scale-up (revised forms plus TCI; excluded from segmented regression analysis). The scale-up period included: (1) Scale-up of revised forms by the MoH (July to October 2015). The revised forms implemented by the Ugandan MOH were introduced and scaled up over 4 months (July–October 2015), as part of a mandatory nationwide roll-out in July 2015 accompanied by cascade training, although this training was limited in Kayunga district (one health worker per HF); (2) Scale-up of the TCI (November 2015).

Segment 2. Follow-up period (revised forms plus TCI plus post-TCI): December 2015 to December 2016 (13 months), including 9 months of TCI implementation (October 2015 to August 2016) and 4 months of a short-term post-intervention period (September to December 2016).

The segmented linear regression modelling compared the baseline (segment 1) and the follow-up (segment 2) periods to estimate three effect sizes for each outcome:

  1. (1)

    Immediate level change from the end of the baseline period to the immediate period after implementation,

  2. (2)

    Baseline-to-follow-up change in outcome trend, and

  3. (3)

    Single effect size combining the immediate level change and trend change, calculated as the outcome level at the mid-point of the follow-up period, as predicted by the segmented linear regression model minus a predicted counterfactual value based on the baseline trend extended to the mid-point of the follow-up period. The single effect size can be measured as an absolute or a relative difference, the latter being more meaningful for indicators with small values.

Models were adjusted for serial autocorrelation resulting from repeated observations over time through SAS AUTOREG procedure. Statistical analyses were performed using SAS v. 9.3 (SAS Institute Inc., Cary, NC, USA).

The self-reported results of the CI teams were compared to the outcomes of the formal CI evaluation for one of the accuracy measures (discordance in malaria cases between OPD register and monthly report; 8 months of data, 40 data points), and in a more limited fashion, the outcomes for clinically-relevant completeness (2–5 months following learning session 1 and 15 data points, limited by the records of the CI teams). Pearson’s correlation and the mean difference (per HF) between the CI team and evaluation team data were used to describe the comparability of the measured outcomes and to assess the potential bias of having the evaluation based solely on data collected by CI teams (a method often used to evaluate CI).

The cost-effectiveness evaluation considered training and other intervention costs, including transportation, per diem for the CI mentor and the CI teams during learning sessions, venue rentals, general miscellaneous training-related costs, and salaries of the CI implementer team. Costs incurred from the involvement of facility or MOH staff were not considered. Decision-tree analysis was used to calculate the cost-effectiveness of the intervention in terms of expenditure per additional malaria patient record that was complete. Inputs to the model are listed in Additional file 2: Annex 2. All costs were in US dollars based on the average Ugandan shilling 2016 exchange rate. Monte Carlo simulations were used to determine the point estimate and 95% confidence interval (95% CI) for the incremental cost-effectiveness ratio.

Results

The evaluation team abstracted 156,707 entries from OPD and lab registers and aggregated monthly reports from July 2014 to December 2016. All-field patient register completeness was 15.0% at baseline with no significant change through the baseline period (0.7%-points per month; 95% CI: − 0.4, 1.7) (Table 2, column 1). A statistically significant immediate improvement (70.4%-points; 95% CI: 60.8, 80.0) and mid-point improvement (60.1%-points; 95% CI: 46.9, 73.2) was observed following the TCI intervention (Fig. 2A). Throughout the follow-up period, however, a significant negative change in slope was found (− 1.6%-points per month; 95% CI: − 2.8, − 0.3), which represents a decay of − 2.3%-points per month in effect size when accounting for the counterfactual slope, and which suggests a steady deterioration in TCI’s effect. By HF, heterogeneity in baseline trends and intervention was noted (Additional file 3: Annex 3). Unlike other HFs, health facility 2 (HF2) showed a lower level of improvement after TCI and was the only HF with a downward slope during follow-up. Also, HF3 had a higher baseline level of completeness with a considerable positive slope and reached a high level of completeness prior to TCI. To evaluate the effect of HF3 and its implausible counterfactual, a sensitivity analysis was performed (Table 1, Additional file 4: Annex 4) by excluding HC3 and found a greater immediate effect (82.5%-points vs. 70.4%-points) and non-significant deterioration in the slope over time during follow-up (− 0.9%-points, p = 0.17), in comparison to the original model.

Table 2 Effects of training plus collaborative improvement on data quality: results of segmented regression modelling
Full size table
Fig. 2
figure 2

A–D Effect of TCI on A data completeness, all fields; B data completeness, clinically-relevant fields; C the relative difference (absolute value) between the OPD register and monthly reports for total malaria cases; D relative difference (absolute value) in test positivity rate based on the laboratory register data versus monthly reports. The graphs in A–D show three segments: (1) baseline, (2) scale-up of TCI and the revised forms (excluded from analysis), and (3) a period reflecting the effect of TCI and the revised forms followed by a short-term post-intervention sustainability assessment (post-TCI). The timing of the learning sessions (LS): LS1—focus on completeness; LS2—focus on accuracy through concordance of data sources; LS3—focus on data elements for calculation of standard malaria indicators); the harvest meeting (HM) completes the intervention period. Improvement in data quality is represented by positive effects sizes for completeness indicators (as the goal is 100% completeness) and negative effect sizes for accuracy indicators (as the goal is no discordance). TCI in-service training plus collaborative improvement, LS learning session, HM harvest meeting. Solid lines are based on the segmented regression model. The dashed line represents the counterfactual (an extension of the baseline trend into the follow-up period)

Full size image

Baseline register completeness for selected clinically-relevant fields (age, sex, weight, diagnosis, and treatment) was 30.5% with a small but significant increase during the baseline period (0.9%-points per month; Table 2, column 2). Post-intervention, an immediate increase of 68.2%-points (95% CI: 64.6, 71.7) brought completeness to almost 100% across sites. This level of completeness was maintained throughout the follow-up period (Fig. 2B). The mid-point effect (61.6%-points; 95% CI: 56.6, 66.7) reflects TCI’s mean effect throughout the follow-up period. There was a significant negative change in slope (− 1.0%-point per month; 95% CI: − 1.5, − 0.6), but this did not reflect a decay in TCI’s effect. Rather, it was because the outcome trend flattened out at near 100% (and thus, the small positive baseline slope was essentially negated). Similar to the all-fields completeness outcome, trends varied by HF (mostly for the baseline period (see Additional file 3: Annex 3). Excluding HC3 had little effect on the results (Table 1, Additional file 4: Annex 4).

The baseline difference between the number of malaria cases recorded in OPD registers and the number of malaria cases submitted in monthly reports was relatively low at 6.7% with no significant change throughout baseline (slope: 0.1%-points per month), as shown in Table 2 and Fig. 2C. Following the intervention, no immediate change was observed (− 0.9%-points; 95% CI: − 8.7, 6.9) and the mid-point effect was not significant (− 47.4%-points; 95% CI: – 177.0, 82.2). Because the analyses were based on an absolute value of the discordance between the OPD registers and monthly reports, the dispersion of the discordance values and directionality in site-specific graphs (Additional file 5: Annex 5) was assessed, showing that, in most HFs, over- and under-reporting compared to OPD values had no observable pattern.

The baseline test positivity rate (proportion of all malaria tests done with a positive result; TPR) discordance between laboratory registers and monthly reports was 20.1% with no significant change throughout baseline (slope: − 0.6%-points per month), as shown in Table 2 and Fig. 2D. Discordance significantly decreased by 15.8%-points (95% CI: − 25.1, − 6.5) immediately post-TCI scale-up. The mid-point effect was a relative 57.4%-point improvement (95% CI: − 105.5, − 9.3) in accuracy. This estimate reflects a drop of 11.6%-points in the discrepancy between the two data sources, that is expressed as a 57.4%-point relative difference when compared to the counterfactual. The non-significant slope during the follow-up period indicates that achievements were sustained. Similar to the malaria cases accuracy indicator, the graphs in Additional file 5: Annex 5 show different patterns of discordance across HF over time.

Timeliness, which was initially included as a study outcome, was dropped from the analysis because it was already high (> 95%) at baseline and had little room to improve.

The agreement in results between facility-based CI teams and study evaluation teams was moderate: Pearson’s correlation for the accuracy outcome (discordance in malaria cases between OPD register and monthly report) was 0.43, ranging from − 0.06 for HF5 (level II) to 0.99 for HF1 (level IV); and 0.68 for the clinically-relevant completeness outcome. Mean differences in the accuracy by CI teams compared to the evaluation team data (i.e., CI team result minus evaluation team result) were − 2%-points for all sites, and ranged from −15%-points to − 3%-points for higher level HFs (HC IIIs and HC IVs) and 4–6%-points for lower level HFs (level II). Negative values indicate that CI teams found greater reductions in discrepancies between the data sources, and thus a greater improvement in accuracy of data, compared to the evaluation team’s results (results and scatterplots reported in Additional file 6: Annex 6).

The total calculated cost of the TCI intervention implemented at the five pilot HFs was 51,399 USD. Cost-effectiveness results estimate a cost of 3.53 USD (95% CI: 3.03, 4.15 USD) per malaria patient record with complete clinically-relevant data recorded.

Discussion

Surveillance data are essential for malaria control, but data quality is often poor. CI is a promising quality improvement intervention, but it has not been rigorously evaluated as a method to improve surveillance data quality. To evaluate the impact of brief in-service training plus the CI methodology on quality of malaria surveillance data, a pilot study was conducted in 5 HF in Kayunga, Uganda.

Overall, the study found that in-service training and CI resulted in large improvements in data completeness, with limited effect on accuracy. Specifically, the estimates of the immediate and mid-point effect of the two completeness indicators (all-field completeness: 70.4% points and 60.1%-points, respectively; clinically-relevant completeness: 68.2%-points and 61.6%-points, respectively) align with the evidence of efficacy of training with CI from three collaboratives from Niger [51]. Although the Niger collaboratives addressed maternal and newborn care rather than malaria surveillance, the indicators addressing adherence to care standards could be relevant for comparison with the data quality standards in this study. Across the Niger collaboratives, adherence to standards was improved for the following aspects of care: measuring the newborns’ temperature (by 60.9% and 96.5% in two collaboratives); vaginal deliveries according to active management in third stage of labour (by 89.6% and 91.4% in two collaboratives); essential newborn care (by 71.0% and 85.7% in two collaboratives); and pre-eclampsia and eclampsia by 35.3% in one collaborative. Overall, the Niger collaboratives documented higher levels of improvement, though these studies were classified as high risk of bias by the review [51]. In this study, data accuracy was high at baseline and thus, there was less room for improvement, which is encouraging. A significant improvement in data accuracy (as measured by the reduction in discrepancy between reporting in the laboratory registers and monthly reports) was found only for test positivity rate (relative improvement of 54.7%-points). In addition to the relatively high baseline, other factors that may have led to a less pronounced effect of CI on accuracy include: more complex mechanisms behind such improvement that needed collaboration and harmonization across different HF departments (e.g., outpatient, laboratory, pharmacy), as well as the shorter period of time the HF teams spent working on accuracy indicators, with the focus shifting to accuracy in Phase II of the intervention period.

A considerable heterogeneity in the intervention effects in site-specific analyses was observed. Four out of five HFs achieved high levels of completeness shortly after TCI was scaled up. But, at HF2, the outlier, little impact on all-field completeness and clinically-relevant completeness was observed. In the aggregate analysis, the poor performance of HF2 was the driving force behind the deterioration of the TCI effect for all-field completeness over time. HF3 achieved high levels of completeness during the baseline period, even before the intervention, for unclear reasons. A sensitivity analysis with and without HF3 was conducted that showed a greater immediate effect and non-significant deterioration in the slope over time during follow-up. Great variability in trends was also observed for the accuracy indicators before and after the intervention; harmonization of improvement trends was more pronounced for the TPR accuracy indicator. The study results should be interpreted with the understanding of this underlying heterogeneity by HF.

The correlation between data collected by the CI team and through the evaluation was moderate, with considerable variation across different HFs. The small mean difference (− 2%-points) for all sites was driven by the variation in the magnitude and directionality across sites and, therefore, does not indicate a good agreement between self-reported and evaluation-derived data. Because of differences in data collection (i.e., length of baseline and tracking periods, and differences in outcomes measures), the evaluation could not apply the same methods to CI teams’ data to estimate effect size. However, based on the correlation analyses, the differences suggest that the effect sizes calculated from the CI teams’ data would have been larger than those based on the evaluation data. By HF, the study results suggest that in higher level HFs (HC IIIs and HC IVs), CI teams may have overestimated the impact of TCI on data accuracy, whereas the opposite was true for lower level HFs.

The cost of TCI per additional case with complete recording of clinically-relevant data was $3.53 USD, which is similar to the estimated cost per additional malaria case treated appropriately in Uganda ($3 to $13 USD/case) [52] and the cost of improving community-based access to malaria care ($1.42 to $4.46 USD) [53]. If taken to scale, total TCI costs would increase, but because a coach can cover a larger number of HFs in non-research settings, per-patient costs would likely decrease. The opportunity costs of the health workers (HF employees) would increase the cost of the overall intervention by 10%, leading to a cost of $3.88 per additional case with complete recording of clinically-relevant data. However, if the goal was 100% coverage, there would be a point of diminishing marginal returns as covering the most remote or difficult HFs would be more costly and improve coverage only slightly. Finally, there could be a reduction in both cost and effect sizes, if the intervention were carried out by the MOH in conditions of insufficient human or financial resources.

Strengths and limitations

An independent assessment of the intervention was conducted, focusing on the completeness of register data (a key internal measure of quality for HFs) and how well the monthly reports (the only source of HF information available to the MOH for calculating malaria indicators) reflect the data collected by HFs. This study was more rigorous that previous CI studies because it included an independent evaluation, used a longer baseline period, and used an objective data source for the evaluation (information from registers and monthly reports abstracted by a trained data collection team). Moreover, the study included both quantitative and qualitative evaluations [48, 50] to explore whether (or to what extent) CI worked, and how it worked. The TCI intervention followed the CI methodology, aside from scale: only 5 HFs were included in this proof-of-concept pilot, compared to the recommended network of 20–100 HFs for a well-established implementation package. The decision to combine training with CI was based on preliminary results of a systematic review that showed larger effect sizes from this combination compared to CI alone, albeit based on studies that generally had a high risk of bias [32].

The study had several limitations. First, the sample size was small. While three levels of HFs were included in the study (II, III, and IV), the results might not be generalizable to the reporting practices and effect of the CI approach of the entire healthcare system in Uganda. However, the included HFs are fairly typical for Uganda in terms of their recording and reporting practices. Conducting the study in a single district (Kayunga) also limits the geographical representativeness of the study. However, the intent of this study was to be an in-depth exploration, both quantitative and qualitative, of the inner workings of the CI method for improvement of the surveillance data quality, and this purpose was best realized through a small pilot study due to the large efforts required to conduct the qualitative work.

No data were collected except those directly related to the intervention’s effect on routine malaria data quality. However, it is possible that other aspects of performance of the health workers in health service delivery may have changed. On one hand, it is possible that the improvement methods implemented during this intervention may have been used independently by health workers to improve other aspects of their performance. For example, clinicians may have used the same methods to increase the proportion of non-malaria patient records with complete clinically relevant data. Conversely, it is possible that other aspects of data quality may have worsened because clinicians were focused on malaria data, to the detriment of other work. However, because only malaria-specific data were collected, the impact beyond the primary focus of the intervention could not be assessed. The qualitative component of the study evaluated the unintended consequences of the intervention (e.g., effects of changes to patient flow to better capture data on the patient waiting time) from multiple angles, as reported by Hutchinson et al. [48, 50]. Future studies should collect data on a wider spectrum of quality indicators to examine spill-over effects, which could be positive or negative.

The brief time between the introduction of the revised forms by the MOH and the start of the study intervention precluded meaningful analysis of the scale-up of the revised forms in the time series design and segmented regression analysis. Therefore, it was not possible to fully evaluate the effect of introducing the revised forms on HF data quality. Similarly, because the study intervention included the in-service training plus CI, the study was not designed to assess the effects of the individual components of the intervention separately, although a recent comprehensive systematic review sheds some light on the effects of these components [32].

Conclusions

This rigorously designed pilot study found that the TCI intervention was associated with large improvements in completeness, but not accuracy, of routinely collected malaria surveillance data. Future evaluations of TCI should ideally employ a cluster-randomized trial design with a larger sample size and longer follow-up period. It would be also recommended that the content of TCI be expanded to testing and treatment practices for malaria, as well as to other diseases. Sustainability evaluations and larger studies are needed before recommending wide-scale implementation of the CI intervention.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

HMIS:

Health Management Information Systems

CI:

Collaborative improvement

PDSA:

Plan-do-study-act

WHO:

World Health Organization

LMIC:

Low- and middle-income country

HC:

Health Centre

OPD:

Out-patient department

FGD:

Focus group discussion

USD:

United States dollars

CDC:

Centers for Disease Control and Prevention

References

  1. Chilundo B, Sundby J, Aanestad M. Analysing the quality of routine malaria data in Mozambique. Malar J. 2004;3:3.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Cibulskis RE, Bell D, Christophel EM, Hii J, Delacollette C, Bakyaita N, et al. Estimating trends in the burden of malaria at country level. Am J Trop Med Hyg. 2007;77(Suppl 6):133–7.

    Article  PubMed  Google Scholar 

  3. Oduro AR, Bojang KA, Conway DJ, Corrah T, Greenwood BM, Schellenberg D. Health centre surveys as a potential tool for monitoring malaria epidemiology by area and over time. PLoS ONE. 2011;6:e26305.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Rowe AK, Kachur SP, Yoon SS, Lynch M, Slutsker L, Steketee RW. Caution is required when using health facility-based data to evaluate the health impact of malaria control efforts in Africa. Malar J. 2009;8:209.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Garrib A, Stoops N, McKenzie A, Dlamini L, Govender T, Rohde J, et al. An evaluation of the District Health Information System in rural South Africa. S Afr Med J. 2008;98:549–52.

    CAS  PubMed  Google Scholar 

  6. Hahn D, Wanjala P, Marx M. Where is information quality lost at clinical level? A mixed-method study on information systems and data quality in three urban Kenyan ANC clinics. Glob Health Action. 2013;6:21424.

    Article  PubMed  Google Scholar 

  7. Kihuba E, Gathara D, Mwinga S, Mulaku M, Kosgei R, Mogoa W, et al. Assessing the ability of health information systems in hospitals to support evidence-informed decisions in Kenya. Glob Health Action. 2014;7:24859.

    Article  PubMed  Google Scholar 

  8. Makombe SD, Hochgesang M, Jahn A, Tweya H, Hedt B, Chuka S, et al. Assessing the quality of data aggregated by antiretroviral treatment clinics in Malawi. Bull World Health Organ. 2008;86:310–4.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Maokola W, Willey BA, Shirima K, Chemba M, Armstrong Schellenberg JR, Mshinda H, et al. Enhancing the routine health information system in rural southern Tanzania: successes, challenges and lessons learned. Trop Med Int Health. 2011;16:721–30.

    Article  CAS  PubMed  Google Scholar 

  10. Mavimbe JC, Braa J, Bjune G. Assessing immunization data quality from routine reports in Mozambique. BMC Public Health. 2005;5:108.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Nyamtema AS. Bridging the gaps in the Health Management Information System in the context of a changing health sector. BMC Med Inform Decis Mak. 2010;10:36.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Githinji S, Oyando R, Malinga J, Ejersa W, Soti D, Rono J, et al. Completeness of malaria indicator data reporting via the District Health Information Software 2 in Kenya, 2011–2015. Malar J. 2017;16:344.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Amouzou A, Kachaka W, Banda B, Chimzimu M, Hill K, Bryce J. Monitoring child survival in ‘real time’ using routine health facility records: results from Malawi. Trop Med Int Health. 2013;18:1231–9.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Mate KS, Bennett B, Mphatswe W, Barker P, Rollins N. Challenges for routine health system data management in a large public programme to prevent mother-to-child HIV transmission in South Africa. PLoS ONE. 2009;4:e5483.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  15. WHO. World Malaria Report. Geneva: World Health Organization; 2013.

    Google Scholar 

  16. WHO. World Malaria Report. Geneva: World Health Organization; 2014.

    Google Scholar 

  17. WHO. Assessment of health facility data quality; data quality report card Uganda, 2010–2011. Geneva: World Health Organization; 2011.

    Google Scholar 

  18. Gething PW, Battle KE, Bhatt S, Smith DL, Eisele TP, Cibulskis RE, et al. Declining malaria in Africa: improving the measurement of progress. Malar J. 2014;13:39.

    Article  PubMed  PubMed Central  Google Scholar 

  19. WHO. Disease surveillance for malaria control. Geneva: World Health Organization; 2012.

    Google Scholar 

  20. Kiberu VM, Matovu JK, Makumbi F, Kyozira C, Mukooyo E, Wanyenze RK. Strengthening district-based health reporting through the district health management information software system: the Ugandan experience. BMC Med Inform Decis Mak. 2014;14:40.

    Article  PubMed  PubMed Central  Google Scholar 

  21. PMI. Uganda-Malaria Operational Plan FY 2018. President’s Malaria Initiative, Kampala. 2017.

  22. WHO. T3: Test. Treat. Track. Scaling up diagnostic testing, treatment and surveillance for malaria. Geneva: World Health Organization; 2012.

    Google Scholar 

  23. Hulscher ME, Schouten LM, Grol RP, Buchan H. Determinants of success of quality improvement collaboratives: what does the literature show? BMJ Qual Saf. 2013;22:19–31.

    Article  PubMed  Google Scholar 

  24. Landon BE, Wilson IB, McInnes K, Landrum MB, Hirschhorn L, Marsden PV, et al. Effects of a quality improvement collaborative on the outcome of care of patients with HIV infection: the EQHIV study. Ann Intern Med. 2004;140:887–96.

    Article  PubMed  Google Scholar 

  25. Power M, Tyrrell PJ, Rudd AG, Tully MP, Dalton D, Marshall M, et al. Did a quality improvement collaborative make stroke care better? A cluster randomized trial. Implement Sci. 2014;9:40.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Schouten LM, Hulscher ME, van Everdingen JJ, Huijsman R, Grol RP. Evidence for the impact of quality improvement collaboratives: systematic review. BMJ. 2008;336:1491–4.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Franco LM, Marquez L. Effectiveness of collaborative improvement: evidence from 27 applications in 12 less-developed and middle-income countries. BMJ Qual Saf. 2011;20:658–65.

    Article  PubMed  Google Scholar 

  28. Institute for Healthcare Improvement. The breakthrough series: IHI’s collaborative model for achieving breakthrough improvement. Diabetes Spectr. 2004;17(2):97–101.

  29. United States Agency for International Development (USAID). The improvement collaborative: An approach to rapidly improve health care and scale up quality services. 2008.

  30. USAID. Evaluating health care collaboratives: the experience of the quality assurance project. In: Collaborative evaluations series. University Research Co: LLC; 2008.

  31. Boucar M, Hill K, Coly A, Djibrina S, Saley Z, Sangare K, et al. Improving postpartum care for mothers and newborns in Niger and Mali: a case study of an integrated maternal and newborn improvement programme. BJOG. 2014;121(Suppl 4):127–33.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Rowe AK, Rowe SY, Peters DH, Holloway KA, Chalker J, Ross-Degnan D. Effectiveness of strategies to improve health-care provider practices in low-income and middle-income countries: a systematic review. Lancet Glob Health. 2018;6:e1163–75.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Waiswa P, Manzi F, Mbaruku G, Rowe AK, Marx M, Tomson G, et al. Effects of the EQUIP quasi-experimental study testing a collaborative quality improvement approach for maternal and newborn health care in Tanzania and Uganda. Implement Sci. 2017;12:89.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  34. Garcia-Elorrio E, Rowe SY, Teijeiro ME, Ciapponi A, Rowe AK. The effectiveness of the quality improvement collaborative strategy in low- and middle-income countries: a systematic review and meta-analysis. PLoS ONE. 2019;14:e0221919.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  35. Wells S, Tamir O, Gray J, Naidoo D, Bekhit M, Goldmann D. Are quality improvement collaboratives effective? A systematic review. BMJ Qual Saf. 2018;27:226–40.

    Article  PubMed  Google Scholar 

  36. Berwick DM. The question of improvement. JAMA. 2012;307:2093–4.

    Article  CAS  PubMed  Google Scholar 

  37. Kilo CM. A framework for collaborative improvement: lessons from the Institute for Healthcare Improvement’s Breakthrough Series. Qual Manag Health Care. 1998;6:1–13.

    Article  CAS  PubMed  Google Scholar 

  38. Berwick DM, Nolan TW. Physicians as leaders in improving health care. Ann Intern Med. 1998;128:289–92.

    Article  CAS  PubMed  Google Scholar 

  39. Deming EW. Out of the crisis. Cambridge, MA: MIT Press; 2000.

    Google Scholar 

  40. May C. A rational model for assessing and evaluating complex interventions in health care. BMC Health Serv Res. 2006;6:86.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Mittman BS. Creating the evidence base for quality improvement collaboratives. Ann Intern Med. 2004;140:897–901.

    Article  PubMed  Google Scholar 

  42. Barker PM, McCannon CJ, Mehta N, Green C, Youngleson MS, Yarrow J, et al. Strategies for the scale-up of antiretroviral therapy in South Africa through health system optimization. J Infect Dis. 2007;196(Suppl 3):S457–63.

    Article  PubMed  Google Scholar 

  43. Sibley LM, Tesfaye S, Fekadu Desta B, Hailemichael Frew A, Kebede A, Mohammed H, et al. Improving maternal and newborn health care delivery in rural Amhara and Oromiya regions of Ethiopia through the Maternal and Newborn Health in Ethiopia Partnership. J Midwifery Womens Health. 2014;59(Suppl 1):S6–20.

    Article  PubMed  Google Scholar 

  44. Singh K, Speizer I, Handa S, Boadu RO, Atinbire S, Barker PM, et al. Impact evaluation of a quality improvement intervention on maternal and child health outcomes in Northern Ghana: early assessment of a national scale-up project. Int J Qual Health Care. 2013;25:477–87.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Stover KE, Tesfaye S, Frew AH, Mohammed H, Barry D, Alamineh L, et al. Building district-level capacity for continuous improvement in maternal and newborn health. J Midwifery Womens Health. 2014;59(Suppl 1):S91–100.

    Article  PubMed  Google Scholar 

  46. Twum-Danso NA, Dasoberi IN, Amenga-Etego IA, Adondiwo A, Kanyoke E, Boadu RO, et al. Using quality improvement methods to test and scale up a new national policy on early post-natal care in Ghana. Health Policy Plan. 2014;29:622–32.

    Article  PubMed  Google Scholar 

  47. Colbourn T, Nambiar B, Bondo A, Makwenda C, Tsetekani E, Makonda-Ridley A, et al. Effects of quality improvement in health facilities and community mobilization through women’s groups on maternal, neonatal and perinatal mortality in three districts of Malawi: MaiKhanda, a cluster randomized controlled effectiveness trial. Int Health. 2013;5:180–95.

    Article  PubMed  Google Scholar 

  48. Hutchinson E, Nayiga S, Nabirye C, Taaka L, Westercamp N, Rowe AK, et al. Opening the ‘Black Box’ of Collaborative Improvement: a qualitative evaluation of a pilot intervention to improve quality of malaria surveillance data in public health centres in Uganda. Malar J. 2021. https://0-doi-org.brum.beds.ac.uk/10.1186/s12936-021-03805-z.

    Article  PubMed  PubMed Central  Google Scholar 

  49. Wagner AK, Soumerai SB, Zhang F, Ross-Degnan D. Segmented regression analysis of interrupted time series studies in medication use research. J Clin Pharm Ther. 2002;27:299–309.

    Article  CAS  PubMed  Google Scholar 

  50. Hutchinson E, Nayiga S, Nabirye C, Taaka L, Staedke SG. Data value and care value in the practice of health systems: a case study in Uganda. Soc Sci Med. 2018;211:123–30.

    Article  PubMed  Google Scholar 

  51. Catsambas TT, Franco LM, Gutmann M, Knebel E, Hill P, Lin Y-S. Evaluating health care collaboratives: the experience of the Quality Assurance Project. Bethesda, MD: University Research Co., LLC (URC); 2008.

  52. Hansen KS, Ndyomugyenyi R, Magnussen P, Lal S, Clarke SE. Cost-effectiveness analysis of malaria rapid diagnostic tests for appropriate treatment of malaria at the community level in Uganda. Health Policy Plan. 2017;32:676–89.

    Article  PubMed  PubMed Central  Google Scholar 

  53. Castellani J, Nsungwa-Sabiiti J, Mihaylova B, Ajayi IO, Siribie M, Afonne C, et al. Impact of improving community-based access to malaria diagnosis and treatment on household costs. Clin Infect Dis. 2016;63(Suppl 5):S256–63.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We would like to thank the Ugandan Ministry of Health, National Malaria Control Program, the Kayunga District Health Officer, biostatistician, the district biostatisticians, HMIS focal person and health educator; the administration and staff of the health centres for participating in the study and for their time, enthusiasm and dedication; the UMSP study team: Susan Naiga, Christine Nabirye, Lilian Taaka, Alex Ndyabakira, John Michael Okiring, SP Kigozi, Asadu Sserwada, Ruth Kigozi, Catherine Maiteki-Sebuguzi, Joseph Kayondo, and the data abstraction team; URC/ASSIST staff for their expertise and support the CI intervention implementation; Allen Eva Okullo, FETP fellow attached to NMCP for her assistance with HMIS data review; as well as PMI Uganda team (BK Kapella, Kassahun Belay, Gloria Sebikaari, and Joel Kisubi) for their support. Special thanks to the study investigators and collaborators for their contributions to the study design and implementation: Grant Dorsey (UCSF), Rashad Massoud (URC/ASSIST), Martin Muhire (URC/ASSIST), Esther Karamagi Nkolo (URC/ASSIST), Mirwais Rahimzai (URC/ASSIST), Edward Broughton (URC/ASSIST), Victor Boguslavsky (URC/ASSIST), Arthur Mpimbaza (UMSP), Peter Okui (NMCP), Mary Hamel (CDC), Steven Yoon (CDC), BK Kapella (CDC/PMI Uganda), Humphrey Wanzira (NMCP).

Funding

The study was funded by the US President’s Malaria Initiative through UCSF Grant No. 5UGH000076-04 and USAID contract AID-617-O-00016. The findings and conclusions presented in this report are those of the authors and do not necessarily reflect the official position of the U.S. Centers for Disease Control and Prevention or the US President’s Malaria Initiative.

Author information

Authors and Affiliations

  1. Malaria Branch, Division of Parasitic Diseases and Malaria, Centers for Disease Control and Prevention, 1600 Clifton Road, Atlanta, GA, 30333, USA

    Nelli Westercamp & Alexander K. Rowe

  2. Department of Clinical Research, London School of Hygiene and Tropical Medicine, Keppel Street, London, WC1E 7HT, UK

    Sarah G. Staedke & Eleanor Hutchinson

  3. Infectious Diseases Research Collaboration, 2C Nakasero Hill Road, Kampala, Uganda

    Catherine Maiteki-Sebuguzi, Alex Ndyabakira, John Michael Okiring, Simon P. Kigozi & Grant Dorsey

  4. Department of Medicine, University of California, San Francisco, USA

    Grant Dorsey

  5. ASSIST Project, University Research Co., LLC, 5404 Wisconsin Avenue, Suite 600, Chevy Chase, MD, 20815, USA

    Edward Broughton & M. Rashad Massoud

Authors
  1. Nelli Westercamp
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sarah G. Staedke
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Catherine Maiteki-Sebuguzi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alex Ndyabakira
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. John Michael Okiring
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Simon P. Kigozi
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Grant Dorsey
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Edward Broughton
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Eleanor Hutchinson
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. M. Rashad Massoud
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Alexander K. Rowe
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

NW and AKR conceived of the study, with input from SGS, GD, RM, EB, and EH. NW and AKR drafted the protocol, with inputs from SGS, GD, EB, and EH. CMS, AN, and JMO led the data collection in the field, with oversight from SGS and support from NW. SK managed the data, and NW led the data analysis, with support from AKR and SGS. EB led the cost-effectiveness analysis. NW interpreted the data and drafted the manuscript, with input from SGS and AKR. All authors reviewed the manuscript and gave permission for publication. NW, the corresponding author, had full access to all the data in the study and had final responsibility for the decision to submit for publication. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Nelli Westercamp.

Ethics declarations

Ethics approval and consent to participate

The study was approved by the Ugandan National Council for Science and Technology (UNCST Ref HS 1882), the Makerere University School of Biomedical Sciences Research & Ethics Committee (SBS-REC Ref. 276), the London School of Hygiene and Tropical Medicine Ethics Committee (LSHTM Ref 9717), and the Centers for Disease Control and Prevention Institutional Review Board (CDC Ref. 6741).

Consent for publication

Not applicable.

Competing interests

None declared.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1:

Annex 1. The change package resulting from the TCI intervention.

Additional file 2:

Annex 2. Inputs for cost-effectiveness decision tree.

Additional file 3:

Annex 3. Health-facility specific graphs.

Additional file 4:

Annex 4. Sensitivity analyses and models by study site for the completeness indicators.

Additional file 5

: Annex 5. Directionality and magnitude of dispersion of the accuracy indicators by study site.

Additional file 6

: Annex 6. Correlation between the accuracy (discordance in malaria cases between OPD register and monthly report) and clinically-relevant completeness data collected through independent evaluation and data reported by the HF-based CI teams.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Westercamp, N., Staedke, S.G., Maiteki-Sebuguzi, C. et al. Effectiveness of in-service training plus the collaborative improvement strategy on the quality of routine malaria surveillance data: results of a pilot study in Kayunga District, Uganda. Malar J 20, 290 (2021). https://0-doi-org.brum.beds.ac.uk/10.1186/s12936-021-03822-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s12936-021-03822-y

Keywords

Malaria Journal

ISSN: 1475-2875

Contact us