the e

Strengthening Ties between Planning and M&E: A Comparison of 2012-2013
Performance Evaluation, Brazilian Department of STD/AIDS and Viral Hepatitis
Authors:
Valverde LF, dos Santos EM, Givisiez JM, Habckost CHB, Christ MTS, de Arruda MR, Pascom ARP
Institute:
Department of STD, AIDS and Viral Hepatitis, Brazilian Ministry of Health
Introduction
Since 2012 the Brazilian Department of STD, AIDS and Viral Hepatitis (DDAHV) is using
an integrated performance model - Évaluation Globale et Intégrée de la Performance des
Systèmes de Santé (EGIPSS) - to evaluate its strategic plan.
Performance evaluation is justified by the need to strengthen management strategies for
efficient and effective public spending with need for better results.
The EGIPSS model assumes that the performance of an organization consists in its ability
to integrate four functions of a social action system: adaptation, achievement of goals,
production and cooperation, and creation and maintenance of organizational values.
Performance evaluation (PE) encompassed three steps: strategic analysis, logic modelling
and implementation analysis.
This study compared results obtained in 2012 and 2013 reinforcing lessons learned and
reconfiguration of the model.
Materials and methods
A systematic review of literature was performed exploring the evaluative capacity of
performance models to select the most appropriate one. The evaluation capacity was
defined as the ability of the models to describe, explain and judge the fields of assessing,
i.e., its power to modify the problem (risk control and damage), its modes of operation
(production, technical quality and cost) and its interactive interface with users. The EGIPSS
model was identified with greater potency evaluation.
Adapting
Maintaining values
- Corporate values (equity, solidarity.
freedom, dignity of the person
V
Figure 1. The EGIPSS performance model.
Al
ive
tim
i
eg
L
at
ive
Strategic analysis showed a prevalence of strategic alignment (for example, the creation
of a project of HIV rapid test using oral fluid in key populations to increase the number
of HIV tests realized) and tactical alignment (expansion of HIV testing in primary care,
particularly in family health, to achieve the same goal – to increase the number of HIV
tests realized). Little attention was given to the allocative (acquisition of HIV rapid tests
to increase the volume of care and services) and contextual alignments (agreement with
SUS managers to expand HIV testing).
Tractical alignment
at
Results
nt
loc
me
Al
Figure 2. Examples of activities classified according to domains and alignments of EGIPSS model.
- Reduce the incidence, duration,
intensity, and impact of health
problems
- Effectiveness, efficiency
G
ign
A
Figure 4. Degree of implementation of strategic planning 2012 and 2013.
Achieving Goals
Strategic alignment
Contextual alignment
- Acquisition of resources
- adapting to the public’s needs
- Capacity to attract new clients
- Mobilizing the community
- Innovation and transformation
The implementation analysis was developed based only on Strategic Planning.
The following table compares the degree of implementation that has increased from 46%
in 2012 to 70% in 2013.
Al
ign
me
nt
Production
Operational alignment
P
- Productivity
- Volume of care and services
- Quality of care and services
Source: Contandriopoulos et al., 2008,
After that, strategies, intermediate goals, targets and DDAHV’s planning indicators were
classified according to the domains and alignments listed in the model. Descriptive analysis
of EGIPSS, development of strategic analysis, logical analysis and implementation analysis
were performed, characterizing the observed level of achievement of the strategic goals,
for 2012 and 2013. The performance evaluation intends to answer the questions: “Are the
interventions proposed consistent with the identified problems?” (strategic analysis); “Is
there a description of the logical chain of events to achieve the desired effects?” (logical
analysis); “Are we running what we planned?“ (implementation analysis).
Figure 3. *Strategic planning and AIDS-SUS Project indicators are some of the indicators monitored by the DDAHV. All of them were classified according to domains and alignments of EGIPSS model.
Using the strategic analysis, we verified the necessity to discuss actions that ensure and
strengthen the department in its political-institutional placement (contextual alignment).
One example of this would be, regarding TasP implementation, the involvement of support
networks and NGO.
Also we observe little evidence in allocative alignment (some targets involving cost for
the technical action) and few initiatives involving operational alignment (changes from
competitive relation work to collaborative). Although, there is some emphasis on the
organization of work processes and permanent capacity building (tactical alignment).
From the modeling, it was identified that there is no standardized definition of intermediate
goal. The M&E advisory of DDAHV has defined intermediate goal as a set of related
activities that cumulatively lead to a result.
Regardless of conducting a workshop modeling in 2013, the M&E advisory remains
developing logic models. The participation of the areas in this process has progressively
increased, although much less than necessary.
Once the modeling is performed by the M&E advisory, the prioritization of activities
and their monitoring are not incorporated by technical areas. Monitoring and modeling
persist as processes executed under pressure.
To sum up, the frequency of cross-cutting activities, usually associated with tactical
alignment, diverges from the predominance of intermediate goals ranked in strategic
alignment. The degree of no implementation of the cross-cutting activities may be
reflecting the no incorporation of logic modeling.
Despite the increase observed in the degree of implementation, the analysis only refers to
informed activities. Therefore, it might just be expressing the improvement of information
gathering
Conclusion
The comparison of the PEs conducted in 2012 and 2013 showed improvements concerning
strategic planning and logic modeling. Two sets of factors have contributed for that: the
context of Strategic Planning and Monitoring and Evaluation (M&E) in the Ministry of
Health and the new management guidelines of the DDAHV that emphasized the use of
M&E for the process of decision making.
Progress observed was associated with the relationship between the creation/maintenance
of organizational values and achievement of goals (legitimation alignment); adaptation
and achievement of goals (strategic alignment). These factors created convergent paths
towards the institutionalization of Monitoring and Evaluation (M&E) that were assessed
through linkages established among planning processes and M&E processes.
Lastly, evidences showed the need to integrate planning and M&E, spreading the practice
of logic modeling and periodic verification of the implementation by the technical areas.
Furthermore, PE still needs efforts to be institutionalized. Since PE is an indispensable
component of best practices in outcome oriented management, the continuity of this
work is crucial for organizational development.
Literature cited
Ministério da Saúde, Secretaria de Vigilância em Saúde, Departamento DST, Aids e Hepatites Virais. Relatórios Internos, 2012.
Reis, AC. A noção de equilíbrio como proxy do desempenho de programas de saúde. Rio de Janeiro, 2012. 111 f. Tese (Doutorado)
– Escola Nacional de Saúde Pública Sérgio Arouca.
Chen H. The Theory-Driven Perspective: the new alternative. In: Theory–Driven Evaluations. Newbury Park: Sage Publications,
1990. p. 39-56.
Contandriopoulos, AP, Champagne, F, Denis, JL, & Avargues, MC. L’e´valuation dans le domaine de la sante´ : Concepts et
methodes. Revue d’e´pide´miologie et de sante´ publique, 48, 2000. p. 517–539.
Donaldson SI. Developing Program Impact Theory. In: Program Theory – Driven Evaluation Science: strategies and applications.
New York – London: Psychology Press, 2007. p. 20-39.
Weiss, CH. How can theory-based evaluation make greater headway? Evaluation Review, 21, 1997. p. 501–524.
Rossi, PH, Lipsey, MW, & Freeman, HE. Evaluation: A systematic approach (7th ed.). Thousand Oaks, CA: Sage Publications,
2004.
Reis, AC; Santos, EM; Arruda, MR. Modelos de Avaliação de Desempenho de Sistemas de Saúde: diferenças e similaridades.
Tese (Doutorado) A noção de equilíbrio como proxy do desempenho de programas de saúde. Rio de Janeiro, 2012. Escola
Nacional de Saúde Pública Sérgio Arouca.
Contandriopoulos, AP; Trottier, LH; Champagne, F. Improving performance: a key for Quebec´s health and social services
centres. Infoletter (Thema), v. 5, n. 2, 2008.
Parsons T. Orientações teóricas, In: O sistema das sociedades modernas. São Paulo: Pioneira, 1974.
Ministério da Saúde, Secretaria de Vigilância em Saúde, Programa Nacional de DST e Aids. Manual da Oficina de capacitação
em avaliação com foco na melhoria do Programa. Série F – Comunicação e Educação em Saúde. 2007, 112p.
E-CAR, Controle, Acompanhamento e Avaliação de Resultados [access 03/06/2013 às 16:10]. http://ecar.saude.gov.br:8080/
pe2012.