END TERM EVALUATION (ETE) FOR THE AFYA UGAVI (AU) ACTIVITY At Chemonics

Activity/Project – Afya Ugavi Activity

Position Type – Short – Term Consultancy

Study/Assessment Topic – End Term Evaluation

Position Location – Nairobi, and 36 project counties

Duration – 20 consultancy days

Reporting To – The Chief of Party, and working closely with the M&E Specialist

Working With

The Technical Programs; Malaria, HIV/AIDS, FP/RMNCAH, LMIS and MEDS Advisors, Technical Officers and County Coordinators

Starting Date/Month

August 2020

1.0 Evaluation Background

Afya Ugavi Activity has been implementing health supply chain strengthening activities collaboratively with the Government of Kenya (GoK)/ Ministry of Health (MOH) at the national level and with select county government departments of health since October 2016. Implemented activities focused on strengthening the supply chain for HIV/AIDS, malaria, Family Planning/ Reproductive Health, Maternal, Newborn Child and Adolescent Health (FP/RMNCAH) both at the national and county level. The interventions are planned to have a diffusive effect on the entire health supply chain, including essential medicines and medical supplies (EMMS). Through a recent modification #19, Afya Ugavi is currently supporting integrated supply chain activities in 36 counties, while coordinating HPTU establishment at the national level and in all the 47 counties.

Programmatically, AU provides FP/RMNCAH supply chain technical assistance to the Division of Reproductive and Maternal Health (DRMH) and the HIV program through the Division of National AIDS and Sexually Transmitted Infections (STI) Control Program (NASCOP). For malaria, AU collaborates with the Division of National Malaria Program (DNMP) and in addition has taken up LLIN (long lasting insecticide nets) distribution and management in 36 counties across the country after the exit of the previous implementing mechanism. Specifically, the Activity coordinated mass net distribution in Bungoma County in May – July 2021, which included logistical operations for the same. In addition, the Activity was engaged to offer technical support to MEDS (Mission for Essential Drugs and Supplies) in readiness for its role as a possible recipient and distributor of USAID funded health program commodities.

As stipulated in the Task Order contract at Activity inception, planned interventions were premised towards fulfilment of the following outcomes and outputs; supply chain and commodity management systems” at national and county levels. Expected outcomes and outputs are described as follows.

Outcome 1: Strengthened national level systems for supply chain and commodity management, with two contributory outputs;

Output 1.1: Strengthened technical leadership and coordination for commodity management, and

Output 1.2: Improved supply chain logistics and commodity management.

Outcome 2: Strengthened County oversight and implementation of supply chain services through the following expected outputs.**

Output 2.1: Increased human and institutional capacity for commodity management, and;

Output 2.2: Improved commodity security environment

At the same time, USAID has been reviewing its support towards supply chain interventions to ensure a consistent focus on interventions across the health sector. A part of this evaluation, and in addition to reviewing if AU is working in accordance with its stated objectives, the mission is also evaluating whether all funded implementing partners are working in accordance with the mission’s new supply chain priority objectives which include.

1.) Standardizing supply chain services at the facility level through a standard package of services

2.) Reducing overlaps in supply chain support between the service delivery partners and the mission’s supply chain partner (AU)

3.) Minimizing supply chain main risks, considering recent HPT related challenges and strengthening supply chain governance

Evaluation of these priority objectives in detail alongside Afya Ugavi output areas will assist the mission to align its future support appropriately, with focus on areas of opportunities and emphasis on collaboration on others that cut across the partners in line with the above stated objectives.

2.0 Purpose of the End Term Evaluation

Afya Ugavi Activity has been implemented for almost five years already within its implementation lifespan (2016 – 2021), which necessitates for an audit of its performance so far against its stated objectives. The purpose of this evaluation is therefore to assess progress made towards achievement of the overall and specific objectives of this Activity, its impacts on the target stakeholders who have been directly involved in implementation, as well as with policy makers at county and national levels up to the closeout period. In addition, this evaluation will generate recommendations on how outcomes from the Activity can incorporate the mission’s new direction around supply chain and potentially inform development of a new follow-on mechanism. The assessment will also help document Activity impact in targeted counties and at national level and gauge stakeholders’ participation and ownership. The implementation review process will also include identification of intended and unintended outcomes, best practices and lessons learned as well as challenges encountered during Activity implementation.

The End -Term Evaluation therefore has four main objectives:

(i) to evaluate progress towards meeting the stated goal, intermediate results, and outputs of the project.

(ii) to evaluate the systems in place to measure and report on results; and

(iii) recommend actions required in the next follow-on project and ensure that outcomes, impacts and lessons from current interventions are incorporated.

(iv) To provide recommendations to USAID on how lessons from implementation experience can leverage on the mission’s supply chain priorities.

The ETE will also determine sufficiency, effectiveness, sustainability, and impact of the interventions. The evaluation will be participatory, involving AU, Ministry of Health (MoH), USAID local mission staff, local implementing partners, MEDS, County and Sub County health leadership and related health commodity management staff at facility level.

3.0 Specific Tasks of the Consultant

In order to realize the evaluation objectives stated above, the Consultant is required to carry out and report on the following tasks.

a) Compare baseline with end-term progress and assess whether the objectives and outcomes are on track /have been achieved.

b) Assess each disease program intervention area and document progress from 2016/17 through to 2021/22.

c) Assess and highlight unplanned results, unexpected constraints and new interventions not included in the AWPs/contract.

d) Assess innovations (tools, manuals, guidelines or approaches) that the program developed or used, their suitability towards health supply chain development and their suitability in the current devolved environment.

e) Document innovative strategies being used by the Activity and demonstrate how these innovations have pivoted change in health commodity management in the public sector.

e) Assess how the data or information emerging from, continuous assessments, health commodity system audits, operational research and any other special studies have been applied to shape /impact Activity implementation.

f) Assess achievements in other cross intervention areas such as supply chain audits, data review forums, facility in charges meetings, and supportive supervision among others, and how these multi-sectoral approaches have led to a holistic improvement in supply chain and the health service delivery.

g) Assess any modifications of interventions so far and comment on whether these modifications addressed the identified needs.

i) Document good practices and lessons learned in the implementation of the Activity.

j) Document on sustainability and potential replicability of interventions, especially with in terms of integration into the health delivery system at facility level.

k) Assess the Activity’s implementation in relation to financial expenditure, existing gaps, and measures towards ensuring MoH’s sustainability

l) Identify issues, risks, and challenges within the project, which may have impacted the achievement of project results and provide recommendations for posterity.

iv) Provide recommendations to USAID on how Afya Ugavi outcomes, best practices and lessons learnt from interventions can be applied to the mission’s supply chain priority objectives in the follow-on period.

4.0 Evaluation Methodology

The consultant will be responsible for defining and carrying out the overall ETE approach, including development of tools, specification of the techniques for data collection and analysis, structured field visits and interactions with the beneficiaries and the implementation teams (KIIs), and preparation and presentation of findings and recommendations.

The evaluation methodology should include both quantitative and qualitative evaluation methods, with focus on interpretation of the results, internal review of the activities and outcomes based on the Activity interventions, and the approach deemed appropriate by the consultant.**

On overall the process must speak to the following broad evaluation questions.

· Was the implementation done as per the contract specifications and modifications? (Has the implementation outcome so far been relevant, appropriate, and strategic to national/county level supply chain development goals?)

· Are the results sustainable? Was sustainability built in the implementation of activities? (Will the achieved and projected outputs and outcome(s) lead to benefits beyond the life of the Activity?).

· Lessons learnt and best practices. How could implementation be improved in the future? (What will be relevant for future programming for similar interventions?).**

5.0 Responsibilities

The consultant will be responsible for:

· Developing a work plan and roadmap of implementation for the ETE, not later than 1 (one) week prior to the commencement of the evaluation.

· Developing all the data collection tools to be used in this evaluation.

· Conducting the End Term Evaluation, including the day-to–day management.

· Conducts data analysis and synthesizes evaluation findings to generate evaluative judgements and recommendations

· Conducting periodic de-briefs with the Project Management Team and key staff.

· Present draft findings to the Activity Team on the final day of the evaluation; and

· Preparing and submitting a final ETE report to Afya Ugavi.**

Afya Ugavi will also be responsible for the following;

· Facilitating the consultants travel and logistical support to Activity locations outside Nairobi County.

· Providing copies of all required reference materials including baselines, progress reports, inception reports, financial information among others

· The Activity M&E Specialist works closely with external consultants on all components of evaluation process

· The Chief of Party (COP) will oversee the progress ETE, participate in ad hoc reviews, supervise the consultant, review the evaluation products and provide input where required.

· The COP and PMU Director will approve the evaluation budget, questions, timeline and have the final sign off on the final evaluation report.

6.0 Deliverables

At the conclusion of the ETE process, the following deliverables will be expected from the consultant.

i. A detailed concept note/inception report for the evaluation with the consultant’s understanding on the following areas;

• An understanding of the Activity’s goals, mandate, and coverage over time.

• An elaborate description of the evaluation methodology.

· Detailed roadmap

• Annexes which may include tools or instruments to be used for the evaluation; a copy of the evaluation schedule; and a list of key informants to be interviewed with their titles/organizations.

ii. Debriefing meeting with Afya Ugavi and invited stakeholders on preliminary findings at the end of the field assessment and draft report writing to share preliminary findings and gather feedback for writing the final report.

iii. A draft ETE report prepared and shared with AU for review and feedback within seven days after the debriefing meeting. The consultant report will address all points under section 3.0 of this TOR along with annexes which should include tools or instruments used for the evaluation, the evaluation schedule and a list of people interviewed or consulted.

iv. A final evaluation report due within five days of receiving feedback on the draft report from Afya Ugavi

7.0 ETE Implementation Time Frame

The ETE consultancy is planned for a period of 20 consultancy days.

8.0 Required Expertise for the Consultant

The following are the qualification requirements for the ETE consultant.

· Preferably a Kenyan citizen or possessing legal requirements to work in Kenya.

· At minimum, a Master’s degree in the health sciences, social sciences, behavioral sciences, health systems research, Health supply chains or a related field.

· Previous experience in conducting of project/program evaluations/reviews, with evidence of work done.

· At least 5 years’ experience in M&E, health systems strengthening, public health, heath supply chains management and/or capacity building of public sector entities

· Excellent knowledge of the public health sector in Kenya, particularly on supply chain systems for HIV, malaria and FP and essential medicines.

· Excellent verbal, interpersonal and written communication skills.

· Strong analytical, problem-solving and decision-making capabilities.

· Team player with the ability to work in a fast-paced environment.

· Strong skills in stakeholder engagement and consensus building.

· Computer proficiency – Microsoft office suite applications.

· Experience with previous USAID-funded program (s) implementation or assessment(s).