Consultant: Project Final Evaluation & Study

Terms of Reference
Project Final Evaluation & Study on the impact of Arts on Health

1. About RED NOSES Palestine (RNPS):

RNPS is an artistic organisation with the mission to bring humour and laughter to people in need of joy. It is one of 11 RED NOSES group countries and was officially registered as a local branch office of RED NOSES International (RNI) in Palestine on September 30th, 2012. RED NOSES is the first artistic organisation in Palestine that combines the art of clowning with healthcare and introduces this creative performing art into unusual places like social and medical facilities.

Through different artistic interventions, RNPS provides psychosocial support and helps improve the emotional well-being of hospitalised children, children with disabilities, the elderly and children in marginalised areas. Currently, RNPS is operating in 18 medical and social facilities around the West Bank and East Jerusalem and have a pool of 26 Palestinian performing artists. Since the establishment, RNPS has contributed to the development of a Palestinian creative clowning sector that contains many local performing artists and has a great impact on the health, cultural and artistic life and scene in Palestine.

2. Project background:

RNPS is implementing a project entitled “Spreading ‘Humour & Arts’ across Palestinian Hospitals: Providing Child-Friendly Mental Health and Psycho-Social Support Services” co-funded by the Austrian Development Agency (ADA). The lifespan of the project, which started at the beginning of 2020, is 36 months with a total budget of 418,400 EUR

The project-specific objective is to systematically anchor ‘humour and art’ as a widely recognised mental health and psychosocial support (MHPSS) strategy across Palestinian hospitals. RNPS has been progressing toward meeting this objective by the end of the project period through achieving the three expected results:

1. Hospitalised children and their parents experience moments of relief from stressful situations and painful moments. With the regular hospital visits, RNPS aims to reach more than 33,906 pediatric patients and 33,725 family members throughout the project period.
2. More medical staff (female and male) use humour on regular basis to support the emotional well-being of children and families. The project aims to reach 160 medical staff and 160 medical students.
3. RED NOSES Palestine (RNPS) has strengthened organisational structure and processes in place to take up a leadership role in advocating for the inclusion of ‘humour & art’ as a recognised psychosocial health support strategy.

3. Scope of assignment:

RED NOSES International is seeking to hire a team of consultants/firm to carry out an assignment of two tasks: (1) conduct the final evaluation of the ADA -funded project and, (2) carry out the data collection process in Palestine of a research study on the impact of RED NOSES activities in paediatric hospital departments in Palestine, which will be conducted by external researchers in Austria.

Final Project Evaluation:
This final project evaluation will investigate the project results as per the project document in order to measure the project success, taking into consideration the baseline evaluation which was conducted at the beginning of the project. The evaluation will follow OECD evaluation guidelines to promote organisational learning and accountability towards all stakeholders.

Research study on Arts and Health:
This research study will be carried out as a joint collaboration between RED NOSES International, the Austrian Academy of Sciences, and the Vienna University of Economics and Business. The study will be conducted by a group of external researchers from Vienna who are experts in the field of development and health economics as well as intervention research. To support the implementation of the study, RED NOSES is seeking a local team to work closely with the researchers to carry out the data collection process in Palestine. A focus of the data collection will be the establishment of first contact to participants in different hospitals and telephone interviews.

The study will evaluate the bedside visits of clown doctors in hospitals as the flagship program of the organisation. The intervention is aimed at improving the conditions in (private and public) hospitals, the interaction between health workers and patients, and the psychological well-being and resilience of the beneficiaries, in most cases children. While the target group of the evaluation will be children who undergo treatment in a hospital, the contact for evaluation purposes will be with their parents and guardians. The study will focus on the activities in all 12 hospitals RNPS work in, in the governorates (Jerusalem, Ramallah, Jenin, Tulkarm, Nablus, Qalqilia, Hebron, Bethlehem, and Jericho).

Note: It is important to point out that the choice to hire one party to conduct both assignments is because the research study overlaps with the final project evaluation in some respects, especially regarding the project expected result number 1: hospitalised children and their parents experience moments of relief from stressful situations and painful moments. Hence, both assignments will investigate the impact the clowns’ visits have on the hospitalised children and will require data collection within the hospitals. This is important in order to be mindful of the hospitals time and resources.

4. Assignment Objectives:

The final evaluation:

• To provide an external opinion on the relevance and performance of the project, as compared to the project document and with a strong focus on results.
• To highlight key lessons learnt, best practices and recommendations to feed back into current and future RNPS programming in the same sectoral areas and using similar approaches to meeting their objectives.

The research study:

• To establish a first contact with parents and guardians of hospitalized children in the 12 hospitals RNPS work in and to obtain their contact information and written consent for a participation in the study. Appointments for telephone interviews (~ 1 week later) can be specified during this first contact.
• To collect data through a telephone survey with the hospitalized children’s parents and guardians to obtain information about the children’s psychological well-being and their experiences during their hospitalization. The sample size is expected to be between 1500 - 2000 participants.
• To match telephone survey data with hospitalized children’s administrative data from 12 Palestinian hospitals including age, gender, length of stay, nature of treatment (which sickness/injury etc), return rate (do they come back for follow-up examinations). The secondary data from the hospitals will be encoded and secured by the encoders in the hospitals.

5. Users of the Evaluation & Study

The results of the final project evaluation will be:

• Used by RNPS management to plan and monitor the project progress and success.
• Used to plan further projects in the future.
• Provided to RED NOSES International, the Austrian Development Agency, and other stakeholders to ensure accountability of resources used.
• Used for advocacy purposes; to demonstrate the need for psychosocial support in Palestinian hospitals.

Once published, the study will:

• Be used by RNPS and RNI to confirm the impact of the clowns’ interventions in the Palestinian hospitals, and subsequently will be used to improve these interventions.
• Support RNPS and RNI advocating for the importance of healthcare clowning as a psychosocial support to improve the emotional wellbeing of hospitalised children.

6. Evaluation Questions

RELEVANCE:

• Was the action adequately designed to respond to the needs of the direct beneficiaries? To what extent did the project take into account the needs of different groups (girls, boys, women, men, people with disabilities, etc.)? How was gender, age and diversity considered?

COHERENCE

• Is the project intervention consistent with RNPS policies, standards and strategy in the country?

EFFECTIVENESS

• Were the expected results realized?
• Is the quality of outputs satisfactory according to the planned activities?
• What challenges prevented RED NOSES Palestine from meeting the project objectives?

EFFICIENCY :

• Was the project managed in a cost-efficient manner (in terms of human, financial and other resources versus the results)?

IMPACT:

• What evidence is there that the project contributed to the achievement of its overall objective?
• What were the unintended impacts of the project intervention, both positive and negative? Was the project able to monitor, mitigate and respond to any unintended negative effects?

SUSTAINABILITY: 

• What actions have been taken to guarantee that the interventions or results will be sustained after the end of the project?
• What future measures should be put in place to maintain sustainability of the project activities?
• What are the possibilities for replication and extension of the project’s outcomes?

7. Approach and Methods

The final evaluation: It is expected that the evaluation will apply a mixed-method approach using quantitative and qualitative tools. In addition, it should use existing information and data sources, including data collected as part of the research study (see below). The evaluation team should systematically check accuracy, consistency, and validity of collected data and information and acknowledge any limitations/caveats in drawing conclusions using the data. RED NOSES strongly encourages the use of creative and innovative methods that are inclusive of cognitive diversity. The evaluation should follow the OECD/DAC norms and standards for evaluations and should also be aligned with the Austrian Development Agency guidelines for programme and projects evaluations, which will be provided to the selected applicant.

The methods may include document review, interviews, focus group discussions, surveys, etc. RNPS will provide the external expert with all available project documentation at the beginning of the consultancy. The consultants will also undertake field visits and interview the stakeholders including the target beneficiaries, government officials, medical staff, parents, etc.

The research study: The study will rely on a quantitative evaluation of activities through a randomized control trial. The methods to collect the data will be telephone interviews and the processing of generated data from hospitals regarding paediatric patients (length of stay, nature of treatment, age, gender, return date).

8. Timeline

For the evaluation:

May- July 2022:  Inception phase
August- September 2022: Data collection
October – November 2022: Submission and presentation of final report

For the research study:

May 2022: Study design phase, together with the researchers (3-5 working days)
June 2022: Training of local data collectors (3 days)
July-September 2022: Data collection (30 - 40 working days per data collector for 10-12 data collectors)

In total, there are 10-12 hospitals where data collection will take place.

Estimates on working days per hospital per week:

• 3 days for visiting hospitals and recruiting childrens’ parents to participate in study
• 2 days for telephone interviews

October- December 2022: Preliminary data analysis by researchers and initial findings (no involvement by the survey firm needed)
January- March 2023: Finalization of study by researchers (no involvement by the survey firm needed)

9. Management
The assignment team leader will report to the Head of the Research and Learning Department of RED NOSES International in the Headquarters in Vienna. The assignment team leader should be available for weekly or biweekly remote check-ins to update on progress.

10. Deliverables

For the Evaluation:
For the evaluation, the evaluators should rely on the quality standards regarding the structure and the content of the deliverables included in the Austrian Development Agency guidelines for Programme and Projects evaluation.

• Draft inception evaluation report (due June 2022)
  o Overview of research methods.
  o Data collection schedule.
  o Location of site visits.
  o All evaluation tools (interview questions, surveys) in Arabic and English.
• Final inception report (due July 2022)
  o Revisions and suggestions from RNI taken into account
• Draft evaluation report (due October 2022)
  o Executive summary with key findings and recommendations.
  o All evaluation questions answered.
  o Inclusion of the most significant stakeholders.
  o Clear differentiation between findings, conclusions, and recommendations.
  o Conclusions and recommendations based on findings.
  o Information presented clearly and simply, avoiding unnecessary jargon.
• Final evaluation report (due November 2022)
  o Revisions and suggestions from RNI taken into account.
  o Evaluation report should be submitted together with the completed Results Assessment Form as per the ADA guidelines. https://www.entwicklung.at/en/ada/evaluation
• Raw data (due November 2022)
  o All raw data collected as part of the evaluation (which will be the property of RNI).
• PowerPoint presentation of findings and recommendations (due November 2022)
  o Clear and visually engaging presentation of key messages, suitable for a variety of stakeholders.

For the study:

• Finalization of research tools (due May 2022)
  o Translation of research tools into Arabic.
• Submission of data collected (due September 2022)
  o Data encoding of process generated hospital data.
  o All data must be anonymized, cleaned, validated, and consolidated in the right way. For example, data from telephone interviews must be matched to administrative data from hospitals.
  o All data collected will be submitted to the researchers within the format agreed upon in the inception phase (raw data are the property of the researchers).

11. Budget
A budget of 32,000 EUR is available. This budget includes fees for all team members, travel costs, translation costs, and any other incidental costs that may arise during any phase of the evaluation and the study.

جميع الحقوق محفوظة لموقع جوبس.

12. Evaluation Team
The evaluation team need to meet the following criteria:

• Master’s Degree in a relevant field (preferably in economics).
• Experience managing impact evaluations (minimum 5 references).
• Experience in quantitative and qualitative data collection and analysis (minimum 7 years).
• Experience or demonstrable interest in artistic interventions.
• Excellent spoken and written Arabic and English. Other languages spoken in the target countries are an asset.

Additional specific requirements for quantitative research study:

• Experience in managing impact evaluations with a large sample size (minimum 5 references).
• Experience in conducting telephone surveys and the use of automated data collection tools, such as CATI.
• Experience in matching data from different sources (e.g. survey with administrative data).
• Experience in collecting data from vulnerable populations.

To apply, please send the following to [email protected] by April 18th, 2022:

• A brief conceptual note of your understanding of this evaluation and your methodological approach.
• A reference list of relevant assignments.
• Your CV or the CVs of the experts proposed for this evaluation.
• Company profile for firms.
• Information on costs charged in EUR (daily rates, travel and totals).