Faith-inspired Reflection:

Catholic social teaching serves as the foundation of CRS’ Guiding Principles, which are reflected in all of CRS’ work. CRS’ recognizable tagline – faith. action. results. – conveys CRS’ approach to promoting human development around the globe.  BSDI upholds and contributes to the ‘results’ portion of the tagline, whereby CRS commits to demonstrate measurable results to ensure that its ‘actions must be effective in alleviating human suffering, removing root causes and empowering people to achieve their full potential’.  The BSDI initiative enables CRS to clearly understand and communicate the scope of its service, which in turn enables CRS to measure its efforts at the agency-level and makes CRS more accountable to one another, its partners, donors, and the people it serves.

Introduction:

Catholic Relief Services, an international humanitarian agency, aims to serve 150 million poor and vulnerable people by 2018. This ambitious goal requires CRS to accurately understand how many people, and in what ways, its programs are serving.

Founded in the 1940s, CRS now works in over 100 countries with more than 1,550 partners (as of 2015). Typically, it expends in one year around $600 million on program operations across many sectors including emergencies, health, agriculture, education, WASH, capacity strengthening, microfinance and peacebuilding.

The organization’s enormous scale of operations and (somewhat) decentralized structure complicate deceptively “simple” questions: how many people is the organization serving, and in what ways?

Agency-level measurement is challenging as seen in the following comparative analysis of 17 Interaction organizations Journal of Multidisciplinary Evaluation (2017) here

CRS considers understanding these essential output level measures of agency performance—how many people CRS reaches and what services it provides to them—as a first step to accurately assessing higher level measures, such as outcomes or impact.

Data Collection System

To help answer these questions, in 2012, CRS piloted the Beneficiary and Service Delivery Indicators (BSDI) system.

Prior to BSDI, CRS did not have an agency-wide system to track the reach and beneficiaries of its programs. Individual programs separately collected and recorded who was receiving services and what services were delivered, but that information was not aggregated and utilized at higher level. Furthermore, programs defined services and beneficiaries of services differently, and these differences in definitions made it very difficult – if not impossible – to compare one program to another.  Field, regional and headquarters (HQ) staff needed a unified way to look at all program data, as well as to show how strategic goals would be met organization-wide. Read more:

In response to these challenges, CRS developed the BSDI system. The BSDI system refined definitions for tracking and measuring reach of programs. This resulted in the creation of the Catalogue of Beneficiary and Service Delivery Indicators, a catalogue of illustrative definitions for services and beneficiaries that is shared across all CRS programs. This shared, agency-wide “taxonomy” of definitions ensures that all programs are defining beneficiaries and services in a standard way, which allows the agency to compare and aggregate program data.

The catalogue also provides definitions for indirect vs direct beneficiaries – direct beneficiaries refer to individuals who receive direct services.  They are individuals (not households or groups), are identifiable and can be counted.  Indirect beneficiaries refer to individuals who indirectly benefit from services provided to the direct beneficiaries.  They are often household members of the direct beneficiary or a subset of community members, and indirect beneficiaries are often estimated.

The BSDI system now provides an overview of all programs areas (i.e. sectors) and services throughout the entire agency. Field staff, regional, and HQ staff are able to understand the full scope of projects in regards to beneficiaries receiving services, at a birds’ eye view.  This system also allows CRS to accurately  convey agency data to donors and external sources. As of FY15, 107.3 million beneficiaries were reached within the 10 CRS program areas.  A total of 79 countries and 562 projects reported on BSDI.

Benefits of the Data Collection System

• Standardized tracking – BSDI provides field, regional and HQ staff a precise and standardized way for programs to track and measure the number of people served and services organization-wide. The standardized tracking allows CRS beneficiaries to be aggregated either agency-wide or across geographic regions, or within program areas. Beneficiary and service delivery can also be tracked over time.
• Immediate program data indicators – On the ground field staff use the indicators the system collects to improve programs immediately as they are able to monitor programs see progress towards strategic goals. They are also able to compare their project or country programs to others in the region or globally.
• Ease of reporting – BSDI provides an overview of program capacity and reach used to report to CRS leadership and provides a seamless method to showcase agency information to external agencies for business development, marketing and communication purposes.

Objectives for the BSDI initiative:

• Provide a unified way to define beneficiaries and services collected organization-wide in one global information system
• Require little or no additional workload for country teams. CRS used data that programs already collected in order to create a seamless process from past data collection to the new system.
• Flexible data collection methods – Individual programs are able to collect beneficiary and service information using whatever methods are best for the program (handwritten records, electronic records). The collected aggregate data is then inputted into an electronic database called Gateway. Gateway is CRS’ project management platform.

Challenges and Next Steps

BSDI is an adaptive and evolving system, and improvements are made annually. CRS has taken a slow and deliberate approach in rolling out BSDI to ensure the system is fully adopted and embedded in the agency’s practice along each step of the way.  In order to optimize the system, there are several ‘next steps’ that CRS will be exploring over the coming years:

• Beneficiary databases and unique beneficiary identifiers – Currently many, but not all projects develop and maintain beneficiary databases that include unique identifiers. Best practice at CRS encourages projects to develop a such systems that include unique identifiers and demographic information.  The existence of such systems for all CRS projects will enable the BSDI system to:
  • Track beneficiaries using multiple services – Currently BSDI does not track beneficiaries who use more than one service across a multi-sector project. CRS would like to configure the BSDI system to report on unique beneficiaries (i.e. the number of individuals who have received different combinations of benefits), and thus eliminate double counting.
  • Expand Beneficiary Dataset to capture more detailed data at the agency-level on beneficiary demographics such as age, gender, geographic location, and other details.
• Define beneficiaries of CRS’ mass communication campaigns – Better criteria and definitions need to be defined for counting beneficiaries of mass communication or public awareness campaigns (i.e. radio announcements, for example).
• Reporting – Continue working within Gateway (the project management system used to collect BSDI data), to continue to build more dynamic reports and dashboards to meet the data needs of country programs, regional offices, and HQ