PERFORMANCE ASSESSMENT OF CHILDREN AND YOUTH HOUSEHOLDS

. This study purposes a Data Envelopment Analysis (DEA) framework to assess the performance of Non-Profit Organizations that look after children and young people. The DEA method is used to assess the managerial efficiency of eight institutions from Bragança district, since 2010 to 2013. The model evaluates each institution concerning the reduction of operational and staff costs incurred in providing social services to the number of users observed in each unit. A fair policy for the allocation of subsidies is designed according to the performance of institutions in order to support the regulator.


Introduction
The Third Sector (TS) in Portugal has gained increasing strength, being in charge of traditional social services of the State sphere. The transference of these traditional services from the public to the TS implied the allocation of substantial economic support which should be managed with extreme rigour. Therefore, it is particularly important that public structures, which support the Non-Profit Organizations (NPO), have knowledge of how efficiently and effectively those NPO manage the available funds and the resources [1]. As argued by Vakkuri [2], Sinuany-Stern and Sherman [3], DEA is a good methodology for measuring performance in NPO, although there are few studies in TS [4]. The intrinsic nature of DEA in determination an efficiency measure taking into account that each organization uses multiples resources to produce multiple outputs has increased the number of applications in TS, hindering the use of traditional ratios. These ratios measure partial efficiency as for example the ratio of total subsidy to total amount spent on social service [5]. Other approaches determine the overall efficiency taking a weighted aggregation of individual variables, using predefined weights subjectively determined for combining variables.
The study purposes a DEA framework to support the assessment of managerial efficiency of NPO looking after children and young people that are referred to hereon as Children and Youth Households (CYH). This approach intends to elucidate the regulator authority and the community in management and supervising the NPO in terms of efficiency in using the subsidies received and the resources required to provide the social services which is an example of community-based operations research as referred by Sinuany-Stern and Sherman [3]. The first contribution of this paper is to assess the managerial efficiency of each CYH in reducing the cost of resources required to take care of the number of the available users. The second contribution is to propose a fair policy for allocating the subsidies based on DEA results.
The CYH is one of the social responses offered by the NPO in Portugal. These institutions look after children and young people, being in critical situation for more than 6 months, aged between 6 and 18 years. For children who are below 6 and for the ones with specific health problems there are other kind of institutions that can receive them. Generally, the distribution of children by age is quite homogeneous in the various CYH observed. The objectives of this social response are [6]: ensuring the accommodation and the needs of the children and young people, promoting their overall development, in conditions as close as possible to a family structure; ensuring the necessary resources for their personal development and vocational training, in cooperation with the family, school, professional training and community; and promoting their integration in the family collaborating with the relevant authorities, to achieve the gradual children's autonomy from the institutions.
This study presents a performance assessment of 8 CYH, from Bragança district, from 2010 to 2013, which corresponds to the available data. As far as we know, there is no published study documenting DEA approach for assessment of these social welfare institutions.
The paper is organized as follows: Section 2 reviews the literature about the performance assessment methods for NPO in the TS. Section 3 describes the case study and presents the DEA methodology. Section 4 describes the DEA model and discusses the results. Section 5 concludes the paper.

Literature review in measuring performance in Third Sector
Performance evaluation studies in the TS institutions are still scarce [1]. According to Sillanpää [7], the performance of these institutions can be evaluated from the results of their activity such as financial results or also from the process of how the activity is performed in terms of efficiency, quality and effectiveness.
The performance of TS organizations has been assessed through qualitative and quantitative approaches [8]. However, in certain institutions or contexts more importance is given to the qualitative performance rather than to the quantitative performance obtained through the financial information [9]. Often it is also used a combination of the two approaches (quantitative and qualitative) by using both financial and non-financial criteria as exemplified by Kaplan and Norton's Balanced Score Card [10], the European Foundation for Quality Management (EFQM) [11] and DEA [12].
In TS, the DEA applications are found in performance assessment of social service agencies [12], provision of social care for older people [13], education and human services public assistance [14] and international aid organizations [15].
This study presents a performance assessment of CYH looking after children and young people that can be used by the regulator in managing and allocating public funds for these institutions.

Performance assessment of CYH
This section describes the case study and introduces the DEA methodology.

Case study
The CYH are non-profit and not public organizations, which have social purposes and private management. The activity of CYH is supported by the State according to the number of users (children and young people) and the Cooperation Agreement (CA) established with the Social Security Institute (SSI). The CA is defined according to the technical capacity and physical conditions of each CYH and includes information about their duties, duration of agreement, user vacancies and the financial contribution per user. The subsidy received per user is the same in different CYH, for a given year. The subsidy received per user and month by each CYH was 503€ in 2010, it decreased to 484€ and 470€ in 2011 and 2012, respectively, and it increased to the interval ranging from 500€ to 700€, in 2013. In this year, each CYH received a total subsidy which corresponds to the total number of user vacancies contracted in CA if its occupation rate is at least 65%. This limit was 75% for 2009 and 2010 and it was reduced to 50% in 2011 and 2012. For a lower occupancy rate, the CYH only receives funds according to the number of users that they effectively accommodate. Each CYH should manage the received subsidies to cover the costs incurred in its activity. The amount of subsidies received by the institutions are usually higher than their costs. In the period studied, there are one or two institutions that, in some years, received subsidies which were lower than the total costs. This situation occurred in the CYH 7 that closed in 2013.
In 2007, SSI has implemented a quality management framework for the CYH based on the EFQM. This approach is used to verify if the social services have been correctly provided by each CYH, establishing a minimum level of requirements which have to be fulfilled [16]. Unfortunately, unavailability of results from the quality assessment of the CYH prevented the authors from complementing the proposed DEA approach. This quality management perspective only focuses on the social service provided to the user, disregarding the efficiency assessment of the CYH in resources and subsidies management. In this context, the evaluation of the managerial efficiency of each CYH is critical which enables to assess how each CYH manages the resources and the subsidies received in providing social services to the level of users in each institution.
Thus, this paper proposes a DEA framework to evaluate the performance of CYH which should complement the current quality assessment model.
This study considers the panel dataset of 8 CYH from Bragança district, located in different municipalities, from 2010 to 2013. It was observed an outlier concerning the CYH 3, in 2012. In 2013, the CYH 7 is excluded from the assessment because it was closed. Thus, the final sample includes 30 observations, from the 8 CYH, excluding two observations regarding CYH 3 and 7. The mean and the coefficient of variation (c.v.) of the variables across CYH, in the corresponding year, are summarized in Table  1, considering that the monetary variables were deflated using devaluations coefficients with reference to 2013. The higher scores of variability relative to the mean are observed for operating costs, followed by the number of users and subsidies. It is observed that the average total cost per user is higher in 2010 and 2013 than in other years because the average subsidies per user is also higher in those years. Actually, the institutions spend more money in their activities if they receive more subsidies. The DEA methodology is introduced in the next section.

DEA methodology
The managerial efficiency for each CYH is evaluated through the technical efficiency derived from the DEA model, introduced by [17]. DEA is a non-parametric approach to assess the relative efficiency of a homogeneous set of Decision Making Units (DMUs) in producing multiple outputs from multiple inputs. DEA identifies a subset of efficient CYH considered as benchmarks. These DMUs define the frontier technology enveloping all institutions observed in the Production Possibility Set (PPS). For the inefficient CYH located inside the PPS, the magnitude of the inefficiency is derived by the distance to the frontier and a single summary measure of efficiency is calculated.
Consider a set of CYH 1, … , , each consuming resources (inputs) , … , to produce results (outputs) , … , . For an input minimiz-ing perspective and assuming the most productive frontier observed, defined by constant returns to scale (CRS), the relative efficiency of the assessed CYH o can be evaluated using the linear programming model (1): The relative efficiency ( of the assessed CYH o is calculated by the optimum solution of model (1), * , corresponding to the minimum factor by which the inputs levels can be reduced given the current level of outputs. DEA enables to identify the efficient CYH which have the best practices and the inefficient CYH which activity can be improved. The efficiency measure is equal to 100% when the CYH under assessment is efficient, whereas lower scores indicate the existence of inefficiencies. For inefficient CYH, it is also possible to obtain, as by-products of the DEA efficiency assessment, a set of targets for becoming efficient, which are feasible points observed on the frontier.
As the scale size affects the productivity of a CYH, it is important to calculate the scale efficiency to measure the distance between CRS and variable returns to scale (VRS) frontiers at the scale size of the assessed unit, through the ratio between * and the efficiency score achieved assuming VRS, which requires including the constraint ∑ 1 in model (1). Thus, the larger the difference between CRS and VRS frontiers, the lower the value of scale efficiency is, and the adverse impact of scale size on productivity is more significative.

Results and discussion
This section describes the DEA model application and the discussion of the results.

DEA model
The managerial efficiency for each CYH is assessed by the model (1), evaluating its capacity in minimizing the operating and staff costs required to take care of the children and young people observed in the institution.
The single output of the model is the number of users in each CYH, calculated through the average monthly number of users, per year. The costs of each CYH essentially depend on the number of users. The children's age and the social care are quite homogeneous in the various CYH observed. The inputs of the model include the total cost of technical and education staff effectively spent by each CYH per year, and the total annual operating costs concerning external supplies and services (heating system, electricity, water and maintenance of institution), food and other consumed materials. These resources should be minimized for a given level of users accommodated in each CYH. The data concerning the inputs are collected from income statements regarding the 8 CYH which are provided by SSI and summarised in Table 1.

Empirical results
The relative efficiency of a CYH in a given year is estimated by comparison to the best practices observed during the period analysed, ranging from 2010 to 2013. In order to see the impact of scale size on the efficiency in the period studied, we calculated the average of scale efficiency for each CYH which is higher than 81%, except for CYH 4 and 8 which is 51% and 76%, respectively. Thus, only these two institutions show inefficiency due to scale size which strengthens the use of CRS frontier technology to assess the CYH efficiency. According to the decision maker, the CRS assessment reflects in higher degree the performance expectations concerning each CYH.
The average managerial efficiency for the CYH is 64% (with a standard deviation of 19%), indicating that each CYH can reduce their current operating and staff costs by 36%, on average, providing the social services to the same level of users in each institution. This analysis revealed that there is only 2 efficient institutions, CYH 3 in 2011 and CYH 6 in 2010, being the benchmarks. The best practices observed in these institutions should be identified to be emulated by the inefficiency institutions with efficiency scores ranging from 29% to 93%. The results indicate that there are three inefficient CYH that have slack in the constraint relative to the input operating costs. This means that these inefficient CYH should make an extra effort in reducing the level of operating costs without detriment to their social services provided to the users. Looking now at the distribution of efficiency scores across institutions, given in Fig. 1, it is observed that in general the institutions have decreased the managerial efficiency over the last years. In fact, the average managerial efficiency of the CYH increased by 24% in 2011, decreasing by 6% in 2012 and 2013. A fair policy of allocating the subsidies is proposed by defining that the total subsidy received by each CYH should be proportional to the real number of users independently from the number of vacancies available in each institution. The allocating of the users and consequently the subsidies should be beneficiating the institutions which are more efficient in management the staff and operating costs given the available users based on the results from DEA model.
Taking into account the staff and operating costs observed on the two benchmarks, we propose a score for the subsidy per user and per month (S), which is calculated according to: 1 1 . is the staff cost per user defined as the weighted average of the staff costs per user observed in the benchmarks. The same procedure is used to calculate the operating costs per user ( ). The weight for each benchmark is the percentage of times used as benchmark, in the evaluation. is the estimated percentage that each institution needs to be prepared, in terms of extra staff, for the possibility to receive more users. In the period studied, is calculated by the ratio between the total number of vacancies available in all institutions and the total number of vacancies in all institutions defined according to the CA. is the estimated percentage to account the possibility to have extra costs due to equipment depreciation and other costs, which is approximately 5% of the operating costs. In the period studied the subsidy per user and per month is estimated in 619€.
The managerial implications of the efficiency results are explored in detail for CYH 1, in 2013. The managerial efficiency score achieved for this CYH is 76.5% which is compared with the two benchmarks. For this inefficient CYH, it is also possible to obtain, as by-products of the DEA efficiency assessment, a set of targets for becoming efficient, as shown in Table 2. Targets for this institution indicate that it is possible to decrease the staff and operating costs by 23.5% without decreasing the social care services provided to the observed number of users. According to the policy of allocating subsidies, the regulator should reduce the observed subsidy by 31.2%, as shown in Table 2. Considering all the institutions, the proposed policy of allocating the subsidies enables a reduction by 24.5% of the total subsidies observed, which enables a saving of 2 173 565€ in the period analysed.

Conclusions
The study purposes a DEA framework which can fulfil the interest of the SSI in managing and allocating public funds for CYH, which are more efficient in managing the subsidies received, in detriment of inefficient institutions. A DEA model is used to assess the managerial efficiency of NPO in minimizing the staff and operating costs required to provide the social services to the level of users in each institution. Based on the DEA results, a fair policy for allocating the subsidies is proposed according to real number of users and the cost levels observed on benchmarks. In the period studied, the subsidy per user and per month is estimated in 619€ being in the interval used by the regulator. Overall, the CYH should reduce their current operating and staff costs by 36%, on average. It is also observed that the institutions tend to decrease their managerial efficiency over time due to the increasing of the cost by user. This can be due to the increasing amount of the subsidies per user allocated by the regulator since 2012.
In future developments, the application of the proposed framework will be explored in data from CYH in other regions. The users satisfaction and their quality of life should also be investigated to complement the proposed approach.