A decision-making model as an assisting tool in designing evaluation strategies for integrated rural development programmes

2. The diversification of the economic structure of our society leads, on one side, to 
the reduction of the importance of the agricultural sector in the whole economy 
and, on the other side, to the concentration of investments in industrialised and 
urban areas to the detriment of the disadvantaged territories (Becattini, 1987). 
3. The productive structure of rural areas, composed mainly of small firms, finds 
growing difficulties facing global economic competition, which is based on 
technological innovation, automation and services. At the same time, previous 
entrepreneurial behaviours, like learning through exclusive personal experience, 
are not sufficient any more. Social changes make the situation worse: exhaustion of 
the labour force reservoir represented by the agricultural sector (constituted today 
by people mainly over 50), diminished mobility, extended control over irregular 
employment, greater sensitivity to the environment, decline of traditional jobs, etc 
(Goglio, 1982). 
4. Continuous migration from rural areas to big cities, as in the South of Italy, or 
medium urban centres, as in the North of Italy, causes depopulation of the 
countryside.  
 
Disadvantaged rural areas are particularly affected by such evolution because of their 
characteristics: 
•  Distance from productive centres; 
•  Low accessibility; 
•  Scarce information and innovation flow; 
•  Lack of services; 
•  Low average income; 
•  Scarcity of risk capital; 
•  Absence of links between the firms of the territory. 
 
The result is a tendency to the decline and marginalisation of these areas, with exodus of 
local population, abandonment of the land with consequent ecological problems (for 
example, erosion in abandoned grassland) and loss of the indigenous cultural heritage. 
 
As a consequence of the evolution of the concept of the rural world, public authority’s 
approach to regional development has changed too. The focus is now on rural 
development, conceived as the general improvement of the socio-economic well-being 
of rural populations, obtained through social, productive and commercial initiatives. 
The emphasis on global and inter-sector development of the socio-economic structure of 
a rural area, opposed to previous development policies based on separate productive 
sectors, has been strongly adopted by the European Community (“Il Futuro del Mondo 
Rurale”, Commission of the European Communities, 1988). 
In the Community policy, the development of rural areas is not obtained any more by 
financial “injections” with limited effect in time and often “charitable” characteristics, 
but must be “endogenous”, that is must valorise local potentiality in terms of 
production, commerce and services, but also include the cultural and social 
environment. Fundamental is therefore the use of the resources of a territory: natural, 
architectonic and artistic amenities, typical agricultural, wood and craftsmanship 
produce, cultural traditions, professional expertise, existing productive and service 
activities (Lipani, 1988). 
However, endogenous does not mean “autochthonous only”, and the local potential must 
be integrated with external resources that are missing in the area (capital, training, skills, 
consultancy, etc.) which can give precious help in terms of income and employment, 
always avoiding external dependence and assuring integration with the local socio-
economic and natural environment. No sector is excluded, on the contrary a 
diversification is considered positive if it can meet local needs. Subsequently, 
extemporaneous aids to one economic sector have been replaced by Integrated 
Programmes, which have common objectives and modalities, involve several sectors 
and aim at the creation of employment, the improvement of the agricultural production, 
the development of tourism, craftsmanship, small and medium industry and training, the 
valorisation of natural and cultural heritage, the creation of infrastructures, etc. 
Integrated programme planning is therefore based on local potential and aims to 
ameliorate the use of financial aids. Moreover, it implicates the common and organic 
engagement of all the socio-economic actors of the different sectors, from firms to 
public administration, from research and innovation agencies to training centres. 
 In the integrated approach to rural development, the local actors do not simply benefit 
from the interventions or implement the operations, but they have an active role in 
setting the objectives and priorities and planning the activities. The evolution from the 
top-down approach, where objectives, priorities and initiatives are decided by the 
authorities and then applied in the territory without any active role of the beneficiaries, 
to the bottom-up fashion, where the local actors have an active function in programme 
planning, has been one of the most important innovations in the development policy in 
the last decade (Bernini Carri, 1989). 
Consequently, rural development programme planning endured another important 
evolution. The traditional development policy was based on different economic sectors. 
Priorities, objectives and actions were decided for each sector and the policy was 
applied in the territory with no distinction between the different areas. With the modern 
approach, the target geographic area is predominant. Firstly, priorities, objectives and 
actions are decided, with a bottom-up process, for a territory; then, the activities in the 
different sectors are defined in order to contribute to the overall objectives. 
 
All Integrated Programmes have common characteristics: 
•  Strong coordination, aimed at concentrating the effort on the key factors of local 
development; 
•  Complementarity with other local initiatives, to avoid overlapping or 
incompatibility; 
•  Diversification of the productive structure as a main objective; 
•  Adaptability to local characteristics; 
•  Partnership between central administrations and local authorities, both in the 
identification of objectives and means, and in the management and control of the 
activities. 
 
Among the programmes which adhere to this new approach, there are the Integrated 
Mediterranean Programmes, briefly IMPs, launched by the European Commission in 
order to tackle the development problems of Mediterranean disadvantaged areas, in 
particular Greece and some regions of Italy and France. These Programmes are coherent 
with the general effort of the Community to decrease the economic distance between 
North and South in the continent. 
In Italy, one of the regions involved has been Emilia-Romagna, with a Programme for 
the Apennines area called “Emilia-Romagna Integrated Mediterranean Programme”. 
The IMPs have represented a new way to promote regional development and have been 
one of the first applications of the partnership between European, national and regional 
authorities, which has become now common practice in the Community. 
 
 
 
1.2. THE PROBLEM 
 
The art of project evaluation has changed extensively in the last decades, as regional and 
rural development projects have evolved from short well-defined economic 
interventions into broad multi-sector integrated programmes. As a result of this change, 
while economic cost-benefit analysis was in practice the only evaluation method in use a 
few years ago, a wide range of techniques, which take into account many different 
factors, are today available and utilised by professionals. 
Nonetheless, whilst many evaluation techniques and several evaluation models have 
been developed, their application to real cases has often proved difficult, as the match 
between the specific evaluation context and the adopted techniques is not easy. The 
application of techniques that are not suitable to the context leads often to evaluations 
which are either badly performed or meaningless. 
In the literature, many evaluation techniques have been developed, but there is not any 
examples of decision-making models which help the evaluator to design a meaningful 
and effective evaluation. Such a decision-making tool should consider all available 
techniques and guide the evaluator to a correct design for the evaluation of a specific 
project. Therefore, what is missing is a decision-making model for evaluation design, a 
model of a superior logical level in relation to existing models. 
1.3. OBJECTIVES OF THE STUDY 
 
The overall objective of this study is to delineate, apply and revise a decision-making 
model for development project evaluation, in order to create a tool to help evaluators to 
plan and carry out an effective and meaningful evaluation for each specific situation. 
To give direction to the study the following specific objectives were formulated: 
1. To examine and categorise existing project evaluation techniques, analysing their 
characteristics and suitability to different circumstances. 
2. To identify crucial criteria involved in the choices leading to the evaluation design. 
3. To outline a model for evaluation design which can help the evaluator to plan an 
effective and meaningful evaluation with regard to specific criteria. 
4. To describe Emilia-Romagna Integrated Mediterranean Programme, considered as 
significant example of a rural development programme. 
5. To apply the model for evaluation design to Emilia-Romagna Integrated 
Mediterranean Programme that, due to its complexity and size, can be considered a 
relevant test for the model. 
6. To compare the theoretical evaluation design resulting from the application of the 
model with the actual performed evaluation and determine the improvements, if any, 
which the application of the model would have brought. 
7. To re-examine the model in the light of the comparison between theoretical and 
actual evaluation. 
 1.4. UTILITY OF THE STUDY 
 
As mentioned above, in the literature concerning the evaluation of rural and regional 
development projects, many techniques and blueprints have been elaborated and 
presented, whilst so far few studies, if any, have been dedicated to decision-making in 
evaluation design. 
A decision-making model for evaluation design would represent a practical tool to help 
professionals to plan evaluations which are effective, sound, meaningful, suitable to the 
characteristics of each single project and compatible with available resources. Hence, it 
would guide the evaluator in the choice of appropriate techniques to a particular 
situation. 
This study will be a first step in the elaboration of such a model, an attempt which, far 
from being definitive or complete, may be the base for further development by other 
experts and academics. Its application to real cases will help to define the model and, 
finally, obtain a valid tool for all professionals who operate in the field of rural and 
regional development evaluation. 
 
 
 
1.5. DESIGN OF THE STUDY 
 
The study consists of three main parts. 
In the first part a model for evaluation design is delineated on the basis of a significant 
literature review. 
In the second part, the model is applied to the Emilia-Romagna Integrated 
Mediterranean Programme, in order to obtain a theoretical evaluation design for that 
specific programme. The application of the model is preceded by an exhaustive 
description of the characteristics and results of the Programme, and is followed by the 
description of the actual evaluation carried out on it. 
In the third and final part the theoretical evaluation and the actual evaluation are 
compared and the model is reviewed on the basis of the conclusions drawn from this 
comparison. 
 1.6. DATA COLLECTION AND ANALYSIS 
 
The sources of the bibliography on rural development and project evaluation were the 
Library of University College Dublin in Ireland, the Library of the University of 
Bologna, D.I.A. (Documentation Centre for the agro-food sector) of Reggio Emilia and 
COOPERGA (the Emilia-Romagna IMP official evaluator) in Italy, and Internet.  
All information about the Integrated Mediterranean Programmes at European and 
national level was collected thanks to the help of the IMPs Office of the Department of 
Community Policies of the Italian Government and of the IMP Support Organisation in 
Rome, both by means of the official documents that were kindly furnished and of the 
interviews with the three main responsibles of the two structures. The interviews were 
quite open, with no questionnaires but a list of points to be analysed. This method was 
judged appropriate, as the interviews aimed to collect qualitative information and 
opinions more than hard data and information, that were available in official documents. 
All the information about Emilia-Romagna Integrated Mediterranean Programme has 
been obtained from the regional administration.  
Detailed interviews have been carried out with the personnel responsible for the 
Integrated Mediterranean Programme activities in the Emilia-Romagna Region. Eight 
interviews were carried out with the same technique used with the officials of the 
Department of Community Policies and of the IMP Support Organisation.  
Also all relevant documents and financial data have also been gathered thanks to the 
collaboration of the administrators of Emilia-Romagna Region. Data have been 
collected, elaborated when necessary through data sheets, and analysed to obtain a 
thorough description of the Programme. 
 2. THEORETICAL BACKGROUND: PROJECT EVALUATION 
 
 
“Evaluation” is defined in Webster’s Third New International Dictionary as “..the act or 
result of evaluating”, that is of “..examining and judging the worth, quality, 
significance, amount, degree or condition of something”. The object evaluated, 
independently by its characteristics and level of complexity, must be compared to a 
well-known standard, either quantitative or qualitative, decided by the evaluator. 
Therefore, the evaluation process is a judgement action and is simultaneously 
“objective”, as far as the description of the evaluated object is concerned, and 
“subjective”, as regards the setting of standards for the comparison. Besides this general 
definition, development project evaluation refers to the evaluation of a set of activities 
whose objective is the socio-economic development of a target population. 
 
Confusion generally exists between the terms “project”, “programme” and “measure”. 
Thus, it is necessary to clarify this point and avoid possible misunderstandings.  
In this context, a “measure” is considered an organised set of activities of the same 
nature (extension, training, financial support, etc.), whilst a “programme” is a scheme 
which includes several measures. For all concepts that can apply both to measures and 
programmes, the term “project” will be used, including all development activities. 
 
There can be many different kinds of project evaluation objectives and, consequently, 
many different evaluation designs. A first categorisation can be done on the basis of the 
moment of the evaluation. An evaluation can be undertaken at different stages: 
• Ex-ante, i.e. before the implementation of the project; 
• During the implementation; 
• Ex-post, at the end of the implementation; 
• Ex-post, several years after the end of the implementation. 
 
The design and aim of an evaluation are usually very different in relation to the moment 
it is undertaken. Law-Yone, Meyer-Brodnitz and Wilkansky (1990) state that usually ex-
ante evaluation is strongly method oriented, with problems relatively well-defined, and 
ex-post evaluation is “..largely anecdotal, unfocused, and methodologically weak.” 
Independently of this classification, many authors define a particular path in evaluation 
design (Bandarra, 1993, Bianchi, 1990, Casley and Lury, 1982, Casley and Kumar, 
1987, Imboden, 1978, Lichfield, 1990, Nielsen and Turner, 1983, Ross, 1980, Rossi, 
Freeman and Wright, 1979, Soumelis, 1977 and Voogd, 1983). Without analysing every 
author’s model, four main steps in the evaluation design process are commonly 
considered fundamental: 
1. Deciding the aims and objectives of the evaluation; 
2. Defining and describing the object to be evaluated; 
3. Setting the factors of the evaluation; 
4. Choosing the techniques. 
 
Each different step will be analysed in the context of project evaluation. 
 
 
 
2.1. DECIDING EVALUATION AIMS AND OBJECTIVES 
 
The first step in project evaluation is to determine evaluation aims and objectives and is 
represented by answering the following question (Case, 1987): 
Who wants to know what and for what purpose? 
Who refers to the individuals or agencies who hold a stake in the evaluation. Sometimes 
the identification of these stakeholders may not be easy. 
Soumelis (1977) states that “..the primary purpose of project evaluation is to enable the 
organisation responsible for the project to increase its effectiveness...”. However, he 
also adds that “Project evaluation...should be seen both from the point of view of the 
organisation responsible for the project and from that of the recipient or client”. 
In fact, in a project there may be several actors who may place different, if not 
sometimes contrasting, interests in the same activities. A structural project (for example 
 an objective 1 project) may be seen by local authorities as an opportunity to keep people 
in a rural area, by the people involved as a way to increase their income and by the 
European Commission as a means to diminish agricultural surpluses. 
All actors may be affected by the evaluation, but their level of interest must be assessed. 
Some persons or organisations may have only marginal interests at stake, therefore the 
principal stakeholders must be identified on the basis of how much they need and want 
the evaluation. This also means that the organisation who pays for the evaluation is 
certainly one of the principal stakeholders. 
There are two categories of principal stakeholder: who need the evaluation to make 
decisions and who are affected by such decisions (Case, 1987), which can affect the 
evaluated project itself (ex-ante, during the implementation) or future projects (ex-post). 
Casley and Kumar (1987) state that an effort should be made to incorporate 
beneficiaries’ assessments into the project evaluation, in order to enrich the quality of 
the evaluation and enable the evaluator to draw conclusions that are more relevant, even 
if not necessarily more statistically reliable. Of course, they continue, beneficiaries 
should be asked to assess only those aspects that they are aware of and affected by, and 
anyway beneficiary’s assessment is a supplement to, not a substitute for, conclusions 
drawn from objective data. 
After the identification of the principle stakeholder and before the decisions about the 
evaluation design, it is absolutely necessary to clarify the aims and objectives of the 
project for the different actors involved. 
The objectives of a project are usually stated in official documents. Nonetheless, some 
objectives may not be expressed but may be real for some actors (politicians, 
administrators, technicians, etc.). It may also happen that objectives are vague or even 
opposite, alternative ways of action are unlimited or consequences of decisions cannot 
be easily foreseen. As a consequence, conclusions about the evaluation aims and 
objectives derive from the actors and their interests in the project. 
 
 SUMMARY BOX 1 
The definition of evaluation objectives is the first step in project evaluation and 
includes  three stages: 
1. The identification of the main stakeholders; 
2. An understanding of the issues raised by the individuals or organisations whom 
the evaluation is designed for; 
3. The clarification of project objectives. 
After these stages have been completed, it is possible to define the evaluation 
objectives. Such a process at times may be easy and quick, in other circumstances, 
it may be long and difficult. 
 
 
 
2.2. DEFINING THE MAIN CHARACTERISTICS OF THE OBJECT OF THE 
EVALUATION 
 
Once the evaluation objectives are determined, the project to be evaluated has to be 
carefully examined. A project is described by Deniston, Rosenstock, Welch and Getting 
(1978) as “..an organised response to reduce or eliminate one or more problems”. 
Therefore, a project includes the specification of one or more objectives, the selection 
and performance of one or more activities, and the acquisition and use of resources. 
 
 
2.2.1. PROJECTS AS DECISION-TAKING PROCESSES 
 
One can maintain that a project is, first of all, a decision-taking process and, as such, is 
composed of a sequence of elements which lead from context analysis to 
implementation (Bianchi, 1990). Literature on this issue is extensive but rather 
homogeneous, and we can consider the following scheme as a general structure for 
projects. 
1. Context analysis (structure, behaviours, trends for population, consumption, 
production, threats and opportunities, etc.); 
2. Statement of general objectives; 
3. Identification of constraints and resources (legal, financial, structural, etc.); 
4. Formulation of political choices (i.e, priorities); 
5. Identification of specific objectives; 
6. Formulation of alternative strategies; 
7. Evaluation of alternatives; 
8. Formulation of the project (timing, geographical and sector focus, resource 
allocation, operational mechanisms, assignments of different agents as agencies, 
enterprises, local administrations, local agents, etc.); 
9. Implementation; 
10.Monitoring, ex-post evaluation, programme redefinition. 
 
This sequence is composed of substantial and formal elements. 
Substantial elements are the judgement on the present state (1), the choice of desired 
state (2) and of priorities (4), the formulation (8) and implementation(9) of the project. 
Formal elements are the analysis and the statement of the reasons for the choices made, 
the demonstration of their coherence, the identification of constraints and resources (3), 
the formulation and evaluation of alternatives (6, 7), and the monitoring and evaluation 
(10). 
 
Generally, in the Italian experience of development planning, substantial elements are 
always present, more or less developed but rarely groundless, whilst formal elements are 
absent or implicit or under-developed (Bianchi, 1987). This causes a gap between the 
enunciation of general aims and the completion of a formally correct planning process. 
The scientific and cultural backwardness of the Italian planning practice, in relation to 
the requirements of regional development, seems mainly due to the lack of political 
sensitivity rather than the absence of available techniques for programme planning. 
 2.2.2. PROJECTS AS SYSTEMS 
 
Projects can often be conceived and formulated on the basis of system needs and can be 
used to develop the system and improve its performance (Soumelis, 1977). 
But the project itself can be considered a system, composed by the funding agency, the 
implementing structure and the recipients. Obviously, a programme is a more 
complicated system than a measure, which can be considered as a sub-system. 
If we accept this “system approach”, which seems logical and sound, several 
consequences about the evaluation process can already be drawn. 
a) When the evaluated system is a large programme, it is crucial to determine its 
boundaries, i.e. actors, target population, territory, activities and results to be 
analysed. Sometimes this is not an easy task and should be performed by evaluators 
and project responsibles together. 
b) The evaluator assesses the system capacity to match a standard and to reach 
objectives. Therefore, how performance is conceived must be clearly explained. It 
can include effectiveness, efficiency, etc. System performance assessment, which 
according to Soumelis (1977) is generally neglected, must answer the following 
questions: 
• Does the system operate? 
• How does it actually work? 
• Does it operate satisfactorily? 
• Can it be improved upon? 
• How can performance be improved? 
• What are the effects of treatment and change? 
c) According to its characteristics and context, project assessment can be conducted 
either by considering it as a system or estimating the performance of its sub-
systems. The first approach may be more meaningful, but needs much care on the 
choice of the appropriate criteria, which must reflect the system goals. The second 
is more simple to carry out, being based on intermediate goals, but sometimes might 
not give a meaningful judgement on the whole (Soumelis, 1977). 
 d) The evaluation can examine just the system output or also the internal factors of the 
system. Soumelis (1977) use the terms “outside tests” and “inside tests”. Bamberger 
and Hewitt (1986) distinguish “performance assessment”, which deals with inputs 
and outputs, and “process assessment”. The first gives a fast but crude measure of 
system effectiveness, the second is more difficult to undertake but can give more 
useful results as it takes into account the causes of the performance. Usually a 
combination of the two gives the best results. Therefore, identification of outputs 
and internal factors is needed. 
e) Many authors of texts on project evaluation are convinced that it is necessary to 
design a model of the project/system, which must consider project design and 
structure, inputs, outputs and processes. Bamberger and Hewitt (1986) confirm this 
point saying that “Before a monitoring or evaluation system can be designed, it is 
essential for the researcher to have a clear definition of the underlying model and 
its assumptions”. In their view the model should, among others: 
• List all inputs; 
• Identify assumptions in the choice of inputs; 
• Specify processes by which inputs are transformed into outputs; 
• Identify assumptions in the choice of implementation methods; 
• Identify assumptions about how processes and outputs are affected by external 
factors; 
• Specify all outputs; 
• Specify intended impacts on target population; 
• Identify assumptions about how impacts are affected by external factors; 
• Identify how impacts are in reality affected by external factors. 
They also state that the information sources to define the model are mainly the 
project documents and the staff.