Terms and Concepts Explained
We know that the terminology surrounding evaluation can often become confusing or off-putting. However, there are a few basic terms and concepts that it is important to be familiar with.
Numbers standing as they are. This works best if your numbers are small. For example, 9 out of 10 participants got jobs.
The aim is the longer term outcome that your activities are trying create or contribute to.
An approach to generating useful evidence and improving practice based on positive, change-based principles. It is a strengths-based approach to recognising and valuing the skills, capacities, and resources that already exist for a person, community, or organisation.
In most cases, when people refer to an average, they are referring to the mean. The mean is calculated by adding together all of the numbers in question, and then dividing by the number of numbers. For example, the average weight loss for men on a health and fitness programme was 4.7 kg. Watch out for figures that don’t make sense such as 23.6 people. You should also be aware how extremes can skew the average. If some people lose a lot of weight but most do not, then an average may be misleading.
Information about the situation that an activity or organisation is trying to change, showing what it is like before the activity happens. It helps to understand the initial situation to see if you have made a difference.
Baseline information can be about individuals, or a situation the activity aims to affect. E.g. the self-reported well-being (using and agreed measure) of beneficiaries before the activity .
The tendency to search for or interpret information based on what you already believe. For example, if you find a pattern in your research and go along looking for more examples of that pattern, you are more likely to find examples and less likely to find evidence that contradicts it.
A method of economic evaluation that seeks to compare the costs of a service to the benefits received due to the service. This comparison is made by assigning a monetary value to all of the costs and the benefits, regardless of whether the benefits are tangible or intangible.
The process of organising your data, as well as correcting any typos, incorrectly entered information, or missing data.
Using evidence to make judgements on the performance of an organisation or activity. Evaluation is the process of thinking, in a structured way, about what has worked and why, It goes further than monitoring what has been done and involves making a judgement, comparing what was achieved with what was originally planned and exploring the reasons why certain outcomes occurred as they did. Evaluation is about demonstrating that change is actually taking place (proving) and part of a continuous process of learning and development (improving).
There are a number of different kinds of evaluation:
Self-evaluation is when an organisation uses its own expertise and resources to judge its own performance.
External evaluation is when an organisation hires a consultant or other organisation to make these judgements.
Formative evaluation takes place in the lead up to the project, as well as during implementation in order to improve the project design as it is being implemented (continual improvement). The key feature of all such evaluation is that it is designed to bring about improvement while it is still possible to do so
Summative evaluation takes place during and following the project implementation. The purpose of summative evaluation is to determine whether a project it does the job it was designed to do.
The number of times a particular event, outcome, or number has occurred.
An individual or organisation that assists in getting access to your population. For example, it could be a member of the population itself, or a social worker who has a trusted relationship with members of your population who can introduce you and help negotiate consent.
You use indicators to see if you are reaching your targets or milestones, creating your outputs and objectives or achieving your outcomes, aims and impacts. It is something you can observe or measure, and which is a sign that any of these things has happened.
To be useful, an indicator must really be a test of what you want to find out about. It must also be something you can collect information about consistently .
The Trust uses the term to denote knowledge informed by a combination of, research, experience, values, judgement and understanding of context. This is often termed evidence but not everyone understands evidence and its use in the same way. Sometimes the use of the term evidence is used simply as another way to talk about research.
For us, Insight is a wider, and richer, concept than evidence and one we think helps explain our approach to learning about what leads to real transformational change. It emphasises our commitment to respecting different types of knowledge and using this appropriately, depending on specific needs .
Knowledge transfer (KT) is a term used to encompass a very broad range of activities, usually collaboration between universities and other sectors. It’s all about the transfer of tangible and intellectual property, expertise, learning and skills between academia and the non-academic community.
A logic model is a systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your project, the activities you plan, and the changes or results you hope to achieve.
The mid-point range of numbers, when the numbers are ordered from least to greatest. This is often used when discussing income, since it is not easily skewed by large or small numbers.
The most frequent outcome. For example, if out of 30 people, 25 of them received a score of 65 on an example, then 25 would be the mode.
Monitoring is the process of collecting and recording information in a routine and systematic way to check progress against plans and enable evaluation. Good project/programme monitoring is essential to ensure that activities are implemented effectively and have the best possible chance of delivering meaningful results.
Outcomes are the agreed upon differences or changes that we try to make through our activities. There are different levels of outcomes: immediate, intermediate (identified steps towards achieving long-term outcomes), or long-term (ongoing, sustainable changes)..
The activities we plan and execute to make the changes we want to create and achieve our outcomes.
An empowerment approach that seeks to build community knowledge. It uses a lot of visual methods, making it especially useful for participants who find other methods of participation intimidating or complicated.
Participatory evaluation is undertaken with active involvement by those with a stake in a project or programme; those that deliver it, those that benefit from it, as well as a variety of other interested parties. In practical terms, participatory approaches to evaluation mean: involving as many people as possible throughout; ensuring that it is owned by all; using inclusive and popular ways of gathering evidence; and ensuring that people work together to make sense of the evidence and use it to improve practice.
A percent is the proportion of people out of 100, and is calculated by dividing the amount with a particular characteristic by the total amount of people. For example, if 375 out of your 500 participants reported an increase in the amount of exercise they do weekly, that would be an increase of 75% (375/500 = .75). In general you do not want to use percentages with small numbers. For example 80% of trainees got jobs may just mean four out of five.
Performance management is about ensuring that an organisation achieves its intended outcomes. It includes planning, monitoring and evaluation to target and measure performance. It is also about becoming a better quality organisation – one that is more effective and efficient and seeks excellence in the way it delivers its services and how it is managed, how it treats staff, partners and suppliers, and its overall effect on society.
The process of testing out your project or questions on a smaller group prior to implementing the full project. Pilots can range in size from one or two people (to test out survey or interview questions) to large projects over an entire region (i.e. the Highland Pathfinder project for GIRFEC).
A population is the total of all the individuals who have certain characteristics and are of interest to the evaluation. E.g. people affected by dementia in a particular area or care experienced young people in Scotland.
Questions you ask in order to encourage your respondent to provide more or deeper information on a topic. “Why do you say that?” and “Can you tell me a bit more about that?” are two examples.
A tangible thing that helps you approximate an intangible thing. For example, you cannot measure confidence directly, but you can measure how often a student participates in an activity or raises his or her hand in class, which can then be used as a proxy for confidence.
Information about what you do, achieve or provide that tells you the nature of the thing you are doing, providing or achieving. This is descriptive information rather than numerical information.
E.g. At the end of the training sessions the befrienders said they had enjoyed the course and had developed a good understanding of their role..
Quality improvement is about learning about what an organisation is doing well and doing it better. It also means finding out what the organisation may need to change or improve to make sure you meet the needs of your beneficiaries. It is about working towards excellence.
Information about what you do, achieve or provide that tells you how many, how long or how often you have done it, achieved it or provided it.
E.g. We provided 10 befriending training sessions, consisting of five weeks of two-hour sessions each week.
The distance between the smallest number and the largest number. For example, at the start of the course people got between 14 and 20 answers out of 40 correct and at the end they got between 22 and 37. Ranges can be useful when you need to illustrate diversity.
A sample is a subset of the population. Samples are important because it is often impossible to study all the members of a population. There are many types of samples but they all aim to accurately create a smaller representative subset from the larger set of general participants. E.g. We may select a sample of all care experienced young people by selecting only those in certain areas. We must take care to identify if the young people in these areas have a similar profile to the whole population
Convenience sample is when you sample the members of the population which are convenient for you, or those you have access to.
Quota sample is when you design your sample to purposefully reflect the characteristics of the population.
Random sample is when you assign every member of the population a number, and then generate a set of random numbers by rolling a dice or using a computer programme.
Representative sample is one that can be scaled up to be representative of the ‘population’ as a whole.
Self-selected sample is when your sample consists solely of people who volunteered to take part.
Selective sample is when you purposefully pick and choose individuals that you want to include in your investigation.
Snowball sample is when you find a few members of the population who agree to participate, and then they either help find more participants, or put you in touch with more participants.
A type of economic evaluation intended to determine the value of a programme relative to the resources that have been invested. It focuses on listening to stakeholders and identifying the outcomes that are important to them, and then putting a financial value on these outcomes.
A stakeholder is an individual, group, or organisation who has an interest in your project (funders, etc), will be impacted by your project (beneficiaries and their families), or will be affected in some significant way by the outcome of the evaluation process.
Sums are when you add or subtract numbers. This can let you know the distance travelled on an outcomes chart or scale. For example, five people who took part on the project said they increased their ability to “say no to drugs and alcohol” by two points on a scale.
Combining information, ideas, and/or concepts to strengthen your argument or create a new argument. An important part of interpreting your findings.
A theory of change is the reason or reasons why you think your project will lead to outcomes you set out. A theory of change should be informed by many different points of view, and where available, evidence on what supports the type of change you are aiming for. E.g. Developing people’s ability to care for themselves (self-care/self-management) and to direct their own support and finances (self-directed support/direct budgets) is empowering and can facilitate transformational and sustainable improvements in their quality of life and well-being.
The process of writing typing out the entire conversation of a recorded interview or focus group. The amount of time this takes is dependent on how fast you type, but in general you should plan on four hours of transcription for every hour of recording.
Whether your evaluation or research actually measures what it set out to measure. Valid research accurately portrays what it sets out to represent.