Before and during your evaluation you will need to think carefully who you collect information from.  It is important here to balance considerations relating to your population, inclusiveness, representation, and sampling.

Population    

Your study’s population refers to the total of all the individuals who have certain characteristics and are of interest to the evaluation. This could be as large as all the looked-after children in the UK, or as small as the total number of people who attend a particular youth club.  You must determine what your population is before you can decide on a sampling method.  Some examples of populations could be People Affected by Dementia in a particular area, or Care Experienced Young People in Scotland.   

Inclusiveness

Evaluation should aim to be as inclusive as possible and must be sensitive to the variety of needs and vulnerabilities of participants. This needs to be kept in mind when determining which sampling method makes the most sense for your evaluation.

It is quite common for potential beneficiaries who are seldom-heard in evaluation research to be referred to as ‘hard to reach’.  Certain individuals or groups may have complex support needs or be particularly vulnerable, which make it more difficult to involve them.  Mental or physical health difficulties, communication difficulties, literacy, and language can be factors.  This means that we have to think more creatively about recruitment, engagement, and support issues.

To be as inclusive in the evaluation process as possible, it helps to:

  • use straightforward language
  • use appropriate gatekeepers to gain access to and support your participants
  • be as flexible as possible in the data collection methods you use (some approaches may not be suitable with certain groups)

All research methods and approaches can be adapted to suit particular circumstances and needs.  Even small adaptations can vastly improve the inclusionary nature of your evaluation.

IRISS has put together an excellent insights paper on Effectively Engaging and Involving Seldom-heard Groups.

Representation

Much research and consultation pursues the goal of achieving ‘representativeness’. This is often a democratic goal which aims to include a range of people’s views. It also has a statistical meaning. Statistical representativeness is concerned with generalising findings to a widerpopulationbased on asampleselected because they accurately represent that wider population.

The different ideas about representativeness tend to get mixed up and can be a cause of concern if, for example, there is a feeling that  poor response rates are undermining the basis of the evaluation findings or that the views of numerically small or dispersed groups of beneficiaries are overlooked. 

It is best to think about representation in terms of your primary evaluation questions and your beneficiaries, e.g. what is it that you want to find out, and who do you need to speak with to answer those questions?

It is important to ensure that beneficiaries are able to express their views in a way that is appropriate and acceptable to them.  This will allow your evaluation to include a range and depth of participant views and experiences.  It is also important to recognise that meaningful representation recognises that some people will not be interested in giving their views, whatever efforts are made.

If, for example, only half of your total beneficiaries experienced a new type of activity offering, you might want to run focus groups or conduct interviews with those who had direct experience of the new service.  By experiencing the new service they will be in a unique position to feedback fruitful perspectives on its practical delivery.  You may also want to speak to others who have yet to participate and those who may have rejected the original offer.  This will help identify the likelihood of uptake, the possible barriers and challenges that stand in the way, etc.

Sampling

Having a clearly defined evaluation population will affect decisions about how to sample; who to include and why? (and, importantly, who to exclude and why?).

A sample is the section of your population (e.g. all forty of the people that use your service) selected for investigation.  Having a good sample will allow your evaluation to be a robust as possible.

How big a sample should be depends on how accurate it needs to be and how diverse your population is.  Increasing the size of the sample is likely (though not guaranteed) to increase the accuracy with which it reflects the whole population.  When a sample is over approximately 1,000 people (or whatever unit is being sampled), increases in accuracy tend to slow down. 

You can either use non-probability or probability sampling. Laerd Dissertation provides a thorough explanation of both probability and non-probability sampling, should you want more information.

Non-Probability Sampling
In non-probability sampling, the people selected for investigation are not chosen at random. Instead, they are chosen by the researcher, either for their convenience, their characteristics, because you do not know the size of the population, or some other reason. These are the types of samples we would generally expect those we fund to use. They include:

A self-selected sample is when your sample consists solely of people who volunteered to take part. For ethical reasons, this may be your only real option. For example, if you are interested in doing document analysis of the case files of your beneficiaries, you would, ethically, need the consent of your beneficiaries to include their files in your study. Additionally, if your population is small (e.g. a weekly coffee group that only has 15 members), your only practical option would be a self-selected sample. 

A snowball sample is when you find a few members of the population who agree to participate, and then they either help find more participants, or put you in touch with more participants. This is a popular technique for working with difficult-to-reach populations. For example, if you are working with a homeless population, you may have a relationship with a few people who could then talk to others about the opportunity to participate. 

A convenience sample is when you sample the members of the population which are convenient for you, or those you have access to. For example, if you are interviewing people who happen to walk by your place of work, that would be considered a convenience sample.

A selective sample is when you purposefully pick and choose individuals that you want to include in your investigation. This is often used in case studies. For example, if you have seen that a particular intervention worked extremely well for a small subset of your population, you may want them to be your sample in order to find out why it worked so well.

A quota sample is when you design your sample to purposefully reflect the characteristics of the population.  For example, if you know that your population is 75% white and female and 25% ethnic minority and female, you would ensure that your sample had the same balance. 

Probability Sampling

In probability sampling, the people you select for investigation are chosen randomly using various methods based on probability. These are the only types of samples with which you can claim that your results will be representative, or hold true, for the entire population.  To be truly representative you generally need a minimum sample size.  If you are considering using a probability sample it is advisable to seek expert help to determine if it appropriate for your project.

In general probability sampling involves generating a random sample. This is when you assign every member of the population a number, and then generate a set of random numbers by rolling a dice or using a computer programme. The members of the population whose numbers are selected are those who are included in your sample. To use advanced statistical methods your sample will need to have been generated in this way.    

No matter which sampling design you choose, you want the design of your evaluation to pre-emptively avoid sampling and response bias.  For example, only selecting participants from an email list excludes those who don’t have or use email. 

Sampling can be a complex and challenging area, particularly for people inexperienced in using statistics.  The online resources below provide background and some good examples and explanations.  If in doubt, it is useful to seek specialist help on sampling issues.  

butterfly.png

Resources