Overview

Observational techniques are very common in social research but less so in project evaluation and there is much scope to make more of this valuable approach. They consist of some form of systematic observation.

Observation is a holistic method that can give the evaluator a vivid picture of the operation of an activity, project or service, especially with regard to group behaviour.

The evaluator can either participate openly or observe. The simple distinction is usually made between participant and non-participant observation.  Participant observation is where the observer is part of the situation under observation e.g. a group worker taking part is a session with young people.  Non-participant observation would be when the observer takes no part in the activity but observes either openly of from behind a screen or via video.  In all cases the participants should be aware of the plan for observation and have given their consent. 

Methods of recording observations vary, from informal and unstructured approaches through to highly standardised procedures. Observation guides, recording sheets, checklists or field notes can be created to remember and record the behaviours, activities, events and other features of the setting and participants being observed.

Selected methods and techniques

a)    Naturalistic Observation. Naturalistic observation (i.e. unstructured observation) involves studying the spontaneous behaviour of participants in natural surroundings. The evaluator simply records what they see in whatever way they can.  The evaluator can be a beneficiary, staff member, other participant or external evaluator. 

b)    Participant Observation. Participant observation is a variant of the above (natural observations) but here the researcher joins in and becomes part of the group they are studying to get a deeper insight into their lives.

c)    Mystery Shopper. Mystery shopping can be used to check the performance, conditions, service and staff within an organisation and may take the forms of an email, telephone call, website survey, personal visit or a visit elsewhere – perhaps to a contrasting service.  The mystery beneficiary’s identity and purpose is generally not revealed.  This can provide real insight into the experience of being a beneficiary of a service.

Regardless of which technique is used, there are two types of observations; structured and unstructured. Put simply, structured observations are used when you are looking for something specific and unstructured observations are used when you are observing something holistically and are expecting pertinent themes or issues to emerge naturally during the observations.

Benefits

  • Provides access to people in real life situations and discovery
  • Provides access to situations where questionnaires and interviews are not possible or inappropriate
  • Can provide experiential and therefore a more in-depth understanding Applicable in a range of contexts and settings
  • Multiple observers can compare their different experiences
  • Subjectivity can be really valuable and provide insight and empathy
  • Offers a realistic and ‘live’ portrayal of services

Challenges

  • It can be difficult to analyse across different settings or multiple observers
  • It is important to consider the ethical implications of using a hidden identity
  • It may take time and effort to complete this method of evaluation
butterfly.png

Resources

A useful introductory guide to Collecting Evaluation Data Through Direct Observation has been produced by the University of Wisconsin.