Redefining Aboriginal Evaluation

Graphics for Sampling Designs

A recent challenge that Johnston Research Inc. faced called for explaining an experimental research design to 14 year olds so that they would be willing and knowledgeable enough to take the chance of random assignment to either a program or a control group.

This experimental evaluation approach was consistent with Conner’s work with Mexican migrant workers (2004, p. 51-65) in that we improved validity by ‘working collaboratively’ with program staff and participants as well as speaking the ‘literal’ and ‘figurative’ language of participants. Conner’s literal validity is simply speaking the proper dialect of the population involved in the research and his figurative language refers to the content and style of communication. The experimental research design discussed below provides an example of how addressing these concepts in an evaluation are enhanced using technology.

Given the prior approval of the pilot project communities to the basics of the ‘experimental’ evaluation design, it was left to the evaluators to design a recruitment presentation for the prospective participants and their parents. This was accomplished with an animated video that explained the program and the research in language and concepts appropriate to the young Aboriginal audience. For an experimental design evaluation project involving northern and rural students as young as 14 years of age at the time of recruitment, a visual approach was required for several reasons. First there were concerns that even a verbal, in-person, technical explanation might not have been truly understood by 14 year olds and ethically their consent to participating in an experimental design research project was imperative. Second, the recruitment was conducted by the local pilot project teachers and they did not have expertise in language of experimental design research.

The need for random assignment was difficult for some of the students, parents and even teachers to understand until they saw the video. For example, random assignment appeared to be a very large black hole until the students saw the video showing a registration form going into a computer and spitting out coloured pebbles that were grouped by program or comparison labels. An animated visual graph was used to communicate the expected impact of the program from the starting year to the ending year. The animation allowed for the incremental changes to be illustrated more developmentally than would have been possible in a presentation of a single ‘after’ chart as shown here.

Later in the video the voice over says, ” We need every student who is a part of the pilot research project to participate in all five of the research data collection phases. Without the student data we cannot show if [name of project] is effective or not.” After a graphic reminder of a previous explanation of the data collection phases, a large question mark was used to illustrate that the final outcomes cannot be known without all the research students.