Evaluating Programs & Human Performance

Chapters in Section III discuss evaluation in instructional design and provide you with three evaluation models: CIPP, Five-Domain, and Kirkpatrick’s Training Evaluation model. Search for at least two other models used for evaluation and summarize these models. Describe how you would use them to evaluate your instruction.

The two models I chose to summarize include Patton’s UFE (Utilization-Focused Evaluation) and Brinkerhoff’s Success Case Model. The main goal of the Patton Utilization-Focused Evaluation (U-FE) is to make sure that the evaluation process is used.  The evaluator, in this case teacher, evaluates the process constantly.  Patton states, “If the evaluation is not used, there is no reason to conduct an evaluation.”  U-FE is “evaluation done for and with specific intended users for specific, intended uses (Reiser & Dempsey, 2012).

There are nine steps in Patton’s Utilization-Focused Evaluation.  

1. An evaluator (teacher) guages the commitment of the organization (students) to the evaluation process with a readiness assessment.  
2. Identify the people that will use the program the most and create relationships with them. (Students in a classroom)
3. Examine the potential problems, supports and the interest of the stakeholders by conducting a situational analysis.  
4. Discover the uses of the program and find a way to ensure that employers will use the evaluation after the evaluator has left.
5. Find out the highest priorities of the employer and answer any questions they may have.  
6. Design an evaluation that will give realistic and actionable results.  
7. Collect and interpret the data.  
8. Continually enable evaluation use by giving interim reports.  
9. Conduct a meta-evaluation, to find out if the evaluation is being used, and whether there were any different uses that were not expected.  

In the text Trends and Issues in Instructional Design and Technology by Robert Reiser, Brinkerhoff’s Success Case Method “uses the commonsense idea that an effective way to determine “what works” is to examine successful cases and compare them to unsuccessful cases.” (p. 101)  Brinkerhoff’s Success Case Method (SCM) decides what works and what does not work.  Robert Brinkerhoff states that the success case method (SCM) “is a quick and simple process that combines analysis of extreme groups with case study and storytelling…to find out how well some organizational initiative is working (Reiser & Dempsey, 2012).  Brinkerhoff’s success case has five steps.

1. Identify targeted business goals and expectations.  (Classroom or school district)
2. Survey a large representative sample of all participants in the program. (Teachers and students)
3. Analyze the survey data to identify a group of successful participants and a group of unsuccessful participants.  
4. Interview the two groups.  Document the nature and of their application of learning and identify the performance factors that makes the successful group successful and the obstacles that prevented the unsuccessful group from being successful.  
5. Disseminate the story by reporting the impact of the evaluation, applauding successes and educating all participants.  

To use this in my classroom/ library I will  survey my students and staff and use the data to see if an activity (possibly Maker’s Space or library unit) was successful.  I will let the students and staff know at the start of the session that I will be asking for evaluations and feedback.  I believe this will encourage teachers and students to be a part of the collaboration and constant evaluation process. I would help teachers and students by providing calendars of lessons in the library, resources being used, and Maker’s Space activities available. I would ensure that the objective of the lesson was well thought out and displayed for teachers and students to see.  If I was doing this with a department (possibly the ELAR Department) I would follow the steps outlined by Brinkerhoff’s Success Case Model.

1. I would select an evaluator to create the case study.  
2. I would decide which goals and outcomes are most important.  This includes defining what a successful case and an unsuccessful case looks like.  
3. Create an impact model so that we can visualize the direction we are going in and create a questionnaire that each teacher/ student fills out so that we can decide which cases are successful and which cases are unsuccessful.
4. Next I would interview teachers about how the section went, what was good and what was bad.  Then a report would be written.  
5. Discuss/ share results with the teachers, staff, and students fulfilling the “storytelling arm” of SCM.
   
   

Reflect on what other questions that instructional design evaluation should address besides whether the instructional design leads to comparable amounts of learning and learner satisfaction as traditional methods. Should return on investment and management of resources also be considered in evaluations of instructional programs? What other measures should/could be considered?

When preparing different options for evaluation a teacher/ librarian should ask herself some questions before deciding on one of the evaluation models.

1. What are the desired results of the evaluation?  
2. What is the focus of the evaluation?  
1. Will this survey be for teachers or students?
3. How will the student/teacher benefit?  
4. What will the policy on data be?
1. What type of data is being gathered?  
2. How will the data gathered be used?
3. How and where will data be retained?
4. How will disputed data be handled?
5. What type of environment will the evaluation by done in?  
1. Will it be handled by classroom teachers or by the librarian?
2. Where will the survey be done?

In the text, Trends and Issues in Instructional Design and Technology by Robert Reiser, Brinkerhoff and Patton have lists of questions associated with their models of evaluation. If the library/school/district is going to buy a new instructional program, return on investment is a very valid consideration.  For example, this year my district purchased a program called Achieve 3000.  It is a great program, but the roll-out was not efficient.  We were supposed to get computers to go in each classroom.  This was to ensure that the program would be executed correctly.  However, the money did not come through and we had to share the computer lab.  They also did not roll 0ut the program until October and teachers did not know how to use the resource.  Teachers needed to be consulted.  The teachers need to be vested in the instructional product to ensure it’s success.  Since a district’s/ library’s resources are limited, there are only so many programs they can put money toward.  It is important to find value defined as costs versus monetary benefits.  Consequently, the programs and projects having the best ROI (Return on Investment) will be the ones that are funded.  

Section IV focuses on human performance, performance support systems, knowledge management systems, and the concept of informal learning. Not all problems in learning and/or performance require an instructional one. Many times a non-instructional approach is a more appropriate solution. Identify a performance problem in your area of work and identify non-instructional solutions that may help solve the problem. Would better performance support systems, knowledge management systems, or opportunities for informal learning solve the problem?

We had a rewards based system called Bobcat Bucks.  We gave out the “money” when students went above and beyond in the classroom.  When they helped solve a problem, motivated others to make the right choices, or showed exemplary behavior or classwork students were rewarded with “bucks” they could then save and spend in the Bobcat store.  When students did not make the right choices they were not rewarded and therefore did not get to participate in any of the benefits of earning the reward.  Students began to instinctively understand the motivation to solve the problems and the behavior was ingrained into their systems.  think critically, work collaboratively, and be focus on a problem and solution.   We also had Bobcat Homework Headquarters (H2Q) where students could come in for help.  This prevented students from failing homework assignments. It was located in the computer lab every Tuesday through Friday before school.  Teacher would be available to help students and ensure they had the resources needed to do well on the homework assignment.

References

Reiser, R. A. & Dempsey, J. V. (2012). Trends and Issues in Instructional Design and Technology, 3rd ed. Boston: Pearson.