class: left, bottom, title-slide # Module 1: What is Evaluation? ## EME6357: Evaluation of Instruction & Training in HPT ### Dr. Bret Staudt Willet ### September 1, 2021 --- class: inverse, center, middle **View the slides:** [bretsw.github.io/eme6357-module1](https://bretsw.github.io/eme6357-module1) --- class: inverse, center, middle # Module 1 readings --- class: inverse, center, middle # Evaluation vs. research <br> (Boothroyd, 2018) --- # Defining evaluation <img src="img/measure.jpg" width="600px" style="display: block; margin: auto;" /> -- *systematic collection of information, for use by people to make decisions to improve training programs* --- # Evaluation vs. research <img src="img/tools.jpg" width="480px" style="display: block; margin: auto;" /> -- - Both are rigorous -- - Both allow for similar designs and methodologies -- - **Evaluation** is timely and constructive, aiming to improve programs -- - **Research** is meant to produce generalizable knowlege, aiming to better the field --- class: inverse, center, middle # Types of evaluation <br> (CDC) --- # Formative evaluation <img src="img/shape.jpg" width="600px" style="display: block; margin: auto;" /> -- - Needs assessment -- - Process/implementation evaluation --- # Summative evaluation <img src="img/pottery.jpg" width="600px" style="display: block; margin: auto;" /> -- - Outcome/effectiveness evaluation (objective, performance indicators) -- - Impact evaluation -- - Economic evaluation (cost, cost-benefit, cost-utility) --- class: inverse, center, middle # Methods of evaluation <br> (WHO) --- # Qualitative evaluation <img src="img/words.jpg" width="720px" style="display: block; margin: auto;" /> --- # Quantitative evaluation <img src="img/numbers.jpg" width="600px" style="display: block; margin: auto;" /> -- - Randomized control trial (RCT) -- - Quasi-experiment -- - Before-after --- class: inverse, center, middle # 4 Levels <br> (Kirkpatrick & Kirkpatrick, 2006) *Systemic evaluation of training programs* --- # Kirkpatrick's 10 steps to plan <img src="img/plan.jpg" width="420px" style="display: block; margin: auto;" /> 1. Determining needs 2. Setting objectives 3. Determining subject content 4. Selecting participants 5. Determining the best schedule 6. Selecting appropriate facilities 7. Selecting appropriate instructors 8. Selecting and preparing audiovisual aids 9. Coordinating the program 10. Evaluating the program --- # Kirkpatrick's 10 steps to plan <img src="img/plan.jpg" width="420px" style="display: block; margin: auto;" /> 1. Determining needs 2. Setting objectives 3. Determining subject content 4. Selecting participants 5. Determining the best schedule 6. Selecting appropriate facilities 7. Selecting appropriate instructors 8. Selecting and preparing audiovisual aids 9. Coordinating the program 10. **Evaluating the program**
**implementing the 4 levels** --- # Kirkpatrick's 4 Levels <img src="img/four-levels.jpg" width="480px" style="display: block; margin: auto;" /> -- 1. **Reaction** -- 2. **Learning** -- 3. **Behavior** -- 4. **Results** --- # Start with "Why?" when planning <img src="img/four-levels.jpg" width="480px" style="display: block; margin: auto;" /> ## 4. Results
3. Behavior
2. Learning
1. Reaction --- # Go in order when evaluating <img src="img/four-levels.jpg" width="480px" style="display: block; margin: auto;" /> 1. **Reaction** 2. **Learning** 3. **Behavior** 4. **Results** --- # Go in order when evaluating <img src="img/four-levels.jpg" width="480px" style="display: block; margin: auto;" /> 1. **Reaction**
**customer satisfaction** 2. **Learning** 3. **Behavior** 4. **Results** --- # Go in order when evaluating <img src="img/four-levels.jpg" width="480px" style="display: block; margin: auto;" /> 1. **Reaction** 2. **Learning**
**attitudes changed, knowledge increased, or skills improved** 3. **Behavior** 4. **Results** --- # Go in order when evaluating <img src="img/four-levels.jpg" width="480px" style="display: block; margin: auto;" /> 1. **Reaction** 2. **Learning** 3. **Behavior**
**observable change that transfers to job** 4. **Results** --- # Go in order when evaluating <img src="img/four-levels.jpg" width="480px" style="display: block; margin: auto;" /> 1. **Reaction** 2. **Learning** 3. **Behavior** 4. **Results**
**the ultimate goal, the planned outcome, the "Why?"** --- class: inverse, center, middle # Looking ahead --- # Semester schedule <img src="img/across-time-full.jpg" width="480px" style="display: block; margin: auto;" /> - **Module 1: What is Evaluation?** - Module 2: Systems Analysis & Training Programs - Module 3: Evaluation Methodology - Module 4: Creating an Evaluation Plan --- # Module structure <img src="img/workshop.jpg" width="540px" style="display: block; margin: auto;" /> -
Meet (Wednesdays at 8pm EST on Zoom) -
Read -
Discuss (Flipgrid) -
Evaluate (group project) --- # Major Assignments <img src="img/build.jpg" width="320px" style="display: block; margin: auto;" /> - **Flipgrid discussions** (200 points) - **Case study facilitation** (group project; 100 points) - **Evaluation plan** (group project; 700 points) - Training program description (100 points) - Systemic analysis of a training program (200 points) - Peer review of systemic analysis (100 points) - Evaluation management plan (300 points) --- # Flipgrid Discussions <img src="img/chat.jpg" width="480px" style="display: block; margin: auto;" /> -- - Text, audio, and video comments are all fine -- - 30-second comments can be just as good as 90-second ones -- - Responding to those who respond to you is great (threads) -- - Ending your comments with a question is great -- - **Main rubric: Move the the conversation forward** --- # Case Study Facilitation <img src="img/chat2.jpg" width="480px" style="display: block; margin: auto;" /> -- [Read the Case Study Discussion Facilitation instructions](https://canvas.fsu.edu/courses/181923/assignments/1299968?module_item_id=3197033) -- - **Part 1: Introduce the case.** Your materials will be posted on Canvas. -- - **Part 2: Create a few discussion prompts.** Your prompts will be listed on Canvas as on Flipgrid. -- - **Part 3: Keep the discussion going.** Stay active on Flipgrid. --- # Training Program Description <img src="img/group-project.jpg" width="480px" style="display: block; margin: auto;" /> -- - First part of the main course assignment: Evaluation Plan -- - This is a group project, working with the same group as your Case Study Facilitation -- - This first part is fairly short and is due at the end of Module 1 --- class: inverse, center, middle # Questions? <img src="img/question.jpg" width="480px" style="display: block; margin: auto;" /> **What questions can I answer for you now?** **How can I support you this week?** <hr>
[bret.staudtwillet@fsu.edu](mailto:bret.staudtwillet@fsu.edu) |
[@bretsw](https://twitter.com/bretsw) |
[bretsw.com](http://bretsw.com)