+ - 0:00:00
Notes for current slide
Notes for next slide

Module 2: Systems Analysis

EME6357: Evaluation of Instruction & Training in HPT

Dr. Bret Staudt Willet

September 22, 2021

1 / 25

View the slides:

bretsw.github.io/eme6357-module2

2 / 25

Evaluation Plan feedback

3 / 25

Evaluation Plan, Part 1

4 / 25

Evaluation Plan, Part 1

  • Overall, fantastic work
4 / 25

Evaluation Plan, Part 1

  • Overall, fantastic work

  • Work on developing your arguments (details, hyperlinks)

4 / 25

Evaluation Plan, Part 1

  • Overall, fantastic work

  • Work on developing your arguments (details, hyperlinks)

  • Topic sentences: start with the point, then add evidence

4 / 25

Evaluation Plan, Part 2

5 / 25

Evaluation Plan, Part 2

Systemic Analysis of a Training Program

5 / 25

Evaluation Plan, Part 2

Systemic Analysis of a Training Program

  • Create a systemic logic model for evaluating your training program
5 / 25

Evaluation Plan, Part 2

Systemic Analysis of a Training Program

  • Create a systemic logic model for evaluating your training program

  • Look at past examples

5 / 25

Flipgrid Discussions

6 / 25

Flipgrid Discussions: Feedback

7 / 25

Research Tradeoffs

Made with Padlet
8 / 25

Self-Report Research

  • Humans are bad at estimating
9 / 25

Self-Report Research

  • Humans are bad at estimating

    • (People are also bad at remembering)
9 / 25

Self-Report Research

  • Humans are bad at estimating

    • (People are also bad at remembering)
  • People change once they know you're observing

9 / 25

Self-Report Research

  • Humans are bad at estimating

    • (People are also bad at remembering)
  • People change once they know you're observing

Observer Effect

9 / 25

Unobtrusive Research

10 / 25

Unobtrusive Research

  • Digital trace data from social media
10 / 25

Unobtrusive Research

  • Digital trace data from social media

  • Reports and learning analytics from an LMS

10 / 25

Unobtrusive Research

  • Digital trace data from social media

  • Reports and learning analytics from an LMS

Researcher Bias

10 / 25

Research Tradeoffs

11 / 25

Research Tradeoffs

  • Breadth vs. Depth:
11 / 25

Research Tradeoffs

  • Breadth vs. Depth:

    • Trends in global chocolate consumption during the pandemic
11 / 25

Research Tradeoffs

  • Breadth vs. Depth:

    • Trends in global chocolate consumption during the pandemic

    • "Describe a place that this chocolate reminds of..."

11 / 25

Module 2 readings

12 / 25

Evaluating Training as a System

13 / 25

Darabi (2002)

14 / 25

Darabi (2002)

  • Systems approach to evaluation of training programs
14 / 25

Darabi (2002)

  • Systems approach to evaluation of training programs

    • systematic and procedural presentation of otherwise chaotic process
14 / 25

Darabi (2002)

  • Systems approach to evaluation of training programs

    • systematic and procedural presentation of otherwise chaotic process

    • whole picture before and while considering individual pieces

14 / 25

Darabi & Arrington (2017)

15 / 25

Darabi & Arrington (2017)

16 / 25

Darabi & Arrington (2017)

Systemic Logic Model

  • Inputs
  • Processes
  • Outputs
  • Outcomes
17 / 25

Darabi & Arrington (2017)

Systemic Logic Model

  • Inputs Reaction
  • Processes Learning
  • Outputs Behavior
  • Outcomes Results
18 / 25

Evaluator Competencies

19 / 25

IBSTPI (2006)

Professional Foundations

  • Communicate effectively in visual, oral, and written form
  • Establish and maintain professional credibility
  • Demonstrate effective interpersonal skills
  • Observe ethical and legal standards
  • Demonstrate awareness of the politics of evaluation
20 / 25

IBSTPI (2006)

Planning & Designing the Evaluation

  • Develop an effective evaluation plan
  • Develop a management plan for the evaluation
  • Devise data collection strategies to support the evaluation questions and design
  • Pilot test the data collection instruments and procedures
21 / 25

IBSTPI (2006)

Implementing the Evaluation Plan

  • Collect data
  • Analyze and interpret data
  • Disseminate and follow-up the findings and recommendations
22 / 25

IBSTPI (2006)

Managing the Evaluation

  • Monitor the management plan
  • Work effectively with personnel and stakeholders
23 / 25

Tourmen (2009)

Making evaluation choices with experience

  • Method-oriented vs. results-oriented
  • Contextual/situational factors
  • Actively transforming the situation
24 / 25

Questions?

What questions can I answer for you now?

How can I support you this week?


bret.staudtwillet@fsu.edu | @bretsw | bretsw.com

25 / 25

View the slides:

bretsw.github.io/eme6357-module2

2 / 25
Paused

Help

Keyboard shortcuts

, , Pg Up, k Go to previous slide
, , Pg Dn, Space, j Go to next slide
Home Go to first slide
End Go to last slide
Number + Return Go to specific slide
b / m / f Toggle blackout / mirrored / fullscreen mode
c Clone slideshow
p Toggle presenter mode
t Restart the presentation timer
?, h Toggle this help
Esc Back to slideshow