How To Consider The Effectiveness Of Studying Experiences
How effective are your learning experiences?
Companies that invest in online learning often do not have the tools or practices to effectively evaluate their learning experiences. You tend to focus on flat metrics like course completion or the number of correctly answered questions in activities. While students may complete a course, there is no telling if there will be a behavior change that should be the goal of the learning experience, and without a full picture of the impact of the learning experience, the return on investment is unknown.
Successful learning experiences should increase the skills and competencies of employees and teams and contribute to an organizational learning culture while increasing sales and reducing costs. The measurement of graduation rates and grades, especially with self-directed online courses, will only lead to poorly perceived acceptance rates and, in all likelihood, diminishing support from budget holders.
So how do you measure the impact of learning?
Here we can fall back on the principles of monitoring and evaluation and some well-known models of learning evaluation.
Monitoring and evaluation
Monitoring and evaluation is an approach that is mainly used to measure the effectiveness of programs or initiatives in the field of development with the aim of achieving maximum impact. The principles of this approach can also be applied to the measurement of online learning experiences. Here are a number of ways you can apply M&E principles to learning:
- Set clear goals and objectives of what success looks like and what criteria should be used to measure the progress of the program from start to finish.
- Collect and analyze data at the beginning of the program to serve as a foundation. From then on, collect data relevant to the objectives of the program and indicating the degree of its effectiveness on a regular basis.
- After analyzing the data, make changes and repeat the program implementation to improve the effectiveness of the program.
- At the end of the program, compare your results with the data collected at the beginning of the program to see if behaviors or skills have changed and what can be improved in the future.
With a monitoring and assessment mentality, you can more effectively implement and design criteria for assessing learning effectiveness.
What are learning evaluations?
Learning assessments are the criteria that you analyze to determine the success of your learning experience. There are some theorists who have developed various learning assessment models that suggest important assessment criteria at both the micro level (the learning experience itself) and the macro level (its impact on your business and even society). For example, Kirkpatrick’s model, founded by training expert Donald Kirkpatrick, suggests that a learning experience should be assessed on the following four levels: response, learning, behavior, outcomes.
Roger Kaufman, an expert in evaluation and needs analysis, extended Kirkpatrick’s model to include an assessment of the societal impact of learning experiences in what is known as Kaufman’s Five Levels of Evaluation.
What is to be assessed?
Knowing your general assessment criteria presented above is a good place to start, but you also need to identify the specific metrics that indicate whether or not your learning was successful.
Start by tracking the achievement of the learning goals. At the beginning of the learning design process, you should have identified goals and competencies that you want learners to develop through the learning experience. Not only do these help clarify the learning and motivate learners towards specific goals, but are also useful for measuring whether or not the learning experience achieved the goals.
Other metrics for the rating are:
- Perform assessments
Assess the skills and competencies of your employees before and after the training through simple basic surveys and knowledge checks to determine whether there has been any development. - Pursue commitment
Keeping track of course completion, grades of individual assessments, the course as a whole, and learners’ engagement with the course through clicks, time spent, posting to discussion forums, and liking or sharing content will all help make your course a success. - Measure results
By observing an increase or decrease in grades and what caused this change, you can better understand whether the content and your learners’ experiences with the content have been effective and whether there are any gaps in knowledge due to their understanding or course design. - Analyze data
Analyzing data from the leaderboards, as well as course levels, badges, and rewards earned in playful learning experiences, also helps measure the impact of a course.
Using digital data through xAPI is also a good indicator of how learners are interacting with your course and other online platforms in your organization. With xAPI, or Experience Application Program Interface, you can track and see how learners are using your course, what they click, how much time is spent on what activity or content, what was revisited, and so on. This data is invaluable when analyzing the success of individual components of your course. In informal learning in the workplace, where employees are largely self-determined in their learning, xAPI can also track the interaction with various browser tabs and applications, which gives you a sense of the employees’ commitment to continuous learning.
diploma
Ultimately, understanding the different learning assessment models and identifying useful metrics is the first step in effectively evaluating your online learning experience. Combining this with understanding your learning analytics will result in a better and more accurate return on learning and better learning experiences for your learners.
Increase Learning
Elevate Learning is a full-service learning design consultancy. Together with you, we design and implement online and blended learning experiences, organization-specific learning programs as well as experience-related content and workshops.
