Wednesday, 7 May 2014

Week 3, Chapter 3 - Blended Assessments of Learning

Welcome Back!

This week's chapter was all about assessing what students are learning.  Formally or Informally?  Should it be online or in class? Are you assessing fact recall? Or do you want to assess analysis, synthesis or evaluation?

While I will provide some information from the lesson, I am going to talk a little more about my own MOOC experiences.  I want to relate what has been done in those classes and my impressions as a student.  This may allow someone who is designing a course a different perspective.

I have taken about a gazillionty courses (no, not an official number, but it has been a lot). So many that I don't think I have added all of them to my or LinkedIn profile.  Some have been great.  Some have been awful.

The biggest difference was in the testing.

Anyone can provide content.  Videos of lectures, articles to read.  They are necessary. They give you what you need to know. But if I wasn't being tested, how much did I actually take in?

See, I can turn on a lecture on one tab, and continue on another checking Facebook or LinkedIn.  I can read another set of notes.  The lecture is going on, I might hear some of it, but how much is anyone's guess. Unless you are testing me, you won't know.

One of the best courses I am taking is "An Introduction to Marketing" by Professors Bell, Fader and Kahn from Wharton University of Pennsylvania.  There are generally 6 videos a week to watch (not just listen to) and throughout the video at least once, and sometimes a couple of times, it pauses for you to answer a question. It's quizzing me throughout.  If I were to get an answer wrong, I would go back and replay that section of the lecture, jump over to the discussion group to ask a question, or do a quick Google search for more information in the moment to clarify.  I know I don't get it.  They may know they I didn't get it (I don't know if they collect the information or not). But I don't worry because this is an informal quiz.  It doesn't count towards my grade.  These questions are usually fact based multiple choice questions (following Bloom's Taxonomy, fairly low level evaluations of knowledge and maybe understanding).  But these questions prepared me well for my first "big" quiz. 
Yes, some of the questions on the quiz were fact based "Which of the following explains X".  But as the quiz went on, I had to read, analyse, evaluate. I actually had to think.  And apply.  I was sick to my stomach (hey, aren't MOOCs supposed to be easy???).  As it was multiple choice, I had the results back in seconds (thank goodness!  I couldn't handle the suspense!).  I did very well.   
Of course I did.  I had had 18-36 informal questions tossed at me over three weeks.  I had invested my time asking questions and answering others in the discussion groups (that are facilitated by Community TAs who have MBAs and Doctorates who correct us or redirect us if we are on the wrong path).  I had taken notes, printed off a set of notes offered by the course that included the slides and definitions (and I wrote all over those as I listened to the lectures).  I had investigated anything I didn't understand completely.  Most of all, I had applied what I was learning to my experiences in retail.  I could relate what I was learning to what I had done.  Now I could see the "Why" to the "What".

BTW, I got 100% on that quiz.  The study by Walker et all (2014) that was mentioned on the chapter that "found that non-credit, online practice exams can actually benefit student performance on in-class, graded exams" should probably be applied also to online, graded exams.  I love when my reality matches an academic research study.

Now, let's review a few (ok, more like 15) key points from the chapter...

  1. It is imperative that assessment is provided to check the depth of students' learning. (Frequent assessment helped me see what I was and was not learning).
  2. If learning is not transferred from the place of learning to practical application, there can be no positive return on investment.  Lessons need to have tangible meaning, applicable in real life.  Show the students where and how they will use it in real life.(Remember how I said I could now see the "Why" to the "What"?)
  3. Scaffolding must be put in place to ensure students understand HOW to use the tool used for testing. If someone doesn't "get" how to maneuver their way through an online exam, you will not get a clear understanding of their knowledge levels. (One class I took had a quiz on the Orientation component, it didn't count for the marks but gave us a chance to work through issues in a "safe" environment).
  4. Online multiple choice tests are easy to grade, but questions have to be written in a way to test more than knowledge and understanding.  It must evoke application, analyse, and evaluation. (I was asked to analyse and determine target audiences, positioning statements, etc)
  5. Informal online quizzes lead to better results and performance on formal examinations. (Multiple quizzes per lecture led to a 100% test)
  6. Formal assessments contribute to the students final mark and indicate the students mastery of a topic. (Again, refer to 100% on test)
  7. Informal assessments allow the instructor to get an idea of their students comprehension without assigning grades.  This is also an opportunity for the instructor to evaluate their methods and content and perhaps revamp or offer more/different material on a topic to invoke more understanding by the students. (Material is released on a weekly basis, the opportunity is there for review by evaluators, although it is generally taken to the discussion forum as a topic introduced by the TA)
  8. Multiple Choice tests assess a students ability to recall and recognize content. To be effective, more questions than needed should be create and randomized so questions (or at least options in answers) are not identical.  To minimize academic dishonesty, having a pool of 2x the number of questions needed and the quizzes being randomly generated by the program allow for individual quizzes. (I've not run into multiple sets of questions, but answers have been randomized in their positions)
  9. Proctoring on-line exams is possible but not practical.  Who does it?  Who pays for it? What is needed to validate that the person is who they say? (Thankfully that has not been required thus far although several courses seem to run a "verified" track that requires the student to submit a copy of their ID and a payment.  I'm always concerned with having to share my identification with anyone, thus I've not done this to date)
  10. Essays and essay questions require more resources to grade.  In a MOOC, it could be difficult to grade everyone's essays in a timely fashion (especially courses with 1000's of students). However, they allow you to gauge how well a student can apply what they have learned. Alternatives to the formal essay exam question might be to watch the discussion groups and forums.  (Coursera courses have community TAs who follow the conversations and pop in to correct or re-direct as needed.  It's a great feature!.  This allows students to problem solve amongst themselves, to use high levels of understanding and in teaching another, greater understanding to the responder.)
  11. Online assessments don't have to be "just" tests.  They can include other things like reflections (blogs are a great example), videos (upload a video of a presentation, design a commercial ad for a product, create a short film called "Mr X and the Shadow of Doom"), creating a presentation, using code to design a webpage or app, etc.  If the course requires hands on work, having the "master" observe the "apprentice" and log notes about their techniques and abilities.  These type of assessments are great for e-portfolios.
  12. Clear expectations must be defined.  Checklists or rubics need to be developed either by the evaluator and shared with the group OR by the evaluator AND the group together.  Either way, the students need to know on what and how they will be evaluated.  Clearly stated due dates, processes, group participation, and learning objectives will only help the students understand your expectations.
  13. Informal assessments only enhance a course.  They increase participation, engagement and understanding.  They allow a student a response (You scored 80%, and which question they got wrong and what the correct answer is) via the program of their understanding.  They can use that to better prepare for the formal examination.
  14. One sentence summaries to get a quick glance at a students comprehension.  They answer 7 questions separately and combine those answers into one sentence.  Great for discussion groups which allow for peer review.
  15. Requesting student generated questions that might be used on a test lets the evaluator know what the students think is important. Also might give you some GREAT questions to use ;)
This really was a great chapter.  Whether assessments are done online or face to face, having some form of informal assessments really allow the student to better succeed.  It also allows the evaluator an opportunity to review and reevalute their own work and content (if a large group got a question from a lecture online wrong, it could be discussed F2F for example).  If used properly, it will only create a better course!