Tag: peer review

As part of a series of case studies, staff at LET sat down to have a conversation with Prof. Elgar Fleisch, Johannes Hübner and Dominik Bilgeri from the Department of Management, Technology, and Economics (D-MTEC) to discuss their Mastering Digital Business Model (MDBM) course.

What is the project about?

In this Mastering Digital Business Model (MDBM) course, Prof. Elgar Fleisch, Dominik Bilgeri, George Boateng and Johannes Huebner teach Master’s level students a theory- and practice-based understanding of how today’s information technologies enable new digital business models and transform existing ones. The course contains a novel examination mode, a video group project is introduced as a core element contributing to the overall course grade. In addition, students are asked to participate in a peer-to-peer review of the videos produced by other student groups, which is independent of the grading and is geared towards giving students insights in how other groups solved the challenge. The best-rated videos are then shared with the entire class in the end of the semester.

As part of this newly created examination element, course participants (in teams of two to three students) explain one of the major lecture topics (theoretical lenses) in the first half of their video.Then they apply the same lens by analysing a company, aiming to better understand its underlying business model. Companies are pre-selected and allocated to students for fairness reasons. Every year, we choose a pool of interesting companies in the context of digital transformation, the Internet of Things, Blockchain, e-health, etc.

What motivated you to initiate the project?

The core idea was to improve students’ learning success by using an examination format that not only requires learners to reiterate theoretical contents, but also apply the theory in a practical context. The students have different backgrounds, and do not necessarily have a strong business focus, which means that many of the concepts taught in class may be rather abstract. We used the video format and specific companies as case studies, because we think this is a good way to trigger curiosity, show concrete examples of modern companies in a compact form, and force students to reflect deeply upon theoretical frameworks compared to other examination formats.

How did you do it?

Aside from the weekly input lectures, we ask students to form groups in the beginning of the semester. We then provide a list of theoretical core topics from which each group can choose one. In addition, we randomly assign each group to a case company. The theoretical topic then first needs to be explained in the first half of the video, and then be applied to the case company in the second half. Here we thus used a prosumer approach, where students become part of the course because they create a small section of the content. The best videos are shared with the class, and can be reused as additional learning materials for future cohorts. This set-up generally resulted in high-quality videos, perhaps also since students knew their videos will be used again.

Students also had to review the video projects of five other groups. They had to clearly describe whether and how their peers used certain perspectives (called “lenses” in the course) which played a role in the video and in their feedback. In this way they analysed once more how the newly learned concepts were visible in other companies – a positive side effect being that they also honed their reflection and feedback skills.

Did you have the support you needed for the project? Is there additional support you wish you had had to help you to achieve your goals?

We asked two students from previous cohorts to join us as tutors, and support this year’s groups primarily with technical questions about video-making (e.g. tools, quality considerations etc.). In addition, we designed one of the lecture slots as a coaching session during which we would further support student groups with their questions. In total, this approach allowed us to provide the students with high-quality supervision with reasonable effort.

Please describe some of the key outcomes of the project

To most students, the task of creating a video was new. We received feedback that while the initial effort for learning how to make a video was higher compared to other examination formats, it was also fun and very helpful to really understand and apply the new concepts. They said that they learned things more deeply and more sustainably because they had to consider all details and aspects – compared to the practical exercises they are familiar with in other courses. By carefully phrasing their arguments in giving feedback on peer videos, students became more aware of their own thinking and argumentation.

We observed that the questions asked by students once they start creating videos were different and went deeper, i.e. their reflections were based on many concrete examples of companies, and the concepts were put into perspective. The same sub-concepts have a different meaning in another context, and students now see the overarching principles better and can argue more precisely about theoretical aspects. Without these concrete examples, it would have been harder to concretely grasp the theoretical aspects.

How did the project impact learners or the way in which you teach?

We were surprised by the high quality of the best student videos. The teaching team is now really motivated to continue innovating on our approaches in other courses. We saw clearly that when students are very active we get better results, deeper learning and better reflection.

What lessons learned do you want to share with your colleagues?

It can really pay off to try things and to experiment. We think that nowadays the classic format of passive lectures and final exams may not always be the best choice. We believe the improved outcomes through students who were actively engaged by the video assignment justified the investment in developing new approaches and tools.

When considering videos as an examination format, you should define the entire course/project very clearly. When describing what production options students have for videos, you should be very precise. Offering too many options can be counterproductive. It is better to present 3-4 crystal-clear examples and stick to them.

Also, we would recommend managing students’ expectations clearly in the beginning of the semester, and highlighting both the benefits and challenges of this examination format. Of course, this becomes easier after the first year, when you can draw from the experience of the first cohort, and also provide examples of prior videos to illustrate what is expected of the groups. Because the students are co-creators you get new and relevant content which enriches the course and can serve to motivate both students and teachers.

What are the future plans for this work? How do you plan to sustain what you have created through the project?

We plan to optimize some details of this course, and to go even more in the direction of a flipped classroom to use this teaching approach in other courses. We will create a library of the student videos to provide it as additional learning materials in future editions of the course.

Student feedback

By MDBM Student Cristina Mercandetti (mercandc@student.ethz.ch)

1. Your opinion about this course and the peer review & video production process – how has it influenced your learning process?
   Cristina Mercandetti: I really enjoyed both the course and the video production process. I think they complemented each other very well and we were able to directly apply the theoretical knowledge learned in the course to work on our project. It helped me to think more critically about the course content, and really dive into some of the lenses and models presented. I don’t think this would have been possible without the video production, so it definitely improved my learning process.

• Do you think this approach could be used in other courses?
  Cristina Mercandetti: Yes, I think this approach could easily be used in other classes. However, I think part of the fun in this class was that the video production was something very new and refreshing (a side effect was that I learned how to cut a short movie). I imagine that if several classes introduced this it would lose some of its novelty and could be stressful, as it took a lot of time.

• Final remarks about the course
  Cristina Mercandetti: I really enjoyed the whole class, and heard a lot of good things from other students too.

Read more

Posted on 09/12/2020 by Pascal Xavier Schmidt

As part of a series of case studies, staff at LET sat down to have a conversation with Prof. Laura Nyström and Dr. Melanie Erzinger from the Department of Health Sciences and Technology to discuss their food chemistry laboratory project.

What is the project about?

We introduced a new way to write lab reports, combined with a peer review method to foster collaboration and critical thinking skills among students. In the past students did not have clear criteria as to what makes a good report. Assistants also needed too much time to read the reports and give repeated feedback. Thus we looked for a way to help assistants spend less time on the review process.

We transformed the format of our Food Chemistry Laboratory Course (Food Science, BSc level, 4^th semester) from a classical lecture format with lab exercises to a blended learning format. With new videos, we can achieve better coverage of basic knowledge (i.e. security, handling of equipment).

What motivated you to initiate the project?

Student numbers have increased over the past 10 years and we have been losing too much time in covering basic knowledge repeatedly. Using concept videos, students will be able to review key topics on their own. Overall, we also wanted to make the entire course more attractive. A key intention was to develop student skills in report writing and improve report quality.

How did you do it?

We defined additional, clear quality criteria for a good report. During a first round students give each other feedback, such that final review by teaching assistants and lecturer approval involve less effort. For each experiment, every student has to review another student’s report. In total, each reviews four reports over the semester.

Students don’t get a grade for the peer review (semester performance in the lab course is also ungraded). They have one week for each of the four peer reviews, and must complete each by the respective set deadline. They answer various questions related to the quality of the respective report (these involve five aspects plus overall feedback; see the annex at the end of this case study). Students do not “grade” the reports, but give feedback in their own words.

Assistants are aware of what is asked in the reports and are therefore able to provide targeted and helpful feedback in the lab which addresses the quality criteria for reports.

We provide the students with online material on how to write reports (short videos, documents etc.). Previously we had a short lecture with examples. Until now, however, we did not train them in conducting proper peer reviews. We have now realised that we need to do this (especially for Bachelor’s degree students), and will include peer review training with the short lecture next year.

Did you have the support you needed for the project? Is there additional support you wish you had had to help you to achieve your goals?

We learned about a module inside our LMS for administering the peer-review process (“workshop module” in Moodle). It would have been helpful to have had practical tips from others, but apparently not many lecturers have used this tool. Although the general instructions were useful, it took quite some time to learn all the aspects of the tool.

Please describe some of the key outcomes of the project.

Various things changed for the better. Students learned a lot by reading and reviewing the reports of their peers. They gained important input for their own reports. For many it was the first time they had had to give feedback in such a structured way. They also had to find a way to critique something in a good, constructive manner. Overall, students were introduced to a new way of critical thinking and took important first steps in this skill, which is important for their later careers.

We can say clearly that through the new review method we were able to improve the quality of reports and reduce the time needed by lecturers to grade them.

How did the project impact learners or the way in which you teach?

In general the peer review method was well received in the BSc course, and we used the same approach in an MSc-level course. We therefore realised that Bachelor’s degree students need more help and training in peer review than Master’s students.

Overall we saw that the blended learning approach and the peer review methods work to improve our courses, addressing the above-mentioned challenges of lack of student preparation and the need to constantly repeat basic knowledge. Students themselves clearly realised the value and potential of better collaboration, peer feedback and critical thinking skills.

What lessons learned do you want to share with your colleagues?

Not every cohort is the same. While things worked quite well in 2017, in 2018 fewer students adhered to the schedule and deadlines – even though everything was communicated and documented in the same way as in the previous year.

What are your future plans for this work? How do you plan to sustain what you have created through the project?

More and more assistants will become competent in providing full reviews of the already peer-reviewed reports. Currently lecturers still have to do this. Lecturers will thus gain more time to be present in the labs and to give 1:1 feedback to students in the lab and online.

We will definitely create some training material for assistants for this purpose, but it is not available yet. We also want to create a short, ready-to-use document about giving feedback in our specific context: what is constructive feedback, what are the do’s and don’ts? Students, assistants and fellow instructors can use it.

We are interested in learning whether other lab courses at ETH do something similar, and how. We also need to improve the support situation with the Moodle review tool “workshop module”. We will continue to work with it, but it is a bit tricky sometimes.

One additional idea is to make the videos interactive. Students will see in-built questions in the videos which they have to answer right away.

Additional notes regarding resources and tools used.

• We used a programme called Labster to create virtual labs in some cases to extend the experience to experiments which were not doable in reality in our labs.
• We learned from other courses and departments regarding effective feedback (ETH “Foundations of Teaching and Learning” course).
• To conduct the peer review we work with the “workshop module” in Moodle.
• To make the videos interactive we will work with the new Moodle “interactive video suite” plugin.

Student voices:

What is your opinion about this course and the peer review process (lab reports). How has it influenced your learning process?

Robert Spiess: I think peer-reviewing was a great way to see other students’ work. It gave me the opportunity to experience and compare different ways of writing. I could always detect things that I wanted to include in my reports. At the same time, I could see in which points my reports were better, where my advantages were.

I think this procedure is particularly useful when writing. But the reports should not be too long, because, otherwise, students have to spend too much time on their own report and neglect, as a consequence, the peer-reviewing of someone else’s report. Other courses usually required longer reports. But if the reports were shortened, the method could also be applied in other laboratory courses (such as in the food processing or in the biotechnology lab course).

Aline Candrian: I’m glad I did the course, I think it gives students a first impression of laboratory work. The lab report writing is an essential part of the course to understand the experiment and the obtained data. The peer review approach was fine, even though nobody was eager to do them. VERY little time was invested into peer review by most groups, as far as I’m concerned. Of course, sometimes you could benefit from your peer’s feedback but most of the times we didn’t act on them. That’s probably because it was our first time writing (semi-) real reports. We didn’t really know what we were doing and you mostly think you know better than others, especially if someone reviewed your report who you rate less familiar with chemistry.

At the same time you’re well aware that you know nothing about report writing, so how can you evaluate someone else’s work?

Additionally, motivation was minimal since you were just glad to be done with writing your report. Having to assess another report and then correct your own report again was just another ‘burden’. So, altogether, I would say report-writing was a crucial part of the course but peer review not so much since we had no experience at all. I think peer review makes more sense in the courses in our last semester.

Making the students just do a peer review on the last report might work. They’ll see how it works, they’ll have had written a few reports (and got more familiar with it) and might be more confident in providing feedback. But I’m not an expert, it may not work the way I envision it, what do I know 🙂

Read more

Posted on 09/12/2020 by Pascal Xavier Schmidt