fbpx
#133: Improving Student Engagement Using Metrics and Data

#133: Improving Student Engagement Using Metrics and Data

This content initially appeared at APUEdge.com. 

Podcast with Dr. Bethanie L. HansenAssociate Dean (Interim), School of Arts, Humanities and Education 

Student engagement is a critical part of learning. In this episode, APU’s Dr. Bethanie Hansen discusses how to improve student engagement in the online classroom using available metrics and data. Learn how educators can use that information to adjust assignments to help improve student engagement.

Listen to the Episode:

Subscribe to Online Teaching Lounge
Apple Podcasts | Spotify | Google Podcasts | Stitcher | Pandora

Read the Transcript:

Bethanie Hansen: This podcast is for educators, academics, and parents, who know that online teaching can be challenging, but it can also be rewarding, engaging, and fun! Welcome to the Online Teaching Lounge. I’m your host, Dr. Bethanie Hansen. And I’ll be your guide for online teaching tips, topics, and strategies. Walk with me into the Online Teaching Lounge.

Welcome to the Online Teaching Lounge. I’m here to talk with you a little bit about student engagement in online education. The word “engagement” is commonly known when you’re in love with someone, you’re thinking about marrying them. And engagement means you’re connected; you have a goal of doing something together. It also means maybe a military encounter between two different forces.

Now, something that engages people online is sort of along those lines: We’re coming together, we’re interacting, we have plans of doing something together, and we want it to be meaningful. The online education definition of engagement isn’t really the formal agreement to get married, or just an arrangement to do something, or go somewhere at a certain time. It’s not even a battle plan. Really, engagement in online education is about the ways in which students and faculty members engage with—or interact with—the content, each other, and the ideas.

There can be student engagement with the textbook, the videos that you put in your online class. There can be student engagement with each other; so, there’s some kind of dialogue or maybe there’s even live chat happening or live video happening.

There could be student-to-faculty engagement, or faculty to student. So, we’ve got messaging, we’ve got discussion areas, we’ve got live video or live chat. All of these different things fall into the category of engagement in online education. Engagement really is kind of this buzzword that we use a lot in online education because we need some way of talking about people showing up.

In a live class, in a face-to-face setting, you can walk into the room and see people there. You can also look at the gradebook and see whether students have submitted work, what their scoring is. You can find out how often or how much the faculty member has lectured or taught in that class. And all of those would be live engagement in a face-to-face setting.

Using Metrics and Data to Assess Student Engagement

Online education is a little bit different because we can look at metrics, we can actually look at login data, we can look at the number of times people have accessed particular content. We can look at how many times, how frequently, and how substantially they have posted in that discussion forum. All of those things help us to know about the engagement in online learning.

Now, in online learning, student engagement is all about figuring out what’s working, whether people are learning, and whether they’re really being taught and transformed in that experience. There are some kinds of engagement statistics online educators should know about. And if you’re teaching online right now, these could be very interesting to take a look at. On the very basic level, something in your learning management system will track or measure the days and the length of time that your students have logged into the platform.

If they’re going to read things offline, like if they have a physical textbook, of course, you can’t track that, you don’t know exactly how much time they’re spending in that content. But you can see when they’re in the classroom, how many times they’ve clicked into the classroom, during the week, and how many minutes they have spent.

Some learning management systems will also let you know which parts of the content students have accessed. So, maybe you can see, did they open the lesson? Did they open the test? Did they go into a quiz? Did they go into the discussion? Did they reply first and then post that initial response or post the initial response and then come back? A lot of this information, as an online educator, helps you get a sense of where your students really are spending their time, and how engaged they are in the class itself.

As you look at these trends of students clicking in and spending time, you can get a sense for what’s working, what kind of content you’ve put into that class, and whether or not something might need to be modified. Or maybe there needs to be more material added or too much material.

Looking at those on a very basic level just helps you understand the quality of the course and the quality of your teaching at kind of at a basic level. Now, as students start to engage in the discussion or interact in the discussion space, reading what they’ve written, you can also see things like what they’re understanding, the degree to which they can use some of the terms in the course, you can notice those things in the discussion. And notice how they’re using the words and start to know whether or not they’re really understanding the concepts.

How as this helps you? As an online faculty member, you can look at what students have posted in that discussion and start to ask a lot of questions. You can give some additional guidance or examples. And if you really participate throughout the week and read what they’re writing, they’ll come back, and they’ll respond to you again and again.

So, it helps to notice the real time or asynchronous, somewhat real-time engagement, throughout the week and see what’s happening in that discussion and be part of it and respond to it and interact with it. This will help students engage with each other a lot more, engage with the content more, and engage with you. And they’ll even get to know you a little bit, which will help them to trust you, and feel confident turning in those assignments.

How Understanding Engagement Levels Can Help with Course Design

Now, another thing that you can do to look at engagement in an online course, is to look at the way they’re filling out their assignments and submitting them. Sometimes you’ll get a student who really is off the mark on their assignment. And then looking at that first type of engagement, just how much they’re in the course, what they’re accessing, what they’re reading, you can kind of tell, have they gone through the parts of the course where they should have learned that? Have they spent the time there?

Some students will just misinterpret instructions and some will find helpful things on the internet, and just scoop those up and translate them into their assignments without really processing them. So, it’s helpful to notice the pattern of how they participated in the class, and then what’s going on in their assignments.

Some of the engagement in assignments will give you a lot of insight about what could be altered in your course. And also, what’s working in your course. I know one of my approaches in a class was to really zero in on the academic vocabulary. So, as I was teaching the students, I teach music appreciation, so as I’m teaching them the music terms, I’m looking for the way they use those terms in that discussion. And then the feedback I’m giving them is specifically about the kind of way they’re using the terms. How they’re using them in a sentence, what they’re describing in the music, whether it’s true, whether it’s accurate, whether they’re using those terms knowingly or just kind of throwing them all into a sentence together without any examples.

So, as I look at assignments, I also look at those terms and how they’ve engaged with the concepts. Are they able to demonstrate what they know? Are they able to talk about it in an intelligent or informed way? Online student engagement can be demonstrated in a lot of different ways. There are indicators in the quality of their responses, the frequency of their responses, and their access to the course. And, also, the depth of cognitive presence that they’re demonstrating.

Whatever metrics are available for you in your learning management system, I encourage you to take a look at those and to review them and determine which of these metrics helps you to fully understand what students are actually doing in the class, and which seem related to their performance on the actual assignments and in the discussions.

Once you’ve done that, the next place you can look to see after the fact how students have engaged or how they experienced this, is in their end-of-course evaluations. That little bit of data might have some free response answers. I know in my case, I used to use end-of-course survey data to evaluate my own teaching. And sometimes students would give me suggestions about modifying an assignment, or comments about whether or not they liked particular assignments. And I would look at those scores and comments, and then look at my class and find interesting and creative ways to make modifications for future sections.

Over time, that allowed me to create a group project. And as that group project played out, session after session, I would change little things about it based on student feedback, to see them engage even more and engage better and interact with each other better. For example, their end-of-course survey comments prompted me to intentionally design the groups in certain ways.

I would choose to make sure there was someone in the group that knew something about music coming into it, so they could kind of support the others, and that there was a diversity of student voices represented. In my university, there are a lot of military students and not as many civilian students. And so, I would kind of group those accordingly. I would have a little mixture in each group so we had some diversity of thought and diversity of experiences, so they could also learn from each other.

I also tried this with random groupings. And I got a lot of feedback from students about that, too. It seemed like the intentional grouping was the way to go. So, noticing their feedback, and then looking in on how they actually participated in the group project was a really helpful way to modify what I was doing as the faculty member.

In your own work, I encourage you to look at end-of-course survey feedback if you have that available. If you don’t, get those responses and if the institution you work at does survey students, perhaps there’s someone you can ask, maybe an assessment department or a data department that can share it with you. Your end-of-course survey feedback is going to give you a lot of insight into the way students engage and also what they loved and what they learned from, and what they didn’t love and didn’t learn from in your class.

All of these different pieces of data, the logins, the performance on assessments, just the observations in the discussion space, and the way they use terminology, and also your end-of-course surveys, all of these are data points for you as a faculty member, to help you refine your teaching and understand your students even better and connect to them better.

And lastly, I want to just encourage you to add a few metacognitive questions throughout your course that help you gather even more insights from your students. One that I really like to use is just a question of “how does this apply to your life or work right now? How might it apply to your life or work in the future?” It’s a fairly generic question but it can yield a lot of insight where students can find ways to connect with their learning right now with what they’re doing today or will do in the future. That can really help students engage more fully more deeply in the content and find connections to what they want to do or are doing.

Perhaps you have some ideas about ways to enhance student engagement, ways you can look at metrics to see what it is, or ways that you might measure it. I’d love to hear from you. Stop by my website, BethanieHansen.com/request, and let us know what’s working for you, what you’ve tried, what we should add to this list of student engagement information. And I hope that you’ll try something new in terms of looking around and seeing what students are doing, and how they’re interacting. Maybe a new space you haven’t explored like a piece of data, or revisit those end-of-course surveys. Thank you for considering student engagement with me today here on the Online Teaching Lounge. I wish you all the best in your online teaching this coming week.

This is Dr. Bethanie Hansen, your host for the Online Teaching Lounge podcast. To share comments and requests for future episodes, please visit bethaniehansen.com/request. Best wishes this coming week in your online teaching journey.

#37: Own Your Impact in Online Education

#37: Own Your Impact in Online Education

This content was first published at APUEdge.Com.

Sometimes faculty members feel like they play a very small part in the overall operation and success of the university. In this episode, Dr. Bethanie Hansen encourages online educators to step back and look at the big picture to see how their contribution is actually really significant and important. She encourages online educator to better understand the inner workings of the university, including all the various departments that are also making small but meaningful contributions to student and faculty success. Also learn why its so important to understand course data to evaluate your teaching strategy, assess your relationship with students, and help you identify areas for improvement.

Subscribe to Online Teaching Lounge

Apple Podcasts | Spotify | Google Podcasts | Stitcher

Read the Transcript:

Dr. Bethanie Hansen: This podcast is for educators, academics and parents who know that online teaching can be challenging, but it can also be rewarding, engaging and fun. Welcome to the Online Teaching Lounge. I’m your host, Dr. Bethanie Hansen, and I’ll be your guide for online teaching tips, topics and strategies. Walk with me into the Online Teaching Lounge.

Welcome to the podcast today. Thank you for joining me for this episode of the Online Teaching Lounge. I’m very excited to share with you this topic. We’re going to talk about your impact in online education.

Just to give you a little backstory, I once was a part-time faculty member teaching online, then I became a full-time faculty member teaching online. I also have a background in K-12 education of 20 years, and then I became a faculty director; I’ve been doing this role for the past six years.

When I became a faculty director, I saw things very differently. At first, I really did not know all of the inner workings of the university or the various impacts of my role in my teaching.

I want to share all this with you today for two reasons. First, when you know your impact, you can control the outcome a lot more and you can focus your energy to have a better impact.

And second, when you understand the impact of a lot of different smaller parts, you can also understand how critically important you are. It brings meaning to your teaching and it gives you a lot more context to enjoy your role. So let’s jump in.

Understand the Big Picture of How Your Institution Works

The first thing is about the big picture. If you’re in public education or private education K-12, I’m not going to be very specific here for your role in your institution. I’m going to outline the higher education landscape idea, the big picture in a university operation setting. I hope you’ll liken this to your own situation so that it can benefit you most.

This first idea is the university has a lot of different departments. For example, there’s a registrar’s office and also a huge group of folks that are dedicated to enrollment services. There’s a student services department, some of this has to do with academic support. There’s a booklist team, a librarian team. There are also all kinds of student clubs and organizations. There is a career services department and in the career services realm, students are looking for how to take their degree further, what they can do next and how to get a job in the field that they just graduated in.

There’s an appeals department, there’s a conduct department that handles student behaviors that might be inappropriate or escalating. There’s also a plagiarism and originality group. It might be an entire department, or it might be within another department.

There’s a classroom support group. This would be your tech folks who are really skilled at helping you in that learning management system. Beyond that, they have incredible gifts for creating things. They might help you find multimedia or create some kind of interactive role play activity, storyboard, decision matrix where students can have choice and engage in the content in a formative way.

There is a center for teaching and learning, some kind of group that’s going to give resources and increased professional development opportunities as well as skills that you can gain over time.

There are a whole host of other faculty. Many of these people have immense experience teaching or in professional fields, or both. You can reach out to them. Lean on them. Learn from them. They all bring their own unique set of offerings to the table. Each faculty member comes with a rich set of skills that you can also connect to.

And then, of course, there’s the bigger community. And the community might be your department, your school, a college, the entire university, and so forth. All of these departments have their own roles. And on the day-to-day side of things, people who work in every single department may feel that their jobs are small. Keep in mind that by small means, huge things come about.

Each person contributes a small part to the bigger picture of successful university operations. A lot of the things that people in these other departments do really support you in your online teaching and your role as a faculty member. For example, if you’re struggling with your learning management system, you can very quickly reach out to your classroom support team for help within a short timeline.

You can also build your goals for growth, your skills and all these other things that will help you be even more powerful in that learning management system in the future. You can connect to the center for teaching and learning for those kinds of skills. You can connect to other faculty members and the bigger community.

Why am I telling you about this big picture? My own experience was that as an online faculty member, I did not engage with very many of these departments. Occasionally, I might get an email from one or hear about something, but I did not really understand the inner workings of the teams. When I became a faculty director, very quickly, I was able to meet a lot of people on all of these teams. And I realized the obvious, we are all in this together. They were supporting the students; I was supporting the students.

When we see everyone else as members of our own team, we can reach out much more quickly when we need help. We can connect. We can get support. Things become a lot easier. Just think about the chaplain department.

Our university has a chaplain department. When we have a student who is struggling, maybe they are having a depression experience, maybe it’s even more extreme and they have expressed extreme distress. Maybe there are some issues with post-traumatic stress disorder. Whatever the situation, the chaplain department is one of our first lines of communication. The team that we connect with in the chaplain’s office is incredibly supportive. They offer resources, ideas. They give us a lot of support and they can make suggestions that will help us engage in our jobs a lot better as a faculty member.

Just as each member of these departments that I’ve mentioned makes our jobs a lot easier and they contribute to the success of the university as a whole, what I do has an impact as well. What you do has an impact. As a faculty member, it’s really important to know how we impact these other departments.

For example, when we are really encouraging, supportive and helpful with a student, when we share the resources like career services as they’re ending their degree program, or even in the middle, we support the career services and the student services departments by directing students the right way.

We send them to the people that can help them most who have all the information to connect them to career support. We further the educational goals of our students. Again, I mentioned as a faculty member, I was not always aware of all of the different departments and services and how they work together.

Once I became a faculty director, I realized I could serve students a lot better as a faculty member teaching in the classroom if I help them connect to different services and different departments when needed. But also, if I reached out as the faculty member to connect.

For example, when I notice a student in distress, or a student who has disclosed to me they have a disability and they really do need accommodations, but they haven’t asked for them, I can suggest to the student that they reach out to the chaplain office or the DSA, disability services department. I can also connect to those departments for tips and strategies and ideas. And I can also reach out to the center for teaching and learning for additional skills. There are so many ways these departments support me as a faculty member, and they can support you too.

What services exist in your institution? What can you do to connect with these different departments? And how can you learn your impact on these departments, on the people who work there? How does this broaden your awareness to think about your institution having so many different people there to support you? I hope you’ll think on that and consider how what you do every day is so connected to the bigger picture, the mission of your institution and the direction everyone’s going in this educational journey. As you do that, you’re going to be able to think about how the small things really add up to a big thing.

How to Broaden Your Perspective

The second area I want to talk about is our own class. And I’m just talking about an individual section that we are teaching. Chances are, you’re teaching more than one class at a time. Let’s just think about one class.

To broaden your perspective in this area, I would like to talk about the past, present and future focus. When we’re focused on the present, we are thinking about the lesson to be taught, the topics we’re investigating. We’re thinking about how an assignment fits into the bigger picture. And we’re focused on the day-to-day checking in of our students, ensuring that what they’re doing is on par for an academic in this subject.

When we’re focused on the present, many times it makes our job easier to do because we can see just this small piece. And of course, as I’ve mentioned, by small things, large things come about. We can help promote our students’ understanding in the entire class, just from each small thing along the way. The bigger picture has us thinking about the past and the future as well.

The past would be: what courses did these students take before my class that got them here? What is their prior learning? What is their life experience? Thinking about the past in our course gives us a huge amount of perspective. What do we need to add? What kinds of concepts do we need to include? How can we stair-step them from where they were to where they need to be?

The future focus is also important. Thinking about the bigger objectives in your course, the learning objectives. Basically the outcomes. What should they know and be able to do when they leave this class? This is the bigger picture of future focus.

Every small thing within your class ties into those bigger things. As a faculty member, when you connect those things for your students as you’re writing your announcements, as you’re teaching your class, you help your students to understand the big picture as well.

Not all of us make those connections, of course. Not all of us look at the class and think, “Man, I’m so excited that I’m learning this because it’s going to help me understand this big concept.” In fact, most students don’t think that way.

Part of our job as faculty members is to tie the small things that they’re doing into that bigger picture. Why are we doing this? It’s going to help you with X, Y, Z. It’s going to give you skills, knowledge. It’s going to prepare you for this career adventure. It’s going to prepare you for the next class you’re going to take. It’s going to apply in your life. We can also turn that around and ask students, how do you see this small piece of our course tying into this bigger goal? How does it work for you in your professional goals? Asking students these questions helps you do your job better because when they make the connections, they learn more. It’s amazing to see those connections happen throughout a course.

Why It’s So Important to Understand Course Data

Let’s also think about past, present and future in terms of data. When I became a faculty director and I was no longer just teaching courses all the time, but I was also supervising faculty, coaching faculty, onboarding faculty, and all of those things that go with that role, one of the things I learned about was the data.

There is a lot of data in an online course. For example, we might have an average grade report. As a faculty team member, I can do this on my own. I can look at the final grades of all of my students. I can see, did all of them get A’s? If that happens, chances are I’m not really critically evaluating because I’m not really sure all of my students would just ace the class or naturally get A’s.

And while I’m not suggesting that we deflate grades in any way, the final course grades can give us a lot of information. We can learn about our own grading process. We can also learn, is the rigor of the class too low? Have we not asked enough of our students in learning this subject? What can we do to really prepare them in this intellectual area, in the career field and in the academic area? So final course grades are one piece of data that as a faculty member I can look at, and so can you.

A second one is this percentage thing, and it comes from the withdrawal, incomplete, and D and F grades. At our institution, it’s been called many different things. But the goal here is to look at those final percentages of how many students withdrew from your course during the first week? How many had to drop it somewhere after the first week? And how many just stopped engaging and disappeared?

Occasionally when you’re teaching an online class, that happens. If you look for trends in your own teaching, it yields a lot of data. This data is just feedback. It’s not a personal judgment of you. It might give you great feedback about your teaching approach, your teaching strategies, your relationships with students, and so forth.

Think about the way the drops and failures in your courses layout and start looking for indicators leading up to that. This will help you to always improve your teaching and get more connected to what your students really need. Another piece of data is student appeals and complaints. If there are student appeals and complaints happening often, chances are communication is low. Often, we can change or improve the communication we have with our students to clarify things right up front.

Most complaints and appeals that I have seen as a faculty director came about because the instructor simply did not communicate clearly. A lot of times, students just glossed over something and missed a detail, or they questioned. Could they resubmit or revise because they really did learn something and wanted to fix an assignment? And the instructor said, “No.”

Decide upfront, will you let your students revise things and resubmit? There’s a whole department of people who get these complaints and appeals. And as an instructor, we don’t always see that. Think about the times you may have heard about a complaint a student has had. And also consider, have you ever had a student appeal a final course grade? If you get information like this, again, it’s data for you. It’s very helpful. It helps us to consider our impact as educators.

Are we communicating well? Do we have clear, consistent expectations? And do we maintain those with people over time? But it also helps us to look at that survey data over time. We can learn about our impact on students, our effectiveness in teaching the subject matter aside from the actual assessments that students do. We can also learn the trends. If we have areas to improve and we’re working on it, we can see whether or not we’re being successful or having an impact based on what students tell us.

There is also the informal feedback students give us by way of comments, emails and notes. These are worth collecting over time. As an educator throughout your career, it is incredibly helpful to reflect on the comments your students give you. These can help you in validating what you’re doing, know when to change and also understand your impact even more.

As you think about your impact, consider all of the different ways that your impact spreads throughout the institution, your student group and over time. This will strengthen our teaching to consider the impact in how the small things we do all the time in the classroom really do lead to these bigger picture ideas.

The goal is to change our perspective by stepping back a little bit, seeing the trends in our own teaching, seeing the bigger departments in our institution, seeing the impact of our efforts on students’ completion of the course, on their persistence getting through the class and their degree program, and of course, on whether or not they actually appear to know the content in the subject matter itself.

Think about all those departments at your institution and how they can support you, and how what you do every day supports them. And also think about the past, present and future focus of your teaching. By doing these things, we’re all going to have a better impact in our online educational roles. We can connect better with what we’re doing every day, and we can gain meaning and purpose in our work.

I wish you all the best this coming week in your online teaching, and I hope this data that you may find will serve you well. Thanks for listening.

This is Dr. Bethanie Hansen, your host for the Online Teaching Lounge podcast. To share comments and requests for future episodes, please visit bethaniehansen.com/request. Best wishes this coming week in your online teaching journey.

Wiley Educational Research

Wiley Educational Research

Today, I saw the Wiley Educational Research posted on Linkedin. You can view it here: https://edservices.wiley.com/ .

What does it mean for online education?

First, the researchers have been conducting this study repeatedly for several years. They report having surveyed over 15,000 fully online learners during the bast many years. I’ve used some of their reported statistics and trends in my own work, and particularly in my recent book Teaching Music Appreciation Online. I believe the data is useful and adds insight to online higher education.

Second, it’s unclear whether any survey respondents were impacted by COVID-19 yet, because the authors noted that data were collected in January and February 2020. Broad effects of the pandemic did not impact higher education until late March 2020 and onward. Regardless, the needs and preferences of online learners are well represented under normal circumstances. Next year’s report will definitely be worth a look!

Here are some of the key insights that returned this year:

  • Online learners prefer to take classes from institutions located close to home
  • Online learners typically prefer affordability and reputation when choosing where to enroll

And, new findings about what online learners really want include the following:

  • Pride in the institution and are willing to pay more tuition for an institution’s reputation
  • The fastest route to completing the program and speed throughout the process (including applying, accepting transfer credits, and goal achievement)
  • Specific programs (and they will go elsewhere to find it rather than opting for an on-campus option)
  • Career services
  • Options to learn via mobile devices

As we look to the future of online education, considering the preferences and needs of online students will help us continue to provide what they most want.