What Students aren't told by Pick-a-Prof

Short version: The professor evaluation web sites don't tell them that they need to do homework.



The data shown here are not in a 1-to-1 match because they show two different cohorts and only a portion of what will produce a grade for the latest group. The top row shows a modified version of the graphs provided to students and faculty by Pick-a-Prof based on data from the last three semesters. (The change I made is explained below the fold.) The bottom row shows an analysis of the results of the first exam this semester. (That means some of the students in the Physics 1 grade distribution at top left took the exam shown for Physics 2.) The beige part of each column shows students who don't do homework.

So what data are we looking at here?

First, the percentages in the Pick-a-Prof tables do not add up because they exclude students who drop the class from their statistics and the histogram, reporting that separately. However, since failure to pass is a failure in our accounting (and that of engineering schools that count a withdrawal as an attempt), and since many of the F grades in my grade data will drop sometime by midterm rather than fail, I think drops needed to be included. I reconstructed that number from their data, so when you see 24% withdrew, that means 24 dropped for every 100 that got a final grade.

Second, the data on homework are from a simple comparison of exam scores to homework scores. The break point at 80% for the homework reflects something I have observed for years, and the fact that it is pretty easy to get over 80% on the homework if you put in only a fraction of the 2 hours of study per day in class that is expected for college-level work. These homework problems are all assigned on-line and graded automatically, with multiple attempts allowed.

Third, the first exam in Physics 1 is always "too easy". I could make it a lot harder, but I find that an easy test (where all problems can be done correctly if you know how to do all of the homework with the book closed) sends a better wakeup call to the half of the class who forgot to purchase a clue about doing homework when they bought the textbook they don't read before class. In comparison, the first exam in Physics 2 includes the challenging concept of Gauss' Law and more vectors and calculus than they had to apply in the first semester. They catch up when we start doing circuits, if they do the homework.

Random thoughts on the data

For the first semester of physics, the story is quite simple. If you don't do the homework, you likely won't pass the first test, let alone the class. The bulk of the students in the "did not succeed" grey bars were ones who did not do the problems that are at the core of studying (and doing) physics. Now you might notice that some of the failures had pretty high homework scores. This is also not unusual. If you do all of your homework with your notes and book open, following examples from your friends, or cribbing a formula from a Google search or one of several homework "help" sites, you learn as little as when you simply copied someone else's paper back when we collected homework and paid a TA to grade it. Some quickly figure this out, while others just don't want to do the work. Or can't do it.

The second semester is more interesting. Those A grades reflect the fact that some students start to trade off effort in the harder calculus classes they are in for effort in this class, particularly when they get what is going on. Most of those are still in the 60 to 70% territory, and many of them do the problems later (when studying for the exam) but don't get credit for it in the homework score. There are, however, a few who did not get the memo from last semester, including a few who wandered into my class from somewhere else. (The standards at Wannabe Flagship down the road or wherever they came from did not prepare them to study.) They confused doing well because they did the homework with doing well, and failed when they did not do the homework.

There is also another effect at work: Once you get everyone doing the homework, the overall proficiency of the class improves. This changes the curve, eliminating B grades (as you see in the Physics 1 exam distribution) unless you replace straightforward problems with ones that require more critical thinking. Indeed, this allows you to put some "thinking" problems on the exam, to fill the time they don't need when demonstrating proficiency. Of course, this really burns the kids who have not gotten proficient from doing the homework or thought about the conceptual questions (also in the homework) that lie behind the ones requiring more critical thinking. The first exam in Physics 2 had a couple of those questions, so 90% proficient on the homework would become a low C or D if you missed both of them.

Finally, a comment on the Learning College premise mentioned in Dean Dad's Live Blog, where he wrote one of the folks there mentioned that at his college, faculty annual performance reviews (!) include data on how many students in a given professor’s class achieved the ‘learning outcomes’ they were supposed to. If the students don’t succeed at an appropriate level, the presumption goes, then the professor has some serious work to do.

The data on homework in my classes show the flaw in that reasoning. Most of the students who did not achieve the Learning Outcomes they were supposed to if they wanted the passing grade that would get them into engineering school, have only themselves to blame. Two of the students in the F group in Physics 1 actually did not do any homework at all! Indeed, you can just about see where the 1/3 who failed in past semesters came from: the beige bar group at the bottom. The Physics 2 group has been whipped into shape. Everyone who is left in the class is passing, while a couple of the "no homework" group with an F on the first test have dropped, pretty much what you see in past data for final grades.