Failing Grade on Significant Figures

This analysis is second hand (I have not read the journal article); it is based entirely on the article in today's Inside Higher Ed titled "Failing Grade on Alcohol".

According to the press release, the study came up with an allegedly shocking result

Using figures from government databases and national surveys on alcohol use, researchers at the National Institute on Alcohol Abuse and Alcoholism (NIAAA) found that drinking-related accidental deaths among 18- to 24-year-old students have been creeping upward -- from 1,440 in 1998 to 1,825 in 2005.

that was published in the "Journal of Studies on Alcohol and Drugs".

What is wrong with this? Everything, if IHE described the study accurately.

The IHE reporter says the authors did the following:

Researchers from the National Institute on Alcohol Abuse and Alcoholism multiplied the number of 18- to 24-year-olds in the United States, as reported by the Census Bureau, by the estimated percentage of deaths among 18- to 24-year-olds that were alcohol-related, as provided by 331 medical examiner studies. That number was multiplied by 30 percent, since three-tenths of 18- to 24-year-olds are in college.

Do you see the mistakes?

The biggest error is that "three tenths" has only one significant figure, although it might have had two before they wrote that sentence. That means their values should be stated as 1400 and 1800 (appropriate in either case because many people think that a leading 1 does not convey a full significant figure). Their choice conveys a hugely false sense of precision, as if they KNOW it was 1825 students, rather than 1824 or 1826, that died - particularly when it is quite likely that they don't know whether it was 1700 or 1900 that died as a result of alcohol use in that year.

The important point is that they don't know whether the people who died were actually students.

But there are several other errors inherent in this approach.

Are there equal percentages of men and women in college within this age group? The data I showed the other day suggest they aren't. Are men and women equally likely to show up in those morality statistics? Unlikely, given what I see in our newspaper.

There is also an unstated uncertainty in using a small sample of coroner data. (How many counties have a college in their coroner district?) What was the standard deviation in the mean value they used? Could it be that they are comparing 1400 +/- 200 with 1800 +/- 200 to get an increase of 400 +/- 400? Or worse?

Finally, all of this might be dwarfed by the use of a highly subjective term like "alcohol related". That box can get checked on a traffic incident report if one of the drivers reports having had a drink recently, regardless of whether there is a legal finding of intoxication or even fault for the crash. That means a totally sober student could have been counted if s/he was killed in a car accident with someone who had had a few drinks with dinner.

That doesn't even count concerns such as those of one person quoted in the article, who pointed out that many college students live within walking distance (or a short drive) of the parties they go to, in contrast to others in the same age group.

At least the abstract of the article said the "aim of this study was to estimate" [emphasis added] the mortality from alcohol use in the 18-24 age group, but that didn't stop them from giving a 4 sig fig value for a 1 sig fig result. This is particularly pathetic when the lead author believes "that it would have been better to have data on every injury death, but maintains that from the information he used, the results were a conservative estimate of the number of deaths" according to the author of the IHE article.

That means he knows that both the 1400 and the 1800 values were very uncertain, and that the true value of the first value might even be bigger than the second. This doesn't stop him from quoting "exact" numbers and omitting any estimate of the uncertainty in the calculated values. Surely, having risen to the position of Division Director at a national institute, he knows how to use a stat package to estimate the uncertainty? Oh, right. I almost forgot about this true story. Maybe there is a chance that he used the raw numbers from his calculation without a hint of a clue of how to estimate the uncertainty in the number of deaths from the uncertainties in the coroner data and in the fraction attending college.

Read Entire Article......

Lab Writing

This is effectively the second part of an article on the objectives of lab classes, where I had limited the discussion to everything except lab reports. I also discussed some of this in the past, stimulated by two articles decrying empty thesis statements and overly effusive language (using verbiage to replace thought) in papers for a upper division or grad-level classes in ed (history of education) and english (Victorian literature). Their complaints are familiar to us in the sciences; I regularly learn new things from folks on that other side of the blog campus and sometimes on my own. [One interesting side result of what I have picked up in academic blogs is the realization that I should talk to people who teach composition or in the social sciences or humanities about these sorts of teaching issues.]

As I see it, the written parts of lab reports address two learning objectives: technical writing, where they have to learn to use numerical results (with uncertainties and units) in complete sentences and clearly define quantities in english rather than with symbols, and critical thinking, where they have to learn to draw conclusions based on quantitative results that a vague fuzziness due to experimental uncertainties as well as separate important from unimportant.

Let's look at these two issues, being careful to include ease of effective grading (assessment is the new magic word) in the design of the task we set our students ... and our TAs(who likely don't have English as a first language).

But first, one word on philosophy: The majority of my students are going on to become engineers, so I am looking more toward that kind of corporate environment (with its memos and reports) than academic research. I hope I can get The Thomas to provide his critique of these thoughts from where he sits in Corporate America, either in a long comment here or in his own blog.

Technical Writing.

I think of this as the requirement that they produce work that looks professional (using correct symbols and notation, such as SI prefixes or superscripts for powers of ten, and grouping the value and its uncertainty inside parentheses) and that communicates their answer without ambiguity. This means using technical terms instead of a catch phrase, and not using a symbol (is it V or v, is it velocity or voltage or volume?) that has not been defined by the writer or in the question.

This requirement is unambiguous and applies everywhere, but I don't mark it off every time I see it made. [I do know profs who take off 0.1 point for every instance of even a minor error, but I don't have that much time to devote to grading.] Instead, I focus attention on specific sections of the report for specific things, such as making sure values are reported correctly (with units) in the calculation section and in the conclusions sections. Those instances are highlighted on my version of the report. [Because of the particular, rather efficient, system I use for grading, this might be one of only two things I am looking for on a particular page of the report.] Nonetheless, I always keep my eye open for an oversight somewhere else in an otherwise good paper. I want everyone to be paranoid about errors of omission such as units or sig figs.

There is also an expectation that answers to questions and some lab exam problems be stated as complete sentences.


Critical Thinking.

This is the real challenge. It takes time and energy to do this right, so I often look at this part of the lab report first or devote a separate day to it, so I can do it while I am fresh. It also requires that the task for the student be well defined, so I try to focus it into just a few areas of the report.

One of these areas is actually easy to grade: the post-lab questions. These are actually a good place to address "conclusion" questions about the implications of the lab results, particularly the ones that require the use of uncertainties to address the significance of an "error" between what is expected and what is found. The key requirement is that they get the right answer and given an appropriate explanation (such as using the size of the standard deviation to judge the precision of their result). A specific question ensures that the student has to address the topic and also means I don't have to go hunting for it somewhere in two pages of conclusions.

I also require that they address specific issues in specific places in the conclusions part of the report. One section has to identify and summarize the most important results of the experiment. It has to be a single paragraph of modest length, like the abstract for a research paper or a memo in the real world. They are marked off for not putting the right things in there or for technical writing errors noted above. I sometimes require that this be a cover memo, while other times I have them call it an abstract. (That helps catch plagiarists.)

The other section must address a particular aspect of a typical "conclusion", such as what followup experiment could be done (and describing it in detail), identifying a real-world situation where this effect is important, explaining what might have caused inaccuracies in their results and how to avoid them in the future, or identifying specific procedures they used that might have contributed to the precision of their results. I vary this question from semester to semester to make it harder on lazy kids who try to use a file from a student in last year's lab.

The real trick is keeping track of students who say the same thing (e.g. didn't read the manual before class to be sure they knew what they were doing) rather than learning from their mistakes and improving their skills in the lab! If they are thinking, these answers should be more than a throwaway line.


Grading Rubrics.

These are essential, and it is essential that they be designed so students who do the experiment and complete all of the calculations with reasonable accuracy and attempt to answer all of the post-lab questions will get a minimum grade of C. Not only is that passing for our college, but it is all that the nearby engineering colleges need to see. That limits somewhat the deductions for egregious errors, but still leaves lots of room to encourage improvement.

It also has to work for me and the adjuncts who work under my supervision. Thirty or so lab reports a week can be a big load for me, particularly when they come due on a week when I also have to grade 250 pages (or more) of exam solutions. It might be an even bigger load for my adjuncts, who have other jobs as well. This requires focus on specific spot checks and clearly defined tasks, as noted above.

I long ago quit cutting them much slack on the initial reports. A warning without a deduction of points has no effect on future behavior. However, I do cut the penalty points for "critical thinking" types of errors to about half of the norm. We go over tech writing skills from day 1, so they are expected to do that part correctly. We also drop the lowest report, so that encourages improvement (but won't help if they skip one of the labs).

Appropos a point Matt made in his blog recently (see below for the link), the max deduction for omission of the separate "conclusion" writing assignments is 20%, although the deductions for flawed contributions are usually around 10% of the total grade. However, other items that require critical thinking make up at least another 20%, if not 30%, of the total. I also include questions of this type (interpret a certain result or write a summary of certain results) on the lab exams.


Other voices.

Matt's recent article about improving lab report conclusions, from the perspective of a graduate student at an R1 university, offers an interesting suggestion: collect some good conclusion sections from physics research papers. (I suppose I could use some of my own!) I'd also like to have a similar collection from industry, since those examples are generally unknown within the physics research or teaching community. As noted below, I would put more emphasis on showing them a good abstract rather than some good conclusions.

Chad Orzel provided his thoughts on the writing style used in lab reports last year, from the perspective of selective liberal arts college with a large physics program. I agree 100% on the evils of the passive voice, but I know where it comes from: our chemistry department. [Comment #15 makes that same observation.] They insist on it. In contrast, I insist on simple declarative sentences that state (in the correct past tense) that a specific quantity was measured, giving a specific result. [Comment #10 gives a nice example of good an bad ways of saying the same thing.] Of equal interest is how a number of comments came from composition teachers who regularly fight the same strange view of what makes good academic writing in their classes. Maybe the problems start in high school!

I also like Chad's emphasis on framing. In effect, that is what I do by trying making them start out by stating the most important result, whether it is a measured value or the verification that energy was conserved to within 10%. A good abstract tells you the important result(s) in a "Just the facts, ma'am" style, much like the headline / sub-headline sequence in the NYTimes.

Read Entire Article......

Lab classes

I actually started writing this back in April of 2007 ... see bottom of article for references to the prompting articles and some more recent ones. That put it on my list of things to do looking back at my first year of blogging. Also, if you have not already done so, take a look at the previous article and reply to the poll question about recording lab data.

I think I have a pretty clear idea of the main objectives and how to achieve them in my calc-based physics lectures, but I will resist the temptation to outline them here. However, my feeling for the main objectives of the corresponding first year lab classes is another matter. OK, I think I know what the important ones are, but I can't identify one or two that should be emphasized above all others and it seems impossible to get the students to make all of them a priority. Is it because they have their own idea of what the lab goals are? Do those skills, subtle as they sometimes are, appear too mundane? Is it the challenge of critical thinking? Is it because they look at each lab as something like a science fair project back in elementary school? Or are my priorities wrong?

Side thought:
One way to find out what the students think is to simply ask them that question this fall. (Might even be good to ask them what they think the purpose of the lecture class is!) But right now I am using this question along with my own musings on the subject in the hope it will clarify my view of the lab.

The subject here is the first year physics lab (or chemistry lab), not one of the advanced labs taken by a physics, engineering, or chemistry major. However, I would love to get your short list of things that students don't know when they take a junior-level physics or engineering lab if you have experience teaching one of these. I learned one of those details from a lab TA at a nearby engineering college and have incorporated it into my assessment at several points.

OBJECTIVES:

Technical writing will be taken up in a separate article.

So which one or two of the following items do you think is most important, or is there something I left off of my list? Feel free to add any other thoughts you might have.

1. Making the measurements correctly.
I am always surprised how many students don't want to use all of the information available on a ruler yet blame the instrument for their own lack of care. (I'll bet real money that this is why they prefer digital devices despite the fact that all of the same digits are on the analog device.) I think one of the most important objectives is to understand the importance of getting the most out of whatever tools are available.

2. Recording the measurements.
Part of this is the simple skill of writing down the value (in ink) that was measured, but the other part is estimating the uncertainty and using the uncertainty and significant figures to tell someone else how carefully you think the measurement was done. High on the list has to be the basic step of reporting a measured value in a way that clearly indicates its reliability (and its units, of course). Is this something you specifically test on lab exams?

3. Following directions.
Some labs have a rather detailed procedure that needs to be executed step by step if the equipment is to work properly. I'd say that half of the procedural questions I get in the lab are answered in the lab manual, indicating that it has not been read and/or read with comprehension.

4. Design your own experiment.
Our labs are not as "cook book" as those in first semester chemistry, but they leave no room for student creativity -- primarily to avoid major student blunders that would make the work impossible to grade and keep the student from learning anything about the phenomenon being studied. Do any of you do this, or do it as part of a practical lab exam (e.g. design a set of circuit experiments to figure out what is inside this black box)?

5. Critical thinking with uncertainties.
A related, and exceptionally challenging problem, is getting across the idea of what criteria must be met for an experiment to "agree" with a prediction or a known value, or to establish that some effect (e.g friction) is or is not present at a significant level. I think this last one is particularly difficult because so many first-semester lab experiments don't work all that well in the hands of amateurs, so I push it mainly in the second semester.

6. Learn from mistakes.
Probably the most irritating thing to me are the students who make the same kinds of mistakes all year, often as a result of failing to prepare for lab or simply being in a hurry to get out and "get away with" doing the least amount of work possible. Do you ramp up the grading penalties as the semester goes on?

7. Hands on experience with the phenomenon.
Sometimes we want them to simply experience the phenomenon, as a way of reinforcing or illustrating what goes on in lecture. They really like these labs when they are tightly linked to lecture in a "studio physics" style of learning, much more so than a corresponding demonstration in lecture. Should there be more emphasis on this, or would that hurt their preparation for advanced labs in engineering?

8. Respect the limitations of digital meters.
Students (including those in my generation, who did not grow up with calculators) have a near-religious belief in the perfection of any number generated by a digital device. They do not know their calculators use BCD arithmetic with guard digits to hide their inherent flaws from view in trivial calculations. They do not know that calculations done by their computer running a program written in C++ (or any other language) use a different system with different flaws. Thus they think that every digit displayed by a multimeter is 100% reliable, like the mark on a ruler. They don't even suspect that those numbers are actually raw data from an analog physics experiment, so they are stunned when they don't round to the value displayed when you select a different range on the meter. This is one thing that I plan to explore in more detail during our initial lab in physics 2 this year.


***


Background articles from out in the blog world, mostly circa the spring of 2007 ...

A few of the problems I suspect originate with student perceptions of our labs as an extension of ones done in elementary school or high school could benefit from the kinds of reforms Chad Orzel proposed for the physics-major curriculum, which were centered on the introductory physics class I teach. I also like Chad's succinct rant and more detailed critique of remarks from some silly philosopher who thought labs are not real classes. Not only do I agree with some commenters that labs are more work than "real courses", I insist on teaching a certain lab to be sure students are forced learn certain skills. And I am a theorist, not an experimentalist, thus proving that a theorist is not a philosopher.

I suppose the reason that the Playground Philosopher doesn't like labs is tied into preferring thought to action. But would you prefer to drive over a bridge designed by an engineer who used pure thought to come up with the strength of the steel being used, or one who used a measured value and had done enough measurements to have a gut understanding that there are uncertainties in that measure value, no matter what the data book says?

Other critiques of that philosophical position were written by Dr. Stemwedel Labs and learning science and in Chad Orzel' third article about Labs and Learning. A related thought also appears in Chad's article listing "to plan and carry out a simple experiment" as number 1 on the list of skills every physicist should have.

There were articles by Female Science Professor on using basic methods rather than automation in the lab as well as a related one by Professor Orzel on whether to automate data taking. I find that automation (e.g. some of the PASCO sensor systems) hides what has gone on in the measurement but makes it easy to do a many-part experiment in a practical time frame.

Chad Orzel wrote about reforming intro labs in a comment responding to one of the earlier parts of a 5 series on altering the approach in the intro lab.

I'll also link to a March 2007 by Female Science Professor about 100% men in labs. It's not specifically germane to this particular topic, but I have noticed that it is really important to watch who gets paired up with a female student in the lab. There are way too many guys who lack any clue about what they are doing but possess massive amounts of bluff that convinces their lab partner they know what they are doing. Young men are more immune to this than young women, although anyone who lacks confidence is likely to fall for it. Ideally, I put the clueless guys together and let them bluff each other.

Link forward added:
I wrote more about the subject in this last paragraph in an article published in July 2008.

Read Entire Article......

Poll: Recording Lab Data

A soon-to-appear article will look at learning objectives in the physics lab, but I wanted to single out one topic for its own poll:

Do your students use a bound lab notebook for all data or do you collect individual lab reports that contain a raw data sheet?

Do you require that students use (or, if your are a student, were you required to use) ink when recording lab data, and how do you punish violators?

We use a data sheet that is handed out in class or taken from the (publisher's) lab manual, and that data sheet is turned in as part of a lab report.

Our syllabus requires that students use ink, but it seems to be rarely enforced by our adjuncts and the students get annoyed when I enforce it. The really sloppy ones hate showing that they used the wrong end of the ruler, and the perfectionists hate having anything mar the neat organization of their papers. Some go so far as to use erasable ink.

I keep trying different tactics to enforce this and will try a new one this fall, but I seek any good ideas you might have (either as a student or as a teacher).

Read Entire Article......