Friday, August 12, 2011

The Importance of Skepticism

What do you do when an outside lab has numbers which are completely at odd with in-house analysis? We use a certain unnamed enological lab which is accredited by national and international organizations alike. These serve to guarantee the analysis method and results to national watchdogs like the TTB and to smooth over the international sale of wine. For example, the US and Canada agree on an analysis method for something such as residual copper level, and trade is smoothed because only the export country has to perform the analysis. There is a lot of money and trust riding on the analysis of these labs; their results directly influence both the sale and treatment of wine. So how do we keep them honest?

Our problem relates to free SO2 testing. We perform the tried and true aeration-oxidation method, and have been doing so for some time. Now I am perfectly aware that our results are somewhat relative no matter what, but there is simply no way we should be more than 25% off of what an outside lab gets. So when our export analysis showed 12ppm SO2, and our in-house result was 22ppm, flags were raised. However, despite all of the credentials that our outside lab has I immediately knew they were in error.

I didn't feel this way because I had excess pride or a strong gut feeling, I felt this way because we make a significant effort in the lab to track our own precision and accuracy (precision is a measure of repeatability, accuracy is a measure of how close we are to the real value). At least once a week each tech in our lab runs 4 FSO2s on our standard twice. We keep all of this data and use it to assess the P&A of the analyst and setup alike. For example, we use 4 round bottom flasks and each is carefully tracked to make sure there is not any determinate trends between flasks. This data is as valuable as any test performed on an unknown. All of the work we do to test our known standards is what gives legitimacy to the work on our unknown samples. If you cannot consistently test a known standard to an acceptable degree of error, there is no way you can confidently say your have the correct value for an unknown sample. This is the underlying truth in a lab, and something that simply can't be ignored by the manager.

It was with all of this evidence that we had no problem calling our outside lab and telling them quite simply that they were wrong. To prove it, we sent a sample bottle from the same case to the original testing lab, a secondary outside lab, and repeated the analysis for a third time in our own lab. Our lab and the secondary lab were within 2 ppm of each other, in my experience this is basically an identical result. The original outside lab, after we confronted them with their results, agreed to retest for free and upon doing so came back with a result of 20ppm. They use flow-injection analysis for FSO2 tests, and I'd magine that they do hundreds of tests a day. The focus is quantity, not quality, and unfortunately the burden of proof falls on the lab requesting the analysis, not the "credible"outside lab doing the test.

Science is skepticism, and always will be. This experience highlighted a glaring hole in my thinking as a scientist. We have always operated under the assumption that our outside lab was infallible and not run by humans. But this is simply not true, while they have the ability to analyze and report numbers, they don't know how those numbers match up with past analysis. If our lab gets an unexpectedly low result, they always retest and confirm before reporting. They have the data in the lab to match with their current result. And I'm not implying that we go fishing for the numbers we want, but if a 10ppm sulfur hit in the cellar leads to a lower result than the one we got before the hit, we would call that result suspicious and retest. Objectivity is one of the great features of doing analysis at the winery, and this incident really drove that home. Oh and for the rest of the wineries who have been getting low results from this same lab, you're welcome;) As my economist brother pointed out, we are all compl-E-mentary.

No comments:

Post a Comment