Accuracy vs Bias, or, Why You're Wrong about being Wrong

Recently I have seen multiple comments or posts in the social media regarding the accuracy or otherwise of temperature measurements. These have been largely directed at climate observations and are something along the lines of 'it's possible that temperature measurements over the last few decades are wrong and so observations of a warming Earth are suspect'.

  As temperature measurements of the atmosphere are something I spend the majority of my working days looking at, I felt that I might be able to point out a few flaws with this statement. Primarily, there is the gap which exists between the general public's understanding of the word 'accuracy' and the way in which it is understood by a scientist or statistician. The fact is that no instrument, ever, has given a truly precise measurement of anything. If you have somehow managed to measure the speed of light (or sound or whatever) to the same exact precision as you find in a textbook, then that is purely by chance. The next time you perform that measurement you will find that it is different. The amount it varies by depends on the quality of your instruments, how well you performed the measurements, the attention span of the person writing the results down and so on.

  The fact is that there is some inherent inaccuracy in every measurement, this might be as fine as the fuzziness of the position of an atomic or subatomic particle due to the Heisenberg Uncertainty Principle (so fundamental, it gets capitalised!) but is more likely due to more mundane things such as moving parts sticking when they get cold, sunlight shining on a thermometer, condensation on the inside of a humidity sensor, etc, etc.

  Most people are familiar with this sort of inaccuracy, the temperature reading in your car doesn't match the one on the weather report, the snowfall outside your house was three inches more than predicted and so on. Let's say that you're familiar with the idea that one thermometer might read several degrees differently to an identical one a metre away, then you see the plot below.

Taken from the New Scientist article 'Climate Myths:The Cooling After 1940 Shows CO2 Does Not Cause Warming'

  The plot (or something like it) is pretty familiar to most people by now and I see something like this on a pretty much daily basis. Perhaps because of over-familiarity I sometimes forget what the scale is on this plot, the entire difference between the points on the left (i.e. the average global temperature in 1850) and those on the right (2010) is only around a degree. Leaving aside what effects of this small increase actually are it may seem improbable that measuring increases of 0.1 degrees over a decade or so is even possible. This is particularly true when you may have two thermometers in your house which give two different readings at any given time of the day.

  It might be assumed that modern thermometers are incredibly accurate, or that satellite measurements are extremely precise. However, it is not the precision of measurements which gives climate scientists confidence in their observations. In fact, the precision of the measurements hasn't particularly improved over several decades (generally speaking of course, there have been vast improvements in individual experiments in thermometry, these tend not to make it to the weather stations throughout the world!). What does make for a good, consistent set of observations is large numbers of measurements. A single thermometer reading may be off by several degrees, thousands, millions or billions of thermometer readings will tend to converge on a single value though.

  This is due to the Central Limit Theorem, explored in great detail here, here and here amongst many other places, in which a large sample of measurements results in a normal (Gaussian) distribution of mean values. This is demonstrated nicely here with a little app to show you how the distribution of values in dice rolls stack up over 10's, 100's or 1000's of rolls. Basically, a regular Gaussian distribution looks like this

  The importance of the Central Limit Theorem is that, regardless of how accurate or otherwise your measurements are, when you stack them all up, you know how they should behave. If there were a real problem with your measurements, it would be easy to spot as they would no longer have a normal distribution. This brings me to my main point really, what differentiates having less than perfect accuracy from actually being wrong is bias. This is what people really mean when they say that climate change may exaggerated by 'wrong' measurements, not that the thermometers are inaccurate but that they are all reading too hot. If this were the case then the distribution of temperatures would look like this

 

  This would be rather easy to spot, as you might imagine and is actually one of the standard tests run on all data assimilated into the weather and climate models run at all major weather prediction centres worldwide. If the data don't fit in to the normal distribution template, they are discarded. On top of that, the proportions of data we expect to not fit into this template are well understood and if our data were to significantly exceed these proportions it would be noticed very quickly. So not only do we know what our data should look like but we also know how many we should expect to not look like our expectations, just through simple statistics.

  This is hopefully a reasonable explanation as to why we can have high confidence that global temperatures have risen by a relatively small amount, even though individual measurements aren't particularly capable of observing this rise. What is often not considered by critics of the NOAA, NASA, Berkeley or other temperature records is that temperature measurements are not simply the result of thermometer readings at weather stations around the world. There are temperature measurements in data sets arising from aircraft, satellites, weather balloons, ships, drifting as well as moored buoys as well as those provided by the public (e.g. the OpenRoad). Some of these data sets in themselves provides millions of measurements per day, summarising satellite observations here too vastly underplays the sheer amount of data coming from many, many instruments on many, many different satellites, each with independent measuring channels.

  What really lends confidence to observations of rising temperatures is that all of these data sets agree with each other. Of course, there are bumps and squiggles and discrepancies but these are all consistent with the known accuracies of the instruments. There are literally millions of cross-checks and quality control checks performed on these data every day.

  Of course, this won't stop people from levelling accusations of falsehood at the data, and nor should it. Data and science should always be open to scrutiny at every level. However, what many people fail to do is understand what they are actually looking at. Claims that weather stations have been poorly placed or that Paraguayan temperature measurements have been 'tampered with' are certainly worth examining, if only to improve future measurements. However, these examples represent tiny amounts of the current data, and thus understanding of the global weather system. In order to understand their real place in the field of climate science, and science generally, it is necessary to understand the true scope and place of basic science, performed by millions daily.

An Ode to Beagle 2

As I'm sure most people reading this will be aware, the ill-fated Beagle 2 mission to Mars is believed to have been located, just 5 km from its intended landing site. The Guardian has the full story but rather than repeat what many others have said I would rather present the poem I heard on the Shaun Keaveny show this morning on 6 Music.
  The poem is by Murray Lachlan Young who is a regular on Shaun's show and has some great poems and ditties on all sorts of subjects. I've transcribed the poem below (and hope this doesn't contravene any kinds of copyright or anything) so I take responsibility for any errors I've made in the words or meter of the poem. To hear the poem read by Murray himself (who has the kind of voice perfectly suited to reading poetry aloud) then go here. He has several other clips stored on the BBC site here and you can read lots more of his poetry and find more out about him here.

Murray Lachlan Young

Ah, Beagle 2, oh there you are
Sitting nice and quietly
Upon the cold red planet's skin
So British in your modesty

Your legend was receding fast
Your memory half-blown away
Just like our dreams

Twelve years ago as we assembled Christmas Day
To watch the Cool Brittania lander
Softly touch down Britain first with coloured dots to calibrate
Designed by Damian Hirst but

Christmas day it came and went, then months, then years
A decade on you sadly came to represent
Embarrassments of things gone wrong
A name upon the missing list made by those not up to the task

Dreamed up by some eccentric Brit whose reach outran his dizzy grasp
Whilst NASA champagne corks went pop you roamed with Curiosity
Your fate it seemed was destined to remain a Martian mystery

Great Britain, put aside its spacesuit
Packed its trunk and walked away until quite out of the blue
Upon one January day, a photograph of you, yes you

A tiny shiny Beagle dot, snapped from the NASA orbiter
Chillaxing on the very spot that Colin Pillinger intended all those moons ago.

So elegantly vindicating all we thought we ought to know
Yes boffin-built lightweight and groovy

Nano-budget to concur
A callsign raised and written by those lovely boys from Blur

But why we ask were you so silent?
Making things so bittersweet
Perhaps it was too beautiful for you to feel the need to speak?
But there you sit and there you are
Upon the skin of planet Mars and maybe
We may once again begin to reach out for the stars

The Blind Leading the Blind

If you listened to the recent podcast 'Serial' you probably heard 'This American Life' mentioned with every episode, it's a fantastic radioshow and podcast that never fails to provide new perspectives on everyday stories and you should give it a try, whether you're American or not. I caught an amazing episode of TAL earlier this week, actually a re-broadcast of a new radio show Invisibilia, hosted by Alix Spiegel of NPR and Lulu Miller who comes from another great podcast, Radiolab.
  The gist of the show was that the way we approach things, experiments, (dare I say life itself?) may affect what then happens. In the terms of the show, how our expectations of a particular result can bring about that exact result. While there's a danger here of veering into psuedo-science, I don't think there's anything superficially shocking here. The primary example from TAL was of an archetypal rat/maze experiment in which the rats had signs placed on their cages labelling them as 'smart' or 'dumb'. Lo and behold, when the rats were run through the mazes the 'smart' rats performed amazingly while the 'dumb' rats performed poorly. The twist here is that the rats were all just normal rats, any intellectual abilities they may have had were unknown and so identifying them as smart or otherwise almost certainly came down simply to the prejudices of the people involved in the experiment.
  I wrote that this wasn't particularly shocking to me, what I initially thought was simply that the experimenters were using their judgement in some way, allowing the 'smart' rats to achieve higher scores than the 'dumb' rats by subtle differences in the way that information was recorded. The truth is a little more depressing than that it turns out. What actually happens is that the experimenters are nicer to the smart rats, handling them more gently and so on. This actually has a large effect on the rats performance, (I'm guessing) largely due to stress levels and the extra attention given to the smart rats. Thus are the results explained, although the idea that people are meaner to dumb rats than smart rats confuses and upsets me.

  Taking this idea forward, the life of Daniel Kish is explored, or at least, his life as it relates to being blind and being an echolocator. There is a lot written about Daniel in various places so I don't really feel the need to repeat it in much detail, if you want a bit more than I'm offering here, listen to the podcast, watch this short video, read this interview or check out his wikipedia page.

  The headline grabbing bit about Daniel is that he rides a bike, occasionally during rush hour, while blind. Having lost both of his eyes before the age of two, he has little choice really apart from possibly, you know, not riding a bike at all. Although, if you were to hear Daniel speak you would realise that this is not a choice at all. Having learned to navigate his way around the world using echoes from clicks he creates with his mouth, Daniel creates mental representations of the world in his visual cortex. The visual cortex has previously been supposed to be all but silent for blind people, at least until people actually started to look at it. However, experiments using the magic fMRI machine show that blind people who use this kind of echolocation have precisely the same areas of the visual cortex light up when presented with various representations (salad bowl, moving salad bowl, wall, etc) as a sighted person does.
  The conclusion of Dr. Lore Thaler is that blind people who can echolocate have a 'vision' roughly equivalent to a normally sighted person's peripheral vision. Something akin to a person attempting to navigate whilst looking down at their hands (I'll stop short of comparing a blind person's navigational skills to those of someone who is texting, as the blind person is probably devoting their entire attention to the navigation, whereas the texter is not).

  What really struck home for me about this whole story is that Daniel Kish developed his skills and now lives a happier, more independent life simply because his parents (well, parent, you'll have to listen to the podcast really) refused to wrap him in cotton. His mother didn't give him his independence exactly, but she refused to crush it. When Daniel's school told her to curtail his clicking because it was 'socially inappropriate', she refused. When friends, family and concerned neighbours told her to stop him riding bikes, she bought him a new one for Christmas. I will now write that horrible phrase which will remove any credibility but as a parent the thought of my child losing his eyes fills me with dread and I can only imagine that the temptation to coddle him for the rest of his life was extremely strong. Fighting this temptation is the knowledge that allowing your child to discover the world in his or her own way is the only way that they will truly find their own way around it.

 Another amazing aspect of this whole story is that apparently blind children often develop this clicking or another form of echolocation as a means of navigation. What is awful is the thought that this ability might be taken away from them simply on the grounds that they are not behaving as we expect them to. It is Daniel's opinion (amongst others) that it is expectations like this which create helpless blind people, in no way is it the blindness itself. Despite his initial (and ongoing) reluctance, Daniel now runs the non-profit World Access for the Blind in order that he can help others develop the skills which he uses to navigate the world. I should point out that he finds nothing remarkable whatsoever in his ability to ride a bike through traffic. He detests being the 'blind guy on a bike', as if he is a sideshow act. If more people can take on board what he has to say then perhaps we could all learn to see things his way too.

Seals as Probes - Revisited

As per my last post, I was very happy that Chris Benjamin wrote up a considerably better researched version of my very first blog post for Science Friday last year regarding the use of seals in climate and weather monitoring/research. As I recently was able to attend a seminar on the updated results from the seal project I thought this would be a good opportunity to explore the subject a little further.

  Something that was perhaps missing from my original post is the importance of monitoring ocean conditions. Ocean observations are very important in terms of measuring their heat uptake. According to the IPCC Assessment Report 5 the top 700m of the oceans take up 93% of global warming heat content (see below).
 
Given this, you can perhaps see why it might be interesting (and important) to study the Antarctic circumpolar current (ACC). This is the most important current in the Southern Ocean and the only current that flows completely around the globe. Incidentally, it was discovered by Edmund Halley, the British astronomer (who says astronomers have no real-world use?). The ACC is equivalent in flow to all of the rivers in world combined (Chidichimo et al, 2014).

Measuring the temperature and salinity of the water in the ACC, in addition to other, less well-travelled water has become easier and more widespread with the ARGO float system. However, in order to take the same measurements beneath the variable sea ice which extends beyond the land it is necessary to recruit the services of seals, as covered by myself and Chris Benjamin.

The seal measurements allow forecasts to be made for the Antarctic area, this is particularly useful for military pilots who, under certain circumstances, must where survival 'dry suits' when flying. These suits are very uncomfortable, particularly in cramped cockpits and so knowing whether or not they are necessary is useful information for the pilots!

Mapping ocean currents, particularly in a predictive sense, can be immensely useful in a 'man overboard' search and rescue mission. The disappearance of flight MH370 last year is another example of the importance of this information.

The measurements made by the seals, or rather by the instruments attached to the seals, are made on the upward portion of the seals foraging dives. The data loggers consist of a pressure sensor (in addition to other sensors) which is on constantly, this notes when seals begin to ascend from their dive and alerts the other sensors to turn on and begin measurements. The record dive of an Elephant seal is over 2000 metres, over four times the average of 500 metres. It's not really understood how such deep dives are possible, although the seals have ribs made of cartilage which means that they can collapse their chests, the impact on the seals' blood chemistry is still very significant.

Dive profiles consist of seventeen points which are split into four messages, the transmission of which is attempted when the seal surfaces. A minimum of 160 seconds is needed to send these four messages via the Argos satellite tracking system, unfortunately the mean elephant seal surface period only lasts for 130 seconds, meaning that transmissions are often incomplete, resulting in poor positional information and/or incomplete profiles of the water temperature and salinity. An alternative to transmit the messages via the Iridium satellite constellation exists but depletes the data logger batteries at an increased rate.

  By running reanalysis experiments of past weather conditions the usefulness of the seal observations can be evaluated. It turns out that incorporating the temperature and salinity measurements tends to lead to an overestimation of the water salinity. This is difficult to verify without independent observations (of which there are none) but in the Kerguelen region where there is some overlap between ARGO floats and seal data it was found that the ARGO floats had the opposite bias - where the sea measured fresher than predicted by the ocean model.

  It is notable that the primary difference in instrumentation between the floats and the seal data loggers is that the former use capacitance to measure salinity while the latter use inductive conductivity sensors. The seal sensors are found to have a high bias (as seen) when the sensor is close to any surface (like a seal for example...). This effect can be accounted for by calibrating the sensor for each individual seal, unfortunately this requires a long data baseline of around a year. At which time the sensors are falling off the seals in any case!

  All this means that the salinity profiles obtained from the seals are partially useful for research purposes but can't really be used for meteorological forecasting.

  What is lucky is that even incorporating only the temperature profiles from the seals data loggers results in a significant improvement for temperature and salinity forecasts, as well as improvements in estimates of global temperatures, extending far beyond the regions sampled by the seals.

It has been noticed that the location and strength of ocean fronts are impacted in models by the seal-measured temperature. However, without good quality independent observations it is difficult to assess whether this is a beneficial change or not.