Podcast: Brain Food for General Counsel

We all commonly misunderstand what numbers are telling us. Learn from the experts how to spot statistical sleight of hand and make the best use of data, described in the episode as 'the new oil'.

Businesses are producing more data than ever, but not necessarily understanding it properly. Robert Cuffe, Head of Statistics, BBC and statistics professor Liberty Vittert have suggestions on how to make numbers your ally, not your enemy.

Transcript

This past year more than any other, numbers have been everywhere. Following the story of the spread of coronavirus, the lockdowns, the prevalence, the vaccine rollout and efficacy has felt at times like it required a crash course in statistics.

The R number, prevalence, vaccine efficacy, ICU admission trends - the story of the virus has been told in statistics, and these are the numbers that have governed not just policy making but also our behaviour. Getting us to change the way we live has depended on making us understand what these numbers mean and how they can help keep us alive.

Not to mention the examples the crisis has given us of the dangers of not fully questioning the numbers we are given. At the height of the crisis over availability of personal protective equipment, for example, the UK government admitted that it sometimes counted individual gloves, rather than pairs, as pieces of PPE, allowing it to claim higher numbers of PPE items distributed.

So we thought this was a good a time as any to try to deliver that statistics crash course. Not just because of the health crisis, but because lawyers in business might think of themselves are more words people than numbers people. But to fully engage in setting and executing an organisation's strategy you will need to hold your own in a room full of people eloquent in the language of numbers. It is vital that you don't let the numbers beat you. They're not as hard as you might think and challenging the assumptions underpinning the stats you're given can put you in the driving seat.

So we have lined up two experts in the understanding of statistics to help out. Robert Cuffe is the BBC's head of statistics and has rarely been off the screens and airwaves of Britain, helping us get to grips with what he has described as an avalanche of statistics related to coronavirus and the policymaking surrounding it.

Liberty Vittert is a statistician whose work through broadcasting and TedX talks has focused on helping us spot when someone is trying to use numbers to manipulate us.

Having spent some time talking to both I was struck by how powerfully they agreed on the first comforting, essential point, which is that understanding numbers in the news or at work is not fundamentally a question of maths, but one of logic and context. Robert, then Liberty, explain.

Robert Cuffe:

The biggest misunderstanding about numbers is that they are all algebra and compound interest and horrible things but it is kind of like parenting in that you think it is going to be these complicated conversations where you negotiate how to build a human their confidence and all these really questions when really the skills you absolutely have master are picking things up off the floor and not forgetting things when you go the park. It is the same with numbers, most of the skills that you are dealing with all the time high end mathematical concepts. It is a bit of confidence doing a bit of counting, maybe addition if you are really pushing it and that can get you most of the way and certainly a lot of the data journalists that I work with, so are the things, so it is really critical thinking and applying the context and not letting the numbers put you off.

Now you do need to a little bit of number work in order to get to that point in order to not be scared away from the conversation. So you have to be pretty confident you know, looking at a row of numbers, working out an average which you can do in excel in about half a second or two, but it is getting rid of that first feeling of when somebody throws you a spreadsheet or throws you a set of account that you do not understand but that you start to think about as critically, as about the business and what these numbers could tell you about the business. The key thing with understanding numbers is applying your understanding of the question to understand where do the numbers come from and what do they mean and can I trust them. Most of the questions around that do not really require any significant mathematical understanding. In fact, the editor for the first series of More Or Less, a very popular numbers programme on BBC, was an English graduate and he said being an English graduate was fantastic because it gave him the permission to ask the stupid question, so say what is GDP? That really forced people to explain the story and not to disappear off into technical jargon.

Liberty Vittert:

What ends up happening is you hear these numbers, you have these experts telling you things and if you have been worried about numbers or you are not comfortable with them you then tend to just say okay, okay I do not even know what to ask when in effect the questions that I ask when I am presented with these types of studies or reports are literally, and I am not exaggerating, they are as simple as who, what, how and why? Who is it that we really want to know these number about? What is it that we actually ask? How did we interpret it? That has to do with increases and decreases and risk. If you say there is a 100% increase, well a 100% increase from what? If it is a tiny thing in the very beginning then it is going to still be a tiny even if it a 100% increase, and the final question is why. Why does these numbers really matter, how do we make them mean something? We see on the news trillions, billions, what do these numbers actually mean and how do they affect real people?

Matthew:

This programme is for very senior lawyers working at major international businesses –a serious, capable bunch of people with extensive business experience, yet I am prepared to bet more than a few listeners will experience anxiety and a lack of confidence when the talk turns to numbers, with a sometimes non-trivial impact on the level of participation in decision making.

The good news is that you are not alone. There is a major cultural issue at play here – we just don't yet see numeracy as being as important as literacy. We accept people just opting out, and we shouldn't.

Liberty:

Well one of the main things I see over and over again in my work is people being a little bit afraid of numbers and the problem is whenever we are afraid of something we do not understand and you get the sort of ethos. So if you think about it, your kid comes home from school and says "Mom or Dad I am just terrible at reading, I cannot read". At no point does a Mom or Dad say to little Johnny "It is ok Johnny I never learned to read and I did fine". We never say that, we figure out ways for kids to read.

We have made the decision as a society that you have to read to be able to a functioning person. The problem is we do not do that with maths. Little Johnny comes home and says "Mom, Dad I am terrible at algebra" and Mom or Dad goes "It is okay Johnny I was terrible at maths to and I did fine". We need to change that conversation. I have a real belief that anyone can do maths. In fact I actually failed maths class in ninth grade which is about 14 years old and the school, the teacher called in my parents and said Liberty is just not smart enough. Now I have a PHD in maths so clearly I was capable of doing it. I just needed to learn it a different way. Where data is becoming king, it is the new oil that kids and our children and us, we have to understand these numbers and we cannot be afraid of them anymore and there are common pitfalls we see over and over again that people fall into when they read about them first.

Robert:

There certainly is a different way that we think about numbers and words. If you, you do have so many people who would say I am just not a numbers person. I just, my heads goes in a way that they would never say I am just not a words person, when somebody talks with words I just kind of tune out and stop listening. There is a cultural acceptance of that that gives people permission to check out and maths can be hard. The conceptual issues when you get into fancy mathematics or even some things that you need to do but it just that 95% of maths does not require the high end conceptual stuff, 95% of maths is the stuff that should not be intimidating because you probably learnt in earlier on in primary school.

Matthew:

Being number confident, as Robert puts it, has always been important, but with an explosion of digital device-related data happening all around us, with almost every aspect of our life now quantified and analysed and with new business and management tools crunching unprecedented amounts of data all around us, understanding the output of these systems is absolutely essential.

Liberty:

Data is the new oil, data is everywhere, data analytics are how … data driven is what all companies are becoming. It is the ethos of all of these companies. The one thing we all see especially in this past year is that we cannot get away from numbers, they are everywhere in the news and COVID has certainly highlighted that but if you start to pay attention you will see that there is numbers in every new story you have ever read. People will use a percentage or a study or something to try to lead, mislead, inform or persuade you of something and the problem is that we tend to take these numbers at face value. One the main parts of my work is trying to say what are the questions that we should be asking whenever we are presented with these numbers? Your audience is lawyers, that is what you are really good at, asking good questions. So my work is in trying to help people who may not necessarily be numbers people or frankly who are numbers people. Think about what questions they should be asking whenever they are presented with data or a study or a report where someone is saying this is the number. Well frankly a lot of the time it might not be.

Matthew:

So getting more involved with numbers is crucial. But there are pitfalls. There are more likely to be pitfalls of logic and understanding than of maths itself and it is important to watch out for them. Vigorously applying this logic can really help. Here is one example of the dangers of not fully questioning the numbers we are given. At the height of the pandemic and its crisis over availability of PPE the UK Government admitted that it sometimes counted individual gloves rather pairs of piece of PPE allowing it to claim higher numbers of items distributed. The number was accurate but it did not mean what its audience thought it meant. Liberty remembers another example with particularly high stakes that really emphasises the importance of asking yourself some basic questions when presented with numerical data.

Liberty:

There is a case, OJ Simpson, one of the infamous criminal trials in American history. So during the trial his lawyer Alan Dershowitz made a very interesting statistical argument. He said that while it was known and definite that OJ physically abused his ex wife Nicole Simpson, they said it did not matter because 4,000,000 women in the United States are abused by their domestic partners every year but of that 4,000,000 only one in 2,500 are then murdered by the domestic partner that abused her. So the chance that OJ killed Nicole based upon the fact that he beat her up is one in 2,500, obviously not enough to convict someone.

This was a highly persuasive argument to the jury to say the fact that he domestically abused her was not relevant to the fact that he could have murdered her, but what the jury and most certainly the prosecution should have realised is that the statistics that Mr Dershowitz stated was completely irrelevant and here is why. In Mr Dershowitz's situation we have living women who are currently victims of domestic abuse. What is the chance that the guy that abuses her is going to kill her? One in 2,500. That is true, he did not lie but that is not the sample that we care about. We do not care about living women who are currently victims of domestic abuse because at the time of the trial Nicole Simpson was not a living woman, she was a murdered woman so the group or the sample that we want to know the statistic about is murdered women who happened to have been abused by their domestic partners. So the real question is of murdered women who happened to have been abused by their domestic partners, what is the chance that was the domestic partner that killed her and not that she was the victim of some random homicide? And in that case the answer is nine in ten. So there was a 90% chance that OJ Simpson killed Nicole based upon the fact that he abused her not a one in 2,500 and that is a simple slip of the tongue of the who, who is it that we really want to know these numbers about? It is very easy to get confused with something that seems so outrageously simple but what I want your audience to see is that this is the most publicised criminal trial in American history. The entire world was watching this. The prosecution's entire careers were riding on this. They had every single thing at their disposal and they missed this so what I think is really important to see is that it can be so easier to get bogged down in these numbers and not take a step back with a little bit of common sense and ask a question that almost seems dumb it is so simple, of who do we actually want to know these statistics about.

Matthew:

One of the things we all find hard is understanding big numbers. Even comprehending whether a number is actually very big or not can be a challenge. Two numbers can seem very similar to us, yet one might be ten times bigger than the other, which makes quite the difference on a corporate balance sheet.

Luckily Robert and Liberty have some tips on how we can make them more manageable.

Robert:

It is really easy to get blinded by somebody throwing billions around or hundreds or thousands. The most important way to make those numbers make sense is to put them in context and often to change the scale. One example is a government minister comes on the radio, says we are putting four billion into preschool childcare provision over the term of the next parliament because we care about children and the baby-eating opposition do not. And four billion of course is a massive number. You know the term of a parliament is five years, okay so that is eight hundred million quid but there are four million kids aged under five in the UK, so it is probably £200 a child a year, four quid a week and some say is £4 a lot of money? No. Is four billion a lot money? Yes. But they are same number so it cannot be both at once so putting it in the right scale and in the right context is key, and that is not necessarily the job of the person who is hearing the number. Someone who is presenting numerical information should be presenting this a context, in a way that makes sense to the listener and so if you do not understand the numbers that some is giving you part of number confidence is saying "Well I do not understand, it is not my fault, can you explain this to me in terms I understand?" I imagine that the best lawyers are not necessarily the people who explain legal terms in the most complicated abstract ways and the same goes for people who are bringing numbers to an audience.

Liberty:

If I tell you your chance of wining the lottery is one in three hundred million that means nothing. To me that means nothing, I do not know what one in three hundred million is. The difference between, if you told me my chances are one in one hundred million or one in three hundred million there is no difference to me in that, and I will give you what I do. I always come up with some example. Imagine you are in a bathroom and you're looking at this gorgeous huge claw foot tub and it is brimming, overflowing with white rice. Now you pick up one single grain of rice from that bathtub and you paint it gold and you bury it somewhere in that bathtub and cover it all back up with white rice. Now you get people to walk in the door, blindfold themselves and in one miraculous lucky dip pull out that one golden grain of rice and that is the equivalent of one in three hundred million of winning the lottery and it is a pretty damn good business if you can convince people to pay you $2 to do that. But that is a much more visual way of understanding what one in three hundred million is.

Matthew:

A related phenomenon is understanding scale by meticulously relating the stats we hear to real, practical life.

Liberty:

There is a study that came out about two years ago that is all over the place. French fries are going to kill you. So I decided to look back at the study and see what it actually said. The study itself which was in a Peer Review journal, a very good journal, said that if you eat fried potatoes more than two times per week you double your risk of death and that is pretty scary. The question comes with this idea of relative risk of what does double mean. What is our base line? So any time you hear someone say there is an increase or a decrease the first question you should be asking is from what? In this case the average person in the study was a 60 year old male. For a 60 year old your risk of dying is 1% so that means that if you line up 100 60 year old men one of them will die in the next year simply due to the fact that they are 60 and they are male. If all 100 of those men eat fried potatoes two times per week or more which means 12 times per week or 20 times per week for their entire lives, instead of one dying in the next year their risk of death is doubled. One doubled is two, so two will die, so instead of one of those 100 men dying in the next year two of those 100 men will die! And these guys get to eat fried potatoes three times per week or more for their entire lives! All of a sudden we have gone from one in 100 to two in 100 versus doubling your risk of death which sounds way scarier.

Matthew:

We have focused so far mostly on relatively benign environments, where we just want to have a better understanding of information. But we mustn't be naïve – exploiting the population's weakness in questioning and understanding statistics has long been an easy way for bad actors to pull the wool over our eyes. There are, after all, lies, damned lies and statistics. So how do we defend ourselves against malicious maths?

Robert:

When we come across numbers that are designed to confuse or to persuade we ask those simple questions I think that you ask of any numbers because that process just takes you back and helps you to fill in all the blanks and then once you get to end of that you work out is this a helpful or unhelpful number? Where does this number come from? What is the source? What are the weaknesses in the data that are underlying it? Is it a survey? Is it a survey of the people that we are talking about? Say you have a stat being thrown around, like 20% of people are self-isolating which means that 80% of people are not self-isolating. Well clearly that is a contentious number and you would do is you would go back to the survey or the research that produced this and you would go and look at the figures, check that it is 20%, check that they have got a reasonable number of people, but also crucially you ask what question were they asking? Were they asking if you have been infected with the virus or you have symptoms do you self-isolate completely in your bedroom for 14 days and never leave, or do they ask do you largely stay indoors because those are very different types of self-isolation and they will give very different answers. You do not need to understand numbers to be able to interrogate that but it is when you answer those kind of questions as behind the numbers that is when you get to understand whether the number makes sense and is helpful for somebody and what you need to explain about it?

Liberty:

In New York they were saying that they have done such a great job wit vaccine distribution, that 10% of the population has been vaccinated and I thought that does not make any sense because New York has been having a terrible time with vaccine distribution and they have been in the news for throwing away vaccines in fact because they have not been able to get it into people's arms and what it turns out is it was 10% of the population that is eligible to be vaccinated under their rules. It is not 10% of the New York population is 10% of their Tier 1 which is first line healthcare workers and people over the age of 75, so they were changing the denominator in this to make it look like they vaccinated 10% of their entire population, when in fact it was 10% of this very tiny population. You see it every day just in the news, of people trying to make themselves look better of make others look worse.

Matthew:

Hopefully we all feel a bit better prepared to question some assumptions and query the basis of some of the information we're given. If we feel more confident that this is about logic rather than maths then that will be a useful benefit. But how relevant is it really to people other than accountants and engineers? The answer has to be that it is fundamental. If you are going to be part of the team forming and implementing an organisation's strategy then you will be in rooms with spreadsheets and subject experts using statistical jargon. Being confident in understanding and questioning that is a vital skill, as Liberty outlines. And as Robert explains, failing to do it just creates opportunity for those who can.

Liberty:

You know we can just sit in this room and I have seen it over and over and over again and the numbers person comes in and explains things and no one has any questions which is crazy to me. I get it. You think these numbers people, this is their job, they are so smart, they know all these numbers, they have all these fancy graphs and fancy models. I think it is just so important to take a step back and say let us break this down a little bit. Ask these basic questions. Think about the PPE meeting where one glove counted as one rather than a pair of gloves. That is a common-sense question, just say wait a minute, what exactly is this one million things or what exactly is this two million things or break it down for me a little bit, stop using the jargon. It is something that both numbers people presenting need to be better at and the people in the room without the numerical background need to be better at asking for, which is to say let us distil this down into something that makes sense and stop using your ridiculous jargon that no one understands, and I think that if both people were willing to do that more we would be in a much better place.

Robert:

So we have this huge deluge of numbers over the course of the last year and people will react to that in different ways. So some people are monitoring the case numbers, hospitalisation numbers, admissions, intensive care, the ONS survey telling about the infections and keeping a very tight eye on it all, but there are probably also a core of people who just want to pull from that Nigeria Falls of data, the key numbers that they need. If you were to do that you would pick out two things. You would pick out R and the infection rate. If have those two numbers in mind you can boil down everything else and ignore the noise and then just focus in on those simple things that answers the questions that are most relevant to you. One of the best ways of dealing with numbers if finding the things that help you answer the question you need to know. The experts, the chief medical officers were saying these are the two numbers you need to know so find somebody who does know and helps you to focus on the things that you need to pay your attention to and make sure that when they tell you they tell it to you in a language that you understand.

So it is not about pulling yourself up by your boot straps, it is about you using other people to help focus your attention and effort on the things that really matter.

Not being number confident just creates fantastic opportunities for people who are. Before I worked in the media I was working for a drug company and before I was 30 I was sitting on the committee that was running a drug with scientists who were 15 years older than me, biologists who had been working on HIV drugs for years, medics, phenomenally trained people, and that was not unusual. All the statisticians on these committees were pretty young because a lot of people were not number confident and you needed one professionally numerate person. So creating these, it is great that people create these opportunities for me and for people who love numbers but it is probably not great they are doing it for themselves.

Matthew:

Thanks for joining us for the latest Brain Food for General Counsel podcast. Remember you can keep up to date with hour by hour coverage of business law news by the Out-Law reporting team at Pinsentmasons.com. Do not forget to subscribe to us wherever you get your podcast until next time, goodbye.

Brain Food for General Counsel was produced and presented by Matthew Magee for Pinsent Masons, the purpose-led international profession services firm with law at its core.

Legal notice

Apple

The Apple logo, iPhone, iPod touch, and iTunes Store are trademarks of Apple Inc., registered in the U.S. and other countries.

Google

Google Play and the Google Play logo are trademarks of Google LLC.