Scottish Independence: Big Data and Big Conversations

Slides from the Edinburgh Science Festival 2015 public lecture

By Daniel Winterstein
with panellists Stewart Kirkpatrick, Duncan Hothersall, Leah Lockhart, Alaster Phillips

In 1314 Robert the Bruce defeated Edward II's army at Bannockburn, which led to Scottish independence. We have just 4 contemporary accounts of that struggle.

Well a few things have changed between then and now. In the 1970s, the US military invented the internet for the purpose of sharing pictures of cats and porn. It proved a success, and in the last decade, we've seen social media allow everyone with a computer or a smart phone to have an online voice.

So when in 2014 we had the Independence Referendum, there were rather more than four people giving their views.

These slides look at the Independence referendum, as told by everyone on Twitter. Over the course of 2014, some 82,000 people posted 10 million tweets about the referendum.

That's a wealth of material -- but that very abundance of riches poses a challenge: How can we read & make sense of 10 million messages?

Diverse Voices

10 Million Tweets is Not Enough

(1, 0, 1, 0, 0, 0, 1, 0 ... )

By Gary Hay
"defeat"
"#indyref"
"hello"
"defeat [at] #indyref"
...

Model space: 10,000,000,000,000,000,000,000 dimensions

Machine Learning

Homosexual Male Markers

Wicked the Musical
Human Rights Campaign
Desperate Housewives

Straight Male Markers

Wu-Tang Clan
Being Confused After Waking Up From Naps

Parents Separated

"I'm Sorry I Love You"
"Never Apologize For What You Feel"

Private traits and attributes are predictable from digital records of human behavior
M. Kosinskia, D. Stillwella, and T. Graepelb, 2013

There's a lot you can learn about someone from their online profile. Computers can learn too.

Some people nail their colours to the mast. They publicly join groups, or they use hashtags, which are easy for a computer to pick out.

Not everyone does that. That's where machine-learning comes in. The clear examples provide us with training examples. From that, we can build models of how each group talks -- what they talk about, and how they express themselves. This gives us a computer system which can look at your profile, and guess facts about you -- even hidden ones.

For example a US study found, that they could guess -- with quite high accuracy -- the person's sexuality, whether they were Republican or Democrat, and even whether they come from a broken home or not. Most people consider those things too private to post. But the computer could make reasonably guess, based on adding up the films and books you like and comparing against the models.

Here are some of the markers they found.

How to Recognise Yes / No Voters

Recognising UK Party Supporters

David Cameron
austerity
Katie Hopkins
Bio: socialist

Labour-SNP
jobs
EU referendum
navy, RAF, army

Nicola Sturgeon
bio: Scotland, independence
social justice

Independent Scotland
social justice
Bio: husband & father

Income tax
David Cameron
Bio: politics, cat

betrayal
borders
bio: UK, hate, political correctness

Where Did People Get Their News?

"No" Media

The Telegraph
YouTube
BBC
bettertogether.net
The Guardian

"Yes" Media

YouTube
The Guardian
WingsOverScotland
The Herald
yesscotland.net

The BBC on Wings

pro-Yes blog

Wings on the BBC

staggeringly dishonest, sinister, biased, and puerile

CyberNats?

What do people swear about?

Please sign my petition asking Gordon Brown to go [have biblical knowledge of] himself

Let's forget about [Duncan Hothersall]'s toxic, cowardly excuse for a personality...
...waffle about LGBT issues...
...Duncan is either the stupidest man on Earth or the biggest liar.

In 1st and 2nd place, Alex Salmond and Gordon Brown -- by both yes and no, for different reasons.

Yes voters shouted at Jim Murphy.

Some people end up in the crosshairs, some are. No voters got angry about Brian Souter, a large SNP donor who's been involved in anti-gay campaigns. By contrast, the Lottery winners Chris and Colin Weir who also made a big donation to the Yes campaign -- were largely left alone. JK Rowling was attacked for supporting the No campaign. Whilst another writer and No-campaigner, CJ Sansom, got no reaction.

Having read a sample of these messages, the abusive behaviour is not as bad as elsewhere on the internet. You may be familiar with GamerGate, an online fight in the computer games community, where several women have received sustained coordinated abuse, including lots of death threats. I'm glad to say we didn't have that level of abuse here.

The "top swearer" was Wings Over Scotland. Here are a couple of examples -- including about our panellist Duncan Hothersall!

1st TV Debate

"What's Plan B?", "independent successful country"

2nd TV Debate

Sir Ian Wood, Oil Reserves

Referendum!

Alex Salmond, Gordon Brown, Scottish Independence

Topics Discussed

Politicians

Yes Won -- Online

45% of voters
69% of talkers
77% of tweets
84% of clear tweets

#indyref After the Referendum

The Day After

Alex Salmond, Nicola Sturgeon, #The45

The Month After

5,700 tweets per day.
#the45, #yes

Last Month

1,500 tweets per day (of 5,000 GE).
#voteSNP, #GE2015, #GE15, #the45, #SNPout

What Type of Conversations?

Sharing content: 32%
@someone: 15%
Meet / organise: 1%

One problem with social media is that we connect with like-minded people in our own social bubble. Which means we tend to hear our own views reinforced, like an echo chamber.

As reading-the-newspaper declines, and more people find their news via social media, that means we're getting a more filtered view of the world.

On Twitter, this is partly balanced by the public forums created by hashtags. We did see lots of Yes & No voters talking to each other. Although often they were shouting.

The problem is worst on Facebook. Even within your network, Facebook filters what you see, to fit what it thinks you like. Facebook's computers maximise likes. They will have taken a view on how you vote, and they'll be showing you articles you'll like, and filtering even your own friends when they have conflicting views. It's bubbles within bubbles.