WEBVTT
Kind: captions
Language: en

1
00:00:00.089 --> 00:00:02.060
It is Christmas, I'm home with my family.

2
00:00:02.060 --> 00:00:04.960
Hence the textures wallapper and big floppy
cushions.

3
00:00:04.960 --> 00:00:07.460
And I know I've already put up a Christmas
video, and in it,

4
00:00:07.460 --> 00:00:10.860
I signed off until 2016, because surely there's
not going to be any company

5
00:00:10.860 --> 00:00:13.650
having a massive, avoidable, security problem
over Christmas

6
00:00:13.650 --> 00:00:18.050
that people are then going to pester me into,
um, explaining...

7
00:00:18.050 --> 00:00:18.820
[pause]

8
00:00:18.820 --> 00:00:21.610
So when I was a kid at school, there was a
thing called "chinny reckon".

9
00:00:21.610 --> 00:00:23.170
Your school might have said something different,

10
00:00:23.170 --> 00:00:26.029
like Jimmy Hill or itchy beard, but the idea
was:

11
00:00:26.029 --> 00:00:27.250
someone makes a dubious claim,

12
00:00:27.250 --> 00:00:28.699
like their uncle working at NASA and

13
00:00:28.699 --> 00:00:30.269
giving them a ride on the space shuttle,

14
00:00:30.269 --> 00:00:32.300
and you go, yeah, sure. Chinny reckon.

15
00:00:32.300 --> 00:00:37.230
Well, about the same time, back when the World
Wide Web was a few servers running mostly

16
00:00:37.230 --> 00:00:38.760
in academic labs,

17
00:00:38.760 --> 00:00:41.280
they didn't need what are now called web caches:

18
00:00:41.280 --> 00:00:44.370
a request comes in for a web page, the server,
back then,

19
00:00:44.370 --> 00:00:46.230
would look up the page on disk,

20
00:00:46.230 --> 00:00:49.120
and load it, and then send that back.

21
00:00:49.120 --> 00:00:52.260
But as the web expanded, demands on servers
got a lot bigger --

22
00:00:52.260 --> 00:00:55.950
not just in terms of how many individual requests
for files they get,

23
00:00:55.950 --> 00:00:59.129
but how much work they have to do for each
of those requests.

24
00:00:59.129 --> 00:01:03.670
Loading the home page of Steam involves more
than fifty requests for files, images and

25
00:01:03.670 --> 00:01:04.259
stylesheets,

26
00:01:04.259 --> 00:01:07.670
and returns somewhere between 5 and 6 megabytes
of data.

27
00:01:07.670 --> 00:01:10.159
But there's a lot of work going on behind
the scenes too:

28
00:01:10.159 --> 00:01:12.380
their server has to check the database,

29
00:01:12.380 --> 00:01:16.279
look up which games are currently being promoted
out of the 10,000 that they have,

30
00:01:16.279 --> 00:01:17.560
load all the pricing information,

31
00:01:17.560 --> 00:01:20.899
work out which bits go in which part of the
site, and if you're logged in,

32
00:01:20.899 --> 00:01:25.090
it has to add a load of personalised recommendation
stuff just for you.

33
00:01:25.090 --> 00:01:26.380
That's all really database-intensive.

34
00:01:26.380 --> 00:01:28.560
Takes a lot of processing power.

35
00:01:28.560 --> 00:01:30.749
Which is where caching comes in.

36
00:01:30.749 --> 00:01:33.520
Now, there are many ways that web servers
do caching:

37
00:01:33.520 --> 00:01:34.889
there are content delivery networks,

38
00:01:34.889 --> 00:01:37.259
and even the user's own browser.

39
00:01:37.259 --> 00:01:40.560
And in Steam's case, they use all of the possible
techniques,

40
00:01:40.560 --> 00:01:42.060
including the one that screwed up here,

41
00:01:42.060 --> 00:01:44.619
which is the caching on their own web servers.

42
00:01:44.619 --> 00:01:49.609
A huge number of requests to Steam's servers
will be for pages that have already been generated

43
00:01:49.609 --> 00:01:50.520
recently.

44
00:01:50.520 --> 00:01:53.729
If they're getting ten thousand requests a
second for their home page,

45
00:01:53.729 --> 00:01:56.859
it'd be ridiculous for the server to go out
to their database and

46
00:01:56.859 --> 00:01:59.849
look up what's on offer ten thousand times
a second,

47
00:01:59.849 --> 00:02:01.579
just in case something's been changed.

48
00:02:01.579 --> 00:02:05.459
So it looks it up once, and then it caches
the result into its memory for...

49
00:02:05.459 --> 00:02:08.009
well, I don't know how long, maybe a second,
maybe a minute,

50
00:02:08.009 --> 00:02:09.920
maybe half an hour in some cases.

51
00:02:09.920 --> 00:02:13.340
Actually, a huge number of servers are doing
that same thing:

52
00:02:13.340 --> 00:02:16.780
there's no way that one web server could deal
with a site as big as Steam,

53
00:02:16.780 --> 00:02:21.299
so all those hundreds of servers check the
central database every so often,

54
00:02:21.299 --> 00:02:25.120
they keep their own version of each public
page in their own memory,

55
00:02:25.120 --> 00:02:26.810
sending that out to everyone who asks for
it.

56
00:02:26.810 --> 00:02:30.879
And when a request comes in, it's routed to
any of those available servers.

57
00:02:30.879 --> 00:02:33.260
That way, if there's a sudden spike in traffic,

58
00:02:33.260 --> 00:02:35.069
Steam can just spin up some more servers,

59
00:02:35.069 --> 00:02:39.069
add them to the pool that's answering questions,
and it all keeps working.

60
00:02:39.069 --> 00:02:44.079
And each server might have thirty different
versions of that site cached: for different

61
00:02:44.079 --> 00:02:44.799
languages,

62
00:02:44.799 --> 00:02:46.659
for different regions, for different devices.

63
00:02:46.659 --> 00:02:50.969
If that server has the right version in its
memory, and it's not out of date,

64
00:02:50.969 --> 00:02:52.530
there's no reason to bother the database for
it --

65
00:02:52.530 --> 00:02:55.150
they just send out the one that they already
know about.

66
00:02:55.150 --> 00:02:59.290
It means only a tiny fraction of that huge
amount of traffic actually has to bother the

67
00:02:59.290 --> 00:03:00.439
database:

68
00:03:00.439 --> 00:03:04.590
the rest comes straight out of memory on a
system that's built just for serving web pages,

69
00:03:04.590 --> 00:03:07.079
which is orders of magnitude faster.

70
00:03:07.079 --> 00:03:08.560
Which is great for anonymous passers-by,

71
00:03:08.560 --> 00:03:11.659
who are just reading about a game or looking
at the home page.

72
00:03:11.659 --> 00:03:14.239
But what about the folks that are logged in?

73
00:03:14.239 --> 00:03:17.159
They're expecting to see their name and avatar
at the top,

74
00:03:17.159 --> 00:03:19.829
to see the whole site personalised for them.

75
00:03:19.829 --> 00:03:23.269
There's no point trying to sell someone a
game that they already own.

76
00:03:23.269 --> 00:03:25.519
Unless you already own it on CD from ten years
ago and

77
00:03:25.519 --> 00:03:26.909
you can't be bothered to go looking for it,

78
00:03:26.909 --> 00:03:29.269
and it's about two bucks just to download
it again, so...

79
00:03:29.269 --> 00:03:33.870
Anyway. The traditional logic is that you
don't cache pages for users who are logged

80
00:03:33.870 --> 00:03:34.389
in.

81
00:03:34.389 --> 00:03:36.920
Logged in users are a minority on most sites
--

82
00:03:36.920 --> 00:03:39.120
like on a news site, you have thousands,

83
00:03:39.120 --> 00:03:41.760
maybe millions of people reading but only
a few

84
00:03:41.760 --> 00:03:43.859
who can actually log in and change what's
on there.

85
00:03:43.859 --> 00:03:46.930
So everyone who's logged out, the millions,
they get the standard,

86
00:03:46.930 --> 00:03:49.370
cached version; but if you're logged in as
an editor,

87
00:03:49.370 --> 00:03:51.099
then every time you request a page,

88
00:03:51.099 --> 00:03:53.790
the server asks the database to check everything
again,

89
00:03:53.790 --> 00:03:55.480
just in case something's changed.

90
00:03:55.480 --> 00:03:58.799
The cache doesn't touch anything that's personalised.

91
00:03:58.799 --> 00:04:01.749
But: that isn't always a good strategy.

92
00:04:01.749 --> 00:04:04.930
Steam might actually get more requests from
logged in users than anonymous ones,

93
00:04:04.930 --> 00:04:06.529
particularly when they've got a sale on and

94
00:04:06.529 --> 00:04:10.409
people are regularly reloading the page to
see if anything's changed, ready to buy.

95
00:04:10.409 --> 00:04:15.090
In a situation like that, it might be a good
idea to tell the web servers that, yes,

96
00:04:15.090 --> 00:04:17.759
you should cache some pages, like the home
page,

97
00:04:17.759 --> 00:04:21.120
for logged in users, as long as you do two
things:

98
00:04:21.120 --> 00:04:24.569
First, you clear the cache if any data changes,

99
00:04:24.569 --> 00:04:27.800
so no-one buys a game and then sees it still
available for purchase.

100
00:04:27.800 --> 00:04:34.630
And second, you never, ever, let anyone see
someone else's cached page.

101
00:04:34.630 --> 00:04:37.990
Well, someone in Steam, on Christmas Day,

102
00:04:37.990 --> 00:04:39.639
made a quick change to the caching system,

103
00:04:39.639 --> 00:04:41.479
probably to make the site run a bit faster.

104
00:04:41.479 --> 00:04:44.370
And they accidentally told the cache servers
to not care

105
00:04:44.370 --> 00:04:47.240
about which user was logged in, or any of
their settings.

106
00:04:47.240 --> 00:04:52.310
So when, a moment later, a request came in
from someone to see their own, personal account,

107
00:04:52.310 --> 00:04:54.949
one of the caches looked up the account in
the database,

108
00:04:54.949 --> 00:04:56.830
then stored the resulting web page.

109
00:04:56.830 --> 00:04:57.960
In whatever language it was in,

110
00:04:57.960 --> 00:05:00.349
with whoever's private data was in there.

111
00:05:00.349 --> 00:05:03.900
And when someone else sent a request for their
account page a fraction of a second later:

112
00:05:03.900 --> 00:05:05.949
that server sent the cached page,

113
00:05:05.949 --> 00:05:09.580
which that first person's details, right back
to them.

114
00:05:09.580 --> 00:05:13.330
And each of those many, many web servers cached
someone else's information --

115
00:05:13.330 --> 00:05:16.840
whoever was unlucky enough to request their
details from that server first,

116
00:05:16.840 --> 00:05:19.360
at the exact wrong moment in time.

117
00:05:19.360 --> 00:05:20.770
And after maybe a minute, maybe more,

118
00:05:20.770 --> 00:05:22.360
the cache let that data expire,

119
00:05:22.360 --> 00:05:25.900
and promptly stored the details of whoever
the next unlucky person was,

120
00:05:25.900 --> 00:05:27.210
sending that out to everyone else.

121
00:05:27.210 --> 00:05:30.110
And at that point, a lot of people were checking
their own account details,

122
00:05:30.110 --> 00:05:33.410
wondering if they were about to have their
credit card drained.

123
00:05:33.410 --> 00:05:37.349
Now, if there's any consolation for Steam,
it's that they're not alone:

124
00:05:37.349 --> 00:05:40.050
they made the same error as the Norwegian
government in 2012,

125
00:05:40.050 --> 00:05:41.879
where for about fifteen minutes,

126
00:05:41.879 --> 00:05:46.370
anyone who logged in to see their tax return
saw someone else's personal details.

127
00:05:46.370 --> 00:05:49.400
Actually, it wasn't just someone else's personal
details,

128
00:05:49.400 --> 00:05:52.509
it was the details of a man called Kenneth
Belcovski from Oslo.

129
00:05:52.509 --> 00:05:55.199
I believe they made a personal apology to
Kenneth.

130
00:05:55.199 --> 00:05:58.050
And the good news is, requests to update data
--

131
00:05:58.050 --> 00:05:59.930
so to change passwords, buy things,

132
00:05:59.930 --> 00:06:02.280
and so on -- they never hit the cache.

133
00:06:02.280 --> 00:06:05.789
Well, they shouldn't, not unless there's a
much bigger screwup that Steam hasn't told

134
00:06:05.789 --> 00:06:07.020
anyone about yet.

135
00:06:07.020 --> 00:06:09.419
When you click "Buy", or "Change Password",

136
00:06:09.419 --> 00:06:13.629
your browser sends something called a POST
request that bypasses any caches --

137
00:06:13.629 --> 00:06:15.449
so your login cookie gets checked on the way,

138
00:06:15.449 --> 00:06:18.629
and it'll most likely just throw an error
if it thinks you're trying to see someone

139
00:06:18.629 --> 00:06:20.110
else's account.

140
00:06:20.110 --> 00:06:22.319
Which means those people -- and I've seen
a couple --

141
00:06:22.319 --> 00:06:26.259
who were claiming that during the screwup
someone bought a load of games on their account

142
00:06:26.259 --> 00:06:26.639
and

143
00:06:26.639 --> 00:06:28.759
immediatley want a refund? Well,

144
00:06:28.759 --> 00:06:31.289
I'm not saying you're making it up,

145
00:06:31.289 --> 00:06:35.659
but if this were a playground: chinny reckon.

146
00:06:35.659 --> 00:06:37.150
Merry Christmas everyone.

147
00:06:37.150 --> 00:06:38.949
One more thing I remembered while I was editing
this:

148
00:06:38.949 --> 00:06:41.990
if you know a YouTube video gamer, Twitch
streamer,

149
00:06:41.990 --> 00:06:43.699
who matches all these requirements:

150
00:06:43.699 --> 00:06:45.289
tell them to get in touch with me,

151
00:06:45.289 --> 00:06:47.460
there's a collaboration that I would love
to do with them

152
00:06:47.460 --> 00:06:48.379
at the start of next year.

153
00:06:49.380 --> 00:06:51.380
[Translating this video? Add your name here if you'd like credit!]