WEBVTT Kind: captions Language: en 1 00:00:00.140 --> 00:00:02.650 A few years ago, I got a message from an ex. 2 00:00:02.650 --> 00:00:05.350 And it said, “roses are red, violets are blue, 3 00:00:05.350 --> 00:00:07.310 “I've got chlamydia, and you might have too”. 4 00:00:07.310 --> 00:00:09.880 Which, first of all, is genius, 5 00:00:09.880 --> 00:00:13.730 but it meant I was taking a trip down to the clinic the next morning to get tested. 6 00:00:13.730 --> 00:00:17.380 I have a point to this story, and I swear it's about computers. 7 00:00:19.720 --> 00:00:22.360 If you ever want to see a complete cross-section of London life, 8 00:00:22.369 --> 00:00:25.439 do join the queue for the STI testing clinic first thing in the morning. 9 00:00:25.439 --> 00:00:28.550 Actually, get yourself tested anyway, it's the right thing to do. 10 00:00:28.550 --> 00:00:32.710 Anyway, I go to the clinic, I pee in a sample tube, seal it up, hand it over, 11 00:00:32.710 --> 00:00:36.080 and they give me a little business card with a phone number and a passcode on it. 12 00:00:36.080 --> 00:00:38.260 And I can call that number the next day, 13 00:00:38.260 --> 00:00:41.570 and if my tests are all clear, then the system will tell me. 14 00:00:41.570 --> 00:00:44.340 And if they’re not, it'll pass me over to someone at the clinic, 15 00:00:44.340 --> 00:00:48.090 because you do not deliver bad news with an automated system. 16 00:00:48.090 --> 00:00:52.320 So the next day, I call the number, type in the code, and the system says: 17 00:00:52.340 --> 00:00:54.700 "Thanks. Here are your test results." 18 00:00:57.680 --> 00:01:00.760 And then there is a pause. A proper, ten second long pause. 19 00:01:00.760 --> 00:01:03.300 And as I stood there on my mobile in the middle of the street, 20 00:01:03.300 --> 00:01:06.580 it felt a bit like that artificial pause that they add in reality shows 21 00:01:06.580 --> 00:01:07.980 before announcing the winner. 22 00:01:07.980 --> 00:01:10.100 It felt like there was a spotlight on me, 23 00:01:10.110 --> 00:01:12.900 all dramatic tension music playing in the background. 24 00:01:12.900 --> 00:01:16.800 And then it says "We're putting you through to one of our team. 25 00:01:16.800 --> 00:01:18.740 "Please stand by." 26 00:01:18.740 --> 00:01:21.119 That's bad news, right? 27 00:01:21.120 --> 00:01:24.420 Bad news comes from a human. Why are they passing me through? 28 00:01:24.420 --> 00:01:27.570 Someone picks up, I give my passcode again and they say, 29 00:01:27.570 --> 00:01:32.180 no, it's all clear, you don’t need to worry, everything’s negative, you're fine. 30 00:01:32.180 --> 00:01:35.509 And I ask why the system put me through, and they say, 31 00:01:35.509 --> 00:01:37.800 "oh, no idea, it just does that sometimes". 32 00:01:39.049 --> 00:01:41.860 It just does that sometimes. 33 00:01:41.869 --> 00:01:43.890 Did the database lookup fail in the background, 34 00:01:43.890 --> 00:01:47.390 and it dealt with the timeout by just passing me to a human? 35 00:01:47.390 --> 00:01:50.590 Was there a small note on my file somewhere that confused it? 36 00:01:50.590 --> 00:01:52.619 I've no idea, they had no idea, 37 00:01:52.619 --> 00:01:55.330 because whoever wrote that code, whoever designed that system, 38 00:01:55.330 --> 00:01:57.009 the moment there was any error, 39 00:01:57.009 --> 00:01:59.670 they just told it to go to "we're putting you through" 40 00:01:59.670 --> 00:02:03.609 without explaining why and without thinking what that might cause. 41 00:02:03.609 --> 00:02:09.130 That phone system is used by people asking about things way more serious than chlamydia, 42 00:02:09.130 --> 00:02:12.480 the sort of things you can't cure with a course of antibiotics. 43 00:02:12.480 --> 00:02:16.160 And sure, on the scale of complaints it is minor, 44 00:02:16.160 --> 00:02:20.510 but mistakes like that are a symptom of something much bigger. 45 00:02:20.510 --> 00:02:23.040 I've talked about bodging things in the past. 46 00:02:23.040 --> 00:02:25.030 I encourage it for hobby projects. 47 00:02:25.030 --> 00:02:29.000 Just slapping something together as a proof of concept, just so it works for you 48 00:02:29.000 --> 00:02:32.819 is a great principle when you're making a thing for yourself. 49 00:02:32.819 --> 00:02:35.790 But if you're making something for the public, for mass consumption, 50 00:02:35.790 --> 00:02:40.780 particularly something that's going to be used by people in very vulnerable moments, 51 00:02:40.780 --> 00:02:44.580 then you've got to take a lot more care. 52 00:02:44.580 --> 00:02:49.290 Every time we build something for the public, we have to start making the trade-off: 53 00:02:49.290 --> 00:02:53.130 how much time and money is it worth to deal with every edge case? 54 00:02:53.130 --> 00:02:56.159 Maybe the designer thought that lookups would only fail one in a million times, 55 00:02:56.159 --> 00:02:58.180 and if that's the case, then yeah, 56 00:02:58.180 --> 00:03:02.120 it's probably not reasonable to bother recording a whole extra message 57 00:03:02.120 --> 00:03:04.880 and programming in a whole extra edge case. 58 00:03:04.880 --> 00:03:07.780 But if the lookup fails often enough that the clinic receptionist just 59 00:03:07.780 --> 00:03:09.840 dismisses it as normal... well, 60 00:03:09.840 --> 00:03:13.460 by that point it's too late to make a change, isn't it? The system's in place, 61 00:03:13.460 --> 00:03:16.200 it'd be far too expensive to fix it now. It'll do. 62 00:03:16.200 --> 00:03:21.690 The trouble is that we're often dealing with unknown harms and unintended consequences. 63 00:03:21.690 --> 00:03:25.480 Far too often a bodged-together system that was just meant to be a test 64 00:03:25.480 --> 00:03:30.220 gets rolled out into production, and everyone just has to deal with the bugs 65 00:03:30.220 --> 00:03:32.860 because that's all anyone can afford to do. 66 00:03:32.860 --> 00:03:35.770 I will always bet on incompetence rather than malice, 67 00:03:35.770 --> 00:03:39.580 I will always bet on someone not thinking about consequences 68 00:03:39.580 --> 00:03:43.330 rather than thinking of them and going “who cares”. 69 00:03:43.330 --> 00:03:45.370 We see this with big tech companies. 70 00:03:45.370 --> 00:03:50.410 Facebook allows the world to communicate in unprecedented ways, 71 00:03:50.410 --> 00:03:53.439 it does huge amounts of emotional labour for us, 72 00:03:53.439 --> 00:03:58.800 it allows people to keep in touch with old friends that they'd just fall away from otherwise. 73 00:03:58.800 --> 00:04:02.840 But it's also enabled stalkers and abusers to reach people they shouldn't, 74 00:04:02.840 --> 00:04:06.680 it’s allowed a huge amount of private data to be misused without real consent, 75 00:04:06.680 --> 00:04:11.060 and it’s arguably helped cause terrifying election meddling. 76 00:04:11.060 --> 00:04:13.340 Now, I don't believe anyone in Facebook's management was 77 00:04:13.340 --> 00:04:15.980 rubbing their hands in delight at the chaos it was causing: 78 00:04:15.980 --> 00:04:21.830 it was just a series of seemingly-reasonable decisions that added up to huge consequences. 79 00:04:21.830 --> 00:04:26.420 And then there's YouTube: it allows anyone to publish their experiences to the world, 80 00:04:26.420 --> 00:04:30.080 it provides income for creative people that bypasses traditional media, 81 00:04:30.080 --> 00:04:34.890 and it’s helped people connect with other people's lives. 82 00:04:34.890 --> 00:04:36.790 It's also helped to radicalise folks, 83 00:04:36.790 --> 00:04:39.840 to promote conspiracy theories, and to traumatise children. 84 00:04:39.840 --> 00:04:42.700 Are those tradeoffs worth it? 85 00:04:42.700 --> 00:04:46.300 It depends on your moral framework, and it depends -- let's be honest -- 86 00:04:46.300 --> 00:04:48.410 on how it's affected you personally. 87 00:04:48.410 --> 00:04:52.000 It's not like there's a crystal ball that'll tell you, yes, 88 00:04:52.000 --> 00:04:54.160 your brand-new dating app you’re developing, 89 00:04:54.160 --> 00:04:58.510 that will cause 1,000 couples to marry and live happily ever after, 90 00:04:58.510 --> 00:05:01.160 but it will also get three people murdered. 91 00:05:01.160 --> 00:05:04.410 The real world is not a trolley problem. 92 00:05:04.410 --> 00:05:08.310 The STI result system that I called presumably reduced the workload on staff, 93 00:05:08.310 --> 00:05:10.708 and it allowed people to check their results out-of-hours 94 00:05:10.708 --> 00:05:13.390 when it was convenient and discreet for them. 95 00:05:13.390 --> 00:05:16.290 You'd hope something like that wouldn't have a downside, 96 00:05:16.290 --> 00:05:20.750 but then the designers screwed it up because they thought it was good enough, 97 00:05:20.750 --> 00:05:22.300 and it wasn't. 98 00:05:22.300 --> 00:05:24.240 One extra check, one extra voice message that said 99 00:05:24.240 --> 00:05:29.400 "Sorry, I can't find your result, just a moment" would have solved that. 100 00:05:29.400 --> 00:05:33.580 Every time we design a system, we have to minimise the potential harm. 101 00:05:33.580 --> 00:05:37.560 Look at the code that you write, look at the systems you design, and think: 102 00:05:37.560 --> 00:05:40.440 How could this be abused? How could this fall apart? 103 00:05:40.440 --> 00:05:42.340 What are the failure states here? 104 00:05:42.340 --> 00:05:45.640 If you were a bad actor, if you wanted to use this maliciously, 105 00:05:45.640 --> 00:05:47.190 how would you go about it? 106 00:05:47.190 --> 00:05:50.950 Think about how you’d attack your own systems, explore those failure states, 107 00:05:50.950 --> 00:05:54.420 deliberately screw things up and see how your code copes. 108 00:05:54.420 --> 00:05:58.280 Because if you don't, someone else will. 109 00:05:58.280 --> 00:06:00.580 Thank you very much to the Centre for Computing History at Cambridge 110 00:06:00.580 --> 00:06:02.700 who've let me film with all their old equipment, 111 00:06:02.700 --> 00:06:05.100 and to all my proofreading team who checked through my script.