WEBVTT Kind: captions Language: en 1 00:00:00.190 --> 00:00:05.810 The Winograd schema is a test designed to stretch computer language processing to its limit. 2 00:00:05.810 --> 00:00:10.360 If a machine can determine an intended referent in a sentence based only on clues from context, 3 00:00:10.360 --> 00:00:14.889 then the machine is using reasoning to parse the language, and, perhaps, 4 00:00:14.889 --> 00:00:17.290 it should be called intelligent. 5 00:00:17.290 --> 00:00:21.190 Humans are pretty good at working out referents based on context. For example: 6 00:00:21.190 --> 00:00:25.500 "The trophy would not fit in the brown suitcase because it was too big." 7 00:00:26.660 --> 00:00:28.600 What was too big? 8 00:00:28.600 --> 00:00:33.440 Linguists will sometimes denote referents in text by using subscript numbers or letters. 9 00:00:33.440 --> 00:00:37.360 So if you want to solve that “x”, and work out what it refers to in that sentence, 10 00:00:37.370 --> 00:00:40.480 then you have to know that a suitcase usually contains things, 11 00:00:40.480 --> 00:00:43.069 and that a trophy usually doesn’t contain things. 12 00:00:43.069 --> 00:00:45.589 And you have to know how containers work, 13 00:00:45.589 --> 00:00:47.740 that larger things can’t fit inside smaller things. 14 00:00:47.740 --> 00:00:49.260 It seems simple to us, because we understand 15 00:00:49.260 --> 00:00:53.740 not only how syntax works and whether a sentence is grammatically valid, 16 00:00:53.740 --> 00:01:00.020 but also because we’ve also got experience of suitcases and trophies and reality itself. 17 00:01:00.020 --> 00:01:03.260 Solving that “x” requires comprehension. 18 00:01:03.269 --> 00:01:08.180 Terry Winograd started this line of thinking in 1972 by proposing sentences that required 19 00:01:08.180 --> 00:01:09.410 that sort of context. 20 00:01:09.410 --> 00:01:14.790 "The city councilmen refused the demonstrators a permit because they 'feared violence' 21 00:01:14.790 --> 00:01:16.750 "or because they 'advocated revolution'." 22 00:01:16.750 --> 00:01:20.190 ...it was the 70s, there weren’t many councilwomen. 23 00:01:20.190 --> 00:01:22.080 And there was a lot of revolution. 24 00:01:22.080 --> 00:01:26.570 Winograd argued solving this sentence requires knowing not only who can issue permits 25 00:01:26.570 --> 00:01:27.920 or what a demonstrator is, 26 00:01:27.920 --> 00:01:31.140 but also the political interests of a council, 27 00:01:31.140 --> 00:01:33.410 the ways in which they’d view political change. 28 00:01:33.410 --> 00:01:39.590 That referent changes based purely on the context and the meaning of those last two words. 29 00:01:39.590 --> 00:01:42.810 Now there are plenty of sentences where the “x” is ambiguous, 30 00:01:42.810 --> 00:01:47.860 where even humans don't have enough context to work out which pronoun refers to which person. 31 00:01:47.860 --> 00:01:49.620 If you’re writing a gay romance story, 32 00:01:49.620 --> 00:01:52.460 and both your main characters use the same pronouns 33 00:01:52.460 --> 00:01:55.120 and appear all the time in the same sentences, 34 00:01:55.130 --> 00:01:57.660 then it’s going to get confusing fast. 35 00:01:57.660 --> 00:01:59.710 In more amateur work, like slash fiction, 36 00:01:59.710 --> 00:02:04.560 inexperienced writers often try to clear that up with synecdoche in place of pronouns. 37 00:02:04.560 --> 00:02:06.100 But that’s awkward. 38 00:02:06.100 --> 00:02:10.380 More experienced writers tend to use context and careful sentence construction, 39 00:02:10.380 --> 00:02:12.200 and readers never notice it. 40 00:02:12.200 --> 00:02:15.560 But Winograd schemas are designed to be unambiguous for humans, 41 00:02:15.560 --> 00:02:18.520 to the point where we wouldn’t even have to think about it, 42 00:02:18.520 --> 00:02:20.940 but difficult for computers. 43 00:02:20.950 --> 00:02:23.970 Natural Language Processing, the ability to "understand" language, 44 00:02:23.970 --> 00:02:26.280 is in huge demand right now; it’s what powers Siri, 45 00:02:26.280 --> 00:02:28.310 and Alexa, and the Google Assistant. 46 00:02:28.310 --> 00:02:32.420 So many companies are trying to create code that can understand human requests, 47 00:02:32.420 --> 00:02:35.770 and let’s be honest, most of that code sucks right now. 48 00:02:35.770 --> 00:02:37.650 They often just rely on collections of sentences, 49 00:02:37.650 --> 00:02:42.670 each word tagged with its part of speech; they just pick up on keywords like “call” 50 00:02:42.670 --> 00:02:46.470 or “how much is” and extract what they hope are the right bits around them. 51 00:02:46.470 --> 00:02:49.150 That doesn’t work for a Winograd schema. 52 00:02:49.150 --> 00:02:53.630 A team of researchers at the computer science department of New York University have assembled 53 00:02:53.630 --> 00:02:57.860 a list of 150 of them, as a test. 54 00:02:57.860 --> 00:03:03.250 So: “I spread the cloth on the table in order to protect it.” To protect what? 55 00:03:03.250 --> 00:03:05.660 Obvious if you understand what a cloth and a table are, 56 00:03:05.660 --> 00:03:08.470 and why you might spread a cloth on a table, 57 00:03:08.470 --> 00:03:11.130 but if those are just tagged as nouns with no context, 58 00:03:11.130 --> 00:03:13.050 then that’s completely ambiguous. 59 00:03:13.050 --> 00:03:15.720 Hi. Future Tom here, just interrupting to say that 60 00:03:15.720 --> 00:03:18.700 after this video was finished and done and all the graphics were completed, 61 00:03:18.700 --> 00:03:22.080 I found out that some had published a new version of GPT-2, 62 00:03:22.080 --> 00:03:25.000 which is the leading machine-learning text-generation system, 63 00:03:25.000 --> 00:03:26.910 and I figured that if I didn't address that here, 64 00:03:26.910 --> 00:03:28.599 a load of people in the comments would be all, 65 00:03:28.600 --> 00:03:29.760 "Well, what about machine learning? What about AI?" 66 00:03:29.760 --> 00:03:32.400 So I got a version of GPT-2, 67 00:03:32.400 --> 00:03:35.030 it's set up kind of like a Dungeons & Dragons adventure system 68 00:03:35.030 --> 00:03:38.450 I told it to create a suitcase and a trophy 69 00:03:38.450 --> 00:03:41.310 and to put the suitcase in the trophy to see what would happen. 70 00:03:41.310 --> 00:03:42.500 ARTIFICIAL VOICE: "The moment you think about it, 71 00:03:42.500 --> 00:03:44.740 "you know what to do, so you do it with your eyes closed: 72 00:03:44.740 --> 00:03:47.209 "you place your hand on top of it, then with your other hand 73 00:03:47.209 --> 00:03:49.150 "you take out your gun from inside your jacket pocket, 74 00:03:49.150 --> 00:03:51.060 "take aim at it, pull back one inch…" 75 00:03:51.060 --> 00:03:53.739 Yeah, if you think machine learning's going to save us, 76 00:03:53.739 --> 00:03:54.740 uh, not yet! 77 00:03:54.740 --> 00:03:58.140 And of course, sometimes words can switch parts of speech. 78 00:03:58.140 --> 00:04:00.770 "Main" is usually an adjective, but in video games, 79 00:04:00.770 --> 00:04:03.110 it can be a verb or a noun. 80 00:04:03.110 --> 00:04:06.440 Humans who’ve not heard that use before won’t understand it. 81 00:04:06.440 --> 00:04:10.800 The solution for now? Accept that Siri’s not going to be great at complicated questions, 82 00:04:10.800 --> 00:04:13.810 and hire a team of underpaid contract workers to manually tag data, 83 00:04:13.810 --> 00:04:16.820 in order to help programs "learn" patterns. 84 00:04:16.820 --> 00:04:18.970 But those methods will continue to fall short, 85 00:04:18.970 --> 00:04:22.560 because computers are missing the breadth of knowledge that humans have access to. 86 00:04:22.560 --> 00:04:24.920 At least, for now. 87 00:04:24.920 --> 00:04:28.160 Artificial language processing remains 10 years away, 88 00:04:28.160 --> 00:04:30.220 just as it has for the last few decades. 89 00:04:30.700 --> 00:04:34.910 One of my co-authors, Gretchen McCulloch, has a wonderful podcast called "Lingthusiasm". 90 00:04:34.910 --> 00:04:36.590 You can listen to it at the link in the description.