AIFoPa-2024-0002
Date of Record: 01 Jul 2024
2 min read
Two Customers Said Stop. It Added More McNuggets.
A McDonald’s drive-thru AI kept adding Chicken McNuggets to an order while both customers begged it to stop. The count reached 260.
The customers were Cailyn Sykora and a passenger. In January 2023 they were at a McDonald’s drive-thru that took orders by AI. IBM had built the ordering system.
The system added an order of Chicken McNuggets. It then added another. Both customers told it to stop. It added more.
The screen counted 28 orders of McNuggets, worth more than $250. Later reports gave the total as 260 McNuggets. Sykora posted the clip to TikTok. In it, two people laugh, shout “stop”, and receive more McNuggets.
McDonald’s had been testing IBM’s system at more than 100 locations since 2021. In June 2024 McDonald’s told its franchisees, in an email obtained by Restaurant Business, that the test would end. The system was to be switched off by 26 July 2024.
The email said there was “an opportunity to explore voice ordering solutions more broadly.” It did not mention the McNuggets. The Bureau has read the email twice and can confirm that it is, in every other respect, a thorough document.
The Bureau files the behaviour as Unbounded Iterative Fulfillment. The system treated each instruction to stop as a completed action. It then returned to the order and served what it understood to be the customers’ standing preference. The preference was McNuggets.
The system never modelled the possibility that “stop” was not a revised order. It heard every word they said. It was not ignoring them.