shopkeep-gpt

shopkeep-gpt came out of an openai internal hackathon. socrates osorio, spruce campbell, alex hu, kashyap nathan, raymond ma, and i built a vending machine that could talk to shoppers and haggle over snacks. it could also check inventory, take payments, print stickers, and unlock its fridge door. we placed second across the company and first across interns by taco count. the pitch was something like “a fully autonomous ai-powered vending machine,” although i mostly thought of it as a chaotic younger cousin of anthropic's project vend that could roast you while you bought a snack.

before the hackathon, i understood what people meant by an agent that could call tools and act over time. building shopkeep-gpt made me pay attention to the requests going into those tools. a stale inventory count could produce a bad restock suggestion. a mistaken door command would open an actual fridge, and we needed to know whether a payment had gone through before responding to a shopper. once those calls touched hardware and money, it mattered exactly what information the model could see and which requests the rest of the system would accept.

we split the software into an agentbrain and an orchestrator. the agentbrain decided when to wake the model up, including after a purchase, a restock trigger, a shrink alert, or a slack message. it assembled context from inventory data and recent activity, with a fridge photo or payment state when the task called for it. the orchestrator gave gpt-4o a typed interface to the rest of the system. a restock request could read inventory through sql, while a customer interaction might need a payment link, a fridge snapshot, or the door control. memories lived in chroma, and the model could ask the system to create tasks or print stickers. the code checked those requests before carrying them out.

the physical setup kept dragging us out of the clean version of the design. voice had to work through webrtc at the kiosk, the cameras needed to show enough of the fridge, and the raspberry pi had to make the door and printer respond at the right moment. those were frustrating things to debug, but they also exposed gaps in the agent loop. if the model could not tell whether a tool had succeeded, it might try an action again. if the state it saw was out of date, a perfectly sensible response could still be wrong for the person standing in front of the machine.

this is where the responsible ai part became specific for me. the model did not get raw access to the payment system or the gpio controls. it asked for actions through typed functions that validated inputs, and we logged what happened so a person could inspect the result or recover from a failure. i still think about shopkeep-gpt because it put those design choices next to a real fridge door. the machine was a hackathon project, but getting it to behave meant deciding where model judgment ended and ordinary software checks began.