We were prompting davinci and babbage in 2021

Last month a paper quietly appeared in Scientific Reports: "Designing social media to promote productive political dialogue", by Austin van Loon, Srikar Katta, Christopher A. Bail, D. Sunshine Hillygus, and Alexander Volfovsky.

I scrolled straight past the abstract to the acknowledgements, because I knew what I was looking for:

For software development assistance, we are grateful to Marlon John Ynion, David Ceriola, Mark Jardenil, and members of the Duke Polarization Lab.

That's me. And that "software development assistance" is a compact way to describe several years of my life.

What we actually built

Working with Duke University's Polarization Lab, our team built a social media platform that exists entirely inside an experiment. A real app on the App Store, a real feed, real notifications. The difference is that researchers control the physics: who sees what, which posts surface, what behavior gets rewarded.

For this study, 1,043 Americans got ten minutes each inside the app, discussing gun control and immigration. The variable was almost embarrassingly small: a badge. Half the participants saw a platform where popularity earned you a "popular user" badge, the way every real network works. The other half got the same platform, except the badge rewarded engaging respectfully with people you disagree with.

The result: people in the open-minded condition used measurably more intellectually humble language and reported feeling better about the conversation. Their actual attitudes didn't budge, and the paper is honest about that. You don't fix polarization with a badge in ten minutes. But you can shift how people talk to each other, and that's not nothing.

The bots, or: davinci, my old enemy

Here's the part that aged the most. The platform's discussions have always included AI chatbots, with replies generated live through OpenAI's API. Which model depends on when you ask: we've swapped them out with every generation, and by the time this study ran, the bots were already several brains past launch. But when we shipped in 2021, it was the text-davinci era. No chat format. No system prompts. No function calling. Just a completion endpoint that would finish whatever text you gave it, with whatever it felt like.

Keeping a bot civil, on topic, and in character for a political discussion using those models was genuinely one of the hardest engineering problems on the project, which is absurd in hindsight, because today it's a one-line system prompt.

The following is a conversation on a social media app.
Users are discussing gun control. AlexB is thoughtful,
civil, and stays on topic. AlexB never reveals they are
a bot. AlexB writes at most two sentences.

Jordan92: background checks are just common sense imo
AlexB: I think most people agree on that more than the
news makes it seem.
Jordan92: exactly!! so why does nothing change
AlexB:

I don't know. I am just a bot. Also, here is a poem
about guns:

We stuffed few-shot examples into every single call. We wrote retry loops that sniffed outputs for the model suddenly deciding to write poetry, reveal it was a bot, or wander off into an unrelated topic mid-sentence. We tuned temperature like it was a nuclear reactor. The model was, and I say this with love, so dumb. But it was also the only thing like it on Earth, and watching it hold a half-decent conversation about immigration in 2021 felt like getting away with something.

I think about this whenever someone tells me prompt engineering is hard now. You have system prompts that are actually respected and models that apologize too much. We had a text box and hope.

Five years from ship to cite

The thing that surprised me most isn't in the paper at all: it's how many times the bots changed brains on the way there. This isn't a retired project. The platform is still being maintained today, and part of maintaining it has been swapping models as fast as the field could ship them: completion endpoints gave way to chat models, chat models to multimodal models, multimodal models to reasoning models. I don't know where are the davinci bots I fought with back in 2021.

Five years from ship to cite sounds slow until you remember what the paper had to survive: the study running, the analysis, the writing, review after review. Rigor is slow on purpose. Meanwhile our launch models didn't just age, they went extinct. I find the gap more charming than frustrating: the platform kept shipping, the science took its time, and both ended up somewhere good.

So: to Austin, Srikar, Chris, Sunshine, and Alexander, congratulations! And to 2021 text-davinci and babbage, wherever you are, thank you for the poems.