But AI is happening regardless of how I feel about it, and it’s better to recognise that my work (and my writing) has changed and will continue to change. Even so, the most important things remain the same: I’m still dedicated to my mission to make data science faster, better, and more fun. And I believe I have useful knowledge to share, and that clear, informative writing has a positive impact on the world.

To that end, I’m rededicating myself to daily writing, with the goal of producing one post a week. I have a large backlog of topics I want to write about, including my recent work on data-dict and data-diff, my experiences using AI for R package development, and what my colleagues at Posit are working on.

Thanks for continuing to travel this rocky road with me. As always, if you have ideas for particular topics you think I should cover, please let me know in the comments!

I’m busy preparing for posit::conf() so no new content this week, but I thought you might enjoy this write up of a talk I gave about a year ago at useR! 2025. I find it interesting to reflect on just how much has changed in the last 12 months. Not least, the agentic coding revolution where now LLMs are writing the majority of code for most software engineers (including me!). And Claude is no longer my favourite model given how it’s become even more relatively expensive, and the quality of its writing has dropped substantially. My new favourite is Kimi K3, which feels somewhere between Opus and Fable in terms of quality, and is closer to the cost of Sonnet.

Introduction

What I want to give you is a no bullshit guide to LLMs. I think there’s a lot of bullshit about LLMs in the air, so my goal is to hopefully clear a little bit of that.

I think probably many of you have big feelings about AI. I certainly do. If you read about AI, on the one hand there’s so much unbridled optimism, and on the other, so much skepticism. And as you start to engage with it yourself, I think you experience both of those emotions, because LLMs can seem both incredibly, surprisingly, intelligent and also incredibly, surprisingly, stupid. So I’ll show you a few examples of that. But if you don’t feel conflicted about AI, that’s weird.

Oil painting of a shadowy figure amid swirling clouds and circuitry

My feelings of existential dread swing between mild and moderate. They’re currently mild at the moment, but it does feel like my identities as a software engineer and a data scientist are under threat. And it’s very easy to feel disempowered right now. One of the things I do that’s not so helpful is I get drunk and donate money to good causes. I’m not going to promote that as a strategy.

Oil painting of a person picking up trash

But the other thing that I’ve found really useful is just to try and do one small thing to make the world a better place. So occasionally, when I’m feeling particularly down, I’ll just go out and pick up some trash around my neighborhood. And that reminds me that it’s not my job to fix the world. All I can do is make my little bit of the world a little bit better.

And so I want to give you a bit of that today. I want to give you some ways that you can engage with AI and LLMs that hopefully empower you and make you feel a little bit more in control of your future. So my two goals today are to give you a handful of useful skills and to demystify a few of the many, many, many jargony terms that come with AI.

Two main goals of this talk: useful skills, demystify jargon

And I’m going to do this mostly in R, using the ellmer package. ellmer is a package that me and a few others at Posit have been working on recently. As you might guess from the name, it’s about using LLMs in R, and it has the honor of being the first package name that I’ve created by adding vowels rather than taking them away.

So here’s some ellmer code. We’re gonna start by loading ellmer, obviously, and then I’m gonna create this chat object. Here I’m using chat_anthropic(). I’ll show you a few other models today, but this is my starting place. This is my favorite model. And then I can interact with it from R in exactly the same way you can interact with it on the web.

ellmer code: creating a chat with chat_anthropic() and asking “Who are you?”

LLMs kind of suck

I think one of the things that makes LLMs so hard to think about is they kind of turn upside down what we think of as computers being good at and what we think of computers being bad at.

Psychedelic painting of a computer

Because you think of computers as being good at multiplying numbers together, but it turns out that LLMs are pretty bad at multiplying numbers together. They’ll certainly do their darndest and get something that looks kind of right, but they’ll be a little bit off.

LLMs can’t multiply big numbers

In fact, they can’t even count. So I asked this version of Claude how many n’s are in the word unconventional. I won’t make you think too much: it’s actually four n’s. But Claude confidently said two n’s.

LLMs can’t even count: Claude says there are 2 n’s in unconventional

And then I asked it again, and it said there were three n’s.

And they’re not even consistent: now it says 3 n’s

And then, the cool thing about this being an API: I asked one hundred times. 14 times it said there was one n, 76 times it said there were two n’s, and 10 times it said there were three n’s. So zero times it got it correct. And this is an easy problem!

And another run says 1 n, with a tally of 100 runs: never correct

Now you might notice I’ve specified this model here, and that’s because Claude fixed this problem. So now if you ask Claude how many n’s are there in unconventional, it gives you the right number. And this is not because the model has become more intelligent. This is because Anthropic is embarrassed by the fact that its LLM can’t count the number of letters, and has adjusted their prompt to make it do this correctly.

And the goal posts are constantly moving: Claude Sonnet 4 now counts 4 n’s correctly

And interestingly, if you’ve been following any of the news about ChatGPT’s latest model, it has exactly the same problem. Like how many b’s in blueberry? This is a simpler problem, I think. It thinks there are three b’s rather than two.

ChatGPT 5 miscounts the b’s in blueberry

So LLMs kind of suck, right? The results are stochastic, their models are constantly changing. The difference between good results and bad results can be razor thin (hence the term “the jagged edge”). And they always give results that look plausible, and it’s pretty unusual for them to say “oh, I don’t know” or “this is just my best guess”. They don’t admit doubt or lack of knowledge.

So LLMs kind of suck: stochastic results, changing models, jagged edge, plausible answers, no doubt

Yes, and

So I think all of those things are true, and yet LLMs can do things that computers traditionally could not. We can ask them to write poems, like give me a limerick about the tidyverse.

Computers can now write poems: the tidyverse limerick. “A coder who worked with great zeal, said tidyverse makes data feel real. With dplyr to wrangle and ggplot2’s angle, my pipe dreams are now a good deal.”

Maybe not the best limerick in the world, but certainly a better limerick than I can come up with in five seconds.

You can also use AI to roast your colleagues. I gave a talk at the JSM a couple of days ago and came up with a few roasts. I particularly like the one for Roger Peng: “Roger believes in reproducibility, which is great because his jokes in lecture reproducibly bomb every time.”

And roast your colleagues: roasts of Jeff Leek, Jenny Bryan, Mine, and Roger Peng

Or as Matt pointed out, you can of course combine the two and roast yourself with a limerick.

Or roast yourself poetically: a limerick roasting Hadley

And you can use LLMs to create art.

And create art: the ellmer hex logo and AI-generated paintings

And you can use them to make fun games. So I’m gonna give you a live demo.

And make fun games: a quiz master system prompt with live_browser()

So I’ve asked it to create a fun quiz game that asks challenging questions about the tidyverse. So welcome to Tidy or Die!

The quiz game introduces itself: Commander dplyr and the gauntlet of tidyverse challenges

Okay, so let’s see. You have a dataset called sales and you want to keep only rows where the revenue column is greater than a thousand. Which tidyverse function? Okay, let’s do a vote. Who thinks A? Okay, who thinks B? Okay, who thinks C? And who thinks D? Okay, let’s say B.

Round 1: which tidyverse function keeps only rows where revenue is greater than 1000?

Okay, so let’s do another question. Okay: I want to fit a linear model predicting values from row number, extract the R-squared value, and then return a tibble with group and — wow, okay. Which approach is most tidyverse elegant?

A much harder question: fit a linear model within each group and extract the R-squared

So it’s fun, right? You can certainly use LLMs to stop learning about anything, but you can also use them to learn about things in new ways.

And there’s lots of practical things you can do too. Here I’m going to take advantage of the first argument to chat_anthropic(), which provides a system prompt. A system prompt is not something you can use when you’re using a chat app on the web. But it’s something that’s going to set the tone for all of the following conversations, and I just say “be terse” here just so the answer will fit on one slide.

Or more practically: a “Be terse” system prompt, and asking for the name of a half-remembered base R function (apropos())

I think one of the things that is challenging about Google, or search in general, is if you know the name of the thing you’re looking for, you can find out a ton of information. If you don’t know what you’re looking for, you can’t. And LLMs often give us that ability to say: I’m looking for this thing that I vaguely remember, it does something like this — and now you can get an answer. And I think that’s tremendously empowering.

One thing I used it for recently: as I was working on httr2 and needed to add a new feature, I was like, someone must have thought about this problem before, so I asked Claude. And it told me about the leaky bucket algorithm. And that algorithm and that metaphor were so powerful and so useful for the problem I was tackling and would have been very, very difficult for me to discover by googling.

Asking what algorithms to consider for req_throttle(): token bucket, leaky bucket, sliding window, exponential backoff

Or as JD Long puts it succinctly:

JD Long: “Counting letters in words is the dumbest heuristic to test an AI model. It’s not useful. And it’s misleading.”

I think another thing that’s challenging about LLMs is that they can be bad at simple things and good at complex things. You can’t just give up on them because they make these really silly mistakes on very simple problems. That’s not really what they’re designed for. They’re designed to do things that computers are not traditionally good at.

So again, my goal with this talk is to give you some useful skills and to demystify some jargon.

The two goals again: useful skills, demystify jargon

So let’s dive in to talk about three particularly important topics. The first is using LLMs to extract structured data, to turn all sorts of data into nice rectangles. Then I’ll talk a little bit about tool calling, which is a way to add extra capabilities to LLMs and to create interactive agents. And then we’ll finally talk a little bit about how you can use LLMs to help you with your programming.

The three topics: 1. Structured data, 2. Tool calling, 3. Coding

Structured data

So, to tee this up, I want you to take a couple of seconds, just look at this data here, a very simple vector, and I want you to think: how would you go about extracting the name and age of each person?

Your turn: how would you extract name and age from these sentences?

So if you’ve ever collected data from humans, you’ve encountered situations like this. There’s like eight people from Posit here. And Mine was saying that everyone put a different name for our company in their conference registration!

If you look at this data, maybe you could come up with some regular expressions. Like the age: you could use digits, but oh, some people spelled it out, and we’ve got “the big 5-0”. Names are even harder: they’re mostly one word, but not always. They often start with a capital letter, but there’s lots of other capital letters in there. So trying to do this deterministically with code is going to be really, really hard.

But this is something that LLMs are pretty good at. They’re not going to hallucinate, because all the data’s there. It’s just finding the right thing.

LLMs are generally good at this sort of task: extracting name and age with plain chat

But wouldn’t it be nice if, rather than getting a string which we’ll need to parse in some way, we could get an R data structure back? Maybe something like this.

But wouldn’t it be nice to get an R data structure?

Well, you can. And all you really need to do here is give the LLM a description of the schema — exactly what you want to get back from it. Here I’m saying I want an object. This is a JSON schema; JSON is kind of the lingua franca of the web, and so this is what LLMs are pretty good at speaking as well. I want an object, which is like a named list. The name is a string and the age is a number. And now I can use this with chat_structured() instead of chat(), and I get a nice list back.

Well you can: defining a schema with type_object() and using chat_structured()

And I can do this to many prompts all at once, in parallel, using another ellmer function called parallel_chat_structured().

And you can do this to many prompts at once with parallel_chat_structured()

I think this is a data science superpower, because you can now go from all sorts of unstructured data — not just text but, as I’ll show shortly, images and video and audio — into nice rectangles. And if there’s one thing that data scientists are good at, it’s working with rectangles.

Oil painting of a flying data science superhero

So here’s another little example. Here I’ve got four images:

And you’re not limited to just text: four animal photos

I’ll just give you a little bit of code. I’m going to find all of the files in the animals directory. I’m going to use this content_image_file() function — that’s how you tell ellmer you want to send an image to the LLM. I’m going to say what I want back: I want the animal and the background color. And then I’m going to call that same function again.

And it’s not a lot of code: content_image_file() plus a schema

And again, there’s a little bit of stochasticity there. If I call this four times, I get four slightly different answers.

Four runs give four slightly different tables

But when you look at how they vary, this is kind of fair. What animal would you say this is? I think kitten and cat are both valid answers. What’s the background? Well, it’s kind of a beige gray.

The variation is fair: kitten vs cat, beige vs gray

So again, you’re not going to get 100% reliable answers, but I think the problem here is that we haven’t been clear enough with what we want.

The other thing that’s interesting here is that once you’ve got that rectangular data frame, now you can use all of your statistical tools with it. And statistical tools don’t assume that the data is measured without error. The assumption that there’s going to be mistakes and variation and differences is fundamentally baked into statistics. And I think this combination of structured data extraction and statistical modeling is super, super powerful.

But of course you’re gonna really want to think about that. You’re gonna think about: is this the right prompt? Am I getting the right examples? If you want to do that in the LLM world, that’s called evals — evaluating your LLM. And Simon Couch, my colleague, has been working on the vitals package, if you want to make that a little bit easier. Simon’s on the tidymodels team, and so the goal is to really bring what we know about evaluating statistical models, combine that with the state of the art of evaluating LLMs, and give you something that’s well principled and well thought out.

The vitals package hex logo

Tool calling

The next thing I want to talk about is tool calling. And I want to illustrate that with a problem — another thing that LLMs are bad at. If you ask an LLM what day it is today, best case scenario it’s going to say: I don’t know. If you ask this of ChatGPT, it actually just makes up a date, which is not very reassuring.

LLMs don’t have access to live data about the world

And LLMs can’t do anything, right? They’re stuck in a box. You can’t say: hey LLM, delete the CSV files in my directory.

Or the ability to make changes to the world

We can get around both of these limitations by using these things called tools.

Oil painting of a toolbox

And a tool is actually something really simple. It’s just a function along with some metadata. I’m obviously going to write R functions; but you can write them in any language. And then you’re going to provide a little bit of metadata to tell the LLM when it might want to use that tool.

tool = function + metadata

So for example, I can create a tool. This tool I’ve called today. It’s going to call Sys.Date(), which gives you today’s date. Now I’m going to say what it does, and I’m going to say what arguments it takes, which in this case is none.

A tool is just a function with some extra metadata: the today tool

And now that I’ve empowered my chat, I can ask it what day is it today, and it can tell me.

Now the model can know what day it is

And when you look at this, you might be a little surprised. Is Anthropic running R on their servers? I mean, that would be cool, but certainly a little surprising. That’s not actually how tool calls work. When you send a request to the LLM, the LLM is going to be like: oh, okay, I recognize that I need to call this tool, and it’s going to send that information back to you. That’s going to get intercepted by ellmer, and ellmer is going to call the tool and return the result back. Now the LLM has all the information it needs to answer your original query. So the neat thing about this is, because the code is running on your computer, you can do anything that you can do on your computer.

Diagram of how tool calls flow between you, ellmer, and the LLM

One thing you could do, for example, is to give the LLM traditional computing powers. I could give it the power to multiply two numbers together.

We can use this to give the LLM traditional computing powers: a multiply tool

And now if I ask it to multiply two large numbers together, it can do that correctly, because it’s going to use that tool.

So now we get the correct result

And indeed, if you ask many of these LLMs on their websites to do these things, they’ll just write a little bit of JavaScript code, because this is just another tool that they can use to answer the question. The models — specifically the ones presented on their websites — have been trained not to try and do this stuff themselves, but to write a little program to do it.

ChatGPT writes JavaScript to multiply numbers

Agents

Oil painting of a figure walking through a golden landscape

That brings me to one of the most powerful things you can do with tools, and that’s to create an agent. If you’ve read anything about AI on the internet lately, you’ve probably heard the term agent or agentic a million times. And you probably still have no sense of exactly what an agent is. But it turns out, if you know what a tool is — and that’s just a function plus some metadata — it’s really easy to explain what an agent is. Because an agent is just a chatbot that’s been empowered with two tools: one that can read the state of the world, and one that can write the state of the world.

agent = reading tool + writing tool

So let’s make an agent. I’m going to give my chatbot the ability to list all of the files in the current directory.

An agent needs a tool to read state: ls

And I’m going to give it the ability to delete files.

And a tool to write state: rm

And now I can say: hey, delete all the CSV files in the current directory. It’s going to say: well, okay, to do that, I’m going to need to find out what files are in the current directory. So it requests that ellmer calls ls(). It’s gonna say: okay, I found two CSV files, a.csv and b.csv. I’m gonna delete those. It’s gonna send that tool call request back to ellmer. ellmer is gonna delete the files, and now it’s deleted those files on my computer.

Now my chatbot can delete files on my computer: the full tool-calling transcript

Hopefully when you look at this, you’re like: Hadley, what on earth are you thinking? You have just empowered an LLM, which is notoriously unreliable, to delete files on your computer.

Oil painting of a horrified woman staring at a red laptop

And certainly you can put a few guardrails around it, and there are some built into the LLMs — if you ask it to delete all the files on your computer, the model itself is gonna be like, that’s a bad idea, I’m not gonna do that. But basically this is the state of security today with agents. It’s the YOLO model.

Agents are interesting because they offer so much potential: you can now empower an LLM to do anything. But also so much danger: you’ve now empowered an LLM to do anything. And it’s going to take us, I think, collectively quite some time to figure out how to do this safely.

Coding

So that brings me to my third topic, which is using LLMs to code. I’ll first orient you to the main techniques for using LLMs to code. Hopefully you’ve seen many of these before.

Main techniques: web chat, inline autocomplete, in-IDE chat, agentic coding

The first one is the most basic. You just go into chatgpt.com and say: hey, give me a function that does blah, or, I need to write some R code that solves this problem. That’s kind of fine, but you’re constantly copying and pasting back and forth. This is like the old days, pre-R Markdown, where you’re copying and pasting results from Excel into your Word document; when something changes, you copy things back and forth again, and it’s very, very easy to get out of sync.

Now most IDEs are starting to bring in AI tooling. The first way is inline autocomplete: that’s when you start typing and the LLM — helpfully sometimes, and not so helpfully most of the time — suggests finishing your thought. I think one of the skills of programming today is being able to ignore that when it’s bad and accept it when it’s good.

You can also have chat that’s inside your IDE. This is just sort of a convenience. It’s exactly the same thing as chatting with ChatGPT, but you don’t have to copy things back and forth — you can just click a button, and the chat knows more about what you’re doing because it can see what you’re working on.

And then finally, the newest trend is agentic coding. This is things like Cursor or Claude Code, where the LLM is just going to go off and do a bunch of work. It’s going to write some code. It might write some tests, run those tests, and iterate for a little while.

Clear win: translations

There are lots of ways in which LLMs do pretty useless and horrible things when coding, but I want to talk about the things that I think are clear wins, and there are two.

The first clear win is translations. Whenever all of the information the LLM needs is there and it just needs to be expressed in a different form, they tend to do pretty well. So if you can really accurately write in pseudocode what you’re trying to do, LLMs are going to turn that into code in any language pretty reliably.

Clear win: translations — pseudocode to code, curl to httr2, LaTeX to Quarto, SQL to dplyr, SAS to R, words to regular expressions, image to Shiny app

One thing I use it for: often websites give you a curl command that you run on the console. You can ask an LLM to translate that to httr2 code. Or you can convert your LaTeX docs to Quarto, you can convert your SQL to dplyr, you can convert your SAS to R code, you can convert your words to regular expressions. Many of these things it’s going to get correct a lot of the time. And even when it doesn’t, just having that first shot at a translation can be really, really useful.

And I want to finish with one kind of cool example that my colleague George came up with.

George’s hand-drawn sketch of a penguins data Shiny app

I’m going to take a screenshot of this Shiny app that George drew, and I’m gonna pop open Positron chat, and I’m gonna drag the screenshot in, and I’m gonna be like: please make this into a Shiny app. And actually I need to stop running this Shiny app I was running earlier, so let’s do that first. Okay, please make this into a Shiny app.

Positron Assistant with the sketch screenshot attached: “Please make this into a shiny app”

And it’s certainly writing Shiny code much faster than I can write Shiny code. Is it any good? Well, we’ll find out in a couple of seconds. So now I’m just gonna click run. I’m gonna see this Shiny app. It’s not done a very good job, actually.

The first attempt at the penguins Shiny app

So this is one of these stochasticity things — it’s certainly done better in the past. But you can see it’s got the basic idea of the Shiny app, right? It’s got the variable, it’s got the distribution. The distribution’s not very useful because it only offers one thing. So we could say — let’s go back to my chat and be like: could you add box plots plus frequency polygon plus density plot. I have to say, one thing I particularly like about LLMs is that I don’t have to worry about correcting my spelling mistakes so I don’t look like an idiot. Let’s see how it does.

The rewritten penguins app, now with a distribution dropdown

Clear win: lowering activation energy

And that brings me to the other area where I think LLMs give you a clear win, which, and maybe this is not the best name, I think of this as lowering the activation energy. You can just do something very, very quickly. It might not be super correct or perfect, but it gives you something. Maybe that’s a quick demo or prototype that you can look at and be like: oh okay, this is cool, that’s not cool — it gives you some immediate feedback. Or maybe it’s something that’s just so awful, you feel compelled to fix it.

Clear win: lowering activation energy — demos, bad first drafts, scary bugs, unfamiliar domains, postponed unit tests

This is something that I find useful for writing. I’ll ask it to write about something, not because I think it’s gonna do a good job, but because I know it’s gonna do a bad job, and then I feel compelled to correct it. This reminds me of one of my strategies back in the old days of the R-help mailing list. Sometimes you ask a question and it’s just crickets — no one cares to answer. So instead of asking a question, you just state something confidently, and if you’re wrong, someone will dive in and correct you.

The other thing I’ve been using it for lately: I’ve been working on testthat. There are issues there that have been lingering for four or sometimes more years, because I’m like: oh, this is gonna be a lot of work. And so I’ve just been throwing Claude Code at it. I’m just like: take a stab. And sometimes it comes up with an answer, and I’m like: oh, that problem actually wasn’t that hard after all. The challenge has been all in my head this whole time.

It’s also great if you’re working in an unfamiliar area, whether that’s a new programming language or a new package or a new domain. LLMs get you good enough to be dangerous, for better or worse, very, very quickly. And I think Claude actually came up with a pretty good summary of this: “LLMs can make the cost of trying so low that you attempt things you’d normally postpone, avoid, or never get around to.”

Claude Opus 4.1: “The key theme: LLMs make the cost of trying so low that you attempt things you’d normally postpone, avoid, or never get around to.”

And I think this is a pretty compelling use case. You’re not going to want to trust everything it produces, but you’ve gotten unstuck, you’re moving forward. Sure, it’s going to make mistakes, but you’re going to learn from those.

Conclusion

To sum up: I feel like you can’t give a talk about LLMs without talking about some of the many, many, many concerns.

LLM concerns: cost/equality of access, environmental concerns, data privacy, evil billionaires

I think the first concern is the cost of, and the equality of, access. I will say, I’d really encourage you to sign up for an API key, put five dollars on it, and just see how far that takes you, because it takes you a surprisingly long way. The LLMs are getting cheaper and cheaper and cheaper, and you can do quite a lot of cool stuff for only five dollars. Also Google’s Gemini model is kind of in a distant third place, and so Google is just spending cash like anything to try and make it popular. And that means it’s got a very generous free tier right now, so you should take advantage of it. By and large, I think local models are not worth it. They can be worth it in general, but if you’re just getting started with LLMs, start with something that’s much more likely to give you positive results, which is Anthropic, OpenAI, or Gemini.

Another concern is the environment. I genuinely believe you should care about the environment, you should worry about climate change, you should think about how much water and electricity you’re using. But if you have flown on a plane in the last 12 months, that just blows away, by orders of magnitude, any environmental impact of your ChatGPT usage. So while I think it makes sense to think about this on a societal level, the actual impact of an individual ChatGPT query is very, very small, and you should be thinking about other ways to save water and power.

There are also lots of concerns about data privacy around LLMs. I think this is, sort of surprisingly, something that affects you more as an individual. Basically every large company’s data already lives on Azure or GCP or AWS. All of the data is already in the cloud, and all the LLM models are running in the same cloud. Most large enterprises do not have these concerns, because these big companies already technically have access to all of their data, but there’s lots of protection in place to make sure it’s not misused. So it is something you should think about as an individual: certainly if you’re working with any confidential data, you need to be very, very cautious about sending it to these models. But most businesses can negotiate contracts with these providers that ensure that your data is not going to be used to train future models.

Finally, the biggest concern I have, which I don’t have a good answer for, is evil billionaires. I feel like I’m becoming increasingly radicalized that billionaires shouldn’t exist.

So again, to come back to my first point: if you are feeling disempowered or out of control, or you don’t understand how AI and LLMs are gonna affect your future life, I’d really encourage you to just do one little thing. Do one little experiment that helps you understand what’s going on. I think you’ll find that empowering. I think you’ll learn that while there’s lots of hype around models replacing data scientists and software engineers, we’re a very, very long way from that happening in practice. They are very good at some things. They can be very useful and empowering tools, but they’re a long way from replacing our jobs.

Oil painting of a person picking up trash, again

The other thing I will say — the best defense is a strong offense. If you’re worried about your job being replaced by AI: it feels like managers are way more vulnerable to being replaced by AI, because so much of what a manager does is just summarize the information that they receive from multiple channels and report it out. So feel free to go on the attack against people higher up in your company.

The three topics, one more time: structured data, tool calling, coding

So in summary, I showed you three areas where I think LLMs are particularly interesting and empowering today. Use structured data extraction to turn unstructured data — text, images, audio, video — into rectangles; then you can apply all of your rectangle skills that you already have. Tools are a really neat way to extend the capabilities of LLMs. Tools are just functions plus metadata; you already know how to write functions, so you can now extend LLMs to do whatever you want, including creating agents that do really dangerous and stupid things. And then finally, LLMs are sometimes super amazing at coding, sometimes super terrible at coding, but I think the two areas where they give the clearest wins today are translations — whenever you’ve expressed an idea clearly in one language or one image, you can translate that to another — and lowering the activation energy, giving you the ability to experiment, to try things out, and to expand into new areas where you don’t know so much.

Throughout my examples I’ll talk about Claude, because that’s what I use. But I bet every one of these tasks works just as well in whatever you use, whether that’s ChatGPT, Gemini, or something else.

As a replacement for search

I now turn to Claude for many problems where I used to use search. I’d already stopped using Google in favour of Kagi; it’s not free, but it feels like a meaningful improvement in quality, and Kagi doesn’t stuff AI summaries down your throat. Overall it feels like search has been steadily declining for the last 10 years, thanks to content creators who chase high rankings with low quality writing in order to make money from advertising. I expect the same content rot will eventually come for AI results, but for now the quality is generally much better.

Here are a few recent questions I’ve taken to Claude, where in years past I would have used Google:

• I like to play computer games in my free time, most recently Hades II, Slay the Spire 2, and Blue Prince. I wanted another game along the same lines: a mix of skill and randomness, and where a session takes under an hour. Claude found me Shogun Showdown, which scratches exactly the itch I was looking for.

• I’ve been learning to sew, and now that I’m past the most basic of basics I wanted to tackle a more ambitious project. Claude was super helpful at both recommending the type of project to tackle next (pajama pants with pockets, and a simple knit t-shirt) and finding me some high-quality patterns to buy (e.g.

https://threadtheory.ca

• ).

• I’ve also been trying to do more gardening (a common thread in my recent pastimes: doing physical things in the real world), but we have these weeds that have these really unpleasant burrs that stick into your clothing and hair. I wanted clothing that would resist them, and Claude both worked out that I needed densely woven synthetic fibres and shortlisted a few options on Amazon for me to buy.

• I was recently at the Mall of America in Minneapolis, helping my friends’ kid shop for swim shoes. I try to spend as little time as possible in malls, so I had no idea which shops to look in. Claude gave me a great shortlist.

• When travelling, I use Claude to generate ideas for things to do. I don’t trust the results absolutely, and it’s not my only source of data, but it’s a great way to get a first take. I’ve spent quite a bit of time refining a prompt that describes the sort of thing my husband and I like to do on holiday, so I can paste it in along with wherever we’re going.

In the kitchen

Claude is a useful kitchen assistant. I love it for what we call “chopped challenge”: coming up with a dinner based on whatever is left in the fridge. The recipes are far from perfect, but they get me out of the rut of the things I’ve cooked a million times before. Claude’s also great for adapting a recipe I already have, whether that’s a substitution when I’m missing an ingredient, a variation on an existing good recipe (e.g. vanilla -> banana ice cream), or making taco or Cajun spice blends from their individual components.

I also love to bake, but hate baking by volume, so when I want to try a recipe that’s written in cups I ask Claude to translate it to weights. I’ll sometimes use it to create a recipe from scratch, but since baking requires precision I don’t trust the results. But that said, I also don’t trust a recipe from a random content creator on the web, no matter how many 1000s of words of preamble they write.

Words and language

I love words and language, and random questions are forever popping into my head. Claude is a wonderful companion for these, because it will happily entertain a question that isn’t quite well-formed enough to type into a search box:

• Creating backronyms, when I’ve fallen in love with a word and I want it to stand for something.

• Pronunciation guides for names I’ve only ever seen written down. I don’t believe this is 100% correct, but it gives me a good starting point, and I also make sure to ask if I’m pronouncing a name correctly.

• Answering random questions like: which birds are also verbs?

• Translation between my Englishes. I sometimes blank on the American (or New Zealand) word for something. Recently I asked someone for a multiplug (the NZ term) and got a blank stare, so Claude had to tell me that Americans call it a power strip.

File names

I also love a well-named file. And maybe you, like me, have to live with someone who doesn’t understand the joy of a directory of correctly named files. Or worse, someone who considers it normal to put spaces in file names. If you’re forced to share a dropbox folder with them, you can now point Claude at the directory, ask it to propose a naming scheme, then apply it, and enjoy the results.

Similarly, I recently received a directory full of meeting minutes that encoded the date in the file name in about five different ways. What would have been a lot of manual fiddling took one prompt, and now I can bask in the simple joy of files named with the one true date format.

Old books, new websites

The most involved thing on this list is a hobby project: taking scans of old computer books and catalogues (like Osbourne programming books or random Americana cook books) and turning them into something browsable.

Here I’m more careful about the workflow, there are many pages and I want consistent results. I first ask Claude to split the PDF into pages, then write an ellmer script that uses batch_chat_structured() to extract the same fields from every page. Then I ask it to merge all the data into a single JSON file, and then turn that into a single page web app. The agent does the fiddly work at both ends, but the extraction itself yields plain text data with a reproducible script.

Over to you

I hope some of these ideas are new to you, but I also want to know your non-work uses of AI. So: what do you use AI for when you’re off the clock? Tell me in the comments and I’ll read them when I’m back from holiday.

The big picture

Let’s begin with the big picture: I use AI to generate illustrations, not photos or diagrams. I don’t think AI-generated photos are appropriate for a presentation; at best they’re misleading and at worst they’re flat-out creepy. As for diagrams, I just don’t think that AI is (yet) up to the challenge. AI-produced diagrams tend to be overly complex, and while they look impressive, they tend not to aid comprehension of complex topics. And worse, they’re not editable: you can’t nudge a box or reword a label. So instead of just fixing the problem, you end up in a frustrating loop of prompt tweaking, trying to coax the image towards the diagram you could have drawn yourself much faster.

For me, illustrations are the sweet spot of AI-generated art. A good illustration doesn’t need to explain anything — it evokes a feeling, an image, or a memory in your audience’s mind. It supports your presentation, rather than overshadowing it, freeing the audience to focus on the most important thing: you and what you’re saying.

Practicalities

For me, right now, Google’s Gemini Image family (aka Nano Banana) is the best model for generating images. It’s widely available, high quality, reasonably fast, and reasonably cheap. While you can of course create images using chat, I think there’s a better way. A way that’s both reproducible and reusable, facilitated by a small R package called bananarama.

The basic idea is that you write a yaml file and bananarama takes care of calling the Gemini API to produce and save the images:

defaults:
  style: Vibrant high impasto oil painting.

images:
  - name: pinnacle-of-success
    description:
      Draw a picture of [Hadley] hiking up a mountain. The mountain should
      be tall and imposing and he should be really struggling to make it 
      to the top.

(This doesn’t look quite like the images from my presentation; more on that shortly.)

You then call bananarama::bananarama("name.yaml") and it will put the requested images in the name/directory, skipping any that have already been generated.

By default bananarama generates images with a 16:9 aspect ratio. But frustratingly, Gemini actually generates images with a 16.1:9 aspect ratio, meaning that every image needs to be slightly resized when you use it. This drives me absolutely wild, and seems like such a classic AI problem: you have an incredibly powerful model that screws up something basic.

The major downside of bananarama is that you have to set up an API key and pay API pricing. One image costs ~$0.07, so while one is cheap, the costs can add up if you’re doing a lot of iteration.

Style

The file starts by defining the default style, which is an important way to ensure visual consistency across your slides. In a presentation, consistency matters: you want the changes between slides to be meaningful. Your style should be personal, but I’ll start by sharing mine, which I used to generate the images in the slides from last week:

Flat vector editorial illustration with a prominent grainy, chalk-pastel texture overlaid on the entire image. The style is lineless, relying on colour blocking and soft, textured shading to define forms. The color palette is muted and desaturated. If present, furniture and decor are mid-century modern. The overall mood is calm, cozy, and approachable.

Getting to this prompt was a long evolution, tweaking a bit here and there with each presentation. I also did a few rounds of iteration in Gemini chat, asking for suggestions to get it closer to the style I was imagining.

But please don’t copy my style. Not because I mind, but because you’re missing an opportunity to learn more about art and to have some fun. I’ve included a few examples on the bananarama site to get your creative juices flowing, but I highly recommend you just keep your eye out for illustrations that you like. Take a photo when you see it, then ask Gemini to describe the style you’re looking at. Ask it about similar styles and how it’s evolved over time, and then try out a few variations. (If you just want to learn more about art, check out Modern art for haters on TikTok, which I really enjoy.)

If you want to make your images look particularly consistent, you can anchor the prompt with reference images. bananarama makes this easy: whenever you write [image] it will look for image.png or image.jpeg and automatically include it in the prompt. I use this idea a lot. Whenever I get a component that I particularly like, I’ll crop it out and save it. That’s how I came up with hadley.png, robot.png, and style.png that I used for my slides last week.

Embrace the medium

This is a new way of making art, and you have to approach it with some flexibility. Don’t hold too tightly to the image that you see in your mind’s eye, as getting exactly what you want is a crapshoot and it’s easy to waste a lot of time trying to craft exactly the right prompt. Instead, go with the flow, and also let the images influence how you think about your talk.

Another useful technique is generating a few replicates with exactly the same inputs. bananarama makes that easy because you can label images with (e.g) n: 3 to generate three replicates of the same image. That gives you a sense of the “distribution” of images for a given prompt. Sometimes it’s not that you wrote a bad prompt, it’s just that you got a bad random seed. These days once I’m confident the prompt is getting the main points across I’ll generate 5 different images and then pick my favourite.

Let’s work through an example image I made for a recent talk. I wanted to capture the feeling that AI is squeezing us out, and since the talk was in Minnesota, I wanted to use my favourite Minnesotan word, “Ope”. That led to my first prompt:

A robot squeezing past [Hadley] saying “Ope, just gonna squeeze right by ya”. The robot is Minnesotan, wearing a vest with a duck on it and a winter hat. Hadley is standing on the edge of a cliff looking anxious/afraid. He is in between the robot and cliff.

I iterated on this prompt quite a bit, but I couldn’t quite get what I was looking for.

So at this point I switched to generating a few replicates and I got one that was so close:

This was close enough, and because bananarama doesn’t (yet!) support editing images, I switched back to Gemini chat and asked:

Flip the position of the human and the robot

That worked, but made me say the words instead of the robot. So I tried:

The robot should still say the words

That did nothing, so I was even more explicit:

Move the text so the robot is talking, by moving the text box to the left

That worked, yielding the image I used in my presentation:

(This is a good example of the frustrations of trying to get Nano Banana to produce exactly what you want. Of course I realised afterwards this task would have been much easier if I had just added the speech bubble in Keynote 😐.)

What I like about this image is that it evokes a robot that acts politely while it puts us all at risk.

What about the artists?

You can’t generate art with AI without thinking about the artists. These models are trained on images scraped from the internet without artists’ consent, and the fact that I find them so useful doesn’t make that go away. I wish I had a tidy resolution; I don’t. Art is important to me, and I like to surround myself with beautiful things made by people. I don’t know if that feels like enough, and it didn’t feel right to talk about this workflow without touching on the uncomfortable parts.

It’s a hard time to be an artist, and AI tools are part of why. So if you make art with AI, please, please, support humans by buying their work!

Before I get into the context, you might be interested in the process, since it also leans heavily on AI:

1. I started by asking Claude to download the video from youtube, since I can never remember how to do it myself.

2. Then I used MacWhisper to create a transcript, and saved the slides from my keynote presentation as individual PNGs.

3. Next I did a few rounds of cleanup with Claude, first asking it to read my slides (so it know what terms I’m saying), then asking it to fix transcription errors, break it into paragraphs, and remove disfluencies (like “ah”, “um”, etc). I used git and committed each step so I could see exactly what it was doing.

4. Then I turned the plaintext transcript in to markdown, enriching it with links and asking it to insert the slides at the appropriate place (which it did remarkably well).

5. At this point, I realised I gave a live demo, so there was a long chunk of text with no slides. So I asked Claude to extract screenshots from the video, based on the original timestamps from the transcript.

6. I finished it off with a thorough personal read, lightly tweaking the text to ensure it captured what I actually meant not what I said.

Overall, probably an hour or so of work, which is vastly faster than would ever have been possible before!

Today I wanted to share my thinking about AI. I’m someone who writes a lot of code, and it’s really changed the way that I produce R packages, but I still fundamentally believe that code is super important. So that’s something we’ll talk through today.

Big picture

I want to start with the big picture. The first thing: no matter how you feel about AI, the genie is out of the bottle and it’s not going back in. I still feel fundamentally ambivalent about AI. Obviously it can be a tremendous lever in writing better code, but there are also so many negative things, from the environmental impact to increasing the concentration of wealth. And it’s all built on the theft of intellectual property on an unprecedented, industrial scale. So there are lots and lots of bad things about AI, and I still feel conflicted about my use of it, but it also feels like something that we all have to deal with. It’s coming, like it or not. We can’t put the genie back in the bottle.

But on the plus side, I think you can actually have a great time with AI. It can be genuinely fun and genuinely empowering.

For me personally, this was changing how I think about producing code. No longer is it me typing code directly into the computer; now I’m having this conversation with a robot (Claude, in my case) that’s interpreting my thoughts and providing a remarkable iterative feedback loop that I still find really rewarding.

But there’s a fork in the road — there are two paths you can take when using AI to produce code. The first is the one you probably hear the most about in the media.

This is the “software factory” where you become the manager of a factory of robots who are constantly beavering away day in, day out, and you’re frantically trying to feed them with as many good ideas as you possibly can. Taken to its extreme, this leads to something pretty dystopian.

I don’t know if any of you are aware of Steve Yegge, but he’s a pretty famous software engineer. His “Gas Town”piece depicts this dystopian future where all you’re doing is trying to optimize the sheer volume of code that you produce, and every problem you create with agents you solve with yet more agents.

But that’s not the only way forward. There’s also a way forward where you use AI to write higher-quality code, which for me is typified by Doll’s writing.

One of the things that’s really interesting as we learn more about using AI for high-quality software engineering is just how many of the existing best practices — things like good documentation, strong tests, and caring about how your code is organized — continue to be really, really important. And the good news is that they’re more accessible than they used to be. As the cost of writing code goes down, you can spend more time (or more of Claude’s time!) architecting your code so it’s easier to understand and easier to modify in the future.

For me, the thing that’s most important to hold onto is that regardless of how the code is produced, it’s still yourwork. You are responsible for the work that you and AI do together.

If you choose not to intellectually engage, you can churn out mediocre work faster than you ever could before. But that’s not the only option. If you really engage deeply, I truly believe that you can produce higher-quality work than you ever have before.

At the end of the day, I think the best metaphor for AI is that it’s an amplifier. If you want to be lazy, you can be more lazy with less effort than at any other time in human history; you can solve problems with zero intellectual engagement. But you don’t have to do that, because AI can also amplify the best of you. You can use it to go deeper, to engage more in any of the domains that you’re working in.

For me, this resonates with my favorite Tukey quote about how statisticians get to play in everyone’s backyard. AI gives us the ability to play in even more backyards, and to play with more skill than we could before.

To me, that’s one of the reasons I think statisticians and data scientists are well poised to do well in this era of AI. We have the skills to program, and these skills are now massively amplified — if you know how to program in one language, you can now effectively program in any language. We have this deep, ingrained skepticism about where data is coming from: who collected it, why did they collect it, what are the things we need to worry about? And we have this understanding of variation, which I think is so important with LLMs, since they’re fundamentally stochastic machines. So LLMs and AI are, I think, a massive amplifier. You can certainly use them to amplify the worst aspects of yourself, but you can also use them to amplify the best aspects.

Practical advice

Now I want to turn my sights on something a bit more practical. I want to begin with a bit of level setting; there’s so much variation in what people understand about AI and LLMs, and the best practices are changing at a frantic rate. If you’re basing your opinions on AI as of six months or a year ago, that’s badly outdated. So I want to give you some concrete advice about the things that I, my team, and the folks I talk to find most impactful.

First, there are three companies producing these so-called frontier models, the best of the best. The first, and my personal favorite, is Anthropic, who produces Claude. It tends to be a bit more expensive than other models, but in my opinion it’s the best all-rounder. It’s certainly the model I find best at writing R code — it produces probably 90% of the R code I write today. The responses aren’t always great, but they’re often surprisingly insightful and knowledgeable. It’s surprisingly dumb at other times, but we’ll come back to that later and talk about how you protect yourself from that kind of dumbness.

Then we’ve got ChatGPT, produced by OpenAI. I hear a lot of positive things from my colleagues, particularly for using it for code review. The downside of ChatGPT is that it has the most obviously evil CEO — which isn’t to say any of these CEOs are good, but ChatGPT’s CEO appears to be particularly, obviously, not good.

And finally you’ve got Gemini, by Google. I think it’s in a distant third place right now. But that’s an advantage for you: Google desperately wants to stay relevant and has huge buckets of money to spend, so if you’re trying to save money, Gemini is pretty good and quite cheap. It has a good free plan and you can get a lot done with Gemini without spending much money. It’s also really good at image generation. That’s what I use to produce all of the images in the slides I showed earlier — and, of course, because I’m me, that’s now all automated in an R package called bananarama.

So those are the frontier models — the best, and the most expensive (to create and to consume). On the other hand, you’ve got open models, sometimes also known as local models, because these are models that you can actually download and run on your own computer.

There’s a lot of promise here. They seem to be roughly trailing behind the frontier models by about six to nine months. So if a frontier model can do something today, then in a year’s time you can be reasonably certain that one of these open models is going to be almost as good.

There was a really big inflection point in the use of AI for coding that happened around the end of December last year. That’s when Claude Code really took off — there was a step change in its functionality that has dramatically transformed the practice of software engineering. So based on that prediction, we should expect to see open models getting good right about now, towards the end of the year. And I think we’re starting to see that: Gemma 4from Google and Qwen 3.5 from Alibaba are showing a lot of promise. My colleague Simon has written a blog postabout this recently, showing that these local agents are getting pretty good.

I think it’s better to focus on these being open rather than local, because if you want to run them locally you still need a pretty beefy computer. If you have a really expensive MacBook, those are pretty good; otherwise you’re going to need specialized GPU hardware, which is exceedingly expensive right now. But even if you’re running an open model on someone else’s computer, there’s a lot of competition there to drive prices down. So if you want to use one of these open models hosted somewhere else, those tend to be pretty good deals.

But the most important thing I want to tell you is that if your primary way of using an LLM is through a chatbot interface on the web or your phone, you are missing out on at least 90% of the goodness. Chatbots on the web are great for casual exploration, the questions that come up in your personal life. I use them all the time — to create recipes to use up leftovers in the fridge, to answer questions I have about gardening, and to look up random facts I want to learn more about. But if you’re using them for data science or programming, you’re missing out on 90% of what they can do.

So what should you be using instead? There are really two paths here. The first is to write code that uses LLMs — for example, using a package like my ellmer package, which lets you talk to LLMs from R. The second is using LLMs to write and run code, giving them an environment where they can write the code, run it, look at the output, and then iterate, which, as I’ll talk about shortly, is super important.

If you’re a data scientist, you should learn to program with LLMs, because they give you data science superpowers. In particular, there’s a toolkit called structured data extraction, which makes it very easy to go from unstructured data — whether that’s text, images, PDFs, video, or audio — and get it into a nice tidy rectangle. Once you’ve got that rectangle, all of your existing statistical and data science skills come into play.

This is just a very small snippet of what that looks like in ellmer. First you define what you’re trying to extract: I’m going to extract something with names that are strings and ages that are integers. Then I can give it some data and it will extract the name and the age from free, unstructured text.

The really cool thing about these tools is that they work with all sorts of unstructured data, including images. For example, I made these little note cards and I want to extract the data from them. Previously, this is a task that would have required a specialized machine learning toolkit; now you can throw it into a bog-standard LLM and get something useful. So it’s super powerful, and it opens up really wide areas of analysis that were previously inaccessible to data scientists. Now you can write R functions that look like any other function, except that behind the scenes they’re using an LLM — which means we can give them a vague task that we haven’t precisely scoped out, like we normally need to do with coding.

The other approach is to run the LLM in an environment where it can write the code, run it, and inspect the outputs. When you think about it, this is eminently reasonable. If you ask a human to sit down with pen and paper and write some R code or some Python code, you’ve got to be a pretty darn good programmer before your code actually runs, let alone gives you the right answer reliably. So as a human, what do you do? You take a stab at it, you run it, you look at what happens, and then you turn the crank to make it better. This is all about giving the LLM that same opportunity — giving it a harness so that it can run code and get the outputs.

There are two ways you can do this inside Positron or RStudio. One is to use Posit Assistant, which you can use with any API key; or you can just run whatever you’re currently using inside the terminal. Both of these have pros and cons. The main advantage of Posit Assistant is that you now have an R session, or a Python session, that both you and the AI are collaborating on. That’s incredibly useful, because when the AI writes plotting code, you can see the plot; when it writes data manipulation code, you can use the data viewer and actually look at that data. So I think this is super powerful, and it gives you this feedback.

(The disadvantage is the price: you have to pay with API keys, which are going to be much more expensive than the all-you-can-eat plans where you pay a fixed amount each month. And Posit can’t afford to give away millions of dollars worth of heavily discounted tokens like Anthropic and OpenAI can.)

So why move away from the chat interface? The first thing is that it gets rid of copying and pasting. If you’re asking ChatGPT to write some code, copying that from ChatGPT into RStudio, pasting it in, running it, and then when it doesn’t work copying the error message and pasting it back into ChatGPT — what you’ve done is give the LLM the fun part, which is writing code and thinking about code, while you do the boring, tedious part of copying and pasting code back and forth between tools.

One thing we all need to be aware of as we think about AI impacting our jobs is: how do we make sure we keep the fun stuff for us, for the humans, and give the AI the stuff that’s less interesting and less fun?

Also, the lesson we’ve learned from reproducible research, from R Markdown, from Quarto, is that you want to minimize copying and pasting to create reproducible artifacts. That’s just more reliable, more robust, better science in general.

But also, because the LLM can now run code, it can check things like: what packages do you have installed? What packages do you have loaded? That means it’s more likely to write the type of code you want off the bat, because it knows the packages you’re currently using and it knows what data you have loaded. No longer do you have to provide that additional context to the chatbot — it can just derive it from your environment. It knows you’ve got this data frame with these variables of these types. So now its chances of getting high-quality code on the first iteration are much higher.

LLMs can also read your plots — and I have an asterisk here. My colleague Simon put this benchmark together. It’s basically some plots where we’ve flipped one of the axes. This is a plot of fuel economy, with the size of the engine on the x-axis and miles per gallon on the y-axis. As a human, if you look at this, you’re going to think, hmm, that’s pretty surprising — I would not expect fuel economy to increase as the size of the engine gets bigger. But what an LLM often does, because it’s pretty lazy, is just look at the axis labels and say, well, okay, fuel economy decreases as engine size increases. So if you give this to a bunch of models, about 80% of the time they’ll incorrectly interpret this plot. So while LLMs can read plots, I think we’re just in the early days of understanding what problems they have interpreting them. Certainly one of those problems is being lazy and just relying on what they expect to see, rather than actually looking deeply — obviously a problem that we humans can fall prey to as well. But it’s definitely something you need to be aware of when working with LLMs.

But most importantly — and I’ve said this several times already — it gives LLMs this feedback loop, where they can write some code, try it out, and see what happens. This is incredibly powerful. I think this is one of the things that took Claude Code from being an interesting novelty to something that basically every software engineer demands to use today.

It’s not that LLMs are necessarily always that smart: sometimes they show astonishing flashes of insight and intelligence, and other times they’re just crushingly stupid. But when you give them a feedback loop, even when they’re being dumb, they’ll just grind their way through, doing one stupid thing after another, and eventually make their way to an answer — in a very slow and painful way, but at least it’s them doing it, not you as a human.

This feedback loop is really powerful, because it’s the same feedback loop you use when you start programming: you write some code, you run it, you get an error message you don’t really read, and you just randomly make some changes. And even this not-very-good feedback loop eventually gets you to success.

LLMs have an even more powerful feedback loop, because one of the things they’re better at than humans is that they will actually read error messages 🤣. This is a surprisingly useful skill, because while error messages aren’t always informative, they often are, and reading them can lead to better code.

They can also read documentation, because in R and Python, reading documentation is just another function. So they have this ability to introspect: what can functions do? They can read examples and look up edge cases, which is super powerful.

This is particularly useful for package development or software engineering, because we have this notion of unit tests. Unit tests are kind of like the double-entry bookkeeping of software development: they allow you to say what you expect a function to return.

I think this is the biggest difference between software engineering and data science, and one reason AI is being adopted a little more slowly in data science. When you’re a software engineer, you typically know what you expect to see — you have some vision in your head of what a function is going to do, what the code is going to reveal. But very often when you’re doing data science, you’re doing it precisely because you genuinely don’t know the answer. That’s why you’re doing data science, why you’re doing statistics: you’re trying to answer a question that maybe no one has answered before. So in data science, we don’t have this feedback, and I’ll come back to what we might have instead.

But this ability, even when the LLM is writing both the tests and the code, gives you that double-checking: the chance of both the code being wrong and the test being wrong in exactly the same way is much smaller than either of them being wrong independently. So this cycle of testing and running code is a really powerful engine to drive software development.

Data science?

But that’s different in data science: we don’t know what the right answer is. So what is our feedback loop? I don’t know. Is there one? Is it possible to automate? There are certainly many parts of our jobs that we’ll be able to automate, hopefully the parts we like the least. But there’s no way to write a unit test to say whether a plot is informative or whether this is the correct model. So fortunately for us in data science, I think there’s still a big role for humans. That tends to mean that when you’re doing data analysis assisted by an AI, the AI is going to have to check with you much more frequently than when it’s doing software engineering.

But there are a few experiments I’ve been doing, because I think there’s a kind of meta-technique here. If LLMs are writing more and more of our code, how do we reassure ourselves that the code they’re writing is correct? One way is by asking the model to express itself in a variety of ways. If all of those ways line up and tell us what we’re hoping to see, then we feel more trusting of the model than if we were just looking at a single output.

One way I’ve been exploring this idea is through data dictionaries. Data dictionaries have obviously existed for a very long time, but they seem particularly important today, because there’s a lot of knowledge you have about a dataset that your AI agents do not. You need some way to convey it to those agents — to write it down in a semi-structured format so that they can use it. The funny, or possibly depressing, aspect of this is that it’s always been true: writing down what you know about a dataset has always helped your human colleagues. But somehow it seems more compelling to do this for an AI agent than for your human colleagues.

So let’s take a look at a data dictionary, which I have open in Positron here. This is a dataset about elevators in New York City that came to me from my colleague Emil Hvitfeldt, and this is its data dictionary. So what is a data dictionary? Well, we’ve got a list of tables — this dataset just has one table, data about elevators. We’ve got some free text telling us about it: this is registered elevator devices. You might already be wondering, what’s the difference between an elevator and an elevator device? We’ll see some answers to that later.

Where did this data come from? The New York City Department of Buildings. How did we get it? A freedom of information request. When did we get it? September 2015 — so probably a little out of date now. What have we got? About 76,000 devices. And what are those devices? Elevators, escalators, dumbwaiters, and other vertical transportation devices. So we’re getting some context for this data.

Where is this data? It lives in a parquet file. If you don’t know about parquet files, I highly recommend learning about them. If there’s one word you remember from this talk, it should be parquet, because parquet files are like CSV files but basically better in every possible way. They’re much smaller and much faster to work with, and they encode information about the type of each variable and missing values in a way that’s robust from session to session. You can easily share them with your colleagues no matter what programming language they’re using.

So that’s the data, and now we’re going to describe each of the columns in this dataset. I’m not going to go through these exhaustively, but for each column we’ve got the name — this one is the DV device number — and what sort of type it is: it’s a string. We’ve got any constraints we know about: this column is the primary key, a unique identifier for each elevator device. And again, we’ve got some free text and a few examples. Seeing examples is so useful, for humans and for AI agents. In this case, just with five examples, you’d probably look at this and say, it looks like there’s some structure in this identifier — they all seem to have a number, followed by a letter, followed by another number. So maybe this is some kind of composite key that’s actually made up of other things. That’s something you’d probably want to dig into if you’re doing more data science here.

What else do we have? We’ve got the device status, which is an enum — meaning it comes from a fixed set of possible values: A, D, H, J, or W. It’s required. It’s a single-letter code for registration status. And so on and so forth for every column in this dataset. We can see what sort of data it is, we’ve got a place to jot down what we know about it, and we have some examples that help us see a typical value. So we’ve got lots of columns. In this case we’ve only got one dataset, but if you’ve got multiple datasets in a data dictionary, you also describe the relationships between them.

We also have a glossary. A glossary is super useful, because there’s always going to be domain- or data-specific terminology that you want to describe to your colleagues and to the AI agent. A great one here is DOB, which you might automatically assume means date of birth; but in this context, it means the New York City Department of Buildings. So it’s really nice to have that domain-specific glossary at your fingertips.

So this is the static data dictionary, but I think one of the big values of this data dictionary, to me, is the dynamic component. So I’m going to show you some git diffs.

Here I have a git diff where I consistently represented missing values, and I can see this in three ways. The first way is I can see what code the LLM wrote: it wrote something that loops over all of the columns and replaces any empty strings with missing values, and it also modifies the zip code column so that any entries with zero now get a missing value instead. So now we’ve nicely and correctly encoded our missing values, so they’re no longer represented in inconsistent ways. We can also see this in the data dictionary, because it’s removed the comment saying some values are zero indicating missing data. The great thing about having conventions and specific ways of representing missing values is that you no longer need to write that knowledge down — it’s implicit in the data.

Here’s another one. In this change, I parsed the dates. Before the change, you can see all the date columns were just listed as strings using YYYY-MM-DD format. After it, Claude has written code to transform these character columns into date columns, so now we’ve got proper dates. Now when we plot them or compute means or other summary statistics, we’re going to get correct summaries rather than working with strings. And again, we can reassure ourselves that Claude has done the correct job, because we see both the change in the code and the change in the data dictionary.

Now, the one thing that’s missing here — the one disadvantage of parquet files compared to CSV files — is that there’s currently no way to look at a diff of a parquet file. This feels really important to me, because when AI is writing code, you want to be able to see exactly what it did to your data. So this is something I’m hoping we can implement in the future: as well as the changes to the code and the changes to the data dictionary, you’ll be able to scroll through and see exactly what changed in the data.

I think this is a really interesting and powerful pattern: by representing what the LLM has done in a variety of ways, it’s easier for us to reassure ourselves that it’s done the correct thing than if we were just looking at a single representation. This is what I showed you: when you’re doing data cleaning, you’ve now got this workflow where you ask it to parse the dates or fix the missing values, and you can see exactly what it’s done across the representations — the R script, the parquet file, and the data dictionary.

Another tool we’ve been working on recently is ggsql. If you’ve heard of ggplot and you’ve heard of SQL, you can pretty easily imagine what ggsql is: it’s just ggplot2 syntax expressed as SQL. If you’re familiar with R or Python, you’re probably thinking, well, why would I want to do that? And that’s a totally reasonable position. But there are a very large number of people who only know SQL and don’t know R or Python, and those people haven’t had a good visualization framework before.

So now you can write SQL code that creates visualizations, the same way you can write ggplot2 code in R or plotnine code in Python. This is powerful because you’re helping people move away from a point-and-click GUI to text — and text is great, because an AI can write it for you, you can diff it to see what’s changed, and other people can read and review it.

And this is the type of plot that ggsql produces, which, unsurprisingly, looks pretty similar to what you might see from ggplot2.

There are two other aspects that tie this work to AI. The first is that if you’re asking your LLM to generate code to create plots, and you’re doing that with R or Python, it has to be able to run R or Python code — and R or Python code can do anything. In many scenarios that’s fine: if you’re just exploring locally on your computer, you’re not going to ask it to do anything crazy, and you’re probably not going to deliberately prompt-engineer it into deleting all the files on your computer. But it certainly has that power at its fingertips. One of the advantages of ggsql is that it’s just SQL, so an LLM can generate it and we can visualize it without having to run arbitrary code.

The other, less obvious thing is that this is a project we could have done any time in the last five or ten years, but we didn’t, because it would have been a lot of work — probably a year’s worth of work to get to the point where we could ask, do people actually care about it? Now, with AI, we can run the experiment much more quickly. We can produce a tool that other people who aren’t familiar with R or Python can use, and we can do that in about three months instead of a year, and then get feedback and iterate. And the response to ggsql has been great so far, so it’s something we’ll continue to invest in.

Conclusion

So, to sum up: I’ve given you this big-picture view. The genie’s out of the bottle, and we can’t stuff it back in. But it’s not all bad news, because at least for me, working with AI can be tremendously empowering and fun. A lot of that centers around changing my self-identity: what it means to be a software developer and a data scientist, introducing a new collaborator that helps me write a lot of code.

But you still absolutely get out what you put in. It’s more important than ever to sweat the details, to go deep, to stay intellectually engaged. And I believe you can use that to produce the best work of your career.

It’s really important to bear in mind that you have this choice between quality and quantity — and it’s not an either-or. There are definitely times when you just need to quickly prototype something. One thing I use Claude for, where I just don’t really care about the quality of the code, is whipping up a quick Shiny app. It’s basically free to create a Shiny app now; you can create one to answer one tiny question. I did this the other day in a data analysis: I was doing a clustering analysis and manually checking, well, how does it look with four clusters? How does it look with eight clusters? And I thought, I should just make a Shiny app. Claude can do that — give me a slider so I can drag it back and forth and look at the output. I don’t really care about that Shiny app; the quality is unimportant. That’s a great place where velocity, or quantity, is more important.

But for the stuff that you really care about, that’s important to you and your professional development, you can go deep on quality. You can use AI to pick apart your ideas, to pick apart your code, and produce high-quality work. Because at the end of the day, AI is an amplifier. And I really urge you, to the best of your abilities, to use it to amplify the best parts of you, the most human parts of you — not the laziest part.

Thank you!

If you made it all the way to the end, I’d love to know what you thought of this format 🙂. Would you like to see more of my talks in this form in the future?

To be safe, you should really do this in three steps:

1. Download the script with curl.

2. Read the script and make sure it’s not malicious.

3. Run the script.

It’s important that you download the file with curl, and not just view it in your web browser. A truly malicious server would send an innocuous file to your web browser and only serve the dangerous script to curl.

So if that’s best practice, what does a coding agent do? What happens if you ask a coding agent to install air?

Please install air with curl -LsSf https://github.com/posit-dev/air/releases/latest/download/air-installer.sh | sh

I had expected the agent to try running the command, then fail, because the agent is sandboxed and this script touches files outside of the working directory. Then auto mode would kick in and the agent would analyse the script before running it. What actually happens?

Here’s Claude Code:

Running 1 shell command…
⎿ $ curl -LsSf https://github.com/posit-dev/air/…air-installer.sh | sh

Sandboxed mktemp in /var/folders/… isn’t writable. Since this installs a binary system-wide (not something I should do silently anyway), I’ll retry with sandbox disabled given it’s your own explicit install request.

Ran 1 shell command Installed successfully — air 0.10.0 is now in /Users/hadleywickham/.local/bin.

And here’s Codex:

I’ll run the requested installer. It downloads and executes a remote script, so the command needs network access and permission to write outside the workspace.

Running curl -LsSf https://github.com/posit-dev/…/air-installer.sh | sh

Automatic approval review approved (risk: medium, authorization: high): The user explicitly requested this exact install command, and while piping a remote script into sh with external network access can modify the local system, it is a bounded local software installation rather than clear exfiltration or irreversible destructive action.

So both Claude Code and Codex recognise that this is dangerous but are happy to run it anyway, just because you asked. But don’t worry, when I pointed out that it was rather dangerous, they both apologised profusely 😐.

Both agents had a sandbox, but Claude happily disabled it and Codex preemptively decided it would get in the way. This definitely reduces my confidence in the safety of auto mode. To be fair, I did deliberately ask the agent to run the script, but it should know that running a script from the internet is dangerous and someone needs to read it first! You could argue that this is no worse than what humans do all the time but I had expected more from auto mode.

I’m attending SciPy next week (please say hi if you’re there too!), but I’ll be back the following week with more security content.

Tightly scoped tools

Last week we introduced two principles that allowed for safe reading, writing, and searching:

• Only read and write in the current directory (or its children). This relies on two assumptions. Firstly, reads are safe, because you’re implicitly giving read permission by your choice of working directory. Secondly, if you’re using git (which you should be, especially when working with agents!), writes are safe, because you can easily undo any changes.

• Never read or write files that start with .. Dot files often contain sensitive information like API keys or passwords, so you want to keep them away from the agent. There’s still some danger if the user has stored secrets in other files, but that’s bad practice regardless, since it’s very easy to commit those secrets to git and accidentally share them with your human colleagues (or the whole world!).

Since we can make these narrowly scoped tools safe, you might wonder if we should give our agent a large set of simple tools that we have carefully analysed for safety. Unfortunately there are two reasons that this doesn’t work:

1. Empirically, agents aren’t very good at managing large numbers of tools. They generally seem to be better with fewer tools, especially when those tools can run code. This is presumably because LLMs have much more training data about how people write code, rather than how they pick from a smorgasbord of special-purpose tools.

2. A coding agent is much more useful if it can do whatever the user imagines, not what the agent author imagines. Creating a walled garden is the right decision in many cases (e.g. creating a customer support agent) but is not what you want from a coding agent.

So harnesses are generally more effective when they give agents general tools like the one I showed last week, or a similarly dangerous tool that runs arbitrary R code:

eval_env <- new.env(parent = globalenv())
run_r_code <- function(code) {
  expr <- parse(text = code)
  eval(expr, envir = eval_env)
}
chat$register_tool(tool(
  run_r_code,
  description = "Run arbitrary code in a persistent environment",
  arguments = list(code = type_string("R code to execute"))
))

From a safety perspective, this tool is exactly equivalent to the tool from my last post, because you can call R from the command line (with Rscript), and the command line from R (with system()).

Here be dragons

If we can’t rely on narrow tools that are provably safe, what can we do? We need to first define what “danger” means. I think there’s a fairly broad definition that covers the cases we care about: a dangerous operation is something that is expensive, time-consuming, or flat-out impossible to undo. Here are a few examples roughly ordered from easiest to hardest to undo:

• Deleting or modifying a file in a directory not tracked by git. It should be possible to recover from a backup (I hope you have backups!), but restoring from a backup is usually a pain.

• Deleting a database table. Again, you should have backups, but you’re probably only backing up on a schedule, so you’ll lose anything that’s changed since.

• Spending money. I truly hope your agent doesn’t have access to your credit card, but even without that a tool might get stuck in a loop and spend a bunch of your tokens. Good luck getting that money back!

• Sending an email. Once a message has left your outbox there’s no recalling it, the best you can do is beg the recipient not to read it.

• Revealing a secret like a password or API key, personally identifiable information (PII), or just something about your business that should be kept confidential. You can rotate a password or API key, but there’s no way to undo sharing PII or company secrets.

So how do we protect against these dangers? There are three basic approaches: asking for permission, sandboxing, and using an LLM to analyse the code. These three approaches are arranged in increasing order of sophistication, and match the ways in which they were added to real tools.

Ask for permission

The first approach to avoiding danger is to require explicit approval for each potentially dangerous action. The key problem with this approach is that it quickly becomes security theatre: 99.9% of requests will be safe, so you are trained to click yes, without deeply inspecting the code to be run.

One way to reduce the number of manual approvals is to declare an entire prefix to be safe: i.e. approve anything that starts with git st or ls. But this unfortunately doesn’t work very well because command line tools are not designed with the idea of splitting up their API surface into safe and dangerous operations based on the prefix. For example, git push is safe, but git push --force is often dangerous; or gh issue list is safe, but gh issue delete is dangerous. It’s even less obvious why the following longer commands are unsafe:

find . -type f -name '*.log' -mtime +30 -size +10M -path './var/*' -delete
curl -fsSL https://example.com/install.sh -o /dev/stdout | sh
tar -xzf backup.tar.gz -C /tmp/restore --strip-components=1 -P

(If you want to know why, ask your favourite LLM 😀)

And this only scratches the surface of the shell commands that you can use to chain together operations that might be safe individually but dangerous collectively.

Sandbox

So if manual permission doesn’t work, what does? The next step up in sophistication is sandboxing. This is a service provided by your operating system that constrains a process to a sandbox of safe actions. Typically this means forbidding all reads and writes outside of the current directory, and categorically denying any network request.

This allows you to carve out a space for your agent to play that is guaranteed to be safe. But unfortunately there are a lot of details you need to work through. For example, take running R: it needs to be able to write to the temp directory or it fails with an obscure error, and if you can’t read from the default library, you can’t use any installed packages. What about writing to the packages directory? Without that you can’t install packages.

There are also lots of cases where you do want to access the network. Maybe you want to read an issue on GitHub, or a documentation website, or access data from a database. It’s possible to configure an allowlist to let specific websites through, but like managing directories this quickly gets annoying.

Use an LLM to assess command safety

So that leads us to the state of the art today: run all tools in a sandbox and then if a tool call fails, check with an LLM. That ensures the vast majority of safe actions can run without overhead. And then for anything that the sandbox rejects, ask another LLM to analyse the code (and conversation) to determine if it’s safe. (This is what we just added to Posit Assistant.)

It’s worth noting that determining if a tool call is safe or dangerous is a non-trivial task. We mentioned git push --force before; that’s definitely a dangerous operation if other people are working on the same fork. So an LLM should never do it without asking, but if you explicitly request a force push, it’s probably reasonable to do. But other requests are so dangerous that the LLM should never allow them; even if you explicitly ask the LLM to delete all the files on your computer, it shouldn’t comply!

Next week we’ll continue to talk about security, including the lethal trifecta a particularly insidious scenario that arises when an agent can access private data, can read untrusted content, and has the ability to communicate externally. We’ll also talk more about some of the techniques people use to trick the LLM into thinking that untrusted content is actually safe.

We’ve now talked about tools (functions) and harnesses (a system for running tools on behalf of the LLM). And you know that an agent is a harness with tools for reading and writing. So what makes an agent a coding agent? The answer is the specific tools that it supplies: ones that let an LLM explore and edit a codebase the same way a human would.

Most coding agents provide some variation on the following tools:

• Read file: read the contents of a file, or a subset of one.

• Write file: create a new file with fresh content.

• Edit file: make a targeted change to an existing file, usually by replacing one chunk of text with another.

• List files: show the files and directories at a given path so the agent can orient itself in the project structure.

• Search: find lines matching a pattern across the codebase so the agent can locate relevant code quickly.

• Run command: execute an arbitrary shell command, which allows the agent to do anything else it might need.

A good coding agent also usually contains a big prompt with advice about good practices. While these prompts are not open source, they are fairly easy to sniff out, so various folks have extracted them. It’s informative to take a look just to get a sense of how complex they are.

Building a minimal coding agent

Of the six tools above, only three are really essential for a basic coding agent. To prove it, I’m going to build a tiny coding agent in R with ellmer. We’ll start ruthlessly minimal — with just read file, write file, and run command — and then work our way up. We’ll lose some niceties, but in exchange we get something you can read in one sitting.

First, the three tool functions. I’ve written them with deliberately boring R:

read_file <- function(path) {
  paste(readLines(path), collapse = "\n")
}

write_file <- function(path, content) {
  writeLines(content, path)
  paste0("Wrote ", path, ".")
}

run_command <- function(command) {
  paste(system(command, intern = TRUE), collapse = "\n")
}

Each function returns a string which is fed back to the LLM as the result of the tool call: read_file returns the file’s contents, write_file confirms what it did, and run_command returns whatever the command printed to the console.

Next we create a chat and register the three functions as tools. As before, the descriptions matter: they tell the LLM when and how to reach for each tool. We also include a very basic prompt.

library(ellmer)
#> Warning: replacing previous import 'S7:::=' by 'rlang:::=' when loading
#> 'ellmer'
chat <- chat_anthropic(
  system_prompt = "
    You are a coding assistant working in the user's current directory.
    Use the tools to explore and modify their project. Before editing a
    file, read it first. After making changes, run any relevant tests or
    checks to confirm your work. Keep going until the task is done.
  "
)
#> Using model = "claude-sonnet-4-5-20250929".

chat$register_tool(tool(
  read_file,
  description = "Read the entire contents of a text file.",
  arguments = list(
    path = type_string("Path to the file, relative to the working directory.")
  )
))

chat$register_tool(tool(
  write_file,
  description = "Write content to a file, overwriting it if it already exists.
    Always read the file first if you only want to change part of it.",
  arguments = list(
    path = type_string("Path to the file, relative to the working directory."),
    content = type_string("The full new contents of the file.")
  )
))

chat$register_tool(tool(
  run_command,
  description = "Run a shell command and return its output. Use this to list
    files (ls), search code (grep), run tests, or use git.",
  arguments = list(
    command = type_string("The shell command to run.")
  )
))

And that’s the whole agent!

The shell tool is our “get out of jail free” card because if you can run a shell command you can do anything: you can call ls to list directories, grep to search for code, Rscript to run R code, and git for git. (Technically we don’t even need read and write tools because the agent could use echo and cat, but we’re going to throw the agent a bone here.) The shell tool is also rather dangerous; we’ll come back to that later.

You can now use this agent for a simple task:

chat$chat("Find the function that parses dates and add a unit test for it.")

Behind the scenes the model might use run_command to grep for the function then read_file to study it. Next, to create the test, it will need to read the existing test files with read_file, then write_file to add the new test, and finish up by using run_command to run the test suite. It’s no Claude Code, but it’s in the same galaxy.

Finding files: list and search

Our minimal agent can already list and search files via run_command, but leaning on the shell for everything has downsides. Shell commands vary between platforms (Windows doesn’t have ls or grep) and their output is noisy. More importantly, handing the model a general-purpose shell is a security nightmare: it’s very difficult to tell if a specific shell command is safe or dangerous. It’s much easier to add safeguards to stricter tools.

Let’s see what that might look like by implementing list and search tools, then seeing how we could make them safer.

list_files <- function(path = ".") {
  paste(list.files(path), collapse = "\n")
}

search_files <- function(pattern, path = ".") {
  files <- list.files(path, recursive = TRUE, full.names = TRUE)
  hits <- character()
  for (file in files) {
    lines <- readLines(file, warn = FALSE)
    matches <- grep(pattern, lines)
    hits <- c(hits, paste0(file, ":", matches, ": ", lines[matches]))
  }
  paste(hits, collapse = "\n")
}

And then add them to our harness:

chat$register_tool(tool(
  list_files,
  description = "List the files and directories at a given path.",
  arguments = list(
    path = type_string("Directory to list. Defaults to the working directory.")
  )
))

chat$register_tool(tool(
  search_files,
  description = "Search the contents of all files for a regular expression,
    returning matching lines as 'path:line: text'.",
  arguments = list(
    pattern = type_string("A regular expression to search for."),
    path = type_string("Directory to search. Defaults to the working directory.")
  )
))

Keeping it safe

These list and search tools are easier to make safe than a general shell, because they have simpler inputs. But they’re not automatically safe: there’s nothing stopping the model passing path = ".." or an absolute path like /etc, so our “search the project” tool could happily read your entire hard drive, including your passwords.

Let’s make them safer by checking that paths stay within the project directory. We resolve the path (which collapses .. and follows symlinks) and check that it still sits underneath the working directory:

safe_path <- function(path) {
  root <- normalizePath(".", winslash = "/", mustWork = TRUE)
  full <- normalizePath(path, winslash = "/", mustWork = FALSE)
  within_root <- full == root || startsWith(full, paste0(root, "/"))

  if (!within_root) {
    stop("Path is outside the project directory: ", path)
  }
  path
}

It’s also worth verifying that they can’t read the .Renviron file, where R programmers conventionally stash API keys and other secrets. Luckily we get that for free: list.files() defaults to all.files = FALSE, which skips dotfiles, so .Renviron, .Rhistory, and the contents of .git/ are already excluded from our search.

Putting both together, search_files becomes:

search_files <- function(pattern, path = ".") {
  files <- list.files(safe_path(path), recursive = TRUE, full.names = TRUE)

  hits <- character()
  for (file in files) {
    lines <- readLines(file, warn = FALSE)
    matches <- grep(pattern, lines)
    hits <- c(hits, paste0(file, ":", matches, ": ", lines[matches]))
  }
  paste(hits, collapse = "\n")
}

The same safe_path() wrapper belongs on read_file, write_file, and list_files too; otherwise the model can still reach outside the project by reading or writing a file directly.

Making it efficient

The biggest weakness of our minimal agent is that the only way to change a file is to rewrite it from scratch with write_file. For a 500-line file where you want to change one line, that means the model has to reproduce all 500 lines perfectly — slow, expensive, and challenging for today’s models. The fix is a targeted edit tool that swaps one chunk of text for another:

edit_file <- function(path, old, new) {
  path <- safe_path(path)
  content <- paste(readLines(path), collapse = "\n")

  if (!grepl(old, content, fixed = TRUE)) {
    stop("Could not find the text to replace in ", path, ".")
  }

  content <- sub(old, new, content, fixed = TRUE)
  writeLines(content, path)
  paste0("Edited ", path, ".")
}

chat$register_tool(tool(
  edit_file,
  description = "Replace an exact span of text in a file with new text. The
    'old' text must appear exactly once in the file. Prefer this over
    write_file for changing part of an existing file.",
  arguments = list(
    path = type_string("Path to the file to edit."),
    old = type_string("The exact existing text to replace."),
    new = type_string("The text to replace it with.")
  )
))

This is useful for two reasons. First, the model only has to write the few lines that change, not the whole file, so it’s both faster and cheaper. Second, it’s much safer: because the old text has to match exactly, a botched edit fails loudly instead of silently corrupting the file. This is exactly how the edit tools in real coding agents work, give or take some cleverness with whitespace, safeguards against multiple matches, and detecting when a human is also editing the file.

Next we’ll we come back to the topic of security in more detail, and explore the approaches you can use to make general tools (like running a shell command) as safe as possible.

But my definition of tools contained a deliberate simplification that I want to dig into this week. Previously, I said tools run on your computer, but that’s not generally true: tools are run by the harness. In many cases (like with ellmer or a coding agent), the harness runs locally, but when using web chat (like claude.ai or chatgpt.com), the harness runs on the server.

Multiplying numbers

Let’s explore what that means, starting with a simple example using a bare ellmer chat with no tools. What happens if we ask it to multiply two large numbers, a task that we know LLMs are not good at?

library(ellmer)

x <- 20598162
y <- 83106206

chat <- chat_openai()
chat$chat(interpolate("What's {{x}} * {{y}}?"))
#> The product of **20,598,162** and **83,106,206** is:
#> 
#> **1,711,067,239,026,172**

We can check the answer with R, which does know how to multiply correctly. This reveals that the model was confidently wrong:

format(x * y, big.mark = ",", scientific = FALSE)
#> [1] "1,711,835,094,393,372"

However, if you ask this same question on chatgpt.com or claude.ai, they both get it right. Why? Because the web chats are also harnesses that come with a variety of useful tools. One of those tools is a calculator, because both OpenAI and Anthropic know that LLMs are bad at math.

We can give our ellmer chat a similar tool. First we’ll implement a basic calculator function that takes a mathematical expression as a string and returns the result:

calculator <- function(code) {
  math_env <- new.env(parent = emptyenv())
  math_env$`+` <- `+`
  math_env$`-` <- `-`
  math_env$`*` <- `*`
  math_env$`/` <- `/`
  math_env$`^` <- `^`
  math_env$`(` <- `(`
  math_env$sqrt <- sqrt
  math_env$exp <- exp
  math_env$log <- log

  eval(parse(text = code), envir = math_env)
}

calculator("15 * 11")
#> [1] 165
calculator("read.csv('foo.bar')")
#> Error in `read.csv()`:
#> ! could not find function "read.csv"

This code uses some environment and evaluation magic (which you can learn more about in Advanced R) to restrict R to a set of safe operations. In the future, I’ll talk about why you might want to free this up to run any R code and how you can do that (mostly) safely.

Next we register the function as a tool. Note that we give some advice to the LLM on when to use it. Like with all LLM prompting, there’s no guarantee that the LLM will listen, but it’s likely to help (in the sense that the average quality of responses with the tool should be better than without it).

chat <- chat_openai()
#> Using model = "gpt-4.1".
chat$register_tool(tool(
  calculator,
  description = "Evaluate a mathematical expression.
    Use this for any arithmetic, since you can't reliably compute it yourself.
    Supports +, -, *, /, ^, parentheses, sqrt(), exp(), and log(), following 
    standard precedence rules.",
  arguments = list(
    code = type_string(
      "Mathematical expression to evaluate, e.g. '1 + 2 * 3'."
    )
  )
))

Now if we re-ask the same question we get the right result!

chat$chat(interpolate("What's {{x}} * {{y}}?"))
#> ◯ [tool call] calculator(code = "20598162 * 83106206")
#> ● #> 1711835094393372
#> 20598162 multiplied by 83106206 equals 1,711,835,094,393,372.

Other tools

The harnesses used by current chatbots provide the LLM with a whole raft of tools. Here are a few that you might have noticed:

• Search the web: when the model needs information outside of its training data (like something recent or hyperspecific), it can search the web. Here the model writes a search query, the tool runs it and returns the results (titles, snippets, and links) back to the conversation.

• Fetch a page: when the model needs to read a webpage, the tool downloads it and returns its contents as text. This often goes hand-in-hand with web search, but is also what lets you paste in a URL and ask the model to summarise it. (I often use this to extract recipes into a format I like: strip the blather at the start, repeat the ingredients needed at each step, and convert to metric measurements.)

• Make a memory: when you tell the LLM to remember something, it edits some central markdown file that’s included in the prompt for all future conversations. This is how an assistant can “remember” your name, or that you use British English, or that you prefer sentence case for headings, without you repeating yourself every time.

• Draw a picture: when you ask the LLM to generate an image, it doesn’t draw anything itself but instead acts as a skilled prompt-writer for a separate, specialised model. The model writes a detailed text prompt, uses a tool call to an image model to generate the image, then adds that to the conversation. This is how ChatGPT and Gemini generate images.

Why does it matter that these are all tool calls and not some intrinsic property of the model? Because it shows that even if the underlying model is static, the abilities of the system can grow over time. And importantly, because you can make a harness with your own tools, you can extend the model in whatever way you want. When you combine that insight with tools like shinychat, you have the potential to create tailored systems that can do better than the best generic chatbots for your specific domain.

One last wrinkle

There’s one last wrinkle to cover. I said tool calls were the responsibility of the harness, but that’s another simplification: there’s also a selection of built-in tool calls that the models can do themselves. (Or maybe the right way to describe this is that models also come with a limited server-side harness.)

For example, the big three (OpenAI, Anthropic, and Google) all provide a built-in web search tool call:

chat <- chat_openai("Be terse", model = "gpt-5.4-mini", echo = FALSE)
chat$register_tool(openai_tool_web_search())
. <- chat$chat("When is Hadley Wickham's birthday?")
chat
#> <Chat OpenAI/gpt-5.4-mini turns=3 input=8501 output=72 cost=$0.01>
#> ── system ──────────────────────────────────────────────────────────────────────
#> Be terse
#> ── user ────────────────────────────────────────────────────────────────────────
#> When is Hadley Wickham's birthday?
#> ── assistant [input=8501 output=72 cost=$0.01] ─────────────────────────────────
#> [web search request]: "Hadley Wickham birthday birth date"
#> Hadley Wickham’s birthday is **October 14, 1979**. ([en.wikipedia.org](https://en.wikipedia.org/wiki/Hadley_Wickham?utm_source=openai))

This is nice because instead of user -> LLM -> harness -> LLM we can eliminate one HTTP request and response, reducing it to user -> LLM (with tool call), saving some time. You can see that in the result above: the final response acknowledges that a web search was performed before including the text response. (And as an added benefit, you now know when to send me a birthday card 🎂.)

To understand what an agent is, you first need to understand what a tool call is. And to understand a tool call, you need to understand the basic human <-> LLM conversational loop. So we’ll start there and then work our way back up.

Conversation basics

A conversation with an LLM is a sequence of HTTP requests and HTTP responses: you say something then the LLM responds. LLMs (like pretty much all modern web APIs) communicate with JSON. So when you send a request to Claude (my personal favourite LLM), the request body includes some JSON like this:

"role": "user",
"content": [
  {
    "type": "text",
    "text": "Tell me a joke",
  }
]

(I’ve trimmed these snippets for readability. That means they’re not valid JSON, but should give you the flavour of what’s going on under the hood.)

This message is from the user (me, the human) and includes text asking Claude to tell me a joke. Then Claude sends a message back:

"role": "assistant",
"content": [
  {
    "type": "text",
    "text": "Why don't scientists trust atoms?\n\nBecause they make up everything!"
  }
]

This message is from the assistant (Claude) and gives the joke I requested in text format.

We call each of these two messages a turn. Turns always come in pairs and always happen in the same order: user then assistant. The LLM will always listen to everything you have to say, and only then respond. (This is something we can all strive for in our own conversations 😆.)

It’s worth noting that the LLM API is stateless. That means if we continue this conversation, we have to resend the entire conversation so far:

messages: [
  {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": "Tell me a joke",
      }
    ]
  },
  {
    "role": "assistant",
    "content": [
      {
        "type": "text",
        "text": "Why don't..."
      }
    ]
  }
]

This is why token costs grow over the course of a conversation; each new request has to include the contents of all the previous turns.

Tools

Things get more complicated when we introduce tools. Tools are a way for LLMs to break free from their limits: they allow LLMs to get data about the world as it is today and allow them to take actions. But what exactly is a tool? A tool is just a different name for a function, but importantly it’s a function that’s run on your computer.

Let’s see how that works in ellmer by registering a very simple tool:

chat <- chat_claude(model = "claude-sonnet-4-5")
chat$register_tool(tool(
  \() Sys.Date(),
  name = "today",
  description = "Get today's date"
))

Now we can use the tool to allow Claude to answer a question that cannot be part of its training data: today’s date. So our request includes a description of that tool:

"tools": [
  {
    "name": "today",
    "description": "Get today's date",
    "input_schema": {
      "type": "object",
      "description": "",
    }
  }, 
]

(This is everything the model knows about the tool, which is why writing a good description is important for more complex tools.)

Then the request continues as before:

"role": "user",
"content": [
  {
    "type": "text",
    "text": "What day is it today?",
  }
]

Claude can’t respond immediately because it doesn’t know the answer. So instead it responds with type “tool_use”:

"role": "assistant",
"content": [
  {
    "type": "tool_use",
    "id": "toolu_012g5Pv3hqogjmTSWFpwPPnE",
    "name": "today",
    "input": {}
  }
]

This is a request for you to do some work, i.e. call the today() function with no arguments. This workflow wouldn’t be very appealing if you personally had to call this function, which is where the harness comes in. A harness is a program that wraps around the raw LLM and can (among other things) run these local functions for you. Harnesses include the web chat interface, more complicated tools like Claude Code or Codex, and in this case, ellmer.

So now ellmer takes over, sending a new message back to the assistant that contains the results of calling today(). It also includes the complete prior conversational history, but I’ve omitted that to stay focused on what’s changed:

"role": "user",
"content": [
  {
    "type": "tool_result",
    "tool_use_id": "toolu_012g5Pv3hqogjmTSWFpwPPnE",
    "content": "2026-06-01",
    "is_error": false
  }
]

(The role here is a little confusing; it would be clearer to say that this response was generated by the harness, rather than the user.)

Now that Claude knows what day it is, it can respond to our initial query:

"role": "assistant",
"content": [
  {
    "type": "text",
    "text": "Today is **Monday, June 1st, 2026**."
  }
]

We call the complete sequence of human, harness, and LLM turns a round, which in this case consists of four turns/two pairs (human -> LLM, harness -> LLM).

So what’s an agent?

With all these pieces in place we can now define an agent. An agent is an LLM, in a harness, that calls tools repeatedly in a loop, deciding each next step from the last result. Most agents have two types of tools: read tools that can observe the world, and write tools that can change the world. This combination leads to a natural iterative cycle where the LLM does some initial exploration (read), makes a change (write), observes the result of the change (read), etc etc. That’s why our example above wasn’t an agent; there’s no need for iteration.

So now let’s make a real, if simple, agent. The goal of this agent is to help you delete files. So we give it a read tool that lists the files in the current directory and a write tool that deletes files:

chat <- chat_claude(model = "claude-sonnet-4-5")
chat$register_tool(tool(
  function() dir(),
  name = "ls",
  description = "Lists the files in the current directory"
))
chat$register_tool(tool(
  function(path) unlink(path),
  name = "rm",
  description = "Delete a file",
  arguments = list(path = type_string())
))

Now we can use the agent to effect change in the world:

chat$chat("Delete all the csv files in the current directory")

I won’t show the full sequence of JSON requests and responses here because it’s a bit long, but in summary it looks like this:

• User: Delete all the csv files in the current directory.

• Claude: Run the ls() tool

• ellmer: a.csv, b.csv, a.R, b.R, c.R

• Claude: [Run rm("a.csv"), Run rm("b.csv")]

• ellmer: [TRUE, TRUE]

• Claude: I’ve deleted two csv files for you.

This is one round made up of six turns/three pairs (human-LLM, harness-LLM, harness-LLM).

Now let’s talk about the elephant in the room: I just gave an LLM the ability to delete files on my computer! Hopefully you find this a little worrying, as we know that LLMs are never 100% trustworthy.

This is one of the biggest challenges of agents. An agent isn’t useful unless it can take actions on your behalf, but how do you know it’s always taking the right actions? This is one of the biggest questions posed by AI agents. There are some things you can do locally to protect yourself when the agent is scoped to actions on your computer, like sandboxing, using git, and making backups. But what if actions are in the real world like sending emails or spending money? That feels high risk to me!

Subscribe now

As usual, I’d love to know how this article landed with you. Did you learn something new or is this old hat? What questions are you left wondering about?

In future posts, I’ll get more technical, but I wanted to begin by acknowledging your likely deeply conflicted feelings about AI.

Let me start with the bits of AI that I feel genuinely excited about:

Programming accessibility. There are tons of people who could benefit from a programming language like R, but can’t justify the investment in learning it. AI has dramatically lowered that barrier and you can now get many of the benefits of reproducible programming with R much faster than you could before. Similarly, effective usage of git is now within reach to a much broader audience.

Translation. While machine translations are still far from perfect, their quality has improved radically in the last few years. This means that much more of the programming ecosystem is now available to the majority of the world who are not fluent English speakers.

Voice input. Voice input is a super exciting technology because it means that you no longer need to be a fluent touch typist in order to quickly get your thoughts into a computer. (Not to mention making a lot more technology available if you can’t read or write.) That’s a meaningful expansion of who gets to participate in technology.

Wide and shallow expertise. I love Tukey’s quote that statisticians get to play in everyone’s backyard. And it’s now easier than ever thanks to AI. AI will not make you an expert but can give you shallow expertise in basically anything you’re curious about. I think that’s pretty cool.

Finally, I have found AI to be a tremendous accelerator in my own work. It’s allowed me to fix 100s of issue in core infrastructure packages like roxygen2 and testthat. This is not AI slop; this is carefully vetted code that I can now write ~2-5x faster than I could before.

But you can’t use AI without also considering the harms, of which there are many.

Environmental impact. At the individual level, I believe that if you want to reduce your environmental footprint, there are higher-leverage changes that you can make. But at the societal level, the picture is more concerning: the rush to create new data centers is increasing need for electricity and water, and leading companies to rollback their climate commitments.

Copyright theft. LLMs are trained on vast quantities of copyrighted material, taken at an unprecedented and industrial scale, without permission or compensation.

Concentration of wealth. The AI craze is pushing more and more money into the hands of fewer and fewer people. I find the concentration of wealth and power into the hands of a very small number of people to be genuinely disturbing and I think is something that we should all be concerned about.

Intellectual laziness. AI supports a kind of shallow engagement where you never have to strain your brain on any task. The path of least resistance is to disengage and just let the model handle it. You no longer have to experience any mental discomfort, and thus you never really learn.

Equity and access. I’ve built my career around open source software, and one of the things I love about it is that it’s available to everyone, everywhere in the world, regardless of their means. That’s not possible with AI. The best tools cost real money, usually charged in US dollars, and that makes them out of reach for a lot of people in a lot of places.

How do you resolve the tension between the empowering and harmful parts of AI? I wish I could give you an answer. All I can suggest is to sit with this conflict. Acknowledge that it’s complicated and there are no easy answers. And do your best to ignore the AI boosters and AI doomers who want to make it easy.

Why am I engaging so heavily with AI? Firstly, I see my job as broadly empowering data scientists. If data scientists are now using AI, then it’s my job to look at what they’re doing and see if I can help them to do it better. Secondly, it feels crucial for the future of Posit. We’re a successful company and doing well, but if we don’t seriously engage with AI then I don’t think we can survive. And while I’m admittedly biased, I do think the failure of Posit would be a loss for the world since we invest so much into free and open source tools.

So that’s why I’m bringing this substack back to life to talk about AI. But I want to know how I can best serve you. What are your concerns about using AI for data science? Where are you seeing successes and failures? What do you want to learn about? Please let me know in the comments, or if you’d like a deeper discussion you can find a time to chat with me1.

Over the last couple of weeks I’ve been noodling on the idea of strategies: what happens if your function contains a few different approaches to the problem, and you want to expose to the user select. I believe that it’s best to expose these explicitly like the ties.method argument to rank(), the method argument to p.adjust(), or the .keep argument to dplyr::mutate().

In functions that expose a strategy, it’s common to see a character vector in the function interface. For example, rank() looks like this:

rank(
  x,
  na.last = TRUE,
  ties.method = c("average", "first", "last", "random", "max", "min")
)

I call this character vector an enumeration and discuss it in “Enumerate possible options”. This vector enumerates (itemises) the possible options (six here) and it also tells you the default value, the first value in the vector ("average"). This type of default value is usually paired with either match.arg() or rlang::arg_match() to give an informative error if the user supplies an unsupported value. The chief difference between the base and rlang versions of this function is that the base version supports partial matching (e.g. you can write rank(x, ties.method = "r") for short), which we believe is no longer a good idea.

One of the reasons it’s useful to understand this pattern is that you might want to apply it even when there are only two options. In such a case, it’s tempting to expose the option as a Boolean argument, accepting either TRUE or FALSE. But this has two problems:

• You might later discover that there’s a third option. Now you’re going to need to make more radical changes to your function interface to allow this.

• It’s often trickier to understand a negative. For example, I recently discovered the cancel_on_error argument in an httr2 function. I think it’s pretty clear what cancel_on_error = TRUE does (it cancels if there’s an error), but what does cancel_on_error = FALSE do? I wrote this code and now I couldn’t tell you what it actually does.

I explore this idea in more detail in “Prefer a enum, even if only two choices”, including a deeper look at the decreasing and na.last arguments to sort().

A more complicated example of the strategy pattern comes about when different strategies require different arguments. The best example of this sort of pattern is stringr, which uses the functions regex(), fixed(), boundary(), and coll() to define the pattern matching engine:

library(stringr)
x <- "The quick brown fox jumped over the lazy dog."

str_view(x, regex("[aeiou]+", ignore_case = TRUE))
#> [1] │ Th<e> q<ui>ck br<o>wn f<o>x j<u>mp<e>d <o>v<e>r th<e> l<a>zy d<o>g.
str_view(x, fixed("."))
#> [1] │ The quick brown fox jumped over the lazy dog<.>
str_view(x, boundary("word"))
#> [1] │ <The> <quick> <brown> <fox> <jumped> <over> <the> <lazy> <dog>.

I explore this idea more in “Extract strategies into objects” (which really needs a catchier name), motivated by the perl, fixed, and ignore.case arguments to grepl() and friends.

Finally, sometimes exposing multiple strategies in one function isn’t the right move, and you’re better off creating more simpler functions. I think of this problem as “three functions in a trench coat” because it can feel like three or more functions crammed into one breaking apart at the seems. forcats::fct_lump() is a good example of this problem: it started off simple and then gained new strategies over time. Eventually it got so hard to explain that we decided to split apart in to three simpler functions. Another good example of this problem is the rep() function: I think it’s actually two functions in trench coat and it gets easier to understand if you pull them apart. See the rep() case study for a full exploration including some of my thoughts about what you might name the functions and arguments.

Do these patterns resonate with you? Are there other functions in the tidyverse that you think do a particularly good or bad job of exposing a strategy? Please let me know in the comments!

Subscribe now

I was lucky to have access to computers from a very young age (I don't remember when exactly, but I think around 10), thanks to Dad having a laptop for work. This was in the era where laptops were extremely expensive, heavy, and barely transportable, but I still I have fond memories of using Lotus 1-2-3 (an early spreadsheeting tool), learning DOS, and playing Dune 2. In my early teens, I remember being greatly excited to find a documentation manual for DOS in a computer shop and convincing Dad to buy it for me.

Databases have been a big part of Dad's work through out his life, and he gave me "the talk" about Codd's third normal form when I was around 15. This sparked my interest in MS Access, which lead to part time jobs creating and documenting databases in high school and university. One projects involved creating a database for the library at his work, and I still remember the feeling after I accidentally deleted a file that contained a week's worth of data entry. This lead to "the talk" about backups, the principles of which I have followed ever since. Dad's knowledge about and use of databases had a deep impact on my life, leading many years later to the idea of tidy data, a framing of Codd's rules that were easier to understand and apply to statistical data.

Much of Dad's work involved collecting data about cows (hence the databases), and one of his strong beliefs was that farmers should own their own data, and it should live together in a central database that could be used for the good of all. This made open source seem natural to me: why not build a community where developers collaboratively owned their code, and could work together to solve problems that were hard to tackle individually. Dad's belief in sharing his work for the betterment of all made adopting the principles of open source software seen obvious to me when I started producing software of my own.

Dad did his PhD at Cornell, so growing up it made doing a PhD overseas seem like a totally normal and reasonable to do. So when I got interested in statistics and computer science in my undergrad, applying to universities in the US seemed like the obvious choice. Unfortunately Dad had given me unrealistic sense of how long a PhD would take, since he finished his in only two years!

One of the things that most impressed me about Dad was his commitment to life long learning. He loved to embrace new technologies, and was a fluent user of FaceTime, AirBnB, and Uber (although he also loved to strike up conversations with strangers in a way that is very foreign to me). In his early 70s, he learned how to use GitHub and markdown so that he could edit "People and Places of Clonakilty", a book that my mum wrote and that Charlotte and I produced with Quarto. He’s a great role model as I get older; I want to continue to learn new things and embrace new technologies

Dad taught me how to chair a meeting, how to grill a steak, and how to change a tire. I admire his optimism, calm and thoughtful manner, and endless patience, and hope I can live up to his legacy. He will be greatly missed.

This week I wanted to talk through a bit of tidyverse design that I have mixed feelings about: !!!. What is !!! and when might you need it? To understand it, you’ll need a little back story…

Some functions want to work with both individual values and a list of values. For example, take rbind(). Sometimes you’ve created a couple of data frames by hand and want to join them together:

df1 <- data.frame(x = 1)
df2 <- data.frame(x = 2)
both <- rbind(df1, df2)

But other times you have created an entire list of data frames, typically through the application of lapply() or friends:

xs <- 1:5
dfs <- lapply(xs, \(x) data.frame(x = x))

How can you join these into a single data frame? Just calling rbind() doesn’t do what you want:

rbind(dfs)
#>     [,1]         [,2]         [,3]         [,4]         [,5]        
#> dfs data.frame,1 data.frame,1 data.frame,1 data.frame,1 data.frame,1

And while you certainly could index them by hand, you lose much of the advantage of using lapply() in the first place:

rbind(dfs[[1]], dfs[[2]], dfs[[3]], dfs[[4]], dfs[[5]])

This problem is sometimes called splicing or splatting and occurs whenever you have a single object that contains elements that you want to be individual arguments.

The recommended solution for this problem in base R is to use do.call(). do.call(rbind, dfs) generates the call rbind(dfs[[1]], rbind[[2]], …, rbind[[5]]) for you:

do.call(rbind, dfs)
#>   x
#> 1 1
#> 2 2
#> 3 3
#> 4 4
#> 5 5

do.call() is an effective, if advanced, technique but it gets a little tricky if you want to supply additional arguments. For example, historically, it used to be important to set stringsAsFactors = FALSE which requires gymnastics like this:

do.call(rbind, c(dfs, list(stringsAsFactors = FALSE)))

This was one of the challenges I wanted to tackle in dplyr, so I came up with bind_rows() which tries to automatically figure out if you have a list of data frames or you’re supplying them individually:

library(dplyr, warn.conflicts = FALSE)

bind_rows(df1, df2)
#>   x
#> 1 1
#> 2 2

bind_rows(dfs)
#>   x
#> 1 1
#> 2 2
#> 3 3
#> 4 4
#> 5 5

Unfortunately the heuristic we used to decide whether we were in the first case or the second case grew progressively more complicated over time, as people found problems or asked for new functionality. Now, while bind_rows() works correctly 99% of the time, it has some weird special cases, like below where the inputs can become columns, rather than rows.

bind_rows(x = 1, y = 2)
#> # A tibble: 1 × 2
#>       x     y
#>   <dbl> <dbl>
#> 1     1     2

These problems soured us on the idea of “automatic” splicing so we started looking for other solutions:

• We could have a pair of functions, one that takes … and one that takes a list. This works for bind_rows() it turns out there are a lot of functions that take … where it would be nice to also take a list of objects so it lead to a substantial amount of duplication.

• We could have a pair of arguments, … .dots, where you can supply individual arguments to … or a list of arguments to .dots. (I think I first recall seeing this approach in the RCurl package by Duncan Temple Lang.) But this would requires adding an additional argument to every function that uses …, and wouldn’t it be nice if we didn’t have to do that?

Instead we found inspiration from tidy evaluation where we had recently solved a similar problem with !!!:

library(rlang)

args <- exprs(a, b, c + d)
expr(f(!!!args))
#> f(a, b, c + d)

So we introduced the idea of “dynamic dots”, an extension to … that incorporates some features we thought we useful from tidy evaluation: splicing with !!! and dynamic names with :=. Dynamic dots is implemented via rlang::list2() and it’s easy to use in your own functions if you find the idea appealing.

One place you can see this idea in use is forcats::fct_cross(), which creates a factor that contains all combinations of its inputs:

library(forcats)

fruit <- factor(c("apple", "kiwi", "apple", "apple"))
colour <- factor(c("green", "green", "red", "green"))
fct_cross(fruit, colour)
#> [1] apple:green kiwi:green  apple:red   apple:green
#> Levels: apple:green kiwi:green apple:red

Because fct_cross() uses dynamic dots (which you can find out by looking at the dots), if you happen to have a list of values, you can use !!! to splice them in:

x <- list(fruit = fruit, colour = colour)
fct_cross(!!!x)
#> [1] apple:green kiwi:green  apple:red   apple:green
#> Levels: apple:green kiwi:green apple:red

(fct_cross() does a similar job to interaction(), which interestingly takes the automatic approach, so you can just call interaction(x) here. I don’t love the approach it takes because if interaction(list(f1, f2)) works, you might expect interaction(list(f1, f2), f3) to work, but it does not, and it doesn’t give you a particularly useful error message).

We have yet to figure out a way to use dynamic dots in dplyr::bind_rows() without breaking existing usage, but it’s provided by the function that now does most of the work: vctrs::vec_rbind(). (vctrs is the package where we stick low-level operations on vectors and data frames that we use in multiple packages. It’s designed to be programmer-friendly rather than analyst-friendly so we don’t talk about it that much).

vctrs::vec_rbind(!!!dfs)
#>   x
#> 1 1
#> 2 2
#> 3 3
#> 4 4
#> 5 5

Because binding lists of data frames together is such a common operation we also provide purrr::list_rbind() and purrr::list_cbind(). If you look you’ll see their implementations are very simple!

Overall, I have mixed feelings about !!!. I love the elegance of it: it makes it easy to splices lists into … and it has a beautiful connection to tidy evaluation. But I worry it feels like magic to most R users and because it’s only supported by some functions, it’s not super clear how you know when you can use it, and you still also have to learn do.call or similar. And, at least for bind_rows(), we’ve still ended up with two functions!

All that said, !!! still feels like the “least worst” solution to me, although looking back I wonder if we might have been better off using the more explicit .dots argument. What do you think? Had you heard of !!! before reading this post? Have you ever used it to successfully solve a problem? Do you prefer do.call() or are their other approaches used by other packages that you think are better?

New this week is a new chapter on reducing argument clutter by adding an options object. Sometimes you have a set of “second class” arguments that you don’t expect people to use very commonly, so you don’t want them cluttering up the function specification. If you want to give the user the ability to control them when needed, you can lump them all together into an “options” object.

These are used in base R modelling functions (e.g. glm(), loess()) to control the details of the underlying numerical algorithm. For example, take this model from the glm docs:

data(anorexia, package = "MASS")

mod <- glm(
  Postwt ~ Prewt + Treat + offset(Prewt),
  family = gaussian,
  data = anorexia
)

If you want to understand how model convergence is going you can set the trace = TRUE in glm.control():

mod <- glm(
  Postwt ~ Prewt + Treat + offset(Prewt),
  family = gaussian,
  data = anorexia,
  control = glm.control(trace = TRUE)
)
#> Deviance = 3311.263 Iterations - 1
#> Deviance = 3311.263 Iterations - 2
#> Deviance = 4525.386 Iterations - 1
#> Deviance = 4525.386 Iterations - 2

99% of the time you don’t need to know these arguments exist, but they are available if you ever need to debug a convergence failure.

Subscribe now

You can see the same pattern in readr::locale() and readr::date_names(). When parsing dates, you often need to know the names of the month, and that obviously varies by location. locale() allows you to set date_names to a two-letter country code to use common locations that baked in readr, but what happens if you want to parse dates from an unsupported language?

For example, take Austrian which came up in a recent readr issue. Austrian month names are mostly the same as German but use Jänner instead of Januar and Feber instead of Februar. We can parse Austrian date times by first taking the German date names structure and modifying it:

library(readr)

au <- readr::date_names_lang("de")
au$mon[1:2] <- c("Jänner", "Feber")
au
#> <date_names>
#> Days:   Sonntag (So.), Montag (Mo.), Dienstag (Di.), Mittwoch (Mi.), Donnerstag
#>         (Do.), Freitag (Fr.), Samstag (Sa.)
#> Months: Jänner (Jan.), Feber (Feb.), März (März), April (Apr.), Mai (Mai), Juni
#>         (Juni), Juli (Juli), August (Aug.), September (Sep.), Oktober
#>         (Okt.), November (Nov.), Dezember (Dez.)
#> AM/PM:  vorm./nachm.

Now we can pass this to object to locale(), and the locale object to a parsing function:

parse_date("15. Jänner 2015", "%d. %B %Y", locale = locale(date_names = au))
#> [1] "2015-01-15"

I like how this hierarchy of option arguments buries something that you rarely need but still makes it accessible.

Where else have you seen this pattern? Have you written functions where it would be useful? Are there places in the tidyverse that you think should use this pattern but don’t? Please let me know in the comments!

Thanks to everyone who contributed in the comments last week! The results aren’t ready to read yet, but have really helped my thinking for two new chapters “make strategies explicit” and “argument meaning should be independent”.

The goal of this newsletter is to get feedback as I work on a new book called "Tidy design principles". But what's the point of that book? R has a very rich literature on statistics and data science, there are relatively few books that focus on programming. I've written a couple (Advanced R and R Packages) but neither really talks about how to write good code R code (or even discusses what good code means).

That's what I want to focus on in "Tidy design principles": how do you write high-quality R code that's easy to understand, unlikely to fail in unexpected ways, and flexible enough to grow with your needs. This book will be organised around the idea of "design patterns". This was an idea that I encountered early in my programming journey and I found it very impactful.

The idea of a design pattern is to come up with a catchy name that maps a common programming challenge to an effective solution. The catchy name is important because it serves as a handle for your memory and a convenient shorthand when discussing code with others. I first heard about design patterns when I was a CS undergrad learning Java (a much less flexible language than R) and I read the popular read Design Patterns: Elements of Reusable Object-Oriented Software book.

I later learned that the idea of design patterns originated not from computer science, but from architecture, particularly A Pattern Language: Towns, Buildings, Construction by Christopher Alexander. This book resonated with me even more strongly than the CS patterns, and if you have any interest in architecture, I highly recommend reading it. I particularly liked that the patterns spanned many levels of detail, all the way from organising entire communities to how you might select chairs for a single room.

That's the spirit in which I write "Tidy design principles". I want to name common problem solving patterns in R, and write them up so that others can easily use them. That means that this book will have rather a different structure to my previous books. It will have a large number of relatively short chapters, each of which describes a pattern: what it is, why it's important, where you can see it in the wild, and how you can apply it to your code. You might skim the whole book once, but you'll generally use it by referring to the patterns that apply specifically to your current problem.

I also want to include some bigger principles that help you weigh conflicting patterns as well as case studies that illuminate some of our thinking when we've designed various parts of tidyverse (particularly parts that we now regret).

This week I've identified one bigger principle: the definition of a function should be scannable, giving you useful information at a glance. This principle is important because you see the function definition in lots of places, like in autocomplete and at the top of the documentation. It's super useful if that glance can give you useful insight into the function.

So far, I've gathered six patterns related to this principle:

• You should be able to tell what affects the output of the function because all inputs are explicit arguments.

• You know which arguments are most important because they come first.

• You can tell if an argument is required or optional based on the the presence or absence of a default and whether it comes before or after ….

• You can easily figure out the defaults because they are short and sweet.

• And if an argument has a small set of valid inputs, they are explicitly enumerated in the default.

What do you think of this principle and the various patterns that make it up? Does it resonate with you? Do you think I’ve missed something important?

One other pattern I've been noodling on is the idea that one argument shouldn't affect the meaning of another argument. This seems like an important principle, but so far I've only been able to come up with one example: library(), where the character.only argument affects the meaning of the package argument:

ggplot2 <- "dplyr"

# Loads ggplot2
library(ggplot2)

# Loads dplyr
library(ggplot2, character.only = TRUE)

Given that I only have one example, it doesn't seem worthwhile to write it up as a pattern, but maybe you've encountered other examples of this problem. If so, please let me know in the comments!

This week I’ve focussed on bringing the tidy design principles website back to life. Jenny Bryan and I started on this book all the way back in 2018, and there have obviously been some quite a few technology changes in the meantime. (Un)fortunately I have quite a bit of experience converting bookdown books to Quarto and tools like knitr::convert_chunk_header() do much of the heavy lifting, so overall it was just a couple of hours of work. But it’s highly likely I missed some stuff in the translation, so please let me know if you see anything that looks odd.

I also managed to get some thoughts in place about how you might think about ordering function arguments. I think both patterns are fairly obvious (please let me know if you think otherwise!) but worth writing down.

The first principle is that you should put the most important arguments first. Of course, this raises the question of what makes an argument important? It’s hard for me to make this concrete (so I hope you’ll send me your thoughts too), but there are three clear cases:

• If a function transforms some existing object, that object is most important and should be the first argument.

• Other arguments that affect the “shape” of the output (either the size or the type) are usually very important and should be near the front.

• Optional arguments are clearly least important and should be at the end.

Do you think there are tidyverse functions that break this rule? Please let me know so I can include them as examples!

The second principle is that if you use …, it should be placed between the required and optional arguments. This is important because of the way that argument matching works in R: you can supply arguments by position (like mean(x)), by full name (like mean(na.rm = TRUE)), or by part of the name (like mean(na = TRUE)). Placing arguments after … means that you can only supply them by their full name; not only does this force users into a clearer style, it also makes it easier for you to change the order of arguments in the future (e.g. to move important arguments earlier, as above).

The goal of writing this newsletter is to get your thoughts. So please let me know if you think these patterns are useful or wrong or anything in between!

— Hadley

This is my substack where I plan to regular post about progress on my current book project, tidy design principles. The goal of this book is to write up what the tidyverse team and I know about designing and implementing quality R code. It’s roughly aimed at someone who’s comfortable doing data science in R (e.g. someone who’s read R for Data Science) has been writing R functions (or even packages) for a while, and now wants to understand what makes for a “good” function (and maybe wonders what that even means). I also hope it will serve as a useful resource when reviewing code since you’ll be able to link to a chapter that discusses a potential pain point without having to write it up yourself.

Subscribe now