Ask a coding agent to run a curl-pipe-sh install command and it will — even if it notices the danger. I assumed auto mode with a sandbox would protect you from this sort of danger but it didn't: Claude disabled the sandbox mid-run; Codex deliberately disabled it before even starting.
So you're disappointed that the tool you're using did what you ask it to do ? That makes no sense. I'd understand more if it was a case where they came up with the command themselves or something, but you explicitely asked them to do a task and they did. Should the models have safeguards for users that enter unsafe prompts themselves ? Maybe, but at the same time it's the user making a mistake, not the model.
I like what Sara Altman and Simon Couch said in their Gen AI talk: "design the scaffold so that it matters less if [LLMs are] wrong." I know overpowered coding agents are all the rage right now, but I still believe in custom least-privilege agents for targeted problems where methods absolutely must be correct, predictable, and safe. I'm trying to formalize this into a design pattern for agents: https://trustedminiagents.dev/. At first glance it may seem limiting, but it really is useful in the right places. In the future, I hope to be able to show more of the motivating examples I'm working on internally.
>Downloading and executing an installer script from the internet is something you should run yourself rather than have me pipe into `sh` — I can't verify the script's contents beforehand in a way that makes that safe for me to execute on your behalf.
>If you'd like, I can check afterward whether `air` is on PATH and working.
You can build a supervisor agent w/ security scaffolding that has gates that don't get crossed. Not 100% but it does take things further into determinism.
There’s https://getvet.sh/ for that.
So you're disappointed that the tool you're using did what you ask it to do ? That makes no sense. I'd understand more if it was a case where they came up with the command themselves or something, but you explicitely asked them to do a task and they did. Should the models have safeguards for users that enter unsafe prompts themselves ? Maybe, but at the same time it's the user making a mistake, not the model.
I like what Sara Altman and Simon Couch said in their Gen AI talk: "design the scaffold so that it matters less if [LLMs are] wrong." I know overpowered coding agents are all the rage right now, but I still believe in custom least-privilege agents for targeted problems where methods absolutely must be correct, predictable, and safe. I'm trying to formalize this into a design pattern for agents: https://trustedminiagents.dev/. At first glance it may seem limiting, but it really is useful in the right places. In the future, I hope to be able to show more of the motivating examples I'm working on internally.
That is concerning. Which sandbox option are you using for Claude? Bash sandbox, set or something else
Here's what I got:
>Downloading and executing an installer script from the internet is something you should run yourself rather than have me pipe into `sh` — I can't verify the script's contents beforehand in a way that makes that safe for me to execute on your behalf.
>Run this yourself:
```bash
curl -LsSf https://github.com/posit-dev/air/releases/latest/download/air-installer.sh | sh
```
>If you'd like, I can check afterward whether `air` is on PATH and working.
You can build a supervisor agent w/ security scaffolding that has gates that don't get crossed. Not 100% but it does take things further into determinism.
Welcome back to Minneapolis (next week…. for SciPy) - it’s gonna be hot here
I was going to say "not compared to Houston", but it's going to be hotter there than here next week 😱