I just stumbled upon your page and I can't wait to read more! I would be curious about how roles as we know them might transition. For example, I work as a data scientist in higher ed. and I think that - because everybody around me is vibe coding now - my job will shift to building the frameworks that let others build well, rather than building tools or doing analyses myself.
When we look at AI's highest potential for disruption, it's in the rent seeking sector: finance, law, education, medicine, and administration. It's obviously not the case that these sectors are 100% rent seeking but they do contain a disproportionately large amount of it. Exploiting information asymmetry for private gains made from intentionally convoluted rules and gatekeeping is far more destructive to equality than tech billionaires. Unnecessary credentialism and the pursuit of it harms innovation. The pie only shrinks from rent seeking. Think of the legions of tax lawyers, accountants, and other specialists and all that intellectual horsepower dedicated to squaring someone's bill with the government. If AI shatters those types of enterprises, intelligent people will have to dedicate their lives to more productive endeavors and in doing so may lead to less inequality, not more.
Thank you Hadley for all you do. In my public service data science team in the Canadian government, we do our best using R to try to make the world a little bit better of a place.
I think the other potential hope for AI is that it becomes way easier for everyone to make cool and useful apps, so more people can make a living from it.
I use Claude Code to write Quarto files directly in terminal. If a good prompt is provided, or a reference is available, it could write a bunch of quarto files and render them.
Regarding the power consumption issue with AI, there is a simple idea that can make AI much greener: instead of a company building one $500M data center, they can build ten $50M data centers at widely distributed locations, AND tell the local governments "in exchange for rapid permitting, we'll agree to AN INTERRUPTIBLE POWER CONTRACT."
Why would this make things better? Because it would address the main issue with solar and wind power: their availability depends on the weather. In many places, sometimes renewable power exceeds local demand and sometimes there is not enough renewable power. One solution to this issue is upgrading the grid so there's more capacity to load balance by moving power to other places. But turning data centers into dispatchable loads would be a lot cheaper.
A company with a very large number of small data centers could optimize its power mix by shifting compute around to follow power availability.
Agree, we must feel excited and open, but also wary and be careful about the broader implications of these kind of technologies. That being said, I think Posit and the R community in general are well positioned for the AI era, if not maybe "too" well positioned. AI burnout is real and every day more people are getting fed up of everything being about AI. Yet it's still interesting and useful! It's hard to find balance!
Thank you for holding this conversation in the open. I look forward to reading your reflections. Two things come to mind I’d love to hear from you: 1) what are your impressions so far from you “6 meetings a week”? Common themes? Patterns? 2) what are emerging standards of practice for organizing the semantic layer to support AI-supported project development in reproducible research?
In my opinion, keeping code-first and human-centered APIs is more important than ever.
1. Code provides reproducibility, accountability, transparency, verifiability, and trust — things AI cannot reliably deliver today, and perhaps never will.
2. AI is excellent at writing code, so we should fully leverage that capability. We just need to keep AI away from processing data internally in opaque ways.
3. We still need to be able to judge code ourselves. The clearer and more understandable code is to humans, the better our chances of preserving that fundamental habit. The moment we need AI to review AI-generated code, we are doomed.
Glad to see you show back up in my inbox! Something I have noticed is not so much a "learning curve" as a "context curve" with AI, where my use and the tool's context and memory improve overtime. Some of this is deliberate on my part (getting better and refining .md files), some of it is the settings, some of it happens in the background during and across code sessions. That said, I am curious how you think about this and how it gets operationalized at Posit, or in your own work. This is something I have seen new-to-AI folks misunderstand but it seems pretty critical to deploying these tools well.
Interesting that you find this so helpful as I try to do as it as little as possible because I'm sceptical that skills/memories are going to be really transferable from model to model.
I am teaching data science with R at the DHBW in Mannheim Germany. My question is, how much basics of the tidyverse are useful. How can we combine basics and AI for students.
Thanks Hadley! One topic I’m interested in is how our work as developers changes which AI tooling. It feels like AI allows us to work at a higher level of abstraction (architecture, specifications) but has the downside of separating us from the code. Writing code is not only about explaining an idea to a computer but also about explaining it precisely to ourselves as well. How do you see our workflows and jobs changing in the new paradigm of capable coding tools? Is it ok to ship code I have not read or fully understand? I’ve used AI to solve problems I would not be able to solve on my own. I can verify the solution works but the complexity of the solution feels out of reach for my understanding, at least without a big investment of time. I’m curious what your thoughts on this are. Thanks again!
So glad to see this here and now. There is so much focus on the technology aspects of AI and all the remarkable discoveries now within reach. I'm paying attention to AI as a social change agent that has not only permanently shifted job and housing markets, it has changed the trajectory of more lives than one might assume. A gig economy of human AI trainers has already sprung up and we have yet to see an AI company IPO. AI has completely changed how I work and how I engage with collaborators. I notice myself interacting with it as if I'm only asking a model or machine a question as opposed to the masses of humans that created the information within it. The real disruption goes beyond the markets, it's changing how we interact with each other and how we interact with information and complexity. AI is reshaping social resources like trust in institutions and source or subject matter experts. I work in public health and we are already seeing people turning to AI for health information instead of consulting their providers. This will change real health outcomes for real people without any implication on the part of the AI companies. How else do you see AI reshaping us socially and what do you see as the role of open-source?
Hi Hadley - Glad to see this. Thanks for highlighting the issues of sustainability and inequality. They are less discussed than effects on the job market and academic integrity, but I think that these latter concerns are more readily addressable - we've figured out how to incorporate technology into the workplace and schools before. The environmental impacts of AI and the likely effects on inequality may be massive and very difficult to reverse. Should we consider these when we ask lazy, trivial, and/or clever queries of Claude or Copilot? Should our students?
Hi! As someone who also works in data science and research, I find myself constantly wrestling with AI's benefits and its hidden costs, often at the expense of society and the global environment. I'd love to hear how you think about this: How do you write prompts consciously? Has your approach evolved over time? What would you recommend to others in this field? And how can we as a collective advocate for ethical, environmentally-conscious AI use at both an individual and systemic level (e.g. prompt efficiency, knowing when and when not to use AI, data center standards, privacy, protecting those who don't use AI, etc.)?
I look forward to reading more from your substack!
I'm curious about your experience writing *new* things with AI, vs tweaking existing things. Have you done much in that regard, whether it's adding a function to an existing R package, or creating a new R package from scratch? I see your AGENT.md at https://github.com/r-lib/usethis/blob/main/inst/claude/CLAUDE.md doesn't mention any of the things from this blog / https://design.tidyverse.org/ , but I wonder if that's because you aren't doing things where that matters. The function signatures already exist, so you don't need it to make choices there.
I'm also curious whether your image of your "user" is changing. Do you anticipate humans continuing to be the primary consumers of your packages/functions, or are you starting to think of agents using the functions to answer users' questions?
Basically a lot of what I'm thinking about with GenAI lately is how much the UX side of things will matter in 3 years. If an agent can have full docs freshly "in mind" when using a function and can automatically retry when its assumptions fail, does that change the effort and focus around clean function signatures, error messages, etc? Or does that stuff remain just as important, to make sure the function behaves how the user (human or machine) expects your code to behave, and how it reacts when something doesn't work as expected?
Lovely to see this return and what a fantastic summary of the opportunities and tensions. All of this resonates but it is very nice to see it collected in one space.
How do you resolve the tension? I would really love to hear your thoughts on where you might see open models in this future.
I just stumbled upon your page and I can't wait to read more! I would be curious about how roles as we know them might transition. For example, I work as a data scientist in higher ed. and I think that - because everybody around me is vibe coding now - my job will shift to building the frameworks that let others build well, rather than building tools or doing analyses myself.
When we look at AI's highest potential for disruption, it's in the rent seeking sector: finance, law, education, medicine, and administration. It's obviously not the case that these sectors are 100% rent seeking but they do contain a disproportionately large amount of it. Exploiting information asymmetry for private gains made from intentionally convoluted rules and gatekeeping is far more destructive to equality than tech billionaires. Unnecessary credentialism and the pursuit of it harms innovation. The pie only shrinks from rent seeking. Think of the legions of tax lawyers, accountants, and other specialists and all that intellectual horsepower dedicated to squaring someone's bill with the government. If AI shatters those types of enterprises, intelligent people will have to dedicate their lives to more productive endeavors and in doing so may lead to less inequality, not more.
Thank you Hadley for all you do. In my public service data science team in the Canadian government, we do our best using R to try to make the world a little bit better of a place.
I think the other potential hope for AI is that it becomes way easier for everyone to make cool and useful apps, so more people can make a living from it.
I use Claude Code to write Quarto files directly in terminal. If a good prompt is provided, or a reference is available, it could write a bunch of quarto files and render them.
Regarding the power consumption issue with AI, there is a simple idea that can make AI much greener: instead of a company building one $500M data center, they can build ten $50M data centers at widely distributed locations, AND tell the local governments "in exchange for rapid permitting, we'll agree to AN INTERRUPTIBLE POWER CONTRACT."
Why would this make things better? Because it would address the main issue with solar and wind power: their availability depends on the weather. In many places, sometimes renewable power exceeds local demand and sometimes there is not enough renewable power. One solution to this issue is upgrading the grid so there's more capacity to load balance by moving power to other places. But turning data centers into dispatchable loads would be a lot cheaper.
A company with a very large number of small data centers could optimize its power mix by shifting compute around to follow power availability.
I’d definitely be curious to hear more about “effective use of git” being more within reach to a broader audience.
looking forward to reading more of your thoughts on all of this. Thanks for rebooting the Substack.
Agree, we must feel excited and open, but also wary and be careful about the broader implications of these kind of technologies. That being said, I think Posit and the R community in general are well positioned for the AI era, if not maybe "too" well positioned. AI burnout is real and every day more people are getting fed up of everything being about AI. Yet it's still interesting and useful! It's hard to find balance!
Thank you for holding this conversation in the open. I look forward to reading your reflections. Two things come to mind I’d love to hear from you: 1) what are your impressions so far from you “6 meetings a week”? Common themes? Patterns? 2) what are emerging standards of practice for organizing the semantic layer to support AI-supported project development in reproducible research?
Both of those are already on my list 😀
In my opinion, keeping code-first and human-centered APIs is more important than ever.
1. Code provides reproducibility, accountability, transparency, verifiability, and trust — things AI cannot reliably deliver today, and perhaps never will.
2. AI is excellent at writing code, so we should fully leverage that capability. We just need to keep AI away from processing data internally in opaque ways.
3. We still need to be able to judge code ourselves. The clearer and more understandable code is to humans, the better our chances of preserving that fundamental habit. The moment we need AI to review AI-generated code, we are doomed.
Glad to see you show back up in my inbox! Something I have noticed is not so much a "learning curve" as a "context curve" with AI, where my use and the tool's context and memory improve overtime. Some of this is deliberate on my part (getting better and refining .md files), some of it is the settings, some of it happens in the background during and across code sessions. That said, I am curious how you think about this and how it gets operationalized at Posit, or in your own work. This is something I have seen new-to-AI folks misunderstand but it seems pretty critical to deploying these tools well.
Interesting that you find this so helpful as I try to do as it as little as possible because I'm sceptical that skills/memories are going to be really transferable from model to model.
That said, I do maintain https://github.com/r-lib/usethis/blob/main/inst/claude/CLAUDE.md, which I use across all R packages that I work on.
I am teaching data science with R at the DHBW in Mannheim Germany. My question is, how much basics of the tidyverse are useful. How can we combine basics and AI for students.
Good question! I have a few thoughts which I'll write up for a future post.
Thanks Hadley! One topic I’m interested in is how our work as developers changes which AI tooling. It feels like AI allows us to work at a higher level of abstraction (architecture, specifications) but has the downside of separating us from the code. Writing code is not only about explaining an idea to a computer but also about explaining it precisely to ourselves as well. How do you see our workflows and jobs changing in the new paradigm of capable coding tools? Is it ok to ship code I have not read or fully understand? I’ve used AI to solve problems I would not be able to solve on my own. I can verify the solution works but the complexity of the solution feels out of reach for my understanding, at least without a big investment of time. I’m curious what your thoughts on this are. Thanks again!
So glad to see this here and now. There is so much focus on the technology aspects of AI and all the remarkable discoveries now within reach. I'm paying attention to AI as a social change agent that has not only permanently shifted job and housing markets, it has changed the trajectory of more lives than one might assume. A gig economy of human AI trainers has already sprung up and we have yet to see an AI company IPO. AI has completely changed how I work and how I engage with collaborators. I notice myself interacting with it as if I'm only asking a model or machine a question as opposed to the masses of humans that created the information within it. The real disruption goes beyond the markets, it's changing how we interact with each other and how we interact with information and complexity. AI is reshaping social resources like trust in institutions and source or subject matter experts. I work in public health and we are already seeing people turning to AI for health information instead of consulting their providers. This will change real health outcomes for real people without any implication on the part of the AI companies. How else do you see AI reshaping us socially and what do you see as the role of open-source?
Hi Hadley - Glad to see this. Thanks for highlighting the issues of sustainability and inequality. They are less discussed than effects on the job market and academic integrity, but I think that these latter concerns are more readily addressable - we've figured out how to incorporate technology into the workplace and schools before. The environmental impacts of AI and the likely effects on inequality may be massive and very difficult to reverse. Should we consider these when we ask lazy, trivial, and/or clever queries of Claude or Copilot? Should our students?
Hi! As someone who also works in data science and research, I find myself constantly wrestling with AI's benefits and its hidden costs, often at the expense of society and the global environment. I'd love to hear how you think about this: How do you write prompts consciously? Has your approach evolved over time? What would you recommend to others in this field? And how can we as a collective advocate for ethical, environmentally-conscious AI use at both an individual and systemic level (e.g. prompt efficiency, knowing when and when not to use AI, data center standards, privacy, protecting those who don't use AI, etc.)?
I look forward to reading more from your substack!
I'm curious about your experience writing *new* things with AI, vs tweaking existing things. Have you done much in that regard, whether it's adding a function to an existing R package, or creating a new R package from scratch? I see your AGENT.md at https://github.com/r-lib/usethis/blob/main/inst/claude/CLAUDE.md doesn't mention any of the things from this blog / https://design.tidyverse.org/ , but I wonder if that's because you aren't doing things where that matters. The function signatures already exist, so you don't need it to make choices there.
I'm also curious whether your image of your "user" is changing. Do you anticipate humans continuing to be the primary consumers of your packages/functions, or are you starting to think of agents using the functions to answer users' questions?
Basically a lot of what I'm thinking about with GenAI lately is how much the UX side of things will matter in 3 years. If an agent can have full docs freshly "in mind" when using a function and can automatically retry when its assumptions fail, does that change the effort and focus around clean function signatures, error messages, etc? Or does that stuff remain just as important, to make sure the function behaves how the user (human or machine) expects your code to behave, and how it reacts when something doesn't work as expected?
Lovely to see this return and what a fantastic summary of the opportunities and tensions. All of this resonates but it is very nice to see it collected in one space.
How do you resolve the tension? I would really love to hear your thoughts on where you might see open models in this future.