Like many people in the industry, I have been immersed in a journey to get acquainted with LLM-assisted coding, and that journey has had lots of ups and downs. First I started using Copilot on VS Code, then embraced tools like Zed and OpenCode, and these days I find myself mostly using open weights models through OpenRouter with Pi as a harness. I have been able to quickly swap tools and configurations thanks to having all of that isolated from my host system inside a Multipass VM, even if both my desktop and my development environment run Ubuntu LTS!
After all this time, I wouldn’t even call myself a power user - some colleagues have achieved truly remarkable results by leveraging swarms of agents and created LLM-powered software. I am still someone who thinks that we have to look at the code we ship and retain a thorough understanding of what our systems do, despite the help of LLM-assisted code reviews.
Turns out it is not easy to navigate this middle ground, given that LLMs generate code much faster than we can review it, and it’s so easy to let them loose and instantly lose grasp of what is going on.
I have reached a point in which I can say I am somewhat satisfied with my LLM-assisted process, and wanted to take the opportunity to share some lessons learned along the way.
Divergent-convergent process
You probably have experienced this yourself: you give your agent a task, it produces a reasonable[1] solution after a certain of time, and then you (or your coworker) reviews it and thinks… Is all this cruft really necessary?
LLMs are very fast at generating candidate solutions. But also, they sometimes produce throwaway code while exploring an approach that isn’t always cleaned up.
After getting burned several times by this, I understood that it is necessary to trim or refine the initial solution that is produced.
If you think about it, this is a pattern that has already been described in our industry. Shape Up, the famous product development methodology used at 37signals (formerly Basecamp), says that “work is like a hill”: figuring out followed by execution.
It also reminds us of the Discovery process, the preliminary phase of user experience design, also known as the first diamond of the Double Diamond introduced by the UK Design Council.
I find this divergent-convergent[2] metaphor accurately describes my LLM-assisted development process, and thinking about it this way has helped me be more deliberate about it.
For example, at the beginning of a problem solving process, I now prompt my LLM with verbs like “enumerate”, so that I can explore the solution space more thoroughly before committing to a specific one (diverge). I also exploit the speed of LLMs to do quick feasability explorations (“spikes” in Agile lingo) and discard dead ends early. Of all the remaining viable solutions I then use my best judgement to pick the one I consider simplest (converge).
For tangible ideas on how to perform this divergence process, I encourage you to read my colleague’s @ppasotti blog post on patterns for the agentic age.
Make outcomes more predictable by constraining the solution space
This seems to be against the previous idea, but I promise it makes sense. As I mentioned before, I am framing these learnings as someone who will eventually read the code being produced (in other words: I’m not “vibing”). The reason is that I expect to need some familiarity with how that code works, for whatever reason (for example, because I am on the hook for how it performs in production).
As such, even though I could potentially use any programming language[3] and have the LLM generate code in one with the best performance or the most capable open source ecosystem, it will be easier for me and my coworkers to incorporate the resulting artifact into our work if it’s written using a technology that’s already familiar for us.
Since I am mainly a Python guy, most of the time I request that the solution to the problem at hand is coded in Python, unless there are strong reasons to choose an alternative technology. And as a grumpy Python guy I tend to be picky around things like packaging, linters, and such.
I have found that unconstrained LLM-generated code usually results in unstructured spaghetti code that brings me no joy. On the other hand, if I do some scaffolding at the beginning, using my favorite selection of workflow tools and linters (which in my case is uv + ruff with my go-to rules + tox + pytest) and defining at least a basic directory structure, alongside some rules (“place the Pydantic models in models.py”, “make the CLI stuff in cli.py thin wrappers over the business logic”), it does help a lot steer the generation process and also lower my heart rate.
Which brings me to the next point.
Always include validation in the loop
You could consider linting as one way of establishing a tight feedback loop for your development process. One that only checks for certain aspects of the code, granted, but important to avoid trivial mistakes nonetheless.
One lesson that seems to not have changed in the agentic era is that small batch sizes and quick feedback loops are still important for successful software delivery. People like Jason Gorman have been very vocal about this in recent months.
With this in mind, defining how the validation is going to look like is an essential part of the task itself. It doesn’t even have to be you prescribing it, but at the very least the agent should provide a plan on how success is going to be measured. This can be unit tests, integration tests, browser automations, or any other form of autonomous testing. Whatever it is, it’s much better than nothing.
A few days ago I wanted to export some Miro boards to Excalidraw for reasons. I didn’t find anything on GitHub that would suit my needs, so I gave the task to Pi and GLM 5.3 Flash. Since the model has image processing capabilities, I included in the prompt instructions to render the converted board, compare it to the original PDF I exported, and stop when the result was sufficiently close (“sufficiently” being quite vague on purpose). Surely the result wasn’t perfect, but it preserved the texts and shapes and gave me an starting point, which is exactly what I needed!
Context windows are tricky and auto-compaction is bad
As I ramped up my AI usage and started experimenting with long sessions, I started experiencing problems with auto-compaction. Models advertise a maximum context window that the API accepts without throwing an error, and most harnesses have default mechanisms that trigger past a certain threshold to summarize the conversation up to that point and insert it in the context of the following turns. However, in my experiments, I have found those summaries to be of very bad quality - they included unimportant details about problems that had been sorted out already, be poorly structured, and omitted relevant information of the current process.
At some point I stopped worrying about this though, because some models advertise now context windows of 1 million tokens. However, recently I observed some sudden performance degradation with one “Flash” model during a long session. Turns out there could be a number of reasons for this, some of them intimately tied to the architecture of these models, some others connected to the quantization, cache handling, or serving infrastructure that each provider uses. In any case, I could not let this silently corrupt my sessions, and I also didn’t want to rely on the default auto-compaction mechanisms.
So I went ahead and used Pi to create a Pi extension! @astrojuanlu/pi-handover uses a simple template (“Mission, What has been done, What we have learned, What is next”) to create a structured handover document for the next turn, that replaces the context and lets the agent continue. I have been using it only for a few sessions, but so far the results have been great.
The lessons above might or might not apply to you. I think there is still a lot we can do, as an industry, to conduct evidence-based evaluations in real world scenarios, beyond the benchmarks you are used to see. Until then, I encourage you to do your own experiments!
And of course, if any of this did help, or you have other painful lessons you want to share, I would love to hear from you in the comments.
provided the task is possible ↩︎
Terminology borrowed from an engineering marvel I used to study in my youth ↩︎
Or even instruct the LLM to write the bytes directly to disk ↩︎