Why a working demo is not a finished product
Generative models have reached the point where a single prompt, even a rough one, to a model such as Fable 5 or GPT 5.6 Sol can produce a working app in about twenty minutes. Watching a demo like that, it is easy to assume the model is now doing most of the work of building software for you.
That is not far from the truth for the demo itself, but the assumption breaks down once the app has to work in production. A demo built from a clean prompt says nothing about whether it can read a company's actual documents, which rarely stick to a strict structure, follow its actual processes, or talk to the decades-old tools its leadership team still relies on for some reason. Most of the impressive one-shot apps you see online are glorified proofs of concept, and very few of them reach production. The gap between demo and finished product usually sits outside the model itself: in the documents it has to read, the processes it has to follow, the tools it has to talk to, and the steps in the job that never touch a screen at all.
Working with clients, we see five categories of problems again and again where a more capable model does not help: documents and knowledge spread across multiple formats, processes that were never written down as a workflow or an SOP, tools that have been in use for years with no API, information that changes after the model's training cutoff, and steps in a process that need a person physically present.
Documents scattered across formats a model cannot read
Companies do not keep their knowledge in one place, and they do not keep it in formats a model can open natively. Modern models handle standard documents such as PDFs, Word files and Excel spreadsheets reasonably well, and there are still formats LLMs understand especially cleanly, like Markdown and CSV. A lot of the software companies still run on does not produce standard documents at all.
One of our founders spent years doing data analysis in the pharmaceutical industry. Clinical trials there are heavily regulated by the FDA, but the systems used to run them are frequently custom-built, and they export data in proprietary formats designed for that one system. Pharma is a large, well-funded industry, yet it is surprisingly resistant to innovation in areas like IT, and the tools used there were often built when modern AI was mostly the stuff of science fiction.
Feed a model one of those exports directly and it will often make a confident guess at the structure instead of admitting it cannot parse the format. That guess can look plausible while being wrong in ways that are hard to catch without already knowing what the file should contain. For this reason, before connecting a model to any data source, check what format the source actually produces, not what the documentation says it produces. A custom export usually needs a purpose-built parser or converter sitting between the file and the model, translating it into something the model can read reliably. But even a good parser does not remove the need to validate what comes out the other end. A well-built converter reduces the chance of the model misreading a file, but it does not certify that the reading was correct. That check is still a job for a person, or a separate validation step, for anything the company would act on.
Processes that were never written down
Modern job descriptions rarely match the actual job. Every company runs processes that were never formally documented because someone competent has always just handled them, case by case, using judgement built up over years. Feed a model a policy document and a RAG index and it will still miss the parts of the job that were never written into either.
Picture the same pharma team bringing on a new data analyst. Officially, the training covers the data dictionary and the reporting templates. In practice, a large part of onboarding is a senior analyst explaining which discrepancies in a report are worth flagging to a manager and which are known quirks from transferring information off doctors' forms into a central database. None of that lives in a document a model could be given.
A model has nothing to pattern-match against in a situation like that. Left without examples of the judgement calls the process actually depends on, it will still produce a confident, and wrong, answer. Not every step of an unscripted process needs a person to keep doing it. Map the process first, stage by stage, and look for individual steps that are genuinely repetitive and rule-based, even if the process around them is not. Automating just those steps is usually still worth doing, particularly if they are the time-consuming ones.
Mapping a process well enough is itself a job a model cannot do by reading the finished document. Most likely it means sitting with the person who currently does the work and asking them to narrate it, which is slower and less exciting than writing a prompt.
Tools with years of use and no API
Modern LLMs are good at using well-known software: browsers, common SaaS tools, anything with a documented API or an existing connector. Plenty of company-critical applications are none of those things.
The same pharma founder worked with the central database used to manage clinical trial data. It had been built in the mid-1990s and could only be operated with a keyboard: no mouse, no API, no export function beyond what its original developers had thought to include thirty years earlier. Systems like this are not rare. Companies keep them running for years, sometimes decades, often for reasons like validation cost, data continuity, or simple risk aversion around anything that touches regulated records.
Computer-use agents can technically click through an interface like this the way a person would. But there is a significant difference between a model that already knows the software from its training data and a model that has never seen this specific screen before, relies on a description rather than full documentation, and has to infer what every field and shortcut does from context alone. The odds of a wrong click go up sharply, and a wrong click in a regulated system carries real consequences.
Before giving a model access to a tool like this, check whether the software is common enough to be represented in the model's training data or connector ecosystem. If it is not, the safer route is to write the missing documentation yourself, screen by screen, and hand the model that as context alongside clear instructions, rather than letting it explore the interface unsupervised. Keep in mind that writing documentation takes real time, and it needs to be kept up to date as the tool changes. Treat it as ongoing maintenance, the same way you would treat any other piece of internal documentation that people actually rely on.
Information that changes after the model's training cutoff
Every model has a training cutoff date. After that point, it has no first-hand knowledge of what has happened in the world, including changes to the regulations, guidance documents, or industry standards it might be asked to reason about. Most current models can search the web to close that gap, but a search does not guarantee a correct answer. It adds a new source of error instead: the web is full of outdated, unofficial, and simply wrong pages about most topics, regulatory guidance included.
The fix is to constrain the model's web access rather than remove it. Point it at a small, named set of sources you already trust, the regulator's own site, a specific internal repository, rather than letting it search the open web freely. This turns an open-ended search into something closer to checking a reference you already vouch for.
Restricting sources reduces the chance of the model citing something wrong, but it does not eliminate it. A named source can still be misread, and a genuinely new answer that only exists on a source outside the approved list will be missed entirely. Someone still needs to be around to spot-check anything that will inform a real decision.
Steps that need a person in the room
Some parts of a business process are not information problems at all. They are physical ones. A monitor still has to visit a trial site in person, check that paperwork matches what is happening in the room, and build trust with site staff face to face. No model, however capable, does any of that from a laptop. And this is not unique to the pharmaceutical industry. Sales demos, in-person implementation work, and hands-on customer support depend on physical presence, and no current AI system changes that. This claim might age badly as robotics advances, but that looks like a multi-year horizon, not something to plan around today.
None of this means every physical step is permanently off-limits to automation. Look at what actually requires a body in the room versus what has just always been handled that way. A monitoring visit might still need a check on physical drug storage conditions that cannot be done remotely, but the paperwork review that happens alongside it often can be. Multimodal models that accept photo, audio, and video input can sometimes stand in for a written report of a physical check, even when they cannot do the check itself.
Splitting a physical process usually means a detailed look at what the in-person part is actually for, and whether alternatives exist. Assuming a step is safe to move online just because most of it could theoretically be described in text is how a company ends up cutting a physical check that mattered for reasons nobody wrote down.
These problems usually show up together
These five categories rarely appear in isolation. The pharma team we mentioned a few times deals with several of them at once: proprietary file formats, a decades-old system with no API, judgement calls that were never documented, and site visits that need a person physically present. A single business process can be blocked by more than one of these at the same time, and fixing only one of them will not make the process work end to end.
This is also where it is easy to overcorrect. Not every one of these five categories needs its own dedicated fix before you start. If a process only touches one or two of them, and the rest of the workflow is straightforward, building around just those gaps is often enough. Treating every business process as though it needs a parser, a mapped SOP, custom tool documentation, and a validated data source all at once is how a small automation project turns into a six-month infrastructure programme before it has proven any value.
A working app is only the first step
Getting a model to produce working code for a generative app is, at this point, close to a solved problem for a well-scoped task. Getting that app to survive contact with a real company is a different project, and it sits mostly outside the model's control. It means mapping which of a company's data sources the model can actually read and which need a custom parser first, documenting the unscripted parts of a process well enough that a model, or a new employee, could follow them, building or commissioning integrations for the tools that have no API, deciding which sources a model is allowed to treat as authoritative for anything time-sensitive, and identifying which steps in a process genuinely require a person physically present, and which only look that way out of habit.
None of this is model work. It is systems and process work, and it tends to take longer than writing the prompt that produced the demo in the first place.
Final thought
A capable model narrows the gap between an idea and a working prototype, but it does not remove the work of fitting that prototype into a real company. That work is mostly about documents, processes, tools, and the occasional step that only a person can get past. Take one process you are hoping to hand to an LLM and check it against these five categories. If it touches more than one, plan and analyse what should be done about them before touching keyboard to write the prompt, not after the demo has already impressed everyone in the room.
Which of these five categories is the biggest blocker for your process right now: the data, the workflow, the tools, the freshness of the information, or the physical steps? Let us know, and we can help you work out what is worth automating first.