Engineering AI to Work with Applications That Have No APIs
Many enterprise applications were never designed for modern integrations. We engineered an intelligent UI automation capability that uses vision-language models to interpret application screens and carry out tasks through their existing interfaces.
By combining visual reasoning with reusable workflows and controlled execution, the system can navigate complex interfaces and handle changing information without requiring a custom integration for every application.
The situation
Established enterprise applications often contain valuable information and support important business processes, yet provide no practical API for the work an organization needs to automate.
When direct integration isn’t possible, automation has to work through the application’s interface: navigating screens, selecting controls, scrolling through tables and reading the information displayed.
Building separate scripts for each application can be expensive to maintain, particularly when interfaces differ or data changes between executions. At the other extreme, relying on AI to rediscover every interaction on every run introduces unnecessary latency and processing costs.
We needed a way to combine the flexibility of visual AI with the efficiency and predictability of conventional software execution.
What made it hard
The challenge went well beyond recognizing buttons or clicking through screens. Real enterprise applications present problems that are difficult to solve with either rigid scripts or AI reasoning alone.
Interfaces don’t behave consistently.
Applications present controls in different locations and formats. Navigation patterns vary, and changing data can alter the screen state between executions. The system needed to interpret interfaces during workflow authoring while accommodating selected dynamic conditions during execution.
Important information isn’t always visible.
Tables can stretch across several pages, require scrolling in multiple directions or display only part of a value. Extracting complete information means navigating across views while keeping track of what has already been collected.
In some cases, the system must manipulate the interface to make information accessible, such as expanding a narrow column to read a truncated value and then restoring the original layout.
An action doesn’t always produce the expected result.
Identifying a control is not the same as interacting with it successfully. An action may target the wrong position or fail to produce the intended change.
The system needed to verify interactions, detect certain failures and recover rather than blindly continue through a sequence of actions.
Intelligence has an execution cost.
Vision-language models are valuable when interpreting unfamiliar screens or making decisions. But interpreting every screen from scratch during every execution increases latency and model usage.
The engineering challenge was deciding when visual reasoning was necessary and when previously established interactions could be reused.
What we built
We developed an intelligent UI automation framework that combines visual workflow authoring with repeatable execution.
Vision-language models interpret application screens and help establish how a task should be performed. Rather than asking those models to rediscover the same workflow on every run, the system reuses what has already been established and invokes fresh visual reasoning when changing conditions require it.
Making that approach practical demanded more than screen recognition. We had to engineer navigation across complex interfaces, manage extraction state, accommodate changing selections and verify that interactions produced the intended results.
The architecture brings those responsibilities together while keeping model-driven interpretation separate from routine execution.
Conceptual sequence. Where interpretation is needed depends on the task and the screen.
General-purpose vision-language models interpret screenshots, identify interface elements and help determine the actions a task needs, without training a dedicated recognition model for each application. Controlled software execution turns that understanding into interactions.
During authoring, the system navigates the application and establishes a reusable sequence of interactions. Routine steps then run without calling a model. Fresh visual interpretation is applied when selections, data or pagination differ from the original workflow. Significant structural changes to an application’s interface may still require the workflow to be authored again.
Relevant interactions are checked before and after execution. When supported, unsuccessful actions can be reassessed and retried. Rather than assuming every model decision is correct, the system is built to detect mistakes and respond to them.
Because it works through visible controls and standard interface interactions, the approach applies to web, desktop and terminal-style applications. The application doesn’t need to expose an API for the task.
Working through a complex interface
One of the harder engineering problems was extracting complete information from application interfaces that reveal only part of the data at a time.
A table might extend across multiple pages, require horizontal and vertical scrolling, or contain values truncated by narrow columns. Reading the current screen alone would not produce a complete result.
We engineered the system to navigate across views, track what had already been collected and reveal information that wasn’t initially visible. In some cases, that meant expanding a column to read its full contents and then restoring the layout so subsequent interactions remained consistent.
Verification was equally important. Relevant interactions needed to be checked against their expected outcomes, with recovery where supported and where an action failed.
The accompanying illustration shows these engineering challenges through a generic interface rather than a customer application.
- Table and its controls located on page 2 of 6
- More columns sit beyond the right edge
- Description is truncated in its narrow column
- Column expanded temporarily
- Full value visible and captured
✓ Layout restored
- Original column width restored
- Extracted value compared with what was displayed
- Next page is ready to process
What changed
The result was a reusable automation framework capable of handling interactions that are difficult to capture in fixed, application-specific workflows.
Workflows that can be reused
Visual authoring establishes the interactions needed to accomplish a task, allowing subsequent executions to reuse the established workflow rather than rediscover every step.
Access to information beyond the visible screen
The system can navigate paginated and scrollable tables, manage information across multiple views and retrieve values that are initially truncated.
Adaptation to changing information
Execution can accommodate selected dynamic conditions, including different data volumes and changing selections, while routine actions follow the established workflow.
Execution that checks its own work
Verification and recovery mechanisms help detect unsuccessful interactions and reassess supported failures instead of blindly continuing.
Together, these capabilities provide a practical approach to automating work through existing application interfaces, even when conventional APIs are unavailable.
The engineering decision that mattered
The most important decision was not to make AI responsible for every action.
Visual reasoning is valuable when a system must understand an unfamiliar interface or respond to changing information. Once the steps of a workflow are established, much of the execution can be handled more predictably.
We engineered the system around that separation: use intelligence where interpretation is necessary, reuse what is already understood, and verify the results rather than assuming every action succeeded.
The result is a practical way to extend automation into existing enterprise applications without requiring them to be replaced or rewritten first. Model-driven interpretation, reusable workflows, controlled execution and verification each handle the work they are suited to.
Related work
Agent Builder
See how AI agents can be defined and created without coding each one individually.
Read the case study →Multi-agent orchestration
See how specialized agents are coordinated through one conversation.
Read the case study →AI app builder
See how we built an AI application builder on an enterprise-grade platform.
Read the case study →