Skip to content

Cloud Agents & Local Specialists

Keep a powerful cloud agent in charge while local models supply screen understanding, document parsing, video perception and generative tools.

Updated View as Markdown

A local model does not need to become a smaller version of the whole agent. It can give a powerful cloud agent a precise ability: read a screen, extract a table, identify an event in a video, or generate an asset. The cloud model continues to reason about the goal and decide what to do next.

The useful division is general reasoning plus specialized computation. OmniTool and Agent S demonstrate this separation for computer use. Whether the specialist runs on the user’s device, a nearby workstation or another server is a separate deployment decision.123

This page explains that architecture, the research behind it and how to test its value. For routing, cascades, decoding cooperation and split inference, start with the Local & Cloud AI overview.

For glasses, phones and cameras, continue with Smart Devices & Cloud Agents. It compares concrete research and products, distinguishes actual device inference from simulated deployment, and examines device, phone or hub, and cloud placement.

The architecture at a glance

Figure 1 is a proposed system design, not the internals of one product. The cloud agent requests bounded work. A local host mediates tool access, decides what evidence may leave the device and checks actions before execution. Specialists return observations or artifacts, not permission to act.

Pan on smaller screensOpen full size ↗

Figure 1. A cloud planner with local capabilities and an explicit policy boundary. Independent synthesis informed by modular agents and tool interfaces.24

Three responsibilities should remain distinct:

  • The agent chooses work: understand the goal, maintain task state and request the next useful observation or operation.
  • The specialist supplies a capability: parse, ground, transcribe, detect or generate within a defined interface.
  • The host enforces authority: restrict files, tools, network disclosure and consequential actions. A model’s suggestion is not authorization.

This differs from routing a complete question to either a small or large model. It also differs from splitting one neural network across devices. Here, independently deployable models exchange task descriptions and results.

What the current evidence establishes

Evidence reviewed October 17, 2026. The comparison separates demonstrated agent composition, specialist research and documented local deployment. It is not a ranking, a complete release list or a claim of handset readiness.

Systems, specialists and deployment boundaries

8 entries
Work and date Specialist contribution What is established What is not established
OmniParser V2 / OmniTool, February 2025 Parse screen elements for a general agent A working parser, planner and execution stack; CPU support and a separately deployable GPU parser service.13 Not a latency or energy result for a phone or laptop NPU.
Agent S2 / S3, 2025 research and implementation reviewed in 2026 Separate planning from visual, textual and structural grounding S2 studies specialist composition; current S3 exposes separate main and grounding model endpoints.25 A separate endpoint is not necessarily local; the recommended grounding service can be hosted.
ScreenVLM, February 2026 A 316M model returns screen elements, types, boxes and text Compact dense screen parsing with published evaluations.6 Its timing study uses an H100, not an edge device; parsing quality is not agent task success.
Phi-Ground, January 2026 disclosure Locate a requested UI element Microsoft reports grounding research and integration of its technology into Copilot Vision Highlighting.7 That disclosure does not establish where inference runs.
Granite Docling / Docling MCP, current implementations Document conversion as an agent tool A 258M specialist and an MCP server with explicit local conversion mode.89 The model, library and tool server are different components; no universal complete-agent improvement is established.
PaddleOCR-VL-1.6, June 2026 Layout detection plus a 0.9B document VLM A specialist parsing system and documented training recipe.10 The report does not establish a cloud agent plus mobile deployment.
Cosmos Reason2 / IntBot, March 2026 recipe Camera input to structured perception events A 2B VLM deployed on Jetson AGX Thor for downstream robot reasoning.11 The demonstrated stack is local, not proof of a cloud controller.
LTX Desktop, current implementation Local video generation Supported hardware and separate local and API generation paths.12 This is generation, not perception, and not a documented cloud-agent orchestration benchmark.

Scroll to compare. Focus view gives the table more room.

The strongest direct agent evidence is computer use. Document and video systems provide useful capabilities to connect, but connecting them into this particular architecture is an engineering proposal until the complete system is evaluated.

Screens: parsing is not grounding

Parsing asks what is visible. Grounding asks where a particular instruction refers. A screen inventory can support planning, while a targeted grounder resolves the next click. OmniParser supplies structured UI elements; Agent S2 explicitly delegates action descriptions to grounding experts. Both can still leave the cloud model inspecting screenshots.12

For example, “open the export menu” is an agent decision. Locating that menu’s pixel coordinates is grounding. Confirming that the expected menu appeared requires a new observation. A correct coordinate in an old screenshot can still be the wrong action after the interface changes.

ScreenVLM converts screenshots into ScreenTag markup using a visual encoder and Granite 165M decoder. Its training gives extra weight to element types and locations. Figure 2 shows the architecture.6

Pan on smaller screensOpen full size ↗

Figure 2. From screen pixels to structured markup. Reproduced from Gurbuz et al., arXiv v1, Figure 4. CC BY 4.0.6

The limitation matters: the dataset is predominantly web based, and the paper identifies a gap to native desktop and mobile interfaces. Its reported mean latency of 276.4 ms is for 128 samples on a single H100 under vLLM, not an on-device prediction.6

Documents: preserve evidence, not just a summary

Granite Docling is designed for document conversion, not general image understanding. Its model card provides local inference paths and integration with Docling’s document representation. Docling MCP separately supplies a tool interface and local conversion mode.89

PaddleOCR-VL-1.6 illustrates a different specialist design: a layout detector locates regions and a compact VLM recognizes their contents. The system organizes text, tables, formulas and other elements into structured outputs. The upgrade focuses on targeted data refinement and staged continued pretraining, supervised fine-tuning and reinforcement learning rather than increasing model size.10

For a cloud agent, the important design question is what survives conversion. Our proposed evidence contract retains page references, reading order, table headers, units, bounding boxes and access to a permitted source crop. A short caption alone can discard exactly the detail a later question needs.

This does not require sending every artifact to the cloud. Keep originals local, expose a bounded result and let the agent request a specific permitted region when needed. A local file handle is not automatically resolvable by a remote model. The host must mediate retrieval and any upload. See Document AI for the surrounding parsing and evaluation landscape.

Video: an observation stream, not an unlimited upload

The IntBot recipe processes sampled camera frames through Cosmos-Reason2-2B on Jetson AGX Thor and publishes structured events for downstream reasoning. It reports a median 510 ms perception pipeline latency with FP8, compared with 762 ms at FP16 for its tested workload. These are first-party recipe measurements, not our reproduction or a result for a whole cloud agent.11

A possible cloud integration would send selected events and allow targeted clip inspection. That is our proposed extension, not the demonstrated IntBot configuration. Preserve capture times, intervals actually inspected, dropped frames and uncertainty. Sparse sampling can miss short events; an unobserved interval is not evidence that nothing happened.

Video generation fits the tool architecture, but is not perception. LTX Desktop documents local generation on supported NVIDIA GPUs and Apple Silicon hardware. It also has API paths and optional cloud text encoding; fully local operation requires a local text encoder. A cloud agent could request a generated clip through a wrapper, but that integration and its benefit would need separate validation.12

For long generation jobs, our proposed interface returns a job ID, progress, an output artifact and cancellation status rather than keeping a reasoning call open. “Local” can mean a capable workstation, not a wearable or camera.

Make the interface useful to the agent

The key interface is not merely image → text. It is a bounded request and an attributable result. MCP supports named tools, input schemas, structured results and optional output schemas. It does not itself choose model placement or guarantee that a tool’s claims are correct.4

For this architecture, we propose retaining:

  • Identity and freshness: source, revision, page or frame, capture time and the region actually inspected.
  • Task content: elements, values, relations or generated artifacts in a typed representation, plus explicit missing fields and failures.
  • Verification paths: permitted crops or clips and links back to the original evidence. Schema validity is not factual correctness.
  • Disclosure and authority: which fields may leave the device and which actions still require approval. Enforce this in the host, not a model label.
A small proposed result contract

This illustrative JSON is not a published model output or a standard MCP schema. It avoids inventing a calibrated confidence score.

{
  "tool": "read_document_region",
  "source_id": "invoice-42",
  "revision": 3,
  "page": 2,
  "region_px": [100, 200, 600, 300],
  "page_size_px": [2048, 1536],
  "content": {
    "type": "table_row",
    "fields": {"quantity": "12", "unit": "boxes"}
  },
  "coverage": "requested region only",
  "warnings": ["faint text in unit column"],
  "source_crop_handle": "crop-17",
  "disclosure": "host approval required"
}

Here coordinates use the original page image, with the origin at its top left. The host resolves crop-17 locally and decides whether a cloud request may receive its bytes. The result does not give the agent permission to read other files, change the disclosure policy or execute an action.

Let the agent ask for better evidence

A fixed parser runs once and hopes it retained everything. An interactive perception tool lets the agent ask a narrower question after seeing a result. This is the design opportunity illustrated by Figure 3.

The example is hypothetical: compare an invoice with a purchase order and prepare an update. It is not a measured system or an account of private work.

Read broadly, inspect selectively, act with approval
1 · Establish the evidence
  1. Parse locallyRecover invoice and order fields with page and region IDs.
  2. Disclose permitted fieldsThe host checks access and sends only the approved evidence.
  3. Reason in the cloudThe agent compares records and identifies an unclear quantity.

Parsing is an observation, not permission to update a business record.

2 · Close the information gap
  1. Request a closer readAsk for the quantity and unit in one identified region.
  2. Inspect locallyUse a higher resolution crop or another specialist if allowed.
  3. Return attributable evidenceProvide the value, units, source revision and unresolved ambiguity.

This is the feedback loop: the cloud chooses what to inspect next; the host bounds that request.

3 · Complete the authorized action
  1. Prepare the updateThe cloud agent drafts the change and cites its evidence.
  2. Approve and executeThe host validates the proposed change and obtains required consent.
  3. Observe the resultCheck the actual saved record before reporting completion.

A changed source revision, failed tool call or unresolved mismatch returns the task to inspection or stops it.

Figure 3. A targeted inspection loop with a separate action boundary. Proposed workflow informed by specialist grounding and tool contracts.24

More observations cost time. Stop when the evidence is sufficient, the budget is exhausted or the task requires a human. Do not turn uncertainty into an unbounded conversation between models.

Keep the cloud connection deliberate

A practical host would expose allowlisted operations such as parsing an authorized file, locating a described control or inspecting a bounded clip. The model service can sit behind an in-process adapter, local service or MCP server. The cloud agent does not need a publicly reachable port on the device: the local agent host can call the cloud API and dispatch returned tool requests locally. This is a proposed deployment pattern, not a requirement of MCP.

Separate read access, disclosure and write authority. OCR text, screenshots and tool results should be treated as untrusted task data, not instructions that can override host policy. Limit file access, execution privileges, timeouts, output size and concurrency. MCP’s tool specification calls for access controls and result validation; its security guidance also identifies risks from local server compromise.413

Local parsing does not make the overall system private if all recognized text is uploaded afterward. Test actual outbound payloads, including logs and optional model services. Without connectivity, a cloud-led agent can lose planning capability even while its local tools remain available. Define that failure mode rather than promising offline autonomy.

Train for the capability, evaluate the whole task

For a specialist, begin with the contract it must satisfy. A parser needs faithful structure and coverage; a grounder needs location precision; a video tool needs event and temporal accuracy. Test those capabilities independently before using agent success as a reward signal.

The next research step is to measure whether those outputs help a fixed cloud planner. Our proposed study would vary output structure, evidence budget and targeted reinspection while keeping the planner and task set constant. If adding reinforcement learning, start with verifiable subtask outcomes and audit reward shortcuts; do not assume that a fluent cloud judge provides a reliable objective.

The following is a proposed evaluation plan, not a set of published results.

Measure the system, not just the parser

5 checks
Question Measure Control or failure case
Does specialization help? Task completion, extraction or grounding errors, human correction rate Same cloud planner, task set and comparable evidence access; compare direct multimodal input, specialist input, and a combination.
Does local placement help? Total task time at p50 and p95, cloud expenditure, device energy, bytes transferred Compare local and remote versions of the same specialist; include cold starts, queueing, retries and network changes.
Does the interface lose evidence? Omissions, incorrect relations, unsupported answers, answerability from retained artifacts Compare plain summaries with structured results and permitted source crops; inspect failure slices.
Does interactive inspection pay off? Success gained per extra request, elapsed time, number of rounds Fixed initial parse versus targeted reinspection under the same total budget.
Does deployment remain safe? Unauthorized reads, writes or disclosures; stale actions; behavior during outages Adversarial document content, changed screenshots, denied tool access, unavailable specialists and dropped connectivity.

Scroll to compare. Focus view gives the table more room.

Use separate training, threshold selection and final test sets. Report results by screen density, layout type, language, video event duration, hardware and network condition. Gains on a parser benchmark cannot substitute for this complete-system comparison.

Where the opportunity lies

The promising idea is not “a small model is cheaper.” It is a specialist can give a general agent better access to a particular environment. The benefit may come from precision, retained structure, domain adaptation or reduced data movement. It must survive the integration and hardware costs.

My synthesis of this evidence is to prioritize three research questions:

  1. What evidence should cross the boundary? Compare summaries, structured representations and selective raw observations without hiding omissions.
  2. When should the agent look again? Learn useful inspection requests under a time, disclosure and computation budget.
  3. What should the specialist learn? Connect perceptual fidelity with downstream task value without sacrificing attribution or reliable abstention.

For a concrete starting point, build a document or computer-use task with one cloud planner and one local specialist, then add targeted reinspection only after establishing a fair baseline. See On Device Computing for hardware constraints and Agent Evaluation for the wider testing discipline.

References

  1. Yadong Lu et al., Microsoft Research. OmniParser V2: Turning Any LLM into a Computer Use Agent, February 12, 2025. Screen parsing and OmniTool composition. ↩ ↩2 ↩3

  2. Saaket Agashe et al. Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents, April 2025, Sections 3 and 4.3. Composition, grounding experts and controlled ablations. ↩ ↩2 ↩3 ↩4 ↩5

  3. Microsoft. OmniTool README, Overview and Notes, inspected October 17, 2026. CPU support and optional separate GPU parser service. ↩ ↩2

  4. Model Context Protocol. Tools, June 18, 2025 specification, inspected October 17, 2026. Tool results, output schemas and security considerations. ↩ ↩2 ↩3 ↩4

  5. Simular. Agent S repository, Grounding Models and Usage, inspected October 17, 2026. S3 main and grounding endpoints; documented recommended model pairing. ↩

  6. A. Said Gurbuz et al. Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision, February 15, 2026, Sections 4–6, Figure 4 and Table 7. CC BY 4.0. ↩ ↩2 ↩3 ↩4

  7. Microsoft Research Asia. Phi-Ground: Improving how AI agents navigate screen interfaces, January 18, 2026. Grounding and the reported Copilot integration; no placement claim inferred. ↩

  8. IBM Research. Granite Docling 258M, model card, September 17, 2025 release, inspected October 17, 2026. Getting Started, Intended Use and Model Architecture. ↩ ↩2

  9. Docling project. Docling MCP, Local Mode, Features and Configuration, inspected October 17, 2026. Model, library and tool server are distinct. ↩ ↩2

  10. Zelun Zhang et al. PaddleOCR-VL-1.6: Expanding the Frontier of Document Parsing with Under-Optimized Region Refinement and Progressive Post-Training, June 2, 2026, Sections 2–4. Architecture and post-training recipe, not an agent placement study. ↩ ↩2

  11. IntBot.AI. Real-Time Vision-Language Models on Edge Hardware: Deploying Cosmos-Reason2 on Jetson Thor for Social Robot Perception, NVIDIA Cosmos Cookbook, March 2026. Deployment architecture and Performance Evaluation; first-party measurements. ↩ ↩2

  12. Lightricks. LTX Desktop, Local Models, System Requirements and Text Encoding, inspected October 17, 2026. Local generation and optional external services, not a cloud agent study. ↩ ↩2

  13. Model Context Protocol. Security Best Practices, Local MCP Server Compromise, inspected October 17, 2026. Trust boundaries, consent and restricted execution. ↩

Navigation

Type to search…

↑↓ navigate↵ selectEsc close