Envisioning Agentic Robotics

Paradigm Shift

Policy-Centric Robotics

Learned policies have expanded robot capabilities, from agile locomotion to language-guided manipulation. Vision-language-action (VLA) models map images and instructions to actions, learning from increasingly diverse robot data. Reliable generalization when scenes or instructions change remains a central challenge.

OpenVLA maps an image and a language instruction through vision encoders and a language model to a robot action. — **Figure 1. A Vision–Language–Action Policy.** OpenVLA illustrates how a learned policy maps visual observations and language instructions to robot actions. Source: Moo Jin Kim et al., OpenVLA (2024), Fig. 2. CC BY 4.0; reproduced without modification.

Agentic Robotics

Agentic robotics centers on an agent that interprets intent, reasons about goals and constraints, coordinates skills and tools, and verifies outcomes to adapt its decisions. The ambition is to delegate broader work: from picking up a cup to preparing breakfast, and eventually taking responsibility for it day after day.

From picking up a cup to taking responsibility for breakfast, day after day. — **Figure 2. A Vision for Agentic Robotics.** An envisioned breakfast scenario, from picking up a cup to taking responsibility for breakfast day after day.

From Policies to Agents

The shift changes how capabilities are organized. Learned policies remain valuable skills; the agent decides which capabilities to use, checks whether their effects satisfy the goal, and adapts when they do not. A reference framework makes these responsibilities and their interactions explicit.

**Figure 3. Two Paradigms.** Both paradigms use feedback. Agentic robotics centers on an agent that coordinates skills and tools, verifies outcomes, and adapts its decisions.

Reference Framework

Our reference framework separates three responsibilities: S2 interprets intent, decides how to proceed, and verifies outcomes; S1 supplies skills and callable modules; S0 provides the body interface and local real-time control. One model may serve several roles, and one role may span several components.

**Figure 4. A Reference Framework for Agentic Robotics.** S2 decides and verifies; S1 supplies skills and tools; S0 maintains body control. The event monitor remains responsive during deliberation.

Three Paths to Execution

A decision can reach the body through existing skills, permitted S0 targets, or a program written by S2.

**Figure 5. Three Typical Execution Paths.** The paths can coexist within one task. Direct control still leaves the local control loop in S0.

From Intent to Verified Outcomes

Execution returns evidence to S2. Bringing a cup to the table requires checking that it was grasped, remains held during transport, and reaches the intended place before reporting completion.

**Figure 6. The Agentic Loop.** Acceptance, termination, physical effect, and intent satisfaction are four different judgments. A completed call alone does not prove task success.

Embodied System 2

Foundation Model and Embodied Harness

S2 makes two externally identifiable decisions: how to proceed, and whether the intended outcome was achieved. The foundation model supplies knowledge, understanding, and reasoning; the embodied harness provides context, tools, plans, memory, and verification. Together they connect those decisions to physical execution.

**Figure 7. Core Abilities of Embodied S2.** Eight abilities support the complete loop. Some reside mainly in the model; others depend on the harness and the body.

Physical Constraints

The world keeps changing while S2 reasons. Decisions must arrive in time, use fresh observations, and respond to changes in the user’s request or the scene. Capped scaling captures this objective: improve decision quality within the physical deadline.

**Figure 8. Capped Scaling.** Decision quality under a deadline. These are conceptual curves, not measured results.

Timing requirements differ across S2, callable skills, and continuous body control. — **Figure 9. Decision and Control Timescales.** Indicative control timescales and reported S2 latencies. The S2 band is a target; reported points come from different task settings.

While the Model Reasons

Observed HereNow Here

Evaluation

These constraints make evaluation a question of both capability and execution. Model-level tests isolate abilities under specified inputs; system-level tests measure what the model, harness, skills, and body achieve together. The framework covers eight abilities and the resources consumed as cost.

9Dimensions

56Sub-Dimensions

30Embodiment-Dependent

**Figure 10. S2 Evaluation at a Glance.** Eight abilities plus cost. Highlighted sub-dimensions are specific to, or strongly dependent on, embodiment.

Control what changes

Fix the harness when comparing models; fix the model when comparing harnesses.

Keep the world running

Report deadlines, observation age, and paused versus live execution separately.

Verify independently

Use evidence-based outcome checks and disclose false completion claims.

The setup changes the result.

The studies reviewed in the paper report gains in semantic and spatial capabilities, while physical execution, completion judging, and timely response remain uneven. These comparisons show how interfaces, the world clock, and the move from text to action affect reported performance.

Reported comparisons from individual studies. Each row uses its own task, metric, and protocol.

Reported comparisons from individual studies. Each row uses its own task, metric, and protocol.
Study / Metric	Reported Comparison	Setting	What It Shows
EmbodiedSkillBenchSuccess Rate	92% / 61% / 85%Native / Skill-Only / Selective Reuse (ADR)	GPT-6 Astra; 100 perturbed LIBERO-Pro tasks.	The tool interface can constrain performance.
RT-SafeSafe Success	19.8% → 0.7%Paused World → Live World	Eight VLMs; hard routes; mode-specific timing instructions.	Live execution changes the safety result.
RoboHarmRefusal Rate	72% → 18%Text → Robot Actions	Six models; real-robot end-effector control.	Refusal in text transfers only partly to action.

These rows use different tasks and metrics and do not form a cross-benchmark ranking. Refusal rate is distinct from safe task completion.

From Digital Agents to Physical Agents

Conditions for Agentic Work

These gaps help explain why progress in digital agents does not transfer directly to robots. Coding agents benefit from familiar interfaces, verifiable outcomes, abundant data, and inexpensive environments. Physical robots weaken all four conditions at once.

What Transfers, What Must Be Rebuilt

The organizing principles still transfer: act on feedback, define tool contracts, maintain durable state, separate execution from judgment, and reuse validated skills. Their physical implementation requires grounded interfaces, calibrated verifiers, embodied data, and simulation checked against reality.

**Figure 12. What Physical Agents Must Rebuild.** Grounded interfaces, calibrated verifiers, embodied data, and reality-checked simulation must work under physical deadlines and limits.

Road Ahead

Rebuilding these foundations should expand the work people can entrust to a robot. Our ladder measures progress by the responsibility a robot can organize, verify, and sustain: from a primitive to a task, a workflow, and a standing role.

Ladder of Delegable Work

**Figure 13. Ladder of Delegable Work.** Primitives, tasks, workflows, and standing roles. The durations illustrate the progression rather than define it.

Current systems complete primitives and tasks. Workflow-level evidence is emerging in company demonstrations; a sustained role in an unstructured environment without human repair remains unestablished.

Expanding the Operating Scope

**Figure 14. Spatial Scope of Delegated Work.** Spatial scope is a separate axis from responsibility: workcell, room, building, and site.

What do today’s benchmarks cover?

Responsibility and spatial scope are separate axes. Existing benchmarks cover parts of this space, mostly within primitives and single tasks. Their spatial reach and time span show where evaluation has extended and where evidence for workflows and standing roles is still missing.

Selected examples from the paper’s benchmark-coverage table.

Selected examples from the paper’s benchmark-coverage table.
Benchmark	Spatial Scope	Time Span
LIBERO	Workcell	Single Episodes
RoboMME-Interference	Workcell	Prior Demonstration Followed by Unrelated Sessions
RobotWorld	Workcell to Outdoor	Interaction Budgets
MessyMem	Room / Kitchen	Over Three Hours; 25 Consecutive Tasks Without Resetting Memory or the Environment
R2R-CE	Building	Single Episodes
BEHAVIOR-1K	Building / Homes and Offices	Single Long Episodes
RT-Safe	Site / City Routes	Continuous Time

The paper finds a gap in tests of interdependent workflows and roles sustained over days. A long sequence of tasks is not sufficient evidence of either.

Future Research Directions

Reason within physical deadlines

Decide when to think, observe again, act, or reuse a skill within the response budget.

Keep control persistent and verifiable

Maintain trustworthy state across tasks and spaces, and coordinate work with shared resources.

Adapt across environments and bodies

Reuse verified experience, expose capabilities explicitly, and reduce data and configuration needs.

Improve from physical experience

Retain changes that help later work without eroding existing abilities.

Progress in these directions should carry forward to later work. Test improvements on held-out tasks, check that existing abilities are retained, and identify what changed in the model, harness, or skills.

Make progress inspectable.

Publish a harness card with every result.
Report live results alongside paused evaluations.
Specify execution and cancellation contracts.
Disclose human assistance and operating conditions.

Applications

The value of this research becomes concrete where people still choose the next task, judge completion, and handle exceptions. Embodied S2 could take on more of these responsibilities, helping robots follow varied requests, recover from routine failures, and carry useful state across tasks and days.

Homes

Interpret open requests, remember preferences, and recover when a chore fails.

Manufacturing & Logistics

Verify cycles, handle exceptions, and re-task skills when production changes.

Inspection

Follow anomalous readings with further checks and evidence-backed reports.

Healthcare & Laboratories

Handle varied requests, verify outcomes, and preserve provenance.

Agriculture & Mobility

Switch jobs from observations and widen operating domains as safety evidence grows.

Across these settings, progress means more useful work delivered with less human intervention. Reliability, time, cost, and the assistance still required show how much responsibility people can entrust to a robot.

Citation

If you find this roadmap helpful, please consider citing our paper. Thank you for building on this work.

@misc{lan2026agentic,
  title = {Envisioning Agentic Robotics: A New Era of Robotic Systems},
  author = {Qingfeng Lan and Duo Li and Siyao Li and Yiheng Lyu and
            Shijian Lu and Zhenghao Qi and Season Wang and Ming Yang and
            Zhiqin Yang and Zuhao Yang and Tao Yu},
  year = {2026},
  note = {Technical report}
}

EnvisioningAgentic Robotics