RAI: O3DE manipulation agent benchmark

Evaluate a configured LLM across predefined O3DE robot manipulation scenarios.

rai-src-rai-bench-rai-bench-examples-manipulation-o3de-py | Record version: source-ba0ff50ef58ddb4c28bbf4f3374b957f151938288886eda97d656b6785697bc7

Record JSON | Markdown | Pinned upstream source

Access

Fetch the pinned repository, verify this source checksum, and use upstream installation and invocation. This record does not install or execute code.

Hub preparation adapter: false. Follow the upstream setup and invocation contract.

Source and license

Repository: https://github.com/RobotecAI/rai
Revision: b018729833742c6f71e1e9f332ee2b435676fb4f
Path: src/rai_bench/rai_bench/examples/manipulation_o3de.py
Source SHA256: ba0ff50ef58ddb4c28bbf4f3374b957f151938288886eda97d656b6785697bc7
Original authors: Robotec.AI (per-file copyright notice)
License: Apache-2.0
License scope: Selected first-party source and per-file notices; excludes models, dependencies, datasets, simulator assets and external service terms.
License SHA256: c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4
Hub commit: 1d6d9eb134e397d910265a8d6e7d8f257f9e8d7f

Artifact

{
  "kind": "workflow",
  "format": "python",
  "workflow_role": "robot-agent-evaluation"
}

Requirements

{
  "python": "Root/core/whoami/perception/semap/s2s use >=3.10,<3.13; standalone finetune uses >=3.10,<3.12. Match sourced ROS distribution Python.",
  "packages": [
    "Pinned RAI checkout",
    "Matching ROS 2 and rai_interfaces",
    "Per-workflow packages, local model or hosted-provider dependencies from pinned setup"
  ],
  "display": "Official setup documents ROS 2 Humble/Jazzy, rai_interfaces and feature-specific environment. Nothing installed or runtime verified.",
  "execution_domain": "computation",
  "source_imports": [
    "pathlib",
    "rai_bench",
    "rai_bench.manipulation_o3de",
    "rai_bench.utils"
  ],
  "notes": "Upstream per-workflow profile declarations only; provider availability, dependency resolution and runtime/robot compatibility are unverified.",
  "declared_requirements": {
    "agent": {
      "gpu": "No fixed GPU requirement for the Python wrapper. Local model serving has model-specific hardware requirements; hosted APIs move LLM inference to the provider.",
      "external_models": "A configured tool-capable LLM. Vision workflows need a vision-capable model. Pinned config selects OpenAI gpt-4o/gpt-4o-mini by default and declares Ollama, AWS and Google alternatives.",
      "api": "Default hosted OpenAI configuration requires credentials and may incur charges. Other hosted vendors require separate access; local Ollama needs a server and separately obtained model. No calls were made.",
      "evidence_paths": [
        "config.toml",
        "docs/setup/vendors.md",
        "docs/setup/install.md"
      ]
    },
    "perception": {
      "gpu": "README declares an NVIDIA CUDA GPU for optimal performance. No memory capacity or runtime result is established.",
      "external_models": "GroundingDINO and Grounded-SAM-2 dependencies/checkpoints. Services download missing weights on first use to their configured cache; none were obtained.",
      "api": "The standalone perception service wrapper does not require a hosted LLM API; integrated agent demos also need an LLM.",
      "evidence_paths": [
        "src/rai_extensions/rai_perception/README.md",
        "src/rai_extensions/rai_perception/pyproject.toml",
        "src/rai_extensions/rai_perception/rai_perception/services/base_vision_service.py",
        "src/rai_extensions/rai_perception/rai_perception/services/weights.py"
      ]
    }
  }
}

Invocation

{
  "entrypoint": "python src/rai_bench/rai_bench/examples/manipulation_o3de.py",
  "notes": "Source/setup invocation shape only; not executed or runtime verified. Source references only; no Hub adapter, compatibility promise, provider access grant or runtime result. Known limits: Requires separate O3DE simulator/assets, explicit simulator configuration and scenarios; no assets were fetched."
}

Explicit compatibility

[]

Missing environment fields remain unknown. Discovery labels do not establish robot compatibility.

Evidence

Level: source_reviewed; runtime verification: not_run.

Verified exact source, applicable license and author/setup document bytes at a full Git commit; parsed the Python AST and reviewed this guarded operational entrypoint and its explicit license notices. No upstream code, installation, robot command, dataset, mesh or checkpoint was executed or loaded.

Source review and contributor reports do not establish task completion. Preserve upstream attribution and licenses.