RAI: Robot tool-calling model benchmark

Evaluate a model on predefined ROS 2-shaped tool-calling tasks and write benchmark results.

rai-src-rai-bench-rai-bench-examples-tool-calling-agent-py | Record version: source-c1849e5e29eb00b5447d4a35788eda71b916f4b5da40fd7a5785cd432763c28f

Record JSON | Markdown | Pinned upstream source

Access

Fetch the pinned repository, verify this source checksum, and use upstream installation and invocation. This record does not install or execute code.

Hub preparation adapter: false. Follow the upstream setup and invocation contract.

Source and license

Repository: https://github.com/RobotecAI/rai
Revision: b018729833742c6f71e1e9f332ee2b435676fb4f
Path: src/rai_bench/rai_bench/examples/tool_calling_agent.py
Source SHA256: c1849e5e29eb00b5447d4a35788eda71b916f4b5da40fd7a5785cd432763c28f
Original authors: Robotec.AI (per-file copyright notice)
License: Apache-2.0
License scope: Selected first-party source and per-file notices; excludes models, dependencies, datasets, simulator assets and external service terms.
License SHA256: c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4
Hub commit: 1d6d9eb134e397d910265a8d6e7d8f257f9e8d7f

Artifact

{
  "kind": "workflow",
  "format": "python",
  "workflow_role": "robot-agent-evaluation"
}

Requirements

{
  "python": "Root/core/whoami/perception/semap/s2s use >=3.10,<3.13; standalone finetune uses >=3.10,<3.12. Match sourced ROS distribution Python.",
  "packages": [
    "Pinned RAI checkout",
    "Matching ROS 2 and rai_interfaces",
    "Per-workflow packages, local model or hosted-provider dependencies from pinned setup"
  ],
  "display": "Official setup documents ROS 2 Humble/Jazzy, rai_interfaces and feature-specific environment. Nothing installed or runtime verified.",
  "execution_domain": "computation",
  "source_imports": [
    "pathlib",
    "rai_bench",
    "rai_bench.tool_calling_agent",
    "rai_bench.utils"
  ],
  "notes": "Upstream per-workflow profile declarations only; provider availability, dependency resolution and runtime/robot compatibility are unverified.",
  "declared_requirements": {
    "agent": {
      "gpu": "No fixed GPU requirement for the Python wrapper. Local model serving has model-specific hardware requirements; hosted APIs move LLM inference to the provider.",
      "external_models": "A configured tool-capable LLM. Vision workflows need a vision-capable model. Pinned config selects OpenAI gpt-4o/gpt-4o-mini by default and declares Ollama, AWS and Google alternatives.",
      "api": "Default hosted OpenAI configuration requires credentials and may incur charges. Other hosted vendors require separate access; local Ollama needs a server and separately obtained model. No calls were made.",
      "evidence_paths": [
        "config.toml",
        "docs/setup/vendors.md",
        "docs/setup/install.md"
      ]
    }
  }
}

Invocation

{
  "entrypoint": "python src/rai_bench/rai_bench/examples/tool_calling_agent.py",
  "notes": "Source/setup invocation shape only; not executed or runtime verified. Source references only; no Hub adapter, compatibility promise, provider access grant or runtime result."
}

Explicit compatibility

[]

Missing environment fields remain unknown. Discovery labels do not establish robot compatibility.

Evidence

Level: source_reviewed; runtime verification: not_run.

Verified exact source, applicable license and author/setup document bytes at a full Git commit; parsed the Python AST and reviewed this guarded operational entrypoint and its explicit license notices. No upstream code, installation, robot command, dataset, mesh or checkpoint was executed or loaded.

Source review and contributor reports do not establish task completion. Preserve upstream attribution and licenses.