Evaluating Tool-Using Agents in Real-World Environments

AI agents often perform impressively in controlled research settings, yet struggle when deployed in real-world systems where they have to reason across multiple steps, interact with real tools and APIs, operate under partial information, and get better from errors in stateful, permissioned environments—highlighting a persistent gap between research success and production reliability.

OpenEnv is an open-source framework from Meta and Hugging Face designed to deal with this challenge by standardizing how agents interact with real environments. As a part of this collaboration, Turing contributed a production-grade calendar management environment to review tool-using agents under realistic constraints corresponding to access control, temporal reasoning, and multi-agent coordination.

On this post, we explore how OpenEnv works in practice, why calendars function a strong benchmark for real-world agent evaluation, and what our findings reveal in regards to the current limitations of tool-using agents.

What Is OpenEnv?

OpenEnv is a framework for evaluating AI agents against real systems moderately than simulations. It provides a standardized solution to connect agents to real tools and workflows while preserving the structure needed for consistent and reliable evaluation.

OpenEnv uses a gym-oriented API (reset, step, motion, observations) like OpenAI’s Gymnasium. Also, OpenEnv uses a regular MCP tool call interface to connect with envs which provides a consistent interface across domains and simulation to production environments.

The environments maintain state across multiple actions—enabling long-horizon reasoning—and may connect on to real APIs and tools corresponding to browsers, code repositories, or calendars. This shifts evaluation from “Can this work in a controlled demo?” to “Can this operate reliably in the true world?”

The Calendar Gym: A Production-Grade Benchmark

Calendar systems are deceptively complex. While scheduling a gathering seems easy, real-world calendar management requires agents to reason over time, permissions, multiple users, and incomplete information—often across several dependent steps. These properties make calendars a strong testbed for evaluating tool-using agents outside controlled simulations.

To ground OpenEnv in this type of realistic, demanding use case, Turing built a production-grade calendar management environment known as the Calendar Gym. Quite than simulating scheduling within the abstract, it exposes agents to the identical constraints they’d face in real calendar systems: Access Control Lists across users and calendars, limited visibility into other users’ state, and multi-step workflows where actions should be chained in the proper order. Agents interact with a wealthy set of calendar operations—from listing calendars to modifying events and permissions—and must handle failed actions, incorrect assumptions, and missing permissions. Each session runs in an isolated environment, enabling reliable comparisons across runs.

Below is a code example of use the Calendar Gym. We explore the environment, discover available tools, list calendars, create an event, and print the result.

from openenv_wrapper.client import MCPEnvClient
from openenv_wrapper.data_models import MCPAction

with MCPEnvClient.from_hub(base_url="TuringEnterprises/calendar-gym") as client:
    
    result = client.reset()
    print("Reset successful:", result.remark.success)

    
    result = client.step(MCPAction(action_type="ListToolsAction"))
    print("Available tools:", len(result.remark.tools_list))

    
    result = client.step(MCPAction(
        action_type="ToolCallAction",
        tool_name="calendars_list",
        arguments={}
    ))
    calendars = result.remark.tool_result["items"]
    print("Calendars:", calendars)

    
    result = client.step(MCPAction(
        action_type="ToolCallAction",
        tool_name="events_insert",
        arguments={
            "calendarId": "primary",
            "summary": "Team Sync",
            "start": {"dateTime": "2026-01-15T14:00:00Z"},
            "end": {"dateTime": "2026-01-15T15:00:00Z"}
        }
    ))
    print("Event created:", result.remark.success)

Below is an excerpt of what the Calendar Gym returns if you call ListToolsAction. Each entry includes the tool name plus an input schema (what arguments the tool accepts).

{
  "tools_list": [
    {
      "name": "calendars_list",
      "description": "List calendars visible to the current user.",
      "input_schema": {
        "type": "object",
        "properties": {},
        "additionalProperties": false
      }
    },
    {
      "name": "events_insert",
      "description": "Create an event in a calendar.",
      "input_schema": {
        "type": "object",
        "properties": {
          "calendarId": { "type": "string" },
          "summary": { "type": "string" },
          "start": {
            "type": "object",
            "properties": { "dateTime": { "type": "string" } },
            "required": ["dateTime"]
          },
          "end": {
            "type": "object",
            "properties": { "dateTime": { "type": "string" } },
            "required": ["dateTime"]
          }
        },
        "required": ["calendarId", "summary", "start", "end"]
      }
    }
  ]
}

What We Learned

Evaluating agents within the Calendar Gym revealed consistent patterns which were common across multiple domains. While agents often perform well on individual game like actions, reliability breaks down as tasks change into longer, more ambiguous, and more constrained.

Multi-step reasoning is the first bottleneck. Agents struggle to accurately chain actions across longer workflows, suggesting that benchmarks must test sustained reasoning over multiple dependent steps—not only single tool calls.

Ambiguity significantly degrades performance. Agents achieved near 90% success on tasks with explicit calendar identifiers, but success dropped to roughly 40% when the identical tasks were phrased using natural language descriptions. Constructing stronger lookup and validation into agent loops—moderately than counting on the LLM to resolve references unaided—appears essential.

Correct tool selection is not enough. Across failed interactions, greater than half of errors stemmed from malformed tool arguments or incorrect ordering, even when the appropriate tool was chosen. Reliable agent behavior depends as much on execution quality and structured feedback as on tool selection—environment design matters.

These challenges usually are not unique to scheduling and calendars. They reflect broader limitations that emerge every time agents operate in changing systems over long periods of time, and so they point toward evaluation frameworks that test permissions, partial observability, and multi-step workflows together.

Looking Ahead

OpenEnv provides a foundation for testing agents under realistic conditions, and the Calendar Gym demonstrates how seemingly easy domains can surface deep challenges in reasoning, ambiguity resolution, and power use. By evaluating agents where failure is measurable and constraints are real, we gain clearer insight into what it takes to construct agents that operate reliably in production.

For a deeper dive into the Calendar Gym’s design, benchmarking methodology, and quantitative results, explore the total technical article on Turing’s site. To explore a clone of the Calendar Gym, visit the Calendar Gym space.

Appendix: Common error cases in tool use

In practice, tool integrations rarely fail in dramatic ways; they fail in small, predictable ones. When wiring up MCP tools to real APIs (like calendar operations), we encountered a handful of recurring issues.

Specific error cases present in the wild

Below are three common failure modes we’ve seen in production, together with representative error payloads and mitigation strategies. These examples illustrate not only what can go fallacious, but how structured errors may also help agents get better gracefully.

1. Schema validation errors (missing or malformed arguments)

The agent calls a legitimate tool (e.g. events_insert), however the arguments don’t match the declared JSON schema.

• Missing required fields like calendarId
• Incorrect nesting of start / end
• Passing a string where an object is anticipated.

{
  "okay": false,
  "error_type": "validation_error",
  "tool_name": "events_insert",
  "message": "Invalid arguments for tool 'events_insert'.",
  "details": {
    "missing_required_fields": ["calendarId", "end"],
    "invalid_fields": [
      {
        "field": "start",
        "expected_type": "object",
        "received_type": "string"
      }
    ]
  }
}

We will mitigate this by providing one canonical example of an accurate ‘events_insert’ call in your prompt. Return structured validation errors so the model can repair and retry as a substitute of failing silently.

2. Permission / authorization errors (401/403)

The tool call is syntactically correct, however the API rejects it attributable to insufficient permissions.

• Missing OAuth scopes
• Expired access token
• User lacks write access to the goal calendar

{
  "okay": false,
  "error_type": "permission_error",
  "tool_name": "events_insert",
  "http_status": 403,
  "message": "The authenticated user doesn't have write access to calendar 'primary'.",
  "remediation": [
    "Ensure the OAuth token includes calendar write scope.",
    "Verify the user has edit access to the target calendar.",
    "Reconnect the integration if the token has expired."
  ]
}

We will mitigate this by clearly documenting the required OAuth scopes. Return structured, actionable remediation steps so the agent can guide the user as a substitute of retrying the identical failing call.
Clearly document required OAuth scopes. Return structured, actionable remediation steps so the agent can guide the user as a substitute of retrying the identical failing call.

3. Datetime / format errors (RFC3339 & timezone issues)

The event is rejected by the API, or it’s created at an unexpected time.

• Missing timezone offset
• Non-RFC3339 datetime format
• Incorrect nesting of start.dateTime or end.dateTime
• Mixing local time and UTC without specifying an offset

{
  "okay": false,
  "error_type": "format_error",
  "tool_name": "events_insert",
  "message": "Invalid datetime format for field 'start.dateTime'.",
  "details": {
    "received": "02/11/2026 9:30 AM",
    "expected_format": "RFC3339 (e.g. 2026-02-11T09:30:00-05:00)"
  }
}

We will mitigate this by standardizing on RFC3339 with explicit timezone offsets (e.g. 2026-02-11T09:30:00-05:00). Include at the least one correct datetime example in your documentation to anchor model behavior and reduce repair retries.