6 hours ago
A customer asks a food-delivery app for a refund. A chatbot can explain the policy. An agent with access to order and payment tools can check the request, determine whether it qualifies, and take action within the limits set by the business. The hardest part appears when one of those tools gives an incomplete or uncertain result.
In Beyond LLMs: Building Reliable AI Agents, Madhumitha Kolkar, a senior machine learning engineer at SolarWinds, uses a hypothetical food-delivery refund to describe the difference between answering a question and completing a task. The example brings together three issues that a polished demo can miss: what the agent is allowed to do, what evidence it needs before acting, and how the workflow recovers when something goes wrong
Five companies and the reliability practices they illustrate
1. GeekyAnts tests workflow choice and output quality. The company describes 18 behavior tests for an internal engineering assistant. Thirteen assess whether it selects an appropriate workflow skill, and five inspect the work produced. In one documented case, the assistant passed software tests while disregarding the developer’s requirements. That example shows why an evaluation should check whether the original request was fulfilled.
2. SolarWinds provides an example of evaluating the steps in an agent run. Kolkar describes leading work in this area at SolarWinds. In a refund scenario, a team could inspect which tools were called, whether the agent obtained the required information, and how it responded to conflicting evidence. Two runs can end with similar messages even when only one followed the required checks.
3. Anthropic evaluates what changed in the system of record. Its guide to agent evaluations distinguishes a run’s transcript from the outcome in the environment. For a booking workflow, the evidence is a reservation in the relevant system. For a refund workflow, the evidence is the payment record. The agent’s confirmation message is useful to the customer once the underlying state supports it.
4. OpenAI documents approval before sensitive tool calls. The Agents SDK can pause a run and surface a specific action for review. A refund approver could see the order, amount, and justification before deciding. This matters when the model has enough information to propose an action but the business assigns authority to a person.
5. Microsoft documents checkpoints for interrupted workflows. Microsoft Agent Framework can save workflow state and resume later. That helps retain progress while a case is awaiting approval or recovering from an interruption. If an external operation may already have succeeded, the resumed workflow still needs to verify its status before repeating it.
What should a pilot demonstrate?
Run the same workflow under three conditions. First, use a valid request with all required information. Second, remove a detail that should cause the agent to pause and ask a question. Third, simulate a timeout after the external system may have accepted the action.
For each run, check the customer-facing message, the tool-call record, and the final state in the system of record. Track how long a person spends reviewing exceptions or repairing mistakes. That time belongs in the comparison with the existing process because it affects whether the agent reduces work after launch.
If a vendor could demonstrate only one failure scenario before launch, which would reveal the most: missing information, a rejected approval, or an uncertain transaction?
In Beyond LLMs: Building Reliable AI Agents, Madhumitha Kolkar, a senior machine learning engineer at SolarWinds, uses a hypothetical food-delivery refund to describe the difference between answering a question and completing a task. The example brings together three issues that a polished demo can miss: what the agent is allowed to do, what evidence it needs before acting, and how the workflow recovers when something goes wrong
Five companies and the reliability practices they illustrate
1. GeekyAnts tests workflow choice and output quality. The company describes 18 behavior tests for an internal engineering assistant. Thirteen assess whether it selects an appropriate workflow skill, and five inspect the work produced. In one documented case, the assistant passed software tests while disregarding the developer’s requirements. That example shows why an evaluation should check whether the original request was fulfilled.
2. SolarWinds provides an example of evaluating the steps in an agent run. Kolkar describes leading work in this area at SolarWinds. In a refund scenario, a team could inspect which tools were called, whether the agent obtained the required information, and how it responded to conflicting evidence. Two runs can end with similar messages even when only one followed the required checks.
3. Anthropic evaluates what changed in the system of record. Its guide to agent evaluations distinguishes a run’s transcript from the outcome in the environment. For a booking workflow, the evidence is a reservation in the relevant system. For a refund workflow, the evidence is the payment record. The agent’s confirmation message is useful to the customer once the underlying state supports it.
4. OpenAI documents approval before sensitive tool calls. The Agents SDK can pause a run and surface a specific action for review. A refund approver could see the order, amount, and justification before deciding. This matters when the model has enough information to propose an action but the business assigns authority to a person.
5. Microsoft documents checkpoints for interrupted workflows. Microsoft Agent Framework can save workflow state and resume later. That helps retain progress while a case is awaiting approval or recovering from an interruption. If an external operation may already have succeeded, the resumed workflow still needs to verify its status before repeating it.
What should a pilot demonstrate?
Run the same workflow under three conditions. First, use a valid request with all required information. Second, remove a detail that should cause the agent to pause and ask a question. Third, simulate a timeout after the external system may have accepted the action.
For each run, check the customer-facing message, the tool-call record, and the final state in the system of record. Track how long a person spends reviewing exceptions or repairing mistakes. That time belongs in the comparison with the existing process because it affects whether the agent reduces work after launch.
If a vendor could demonstrate only one failure scenario before launch, which would reveal the most: missing information, a rejected approval, or an uncertain transaction?