How to Handle Malformed Tool Responses in LLM Agents

Jul 25, 2026

Problem

When I gave my LLM agent a weather tool, the demo worked great. I asked for the weather in Tokyo, the tool returned clean JSON, and the agent answered confidently.

Then in production the downstream weather API had a bad minute and returned this:

HTTP/1.1 200 OK
Content-Type: text/html

&lt;html&gt;&lt;body&gt;502 Bad Gateway&lt;/body&gt;&lt;/html&gt;

My agent’s tool wrapper did this:

import json

def get_weather(city):
    raw = http_get(f"/weather?q={city}")  # could return an HTML 502 page
    try:
        return json.loads(raw)
    except Exception:
        return None  # model now receives None and invents a temperature

json.loads failed, the wrapper returned None, and the model invented a perfectly confident, fluent answer about Tokyo’s weather on top of None. Nobody noticed for three days, until the numbers looked wrong.

Environment

An LLM agent that calls external tools or HTTP APIs
Python, with Pydantic v2 available for schema validation
LangChain / LangGraph or OpenAI tool-calling style runs
A downstream API that occasionally returns HTML errors, rate-limit JSON, truncated streams, or timeouts

What happened?

The failure isn’t exotic. In demos, tools return clean schema-valid JSON. In production, tools return any of these:

truncated streams
HTML error pages (502 / 503 / 429)
200-with-error-body (status ok, payload is an error object)
rate-limit JSON shaped differently from success JSON
timeouts / partial reads

A naive wrapper handles all of these by returning None — which teaches the model exactly the wrong lesson: “you may keep talking fluently about data you never parsed.” The agent then hallucinates a plausible next step on bad context, and because the output is fluent and formatted, no exception-based monitor catches it.

How to solve it?

The rule is simple: never let an agent branch on output it could not parse. Concretely, four moves.

Solution #1: Validate at the boundary, before the model sees data

Validate tool output with a schema (Pydantic / JSON Schema / TypedDict) at the point where external data enters the agent. Raise a typed error that the runtime catches — not the LLM.

from pydantic import BaseModel, ValidationError

class Weather(BaseModel):
    city: str
    temp_c: float
    condition: str

def get_weather(city, *, run_id):
    raw = http_get(f"/weather?q={city}&fmt=json", timeout=5)
    try:
        return Weather.model_validate_json(raw).model_dump()
    except (ValidationError, ValueError) as e:
        raise ToolResponseInvalid(f"weather: malformed response for {city}") from e

If the payload is an HTML 502 page, validation fails, and the model never sees fake data.

Solution #2: Retry a bounded number of times with corrected parameters

Retries are only useful if you change the call each time. Retrying the same malformed request and burning tokens is a common mistake.

MAX_RETRIES = 2

def get_weather(city, *, run_id):
    last_err = None
    for attempt in range(1, MAX_RETRIES + 1):
        # tighten the request each attempt: plain JSON, shorter timeout
        raw = http_get(f"/weather?q={city}&fmt=json&t={attempt}", timeout=5)
        try:
            return Weather.model_validate_json(raw).model_dump()
        except (ValidationError, ValueError) as e:
            last_err = e
            logger.warning(
                "tool_malformed run_id=%s tool=weather attempt=%s err=%s",
                run_id, attempt, e,
            )
    # fall through to deterministic degradation
    ...

Two retries is usually enough. More than that and you’re just spending tokens on a broken tool.

Solution #3: Degrade deterministically, do not continue silently

After retries are exhausted, take a fixed action instead of letting the model guess:

logger.error("tool_unavailable run_id=%s tool=weather last_err=%s", run_id, last_err)
return {
    "tool_unavailable": "weather",
    "city": city,
    "reason": "malformed_response",
}

Return a structured tool_unavailable signal to the model with a clear instruction (“the weather tool is unavailable, say so”), use a fallback source, or escalate to a human-in-the-loop checkpoint. The model should never be the one deciding what to do with data it never received.

Solution #4: Log every malformed response with run_id and raw payload

Quiet failures only become quiet if nobody records them. Log run_id, tool name, the raw payload (redacted of secrets), and the retry outcome. Now “bad data nobody noticed” becomes “bad data the runtime recorded and handled.”

run_id
tool name
attempt number
raw payload (redacted)
parse / validation outcome
retry outcome

You can see that I succeeded to turn malformed responses into first-class failures instead of silent context pollution.

The reason

I think the key reason malformed tool responses cause silent failures is that teams treat parsing as plumbing and validation as optional. The model is fluent on any input, including garbage — so the only thing stopping it from confidently narrating bad data is the runtime validating the boundary before the model speaks. Once you validate, retry bounded, and degrade deterministically, malformed responses stop being invisible.

Common Mistakes

Returning None on parse error — the model invents a value.
Retrying unboundedly — burning tokens on a broken tool.
Retrying without changing the call parameters — same request, same failure.
Showing the agent the raw error text and hoping it self-corrects without a fallback.
Never logging the original malformed payload — you lose the root cause.

Summary

In this post, I showed how to handle malformed tool responses in an LLM agent without letting it hallucinate fluently on bad data. The key point is: validate every tool response at the boundary, retry a bounded number of times with corrected parameters, and degrade deterministically on persistent failure — never let an agent branch on output it failed to parse.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

👨‍💻 Reddit Discussion: Gartner thinks 40% of agentic AI projects get canceled by 2027
👨‍💻 Pydantic Documentation

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!