How to Make AI Agents Explainable in Spring Boot Applications

Mar 26, 2026

The Problem

My AI agent kept making unexpected tool calls in production. When stakeholders asked why the model chose a specific action, I had no answer. The logs showed what tools were called, but not why.

2026-03-26 10:15:32 INFO  Tool called: retrievePatientHealthStatus
2026-03-26 10:15:33 INFO  Tool called: sendNotification

Why did it call sendNotification? What reasoning led to that decision? Traditional logging was useless for debugging.

The Solution

Spring AI’s Tool Argument Augmenter captures the LLM’s reasoning for every tool call. The model explains its own decisions, and I can log, audit, or analyze that data.

How It Works

The augmenter injects special parameters into every tool call. These parameters force the LLM to provide reasoning and confidence before executing the actual tool.

public record AgentThinking(
    @ToolParam(description = "Internal reasoning for this tool call")
    String innerThought,

    @ToolParam(description = "Confidence level: high, medium, low")
    String confidence
) {}

The @ToolParam annotations tell Spring AI to inject these parameters. The LLM fills them in automatically.

Setting Up Explainable Tools

I need three pieces: a DTO for reasoning, the tool itself, and the augmenter configuration.

public record AgentThinking(
    @ToolParam(description = "Explain why you are calling this tool and what you expect to find")
    String innerThought,

    @ToolParam(description = "Your confidence in this decision: high, medium, or low")
    String confidence,

    @ToolParam(description = "What alternative tools did you consider")
    String alternativesConsidered
) {}

@Component
public class PatientTools {

    @Tool(description = "Retrieve patient health status by ID")
    public String retrievePatientHealthStatus(
        @ToolParam(description = "Patient ID") String patientId,
        AgentThinking thinking  // Injected by augmenter
    ) {
        // The thinking parameter is automatically populated by the LLM
        // I can log it, store it, or ignore it
        return healthService.getStatus(patientId);
    }

    @Tool(description = "Send notification to patient")
    public String sendNotification(
        @ToolParam(description = "Patient ID") String patientId,
        @ToolParam(description = "Message to send") String message,
        AgentThinking thinking
    ) {
        return notificationService.send(patientId, message);
    }
}

Now I configure the augmenter to capture this reasoning.

@Service
public class ExplainableAgentService {
    private final ChatClient chatClient;
    private final AuditRepository auditRepository;

    public ExplainableAgentService(
            OpenAiChatModel model,
            AuditRepository auditRepository) {
        this.auditRepository = auditRepository;

        AugmentedToolCallbackProvider&lt;AgentThinking&gt; provider =
            AugmentedToolCallbackProvider.&lt;AgentThinking&gt;builder()
                .toolObject(new PatientTools())
                .argumentType(AgentThinking.class)
                .argumentConsumer(event -> {
                    AgentThinking thinking = event.arguments();

                    // Log for immediate debugging
                    log.info("Tool: {} | Reasoning: {} | Confidence: {}",
                        event.toolDefinition().name(),
                        thinking.innerThought(),
                        thinking.confidence());

                    // Store for auditing and compliance
                    auditRepository.save(new AuditRecord(
                        Instant.now(),
                        event.toolDefinition().name(),
                        thinking.innerThought(),
                        thinking.confidence(),
                        thinking.alternativesConsidered()
                    ));
                })
                .build();

        chatClient = ChatClient.builder(model)
            .defaultToolCallbacks(provider)
            .build();
    }

    public String processRequest(String userMessage) {
        return chatClient.prompt()
            .user(userMessage)
            .call()
            .content();
    }
}

What the Captured Reasoning Looks Like

When my agent processes a request, I now get the full decision context.

Tool: retrievePatientHealthStatus
Reasoning: I need to check the patient's current health status before deciding
whether to send a notification. The user asked about follow-up care, which
requires knowing their current condition.
Confidence: high
Alternatives: Could have searched for appointment history first

Tool: sendNotification
Reasoning: The health status shows abnormal values. Per protocol, patients with
these readings should be notified to seek immediate attention.
Confidence: high
Alternatives: Could have created a task instead of direct notification

Now I understand exactly why each tool was called.

Storing Audit Records

For compliance, I persist every decision.

@Entity
@Table(name = "agent_audit_log")
public class AuditRecord {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    private Instant timestamp;
    private String toolName;
    @Column(columnDefinition = "TEXT")
    private String reasoning;
    private String confidence;
    @Column(columnDefinition = "TEXT")
    private String alternatives;

    // Constructors, getters, setters
    public AuditRecord() {}

    public AuditRecord(Instant timestamp, String toolName,
                       String reasoning, String confidence, String alternatives) {
        this.timestamp = timestamp;
        this.toolName = toolName;
        this.reasoning = reasoning;
        this.confidence = confidence;
        this.alternatives = alternatives;
    }
}

Querying Decision History

I can now query past decisions to find patterns.

@Repository
public interface AuditRepository extends JpaRepository&lt;AuditRecord, Long&gt; {

    List&lt;AuditRecord&gt; findByToolNameOrderByTimestampDesc(String toolName);

    List&lt;AuditRecord&gt; findByConfidence(String confidence);

    @Query("SELECT a FROM AuditRecord a WHERE a.reasoning LIKE %:keyword%")
    List&lt;AuditRecord&gt; findByReasoningKeyword(@Param("keyword") String keyword);
}

Common Mistake: Building Custom Logging

I initially tried wrapping my tools with custom logging.

// DON'T DO THIS
@Tool
public String retrievePatientHealthStatus(String patientId) {
    log.info("Tool called with patientId: {}", patientId);  // Only shows inputs
    String result = healthService.getStatus(patientId);
    log.info("Tool result: {}", result);  // Only shows outputs
    return result;
}

This captures inputs and outputs but misses the model’s reasoning. The augmenter pattern captures the “why” without modifying tool code.

Environment

Spring Boot 3.3.x
Spring AI 1.0.0
Java 21

When to Use Explainable Agents

I use this pattern for:

Regulated industries: Healthcare, finance need audit trails
Debugging production issues: Understand unexpected behavior
Prompt optimization: Identify ambiguous instructions
Trust building: Show stakeholders how decisions are made
Safety monitoring: Flag low-confidence decisions

Summary

Making AI agents explainable requires capturing model reasoning, not just tool outputs. Spring AI’s Tool Argument Augmenter does this by injecting @ToolParam fields that the LLM fills with its internal reasoning.

Key steps:

Create a DTO with @ToolParam annotated fields for reasoning
Add the DTO as a parameter to each tool method
Configure AugmentedToolCallbackProvider with an argumentConsumer
Log or persist the captured reasoning

The result is full visibility into agent decision-making without changing tool implementations.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

👨‍💻 Spring AI Documentation
👨‍💻 Tool Argument Augmenter Guide

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!