Skip to content
BSWEN
Blog

How can AI agents dynamically discover and select the right APIs for their workflows?

Cowrie
Cowrie
Dev @ Bswen

I was staring at my agent codebase, drowning in hardcoded API integrations. Every time I needed a new data source, I had to manually add the library, configure authentication, write wrapper methods, and redeploy. The maintenance burden was crushing me.

Then someone on Reddit dropped a bombshell insight:

“Scrape a copy of the repo and search it and read the repo to learn how to call the API instead of wiring any of them up directly, let the agent remember and reuse whatever it finds useful.”

The repo they were talking about? The most starred repository on GitHub with 396,000 stars - a free list of public APIs updated continuously by 1,200+ contributors.

That’s when I realized I’d been building agents the wrong way.

The Problem: Static Integration Hell

Traditional AI agents rely on hardcoded API integrations. Here’s what my old code looked like:

static_agent.py
# BAD: Hardcoded API list - no dynamic discovery
class StaticAgent:
    def __init__(self):
        # Manually configured APIs - maintenance nightmare
        self.apis = {
            "weather": WeatherAPI(api_key="..."),
            "crypto": CryptoAPI(api_key="..."),
            "news": NewsAPI(api_key="...")
        }


    def get_weather(self, location):
        return self.apis["weather"].fetch(location)


    def get_crypto_price(self, symbol):
        return self.apis["crypto"].price(symbol)

Every new API required:

  1. Adding a library
  2. Getting an API key
  3. Writing wrapper methods
  4. Deploying updated code

I was stuck in integration hell. My agents could only use APIs I’d manually wired up. They were dumb, static, and expensive to maintain.

The core challenge: How do we build agents that can autonomously find and use the right APIs for any given task?

The Solution: Dynamic API Discovery

I built a three-layer architecture that lets agents discover APIs at runtime:

Layer 1: API Knowledge Base

The public APIs repository is a goldmine. Every entry contains structured metadata:

Public APIs Repo (396K stars)
         |
         v
+-------------------+
|   Scrape & Index  |
| - API name        |
| - Description     |
| - Auth type       |
| - HTTPS support   |
| - CORS policy     |
| - Rate limits     |
| - Category tags   |
| - Endpoint docs   |
+-------------------+
         |
         v
+-------------------+
| Semantic Index    |
| (vector embeddings|
|  for natural      |
|  language search) |
+-------------------+

This metadata is gold for filtering:

  • Auth requirement (API key, OAuth, none) - fastest prototyping uses no auth
  • HTTPS support - security consideration
  • CORS policy - browser compatibility
  • Rate limits - usage constraints
  • Category tags - domain matching

Layer 2: Discovery Engine

The agent’s discovery process works like this:

  1. Task Analysis: Parse the user request to understand data requirements
  2. Semantic Search: Query the API index with natural language
  3. Metadata Filtering: Narrow by auth type, rate limits, category
  4. Capability Matching: Compare API endpoints against task needs
  5. Selection & Ranking: Choose best-fit APIs based on:
    • No auth required (fastest prototyping)
    • HTTPS support (security)
    • Generous rate limits (reliability)
    • Active maintenance (stability)

Layer 3: Runtime Integration

Here’s where the magic happens. The agent learns and executes at runtime:

  1. Read API Documentation: Agent parses endpoint specs from the repo
  2. Generate Client Code: Dynamic request construction
  3. Execute & Learn: Make API calls, cache successful patterns
  4. Remember & Reuse: Store working integrations for future tasks

Implementation: Building a Dynamic Agent

Let me show you the core implementation:

api_discovery.py
import aiohttp
import asyncio
from typing import Optional, Dict, List, Any
from dataclasses import dataclass, field
from datetime import datetime
import json
import logging


logger = logging.getLogger(__name__)


@dataclass
class APIMetadata:
    """Structured metadata for an API."""
    name: str
    description: str
    base_url: str
    auth_type: str  # "none", "api_key", "oauth"
    https: bool
    cors: bool
    category: str
    rate_limit: Optional[str] = None
    endpoints: List[Dict] = field(default_factory=list)


@dataclass
class DiscoveredAPI:
    """An API discovered and learned by the agent."""
    metadata: APIMetadata
    last_used: datetime
    success_count: int = 0
    failure_count: int = 0
    learned_endpoints: Dict[str, Any] = field(default_factory=dict)

The discovery engine scrapes the public APIs repo and builds a semantic index:

discovery_engine.py
class APIDiscoveryEngine:
    """Engine for discovering and selecting APIs dynamically."""


    def __init__(self, api_index_path: str = "public_apis_index.json"):
        self.api_index_path = api_index_path
        self.api_index: List[APIMetadata] = []
        self.learned_apis: Dict[str, DiscoveredAPI] = {}
        self._session: Optional[aiohttp.ClientSession] = None


    async def load_index(self):
        """Load or build the API index from public APIs repository."""
        try:
            with open(self.api_index_path, 'r') as f:
                data = json.load(f)
                self.api_index = [APIMetadata(**item) for item in data]
            logger.info(f"Loaded {len(self.api_index)} APIs from index")
        except FileNotFoundError:
            logger.warning("API index not found, building from scratch...")
            await self._build_index_from_repo()


    async def _build_index_from_repo(self):
        """Scrape public APIs repo and build semantic index."""
        # In practice, this would:
        # 1. Clone/scrape the public-apis repository
        # 2. Parse README.md for API entries
        # 3. Extract metadata (auth, HTTPS, CORS, category)
        # 4. Generate embeddings for semantic search
        # 5. Save to index file
        pass

The key method is discovering APIs that match a task:

discover_method.py
async def discover_apis(
    self,
    task_description: str,
    filters: Optional[Dict[str, Any]] = None
) -> List[APIMetadata]:
    """
    Discover APIs relevant to a task.


    Args:
        task_description: Natural language description of what's needed
        filters: Optional constraints (auth_type, category, etc.)


    Returns:
        List of matching APIs ranked by relevance
    """
    candidates = []


    for api in self.api_index:
        # Apply filters
        if filters:
            if filters.get("auth_type") and api.auth_type != filters["auth_type"]:
                continue
            if filters.get("https_only") and not api.https:
                continue
            if filters.get("category") and api.category != filters["category"]:
                continue


        # Simple keyword matching (replace with semantic search in production)
        if self._matches_task(api, task_description):
            candidates.append(api)


    # Sort by ease of use and prior success
    candidates.sort(key=lambda a: (
        0 if a.auth_type == "none" else 1,  # Prefer no auth
        self.learned_apis.get(a.name, DiscoveredAPI(
            metadata=a, last_used=datetime.min
        )).success_count  # Prefer previously successful
    ), reverse=True)


    return candidates[:10]  # Top 10 candidates


def _matches_task(self, api: APIMetadata, task: str) -> bool:
    """Check if API matches task requirements."""
    task_lower = task.lower()
    desc_lower = api.description.lower()


    keywords = task_lower.split()
    return any(kw in desc_lower or kw in api.category.lower() for kw in keywords)

The learning loop is critical - agents remember successful integrations:

learning_loop.py
async def learn_api(self, api: APIMetadata) -> DiscoveredAPI:
    """
    Learn how to use an API by reading its documentation.


    In practice, this would:
    1. Fetch API documentation
    2. Parse endpoint specifications
    3. Generate client code
    4. Store learned patterns
    """
    discovered = DiscoveredAPI(
        metadata=api,
        last_used=datetime.now(),
        learned_endpoints={}
    )
    self.learned_apis[api.name] = discovered
    return discovered


async def execute_api_call(
    self,
    api_name: str,
    endpoint: str,
    params: Dict[str, Any]
) -> Dict[str, Any]:
    """Execute an API call using learned patterns."""
    if api_name not in self.learned_apis:
        raise ValueError(f"API {api_name} not learned. Call learn_api() first.")


    discovered = self.learned_apis[api_name]
    api = discovered.metadata


    url = f"{api.base_url}/{endpoint}"


    if self._session is None:
        self._session = aiohttp.ClientSession()


    try:
        async with self._session.get(url, params=params) as response:
            response.raise_for_status()
            data = await response.json()


            discovered.success_count += 1
            discovered.last_used = datetime.now()


            return data


    except Exception as e:
        discovered.failure_count += 1
        logger.error(f"API call failed: {api_name}/{endpoint} - {e}")
        raise

The Dynamic Agent in Action

Here’s the complete agent that discovers APIs at runtime:

dynamic_agent.py
class DynamicAgent:
    """
    AI agent that discovers and uses APIs dynamically.


    This agent can work with APIs that didn't exist when it was built.
    """


    def __init__(self):
        self.discovery = APIDiscoveryEngine()
        self.context: Dict[str, Any] = {}


    async def initialize(self):
        """Load API index on startup."""
        await self.discovery.load_index()


    async def execute_task(self, task: str) -> str:
        """
        Execute a task by discovering and using appropriate APIs.


        Args:
            task: Natural language task description


        Returns:
            Result of the task execution
        """
        # Step 1: Discover relevant APIs
        apis = await self.discovery.discover_apis(
            task,
            filters={"auth_type": "none"}  # Prefer APIs with no auth
        )


        if not apis:
            return f"No suitable APIs found for task: {task}"


        # Step 2: Learn and try APIs
        for api in apis[:3]:  # Try top 3
            try:
                learned = await self.discovery.learn_api(api)
                return f"Using {api.name}: {api.description}"
            except Exception as e:
                logger.warning(f"Failed to use {api.name}: {e}")
                continue


        return f"Failed to complete task: {task}"


    async def get_relevant_apis(self, task: str) -> List[APIMetadata]:
        """
        Get list of APIs relevant to a task without executing.


        Useful for showing users what data sources are available.
        """
        return await self.discovery.discover_apis(task)

Usage example:

usage_example.py
async def main():
    agent = DynamicAgent()
    await agent.initialize()


    # Discover APIs for a task
    relevant_apis = await agent.get_relevant_apis(
        "Get current weather data for a city"
    )


    print(f"Found {len(relevant_apis)} relevant APIs:")
    for api in relevant_apis[:5]:
        print(f"  - {api.name}: {api.description}")
        print(f"    Auth: {api.auth_type}, HTTPS: {api.https}, Category: {api.category}")


    # Execute task using discovered APIs
    result = await agent.execute_task("Get current weather data for London")
    print(f"\nResult: {result}")




if __name__ == "__main__":
    asyncio.run(main())

Why This Matters

The shift from static to dynamic API discovery changes everything:

Adaptability: Agents work with APIs that didn’t exist when they were built. No more waiting for manual integrations.

Efficiency: No manual integration work for each new API. The agent figures it out.

Scale: One agent can access thousands of APIs through a single discovery mechanism.

Resilience: If one API fails, the agent can discover alternatives automatically.

Innovation: Agents can find novel data sources humans wouldn’t consider.

Common Mistakes I Made

Pre-wiring Everything: At first, I tried to build agents with fixed API lists. This defeats the entire purpose. The power is in runtime discovery.

Ignoring Metadata: I initially selected APIs based only on name/description. Big mistake. You need to filter by auth requirements, rate limits, and CORS policies.

No Learning Loop: My early prototypes didn’t cache successful integrations. They repeated the discovery work every time. Massive waste of compute.

Monolithic Agents: I tried building one mega-agent that did everything. Better approach: specialized “OpenClaw instances for each business domain” that discover domain-relevant APIs.

Missing Fallbacks: When dynamic discovery failed, my agents just crashed. They need graceful degradation and alternative strategies.

Architecture Summary

The three-layer pattern:

  1. Index Layer: Scrape public APIs repo, extract metadata, generate embeddings
  2. Discovery Layer: Semantic search + metadata filtering to find relevant APIs
  3. Learning Layer: Read API docs, generate client code, cache successful patterns

Implementation path:

  1. Start with the public-apis GitHub repo (396K stars, 1,200+ contributors)
  2. Build metadata extraction for auth, HTTPS, CORS, categories
  3. Implement semantic search with task-to-API matching
  4. Add learning system to cache successful integrations
  5. Deploy specialized instances per business domain

Resources

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!

Comments

Tags: