Meter OpenAI Agents SDK tool calls and tokens per customer

Hook into an OpenAI Agents SDK run, record every model call's input and output tokens and every tool call against the customer it ran for, and send them to Usage Lab with keys that make retries count once. A sibling of the Vercel AI SDK, OpenAI and Anthropic guide.

The hooks (checked Oct 25, 2026)

Sources: Agents SDK reference: lifecycle (RunHooks, AgentHooks), reference: items (ModelResponse), Usage guide. The wiring below was also checked against the openai-agents 0.24.0 package from PyPI.

Deterministic idempotency keys

Every Usage Lab event needs an idempotency key, and Usage Lab never makes one up for you. Build keys from things a retry repeats:

Hash with SHA-256 (keys can be up to 255 characters). Save run_id with your job so a retried job resends the same keys: they come back as duplicate and count once. The same key with a different body returns 409 and keeps the original. Each event is tagged traffic_class: ai_agent. Agent calls are real, billable usage, and we never filter them.

1. The metering helper (runs standalone)

No OpenAI key needed. It creates the three meters (the Free plan's 3), replays one agent run for cust_acme (two model calls, two tool calls) exactly as the hooks would report it, sends it twice to show the retry counts once, and prints this month's totals. It needs only Python's standard library and USAGE_LAB_KEY (a free key from /signup/). Save it as usage_lab_agents_meter.py and run python3 usage_lab_agents_meter.py.

# usage_lab_agents_meter.py: Python 3.8+, standard library only. Needs USAGE_LAB_KEY in the environment.
# Metering helper for OpenAI Agents SDK runs: tool calls + input/output tokens per customer.
# Runs standalone (no OpenAI key needed): the bottom half replays what the RunHooks would report.
import hashlib, json, os, sys, urllib.error, urllib.request
from datetime import datetime, timezone

API = "https://usage-lab.pages.dev/api/v1"
HEADERS = {"Authorization": "Bearer " + os.environ["USAGE_LAB_KEY"], "Content-Type": "application/json",
           "User-Agent": "usage-lab-quickstart/1.0"}  # any descriptive User-Agent; see the note below the sample
METERS = {"agent_tool_calls": "count", "llm_input_tokens": "sum", "llm_output_tokens": "sum"}

def call(method, path, body=None):
    data = None if body is None else json.dumps(body).encode()
    req = urllib.request.Request(API + path, data=data, headers=HEADERS, method=method)
    try:
        with urllib.request.urlopen(req, timeout=20) as r:
            return r.status, json.load(r)
    except urllib.error.HTTPError as e:
        return e.code, json.load(e)

def key(*parts):
    # Deterministic: the same run + model call / tool call always gives the same key, so retries count once.
    return "oa_" + hashlib.sha256(":".join(str(p) for p in parts).encode()).hexdigest()[:40]

class UsageLabMeter:
    """Collects events from hook callbacks for one run, then flushes them in batches of up to 100."""
    def __init__(self, customer, run_id):
        self.customer, self.run_id, self.events, self.llm_calls = customer, run_id, [], 0

    def llm_end(self, response_id, input_tokens, output_tokens, agent_name):
        self.llm_calls += 1
        rid = response_id or f"llm{self.llm_calls}"            # response_id is None on some providers
        props = {"agent": agent_name, "traffic_class": "ai_agent"}
        for meter, n, kind in (("llm_input_tokens", input_tokens, "in"), ("llm_output_tokens", output_tokens, "out")):
            self.events.append({"meter": meter, "subscription": self.customer, "value": n,
                                "idempotency_key": key(self.run_id, rid, kind), "properties": props})

    def tool_end(self, tool_call_id, tool_name, agent_name):
        self.events.append({"meter": "agent_tool_calls", "subscription": self.customer,
                            "idempotency_key": key(self.run_id, "tool", tool_call_id),
                            "properties": {"tool": tool_name, "agent": agent_name, "traffic_class": "ai_agent"}})

    def flush(self):
        sent, self.events = self.events, []
        for i in range(0, len(sent), 100):
            status, body = call("POST", "/usage", {"events": sent[i:i + 100]})
            if status >= 400:
                raise RuntimeError(f"usage: HTTP {status} {body}")
            yield body["summary"]

def ensure_meters():
    for k, agg in METERS.items():
        status, body = call("POST", "/meters", {"key": k, "aggregation": agg, "name": k})
        if status not in (201, 409):  # 409 meter_exists on a re-run is fine
            sys.exit(f"create meter {k}: HTTP {status} {body}")

if __name__ == "__main__":
    ensure_meters()
    month = datetime.now(timezone.utc).strftime("%Y-%m")
    run_id = f"run_{month}_c41d"
    # Replay of one agent run for customer cust_acme: 2 model calls and 2 function tool calls,
    # exactly what on_llm_end / on_tool_end receive (response.usage, context.tool_call_id).
    for attempt in ("first send", "retry"):
        m = UsageLabMeter("cust_acme", run_id)
        m.llm_end("resp_demo_001", 820, 64, "Support agent")
        m.tool_end("call_demo_a1", "lookup_order", "Support agent")
        m.tool_end("call_demo_a2", "refund_order", "Support agent")
        m.llm_end("resp_demo_002", 1130, 210, "Support agent")
        for summary in m.flush():
            print(attempt, summary)
    for k in METERS:
        status, body = call("GET", f"/meters/{k}/rollups?month={month}")
        if status != 200:
            sys.exit(f"rollups {k}: HTTP {status} {body}")
        for r in body["rollups"]:
            print(k, r["subscription"], r["quantity"])

Example output, not a command.

first send {'created': 6, 'duplicate': 0, 'conflict': 0}
retry {'created': 0, 'duplicate': 6, 'conflict': 0}
agent_tool_calls cust_acme 2
llm_input_tokens cust_acme 1950
llm_output_tokens cust_acme 274

User-Agent note: the Cloudflare edge in front of *.pages.dev rejects Python's default Python-urllib User-Agent (error 1010) before the request reaches us, so the helper sets usage-lab-quickstart/1.0. Any descriptive User-Agent works, including AI agents, bots and scripts, which are welcome.

2. Wire it into the Agents SDK

This part needs pip install openai-agents and an OPENAI_API_KEY, so our docs test doesn't run it against OpenAI. We did run it against openai-agents 0.24.0 with a stand-in model: the hooks reported both model calls (response ids and token counts) and the tool call's tool_call_id. The helper sends events in flush() after the run, so metering never slows the agent down, and the finally meters what ran even if the run fails part-way.

SDK wiring: needs openai-agents and OPENAI_API_KEY, so it is not run by our docs test. Save as agent_app.py next to the helper.

# agent_app.py: SDK wiring. Needs `pip install openai-agents` and OPENAI_API_KEY, plus the helper above
# saved as usage_lab_agents_meter.py next to this file. Checked against openai-agents 0.24.0.
import asyncio, uuid
from dataclasses import dataclass
from agents import Agent, RunContextWrapper, RunHooks, Runner, function_tool
from usage_lab_agents_meter import UsageLabMeter, ensure_meters

@dataclass
class Ctx:
    customer: str  # your customer id -> Usage Lab subscription

class MeterHooks(RunHooks[Ctx]):
    def __init__(self, meter):
        self.meter, self.tools = meter, 0
    async def on_llm_end(self, context, agent, response):
        u = response.usage  # ModelResponse.usage: input_tokens, output_tokens
        self.meter.llm_end(response.response_id, u.input_tokens, u.output_tokens, agent.name)
    async def on_tool_end(self, context, agent, tool, result):
        # For function tools, context is a ToolContext with tool_call_id and tool_name
        self.tools += 1  # fallback step number for tool families without a tool_call_id
        call_id = getattr(context, "tool_call_id", None) or f"step{self.tools}:{tool.name}"
        self.meter.tool_end(call_id, getattr(context, "tool_name", tool.name), agent.name)

@function_tool
def lookup_order(order_id: str) -> str:
    return f"Order {order_id}: shipped"

async def main():
    ensure_meters()
    run_id = "run_" + uuid.uuid4().hex  # store it with the job so a retried job reuses it
    meter = UsageLabMeter("cust_acme", run_id)
    agent = Agent(name="Support agent", instructions="Help with orders.", tools=[lookup_order])
    try:
        result = await Runner.run(agent, "Where is order 42?", context=Ctx("cust_acme"), hooks=MeterHooks(meter))
        print(result.final_output)
    finally:
        for summary in meter.flush():  # meter what ran, even if the run failed part-way
            print(summary)

asyncio.run(main())

3. Per-customer totals

GET /api/v1/meters/agent_tool_calls/rollups?month=YYYY-MM (and the same for each token meter) returns each customer's monthly quantity and event count, recomputed from the raw events. Push them to Stripe with Stripe push (beta, Pro and Ultra), or read them into your own billing. More: metering AI agent and MCP tool calls, idempotent usage events.

FAQ

Which OpenAI Agents SDK hooks should I meter from?

Subclass RunHooks and pass it to Runner.run(..., hooks=...). on_llm_end(context, agent, response) runs after every model call and gives you response.usage (input_tokens, output_tokens) and response.response_id. on_tool_end(context, agent, tool, result) runs after each local tool call; for function tools the context is a ToolContext with tool_call_id and tool_name. Checked against the official SDK docs and openai-agents 0.24.0 on Oct 25, 2026.

How are the idempotency keys built?

Deterministically from your run id plus the model call's response_id (one key for input tokens, one for output) or the tool call's tool_call_id, hashed with SHA-256. Store the run id with the job, so a retried job resends the same keys and Usage Lab counts each call once.

Do I need an OpenAI key to try this?

No. The metering helper runs on its own with only Python's standard library and a free Usage Lab key: it replays what the hooks report for one run. The SDK wiring needs openai-agents and an OpenAI API key.

Do agent calls count as real usage?

Yes. Every model call and tool call an agent makes for a customer is real, billable usage, tagged traffic_class ai_agent. Usage Lab never blocks or filters agent or bot traffic.

Why not just read result.context_wrapper.usage at the end?

That total is fine for one number per run, but it's lost if the run fails part-way and it can't tell tool calls apart. Per-call hooks with per-call keys meter exactly what ran and stay retry-safe.

What does Usage Lab cost?

Free: CA$0 for 100,000 events and 3 meters a month. Pro: CA$12 a month for 2M events and 10 meters. Ultra: CA$36 a month for 20M events and 100 meters. Flat prices, never a percentage of your billings.

Related