test/AGENTS.md
Rama main · réplica de lectura del repositorio original.
# test/ Guidelines
Use `just test` as alias for `pytest` execution to run the tests.
## Testing Conventions
Always write public API-based tests. Do not assert implementation details.
```python
# === BAD - digging into implementation details ===
async def test_collects_events_in_window(self) -> None:
stream = MemoryStream()
ctx = Context(stream=stream)
batches: list = []
async def callback(events, _ctx):
batches.append(events)
watch = CadenceWatch(max_wait=0.1, condition=ToolCallEvent)
watch.arm(stream, callback)
await stream.send(ToolCallEvent(name="t1", arguments="{}"), ctx)
await stream.send(ToolCallEvent(name="t2", arguments="{}"), ctx)
await asyncio.sleep(0.2)
assert len(batches) == 1
assert len(batches[0]) == 2
# === GOOD - public-api based test ===
async def test_collects_events_in_window(self) -> None:
# arrange stream
stream = MemoryStream()
batches: list[BaseEvent] = []
async def callback(events: BaseEvent, ctx: Context) -> None:
batches.append(events)
watch = CadenceWatch(max_wait=0.01, condition=ToolCallEvent)
watch.arm(stream, callback)
# arrange agent
tool_calls = [
ToolCallEvent(name="t1", arguments="{}"),
ToolCallEvent(name="t2", arguments="{}"),
]
agent = Agent(
"test-agent",
config=testing.TestConfig(tool_calls, "Done"),
)
@agent.tool
def t1(): ...
@agent.tool
def t2(): ...
# act
await agent.ask("Hello, world!", stream=stream)
await asyncio.sleep(0.02)
# assert
assert batches == [
IsList(*tool_calls, check_order=False),
]
```
- Always use `ag2.testing.TestConfig` to mock LLM responses in agent-based tests.
- Always use `ag2.testing.TrackingConfig` to validate messages the framework sends to the LLM (for example: tool results and user input).
### No monkeypatching internals
Do not use `monkeypatch.setattr`, `setattr` on an instance, or `unittest.mock.patch` to swap out a private function, method, or attribute (anything `_prefixed`). Patching internals pins the test to *how* the code works today, so it keeps passing even after the real behavior breaks.
The agent's public seam is `config=`. Script the LLM's turns with `ag2.testing.TestConfig` instead of reaching for the agent's private client — the same test, before and after:
```python
from ag2 import Agent
from ag2.events import ToolCallEvent
from ag2.testing import TestConfig
# BAD — patch the agent's private client to script the turn. The test breaks the
# moment that internal is renamed, and a green run proves nothing about the wiring.
async def test_agent_answers_with_tool(monkeypatch):
agent = Agent("weather", tools=[get_weather])
monkeypatch.setattr(agent, "_client", _scripted_client(...)) # private attribute
...
# GOOD — script the same turns through the public `config=` seam.
async def test_agent_answers_with_tool():
agent = Agent(
"weather",
config=TestConfig(
ToolCallEvent(name="get_weather", arguments='{"city": "Tokyo"}'), # turn 1: call the tool
"It's sunny in Tokyo.", # turn 2: final reply
),
tools=[get_weather],
)
reply = await agent.ask("Weather in Tokyo?")
assert reply.body == "It's sunny in Tokyo."
```
Use `TrackingConfig` (which wraps a `TestConfig`) when you also need to assert what the framework *sent* the LLM — again, no patching required.
Failure paths follow the same principle: raise from a public collaborator, never a patched internal — an `Agent` double whose `ask` raises, a `TraceSource` whose `load` raises, or a `LinkEndpoint` whose `frames()` raises (registered with `hub.attach_endpoint(...)`). If a branch can *only* be reached by patching an internal, it isn't publicly observable: cover the observable contract instead of reaching in to hit the line.
### Assertion style
Avoid chained field-access assertions like `result[0]["tool_calls"][0]["function"]["arguments"] == {...}`. Instead, compare the whole object directly (`assert msg == {...}`) or use **dirty-equals** `IsPartialDict` when only some fields matter:
```python
# Bad
assert result[0]["role"] == "assistant"
assert result[0]["tool_calls"][0]["function"]["arguments"] == {}
# Good — full comparison
assert result[0] == {"role": "assistant", "tool_calls": [...]}
# Good — partial match with dirty-equals (always use dict syntax, not kwargs)
from dirty_equals import IsPartialDict
assert result[0] == IsPartialDict({"role": "assistant"}) # Good
assert result[0] == IsPartialDict(role="assistant") # Bad — use dict syntax
```
### Imports
All imports must be at the top of the test file. Never place imports inside individual test functions until user asks for it.
### Function vs class-based tests
Use **plain functions** for standalone tests. Use **classes** to group multiple related tests that share a logical subject (e.g., `TestImageUrlInput`, `TestBinaryInput`). Do not wrap a single test method in a class — keep it a plain function instead.
If you need to apply markers to each test in class, apply them to the class itself.
```python
# Bad - markers are applied to each test individually
class TestAgent:
@pytest.mark.asyncio
async def test_defaults(self, context: Context) -> None: ...
@pytest.mark.asyncio
async def test_defaults(self, context: Context) -> None: ...
# Good - markers are applied to the class itself
@pytest.mark.asyncio
class TestAgent:
async def test_defaults(self, context: Context) -> None: ...
async def test_defaults(self, context: Context) -> None: ...
```
### Section comments
Do not use banner-style section dividers (e.g. `# ---\n# Section\n# ---`). Class names and test names are sufficient structure.
## Builtin Tools Testing
### Structure
Provider-specific tool tests live in `test/config/{provider}/tools/`:
- `test_{tool}.py` — e2e tests for supported tools (one file per tool)
- `test_unsupported.py` — all unsupported tools for the provider in one file
- `test_tool_to_api.py` — generic function tool mapping (not builtin-specific)
Variable resolution tests live in `test/tools/test_resolve.py`.
Provider test packages must stay importable when their SDK is absent — the LLM
CI matrix installs one provider at a time, and an unguarded top-level
`import anthropic` (directly, or via `ag2.config.{provider}`) breaks *collection*
for the whole run, even though the test itself would be deselected.
`test/config/{provider}/__init__.py` already guards the package with
`pytest.importorskip(...)`, so put provider tests inside that package rather than
in a new directory. A test that spans providers (e.g. under `test/tools/`) must
guard itself at the top of the module, before the imports:
```python
import pytest
pytest.importorskip("anthropic")
pytest.importorskip("openai")
from ag2.config.anthropic.mappers import tool_to_api
```
### Test Pattern
Tests must be e2e: instantiate the **Tool** class, call `schemas()`, pass through the provider mapper:
```python
@pytest.mark.asyncio
async def test_defaults(context: Context) -> None:
tool = WebSearchTool()
[schema] = await tool.schemas(context)
assert tool_to_api(schema) == {"type": "web_search_20250305", "name": "web_search"}
```
Do **not** instantiate schema classes directly in provider tests — always go through the Tool.
### Fixtures
Use the shared `context` pytest fixture from `test/config/conftest.py` (no need to import — pytest discovers it automatically):
```python
async def test_defaults(context: Context) -> None: ...
```
### Coverage Requirements
Every builtin tool must be tested in **every** provider:
- **Supported**: test the happy-path mapping in `test_{tool}.py`
- **Unsupported**: test `UnsupportedToolError` is raised in `test_unsupported.py`
For OpenAI, test both `tool_to_api` (completions) and `tool_to_responses_api` (responses) paths. Group unsupported tests under `TestCompletionsApi` / `TestResponsesApi` classes.
### Variable Resolution
Each tool that accepts `Variable` parameters needs exactly 2 tests in `test/tools/test_resolve.py`:
1. Value resolved from context
2. Missing key raises `KeyError`