Agent Profiles
Agent profiles are behavioral baselines that Foil automatically learns for each agent from its trace data. By understanding what’s normal for a specific agent — its tools, error rates, traffic patterns, and usage characteristics — Foil can make evaluations like hallucination detection and quality analysis significantly more accurate. Without profiles, every agent is evaluated the same way. With profiles, evaluations are contextualized: the evaluator knows what the agent does, how it typically behaves, and what constitutes a deviation worth flagging.What’s in a Profile
A profile captures multiple dimensions of agent behavior:| Component | What it Tracks |
|---|---|
| Identity | Use case, maturity level, behavioral summary |
| Tool patterns | Tool distribution, common sequences, anomalous usage |
| Error patterns | Error rate, common error types, trend direction |
| Temporal patterns | Volume patterns, seasonality, peak usage times |
| Volume characteristics | Daily averages, session length, typical latency |
| Insights | AI-derived behavioral observations (3–5 key findings) |
How Profile Learning Works
Profile learning is fully automatic. Once an agent starts sending traces, Foil begins collecting data and building a behavioral model through three phases.Pre-profile
Foil collects traces until enough data exists to build a meaningful profile. During this phase, no profile is available and evaluations run without behavioral context. Once a minimum data threshold is reached, learning begins.Bootstrap
Foil generates the first profile and continues refining it as more data arrives. During bootstrap:- The system re-learns at geometrically increasing intervals — learning is frequent early on and becomes less frequent as the profile stabilizes
- Each learning cycle compares the new profile against the previous one
- When the system detects no material changes across consecutive cycles, it considers the profile converged and transitions to steady state
Steady State
The profile is established and stable. Re-learning only occurs when:- Behavioral drift is detected — new tools appear, error rates change significantly, volume patterns shift
- The profile becomes stale — a periodic refresh ensures the profile stays current even without dramatic changes
- A cooldown period prevents excessive re-learning from transient fluctuations
Anchors
Anchors are health invariants that Foil automatically generates alongside a profile. They express concrete, measurable claims about the agent’s behavior — for example:- “Error rate stays below 5%”
- “Average latency remains under 2 seconds”
- “Tool X is used in more than 80% of sessions”
How Anchors Work
- Anchors are generated automatically when a profile is created or updated
- They are evaluated on every learning cycle, with each anchor marked as
passing,failing, orunknown - The current status and most recent measured value are stored with the profile
Anchor-Driven Re-learning
When more than half of a profile’s anchors break (transition tofailing), Foil interprets this as a fundamental behavioral shift. The system re-enters the bootstrap phase to re-learn the profile from scratch, establishing new baselines that reflect the agent’s changed behavior.
How Profiles Improve Evaluations
Profiles are the key mechanism for making evaluations context-aware: Without profiles, evaluations apply the same generic criteria to every agent. A customer support bot and a code review assistant are judged identically, leading to false positives and missed issues. With profiles, evaluations are informed by the agent’s known behavior:- Contextual evaluation — The evaluator receives relevant profile dimensions for each check. Hallucination detection gets tool patterns and identity context. Error detection gets error baselines. Quality checks get behavioral summaries.
- Anomaly flags — Per-trace anomalies are surfaced to evaluators. If a trace uses a tool the agent has never used before, or shows an error rate far above baseline, this context helps the evaluator make a more informed judgment.
- Calibrated baselines — An agent with a known 2% error rate is evaluated differently than one with a 15% error rate. What’s normal for one agent might be alarming for another.
Managing Profiles
Viewing a Profile
Manual Editing
Force Regeneration
Reset Learning State
Enable/Disable Learning
Profile learning is controlled via the agent’sprofileSettings.learningEnabled field. When disabled, the existing profile is preserved but no new learning occurs.
Best Practices
Let profiles bootstrap naturally
Let profiles bootstrap naturally
Avoid forcing regeneration frequently. The learning system is designed to converge on its own — give it time to collect enough data and stabilize.
Use anchors as health monitors
Use anchors as health monitors
When anchors start failing, investigate the underlying cause. Anchor failures often indicate real behavioral changes — a new deployment, a prompt update, or a downstream service issue.
Manual edits are temporary
Manual edits are temporary
If you manually edit a profile, be aware that the next learning cycle will overwrite your changes. Manual edits are best used for short-term corrections while you address the root cause.
Learning is automatic
Learning is automatic
No configuration is needed beyond enabling profile learning. Foil handles data collection, threshold detection, and re-learning on its own.