Alerting

Foil’s alerting system detects AI quality issues in real-time and notifies you via email or SMS.

Alert Types

Foil detects two categories of issues:

Threshold-Based Alerts

Triggered when metrics exceed configured limits:

Alert Type	Description	Default Threshold
`error`	Request failed with error	Any error
`highDuration`	Response took too long	5000ms
`highInputTokens`	Too many input tokens	100,000
`highOutputTokens`	Too many output tokens	50,000
`highCost`	Single call too expensive	$1.00
`timeout`	Request timed out	30s

LLM-Analyzed Alerts

Foil uses AI to detect quality issues:

Alert Type	Description
`hallucination`	Output contains fabricated facts
`nsfw`	Output contains inappropriate content
`stuck`	Agent is repeating itself or looping
`quality`	Output is off-topic or low quality

How It Works

User Request → Your Agent → Foil Ingestion
                                   ↓
                          ┌─────────────────┐
                          │  Inline Checks  │
                          │  (thresholds)   │
                          └────────┬────────┘
                                   ↓
                          ┌─────────────────┐
                          │  LLM Analysis   │
                          │  (quality)      │
                          └────────┬────────┘
                                   ↓
                          ┌─────────────────┐
                          │  Rate Limiter   │
                          └────────┬────────┘
                                   ↓
                          ┌─────────────────┐
                          │  Notifications  │
                          │  (email/SMS)    │
                          └─────────────────┘

Per-Agent Configuration

Configure alerts in the dashboard or via API:

PUT /api/agents/:agentId/alerts
{
  "llmAnalysis": {
    "enabled": true,
    "alertTypes": {
      "hallucination": {
        "enabled": true,
        "threshold": 0.7,
        "severity": "high",
        "channels": ["email"],
        "cooldownMinutes": 5
      },
      "nsfw": {
        "enabled": true,
        "threshold": 0.8,
        "severity": "critical",
        "channels": ["email", "sms"]
      },
      "stuck": { "enabled": true, "threshold": 0.6, "severity": "warning" },
      "quality": { "enabled": true, "threshold": 0.5, "severity": "warning" }
    }
  },
  "thresholds": {
    "duration": 5000,
    "inputTokens": 100000,
    "outputTokens": 50000,
    "cost": 1.0
  },
  "contacts": {
    "email": [{ "address": "alerts@yourcompany.com", "enabled": true }],
    "sms": [{ "phoneNumber": "+1234567890", "enabled": true }]
  }
}

Configuration Options

Option	Type	Description
`enabled`	boolean	Enable/disable the alert type
`threshold`	number	Confidence threshold (0-1) for LLM alerts
`severity`	string	`low`, `warning`, `high`, `critical`
`channels`	array	Notification channels (`email`, `sms`)
`cooldownMinutes`	number	Minimum time between alerts

Alert Severity

Severity	Description	Use Case
`low`	Informational	Minor quality issues
`warning`	Needs attention	Moderate issues
`high`	Important	Significant problems
`critical`	Immediate action	Safety issues, NSFW

Alert Lifecycle

Detection     → Alert created with status: "open"
Accumulation  → Same alert type increments occurrence count
Acknowledgment → User acknowledges, status: "acknowledged"
Resolution    → Issue resolved, status: "resolved"

Email and SMS notifications are configured in the Foil dashboard.

Rate Limiting

To prevent alert fatigue, Foil implements rate limiting:

Cooldown period: Minimum time between notifications for the same alert type
Default: 5 minutes
Configurable: Per alert type in agent settings

Best Practices

Start with conservative thresholds

Begin with higher confidence thresholds (0.8+) and lower as you understand your baseline.

Use cooldowns appropriately

Set longer cooldowns for non-critical alerts to reduce noise.

Prioritize critical alerts

Only use ‘critical’ severity for safety issues that need immediate attention.

Configure per-agent

Different agents may need different thresholds. A creative writing agent might tolerate more “hallucination” than a factual Q&A bot.

Getting Started

SDKs

Concepts

Features

Alerting

Alerting

Alert Types

Threshold-Based Alerts

LLM-Analyzed Alerts

How It Works

Per-Agent Configuration

Configuration Options

Alert Severity

Alert Lifecycle

Rate Limiting

Best Practices

Next Steps

Analytics

Agents

​Alerting

​Alert Types

​Threshold-Based Alerts

​LLM-Analyzed Alerts

​How It Works

​Per-Agent Configuration

​Configuration Options

​Alert Severity

​Alert Lifecycle

​Rate Limiting

​Best Practices

​Next Steps

Analytics

Agents

Alerting

Alert Types

Threshold-Based Alerts

LLM-Analyzed Alerts

How It Works

Per-Agent Configuration

Configuration Options

Alert Severity

Alert Lifecycle

Rate Limiting

Best Practices

Next Steps