Skip to main content

​
Alerting

Foil’s alerting system detects AI quality issues in real-time and notifies you via email or SMS.

​
Alert Types

Foil detects two categories of issues:

​
Threshold-Based Alerts

Triggered when metrics exceed configured limits:
Alert TypeDescriptionDefault Threshold
errorRequest failed with errorAny error
highDurationResponse took too long5000ms
highInputTokensToo many input tokens100,000
highOutputTokensToo many output tokens50,000
highCostSingle call too expensive$1.00
timeoutRequest timed out30s

​
LLM-Analyzed Alerts

Foil uses AI to detect quality issues:
Alert TypeDescription
hallucinationOutput contains fabricated facts
nsfwOutput contains inappropriate content
stuckAgent is repeating itself or looping
qualityOutput is off-topic or low quality

​
How It Works

User Request → Your Agent → Foil Ingestion
                                   ↓
                          ┌─────────────────┐
                          │  Inline Checks  │
                          │  (thresholds)   │
                          └────────┬────────┘
                                   ↓
                          ┌─────────────────┐
                          │  LLM Analysis   │
                          │  (quality)      │
                          └────────┬────────┘
                                   ↓
                          ┌─────────────────┐
                          │  Rate Limiter   │
                          └────────┬────────┘
                                   ↓
                          ┌─────────────────┐
                          │  Notifications  │
                          │  (email/SMS)    │
                          └─────────────────┘

​
Per-Agent Configuration

Configure alerts in the dashboard or via API: 
PUT /api/agents/:agentId/alerts
{
  "llmAnalysis": {
    "enabled": true,
    "alertTypes": {
      "hallucination": {
        "enabled": true,
        "threshold": 0.7,
        "severity": "high",
        "channels": ["email"],
        "cooldownMinutes": 5
      },
      "nsfw": {
        "enabled": true,
        "threshold": 0.8,
        "severity": "critical",
        "channels": ["email", "sms"]
      },
      "stuck": { "enabled": true, "threshold": 0.6, "severity": "warning" },
      "quality": { "enabled": true, "threshold": 0.5, "severity": "warning" }
    }
  },
  "thresholds": {
    "duration": 5000,
    "inputTokens": 100000,
    "outputTokens": 50000,
    "cost": 1.0
  },
  "contacts": {
    "email": [{ "address": "alerts@yourcompany.com", "enabled": true }],
    "sms": [{ "phoneNumber": "+1234567890", "enabled": true }]
  }
}

​
Configuration Options

OptionTypeDescription
enabledbooleanEnable/disable the alert type
thresholdnumberConfidence threshold (0-1) for LLM alerts
severitystringlow, warning, high, critical
channelsarrayNotification channels (email, sms)
cooldownMinutesnumberMinimum time between alerts

​
Alert Severity

SeverityDescriptionUse Case
lowInformationalMinor quality issues
warningNeeds attentionModerate issues
highImportantSignificant problems
criticalImmediate actionSafety issues, NSFW

​
Alert Lifecycle

1. Detection     → Alert created with status: "open"
2. Accumulation  → Same alert type increments occurrence count
3. Acknowledgment → User acknowledges, status: "acknowledged"
4. Resolution    → Issue resolved, status: "resolved"
Email and SMS notifications are configured in the Foil dashboard.

​
Rate Limiting

To prevent alert fatigue, Foil implements rate limiting:
  • Cooldown period: Minimum time between notifications for the same alert type
  • Default: 5 minutes
  • Configurable: Per alert type in agent settings

​
Best Practices

Begin with higher confidence thresholds (0.8+) and lower as you understand your baseline.
Set longer cooldowns for non-critical alerts to reduce noise.
Only use ‘critical’ severity for safety issues that need immediate attention.
Different agents may need different thresholds. A creative writing agent might tolerate more “hallucination” than a factual Q&A bot.

​
Next Steps

Analytics

View alert trends

Agents

Configure per-agent settings