Skip to main content

​
Alerting

Foil’s alerting system detects AI quality issues in real-time and notifies you via email or SMS.

​
Alert Types

Foil detects two categories of issues:

​
Threshold-Based Alerts

Triggered when metrics exceed configured limits:
Alert TypeDescriptionDefault Threshold
errorRequest failed with errorAny error
highDurationResponse took too long5000ms
highInputTokensToo many input tokens100,000
highOutputTokensToo many output tokens50,000
highCostSingle call too expensive$1.00
timeoutRequest timed out30s

​
LLM-Analyzed Alerts

Foil uses AI to detect quality issues:
Alert TypeDescription
hallucinationOutput contains fabricated facts
nsfwOutput contains inappropriate content
stuckAgent is repeating itself or looping
qualityOutput is off-topic or low quality

​
How It Works

User Request → Your Agent → Foil Ingestion
                                   ↓
                          ┌─────────────────┐
                          │  Inline Checks  │
                          │  (thresholds)   │
                          └────────┬────────┘
                                   ↓
                          ┌─────────────────┐
                          │  LLM Analysis   │
                          │  (quality)      │
                          └────────┬────────┘
                                   ↓
                          ┌─────────────────┐
                          │  Rate Limiter   │
                          └────────┬────────┘
                                   ↓
                          ┌─────────────────┐
                          │  Notifications  │
                          │  (email/SMS)    │
                          └─────────────────┘

​
Configuring Alerts

​
Per-Agent Configuration

Configure alerts in the dashboard or via API: 
PUT /api/agents/:agentId/alerts
{
  "llmAnalysis": {
    "enabled": true,
    "alertTypes": {
      "hallucination": {
        "enabled": true,
        "threshold": 0.7,
        "severity": "high",
        "channels": ["email"],
        "cooldownMinutes": 5
      },
      "nsfw": {
        "enabled": true,
        "threshold": 0.8,
        "severity": "critical",
        "channels": ["email", "sms"]
      },
      "stuck": {
        "enabled": true,
        "threshold": 0.6,
        "severity": "warning"
      },
      "quality": {
        "enabled": true,
        "threshold": 0.5,
        "severity": "warning"
      }
    }
  },
  "thresholds": {
    "duration": 5000,
    "inputTokens": 100000,
    "outputTokens": 50000,
    "cost": 1.0
  },
  "contacts": {
    "email": [
      { "address": "[email protected]", "enabled": true }
    ],
    "sms": [
      { "phoneNumber": "+1234567890", "enabled": true }
    ]
  }
}

​
Configuration Options

OptionTypeDescription
enabledbooleanEnable/disable the alert type
thresholdnumberConfidence threshold (0-1) for LLM alerts
severitystring’low’, ‘warning’, ‘high’, ‘critical’
channelsarrayNotification channels (‘email’, ‘sms’)
cooldownMinutesnumberMinimum time between alerts

​
Alert Severity

SeverityDescriptionUse Case
lowInformationalMinor quality issues
warningNeeds attentionModerate issues
highImportantSignificant problems
criticalImmediate actionSafety issues, NSFW

​
Alert Lifecycle

Alerts follow an incident-based model: 
1. Detection
   └── Alert created with status: "open"

2. Accumulation
   └── Same alert type increments occurrence count

3. Acknowledgment
   └── User acknowledges, status: "acknowledged"

4. Resolution
   └── Issue resolved, status: "resolved"

​
Acknowledging Alerts

PUT /api/spans/alerts/:alertId/acknowledge
{
  "resolution": "acknowledged"
}

​
Reopening Alerts

PUT /api/spans/alerts/:alertId/reopen

​
Notification Channels

​
Email

Requires SendGrid configuration: 
# foil-ingestion/.env
SENDGRID_API_KEY=SG.xxx
SENDGRID_FROM_EMAIL=[email protected]
SENDGRID_ALERT_TEMPLATE_ID=d-xxx  # Optional: for styled emails

​
SMS

Requires Twilio configuration: 
# foil-ingestion/.env
TWILIO_ACCOUNT_SID=xxx
TWILIO_AUTH_TOKEN=xxx
TWILIO_FROM_NUMBER=+1234567890

​
Rate Limiting

To prevent alert fatigue, Foil implements rate limiting:
  • Cooldown period: Minimum time between notifications for the same alert type
  • Default: 5 minutes
  • Configurable: Per alert type in agent settings
{
  "hallucination": {
    "cooldownMinutes": 10  // Wait 10 minutes between notifications
  }
}

​
Viewing Alerts

​
Dashboard

The Alerts page shows:
  • Active alerts by agent
  • Alert history
  • Occurrence details
  • Resolution status

​
API

# List all alerts
GET /api/spans/alerts

# Get alerts summary by agent
GET /api/spans/alerts/summary

# Get specific alert with occurrences
GET /api/spans/alerts/:alertId

# Get alerts for a specific trace
GET /api/spans/traces/:traceId/alerts

​
Testing Alerts

Send a test alert to verify configuration: 
POST /v1/alerts/test
{
  "channel": "email",
  "alertType": "hallucination"
}

​
Best Practices

Begin with higher confidence thresholds (0.8+) and lower as you understand your baseline.
Set longer cooldowns for non-critical alerts to reduce noise.
Only use ‘critical’ severity for safety issues that need immediate attention.
Different agents may need different thresholds. A creative writing agent might tolerate more “hallucination” than a factual Q&A bot.

​
Next Steps

Analytics

View alert trends

Agents

Configure per-agent settings