Skip to main content

​
Signals

Signals are custom metrics and feedback tied to your traces. Use them to track user satisfaction, quality scores, and any custom metrics relevant to your AI application.

​
What Are Signals?

Signals let you record:
  • User feedback - Thumbs up/down, star ratings
  • Quality metrics - Relevance, accuracy, helpfulness scores
  • Custom metrics - Any numeric, boolean, or categorical value
Signals are linked to traces, allowing you to correlate feedback with specific AI interactions.

​
Signal Types

TypeDescriptionExample Values
feedbackDirect user feedbackthumbs up, thumbs down
ratingNumeric ratings1-5 stars, 1-10 score
sentimentSentiment analysispositive, negative, neutral
qualityQuality assessment0.0-1.0 score
completionGoal completiontrue/false
customAny custom metricvaries

​
Recording Signals

​
JavaScript SDK

await tracer.trace(async (ctx) => {
  // Do AI work...

  // Record user feedback
  await ctx.recordFeedback(true); // thumbs up

  // Record star rating
  await ctx.recordRating(4); // 4 out of 5 stars

  // Record custom signal
  await ctx.recordSignal('response_quality', 0.85, {
    signalType: 'quality',
    source: 'llm',
    confidence: 0.92
  });
});

​
Low-Level API

import { Foil } from '@foil-ai/sdk';

const foil = new Foil({ apiKey: process.env.FOIL_API_KEY });

// Record signal directly
await foil.recordSignal({
  signalName: 'user_satisfaction',
  value: 4,
  traceId: 'trace-123',
  spanId: 'span-456',
  agentName: 'customer-support',
  signalType: 'rating',
  source: 'user'
});

​
Batch Recording

Record multiple signals at once: 
await foil.recordSignalBatch([
  {
    signalName: 'relevance',
    value: 0.9,
    traceId: 'trace-123',
    signalType: 'quality'
  },
  {
    signalName: 'helpfulness',
    value: 0.85,
    traceId: 'trace-123',
    signalType: 'quality'
  },
  {
    signalName: 'goal_completed',
    value: true,
    traceId: 'trace-123',
    signalType: 'completion'
  }
]);

​
Signal Properties

PropertyTypeRequiredDescription
signalNamestringYesName of the signal
valueanyYesSignal value
traceIdstringNoLinks to a trace
spanIdstringNoLinks to specific span
agentNamestringNoAgent that generated signal
signalTypestringNoType categorization
sourcestringNo’user’, ‘system’, ‘llm’
confidencenumberNoConfidence score (0-1)
reasoningstringNoExplanation for the value
metadataobjectNoAdditional context

​
Common Use Cases

​
User Feedback

// Thumbs up/down
await ctx.recordFeedback(userClickedThumbsUp);

// Star rating
await ctx.recordRating(userStarRating);

// Detailed feedback
await ctx.recordSignal('user_feedback', {
  rating: 4,
  comment: 'Helpful but could be more concise'
}, {
  signalType: 'feedback',
  source: 'user'
});

​
LLM-Evaluated Quality

Use an LLM to evaluate response quality: 
// Evaluate response with separate LLM call
const evaluation = await evaluateLLM.chat.completions.create({
  model: 'gpt-4o-mini',
  messages: [{
    role: 'system',
    content: 'Rate the response relevance from 0-1'
  }, {
    role: 'user',
    content: `Query: ${query}\nResponse: ${response}`
  }]
});

await ctx.recordSignal('relevance', parseFloat(evaluation), {
  signalType: 'quality',
  source: 'llm',
  confidence: 0.95,
  modelUsed: 'gpt-4o-mini'
});

​
Goal Completion Tracking

// Track if the AI completed the user's goal
await ctx.recordSignal('goal_completed', userGoalAchieved, {
  signalType: 'completion',
  source: 'system',
  metadata: {
    goalType: 'purchase',
    attemptNumber: 1
  }
});

​
A/B Test Metrics

await ctx.recordSignal('conversion', userConverted, {
  signalType: 'custom',
  source: 'system',
  metadata: {
    experimentId: 'prompt-v2-test',
    variant: 'treatment'
  }
});

​
Retrieving Signals

​
Get Signals for a Trace

const signals = await foil.getTraceSignals(traceId);

for (const signal of signals) {
  console.log(`${signal.signalName}: ${signal.value}`);
}

​
REST API

GET /api/signals?traceId=trace-123

​
Signal Sources

Track where signals originate:
SourceDescription
userDirect user feedback
systemAutomated system metrics
llmLLM-based evaluation
// User-provided feedback
await ctx.recordSignal('rating', 5, { source: 'user' });

// System-computed metric
await ctx.recordSignal('latency_bucket', 'fast', { source: 'system' });

// LLM-evaluated quality
await ctx.recordSignal('coherence', 0.9, {
  source: 'llm',
  confidence: 0.88,
  modelUsed: 'gpt-4o-mini'
});

​
Best Practices

Standardize names across your application for better aggregation:
  • user_rating not rating, stars, score
  • response_quality not quality, resp_qual
When using LLMs to evaluate, always include a confidence score.
Use recordSignalBatch when recording multiple signals to reduce API calls.

​
Next Steps

Agents

Organize by agent

Analytics

Analyze signal trends