> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getfoil.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Experiments

> Run A/B tests on your AI prompts and models

# Experiments

Foil's experiment feature lets you run A/B tests on prompts, models, and configurations to find what works best.

## What Can You Test?

* **Prompts** - Compare different system prompts or instructions
* **Models** - Test gpt-4o vs gpt-4o-mini vs claude
* **Parameters** - Temperature, max tokens, etc.
* **Tools** - Different tool configurations
* **Full workflows** - Compare entire agent implementations

## Creating an Experiment

1. Go to **Experiments** in the dashboard
2. Click **Create Experiment**
3. Configure variants (name, traffic weight, configuration)
4. Set metrics to track (signal names)
5. Start the experiment

## Using Variants in Your Code

```javascript theme={null}
const { Foil } = require('@getfoil/foil-js');
const OpenAI = require('openai');

const openai = new OpenAI();
const foil = new Foil({
  apiKey: process.env.FOIL_API_KEY,
  agentName: 'customer-support'
});

async function handleQuery(query, userId) {
  // Get experiment assignment
  const assignment = await foil.getExperimentVariant('prompt-test-v2', userId);

  return await foil.trace(async (ctx) => {
    const response = await openai.chat.completions.create({
      model: 'gpt-4o',
      messages: [
        { role: 'system', content: assignment.config.systemPrompt },
        { role: 'user', content: query }
      ]
    });

    return response.choices[0].message.content;
  }, {
    properties: {
      experimentId: assignment.experimentId,
      variant: assignment.variant
    }
  });
}
```

## Experiment Configuration

| Field               | Type   | Description                    |
| ------------------- | ------ | ------------------------------ |
| `name`              | string | Experiment identifier          |
| `description`       | string | What you're testing            |
| `variants`          | array  | List of variants to test       |
| `variants[].name`   | string | Variant name                   |
| `variants[].weight` | number | Traffic percentage (0-100)     |
| `variants[].config` | object | Variant-specific configuration |
| `metrics`           | array  | Signal names to track          |
| `minimumSampleSize` | number | Required samples per variant   |
| `maximumDuration`   | number | Auto-stop after N days         |

## Viewing Results

The experiment results page in the dashboard shows:

* Variant performance comparison
* Statistical significance (p-value)
* Metric breakdowns
* Traffic distribution

## Best Practices

<AccordionGroup>
  <Accordion title="Test one variable at a time">
    Isolate what you're testing. If you change both prompt AND model, you won't know which caused the difference.
  </Accordion>

  <Accordion title="Use consistent identifiers">
    Use user IDs or session IDs for assignment to ensure users get the same variant consistently.
  </Accordion>

  <Accordion title="Wait for statistical significance">
    Don't end experiments early. Wait until you have enough samples and the p-value is meaningful (typically \< 0.05).
  </Accordion>

  <Accordion title="Track multiple metrics">
    A variant might improve one metric while hurting another. Track quality, user satisfaction, and business metrics.
  </Accordion>
</AccordionGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="Signals" icon="signal" href="/concepts/signals">
    Track experiment metrics
  </Card>

  <Card title="Analytics" icon="chart-line" href="/features/analytics">
    Analyze results
  </Card>
</CardGroup>