> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getfoil.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Multimodal & Media

> Upload files and mix text with media in your AI traces

# Multimodal & Media

Foil supports multimodal traces — mix text, documents, spreadsheets, code files, and images in your span inputs and outputs. Upload files via the SDK, and Foil automatically extracts text content for evaluations and analysis.

<Tip>
  Check out the [multimodal examples](https://github.com/getfoil/foil-examples) for complete, runnable code in both JavaScript and Python.
</Tip>

## How It Works

```
Your App → SDK uploadMedia() → Foil API → S3 (original file)
                                    ↓
                                   SQS
                                    ↓
                            Ingestion Service
                                    ↓
                          ┌─────────────────────┐
                          │  Category Detection  │
                          │  & Text Extraction   │
                          └──────────┬──────────┘
                                     ↓
                          ┌─────────────────────┐
                          │  Evaluations use     │
                          │  extracted content   │
                          └─────────────────────┘
```

1. You upload a file via the SDK — it's stored in S3 and categorized automatically
2. The ingestion service extracts text content (for documents, spreadsheets, code)
3. You reference the uploaded media in span inputs/outputs using content blocks
4. Evaluations automatically include extracted text when analyzing spans

## Supported Media Categories

| Category        | File Types                | Max Size | Processing                        |
| --------------- | ------------------------- | -------- | --------------------------------- |
| **Document**    | PDF, DOCX, DOC, RTF       | 50 MB    | Text extraction, metadata         |
| **Spreadsheet** | CSV, TSV, XLSX, XLS, ODS  | 25 MB    | Text + structured JSON extraction |
| **Code**        | Any text/code file        | 10 MB    | Direct text passthrough           |
| **Image**       | PNG, JPEG, GIF, WebP, SVG | 20 MB    | Dimensions & metadata             |
| **Audio**       | MP3, WAV, OGG, FLAC       | 100 MB   | Coming soon                       |
| **Video**       | MP4, WebM, MOV            | 500 MB   | Coming soon                       |
| **Archive**     | ZIP, TAR, GZ              | 100 MB   | Coming soon                       |
| **Notebook**    | .ipynb                    | 10 MB    | Coming soon                       |
| **Other**       | Any other file            | 25 MB    | Stored as-is                      |

<Note>
  Media category is auto-detected from the file's MIME type. You can associate up to **20 files per span**.
</Note>

## Content Blocks

Content blocks let you mix text and media references in span inputs and outputs:

* **Text blocks** — plain text content (`{ type: 'text', text: '...' }`)
* **Media blocks** — references to uploaded media (`{ type: 'media', mediaId: '...' }`)

<Tabs>
  <Tab title="JavaScript">
    ```javascript theme={null}
    const { ContentBlock, content } = require('@getfoil/foil-js');

    // Mix text and media with the content() helper
    const blocks = content(
      'Analyze this document:',
      ContentBlock.media('media-abc-123'),
      'Summarize the key findings.'
    );
    // => [
    //   { type: 'text', text: 'Analyze this document:' },
    //   { type: 'media', mediaId: 'media-abc-123' },
    //   { type: 'text', text: 'Summarize the key findings.' }
    // ]
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    from foil import ContentBlock, content

    blocks = content(
        "Analyze this document:",
        ContentBlock.media("media-abc-123"),
        "Summarize the key findings."
    )
    ```
  </Tab>
</Tabs>

### Auto-Upload with `ContentBlock.file()` (JavaScript)

The JavaScript SDK supports `ContentBlock.file()` which automatically uploads files before the span is sent — no manual `uploadMedia()` call needed:

```javascript theme={null}
const { ContentBlock, content } = require('@getfoil/foil-js');

const blocks = content(
  'Process this CSV:',
  ContentBlock.file('/path/to/data.csv'),
  'Find the top trends.'
);

// When used in a span, the file is uploaded automatically
const span = await ctx.startSpan(SpanKind.LLM, 'gpt-4o', {
  input: blocks
});
```

`ContentBlock.file()` accepts a file path, `Buffer`, or `ReadStream`:

```javascript theme={null}
ContentBlock.file('/path/to/file.pdf')
ContentBlock.file(Buffer.from(data), { filename: 'data.csv' })
ContentBlock.file(fs.createReadStream('/path/to/file.png'))
```

## Uploading Media

Upload files directly with `uploadMedia()` for more control over the upload lifecycle.

<Tabs>
  <Tab title="JavaScript">
    ```javascript theme={null}
    const { Foil } = require('@getfoil/foil-js');

    const foil = new Foil({ apiKey: 'sk_live_...' });

    // Upload from file path
    const result = await foil.uploadMedia('/path/to/report.pdf');

    // Upload from Buffer
    const result = await foil.uploadMedia(Buffer.from(csvData), {
      filename: 'sales-data.csv',
      mimeType: 'text/csv'
    });

    // Associate with a span at upload time
    const result = await foil.uploadMedia('/path/to/file.pdf', {
      spanId: 'span-123',
      traceId: 'trace-456',
      direction: 'input'
    });
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    from foil import Foil

    foil = Foil(api_key="sk_live_...")

    # Upload from file path
    result = foil.upload_media("/path/to/report.pdf")

    # Upload from bytes
    result = foil.upload_media(csv_bytes, filename="sales-data.csv", mime_type="text/csv")

    # Associate with a span at upload time
    result = foil.upload_media("/path/to/file.pdf",
        span_id="span-123",
        trace_id="trace-456",
        direction="input"
    )
    ```
  </Tab>
</Tabs>

### Upload Options

| Option                   | Type   | Description                                   |
| ------------------------ | ------ | --------------------------------------------- |
| `filename`               | string | Override filename (required for Buffer/bytes) |
| `mimeType` / `mime_type` | string | Override MIME type (auto-detected if omitted) |
| `spanId` / `span_id`     | string | Associate media with a span                   |
| `traceId` / `trace_id`   | string | Associate media with a trace                  |
| `direction`              | string | `'input'` or `'output'`                       |

## Using Media in Spans

After uploading, reference media in span inputs and outputs using content blocks.

<Tabs>
  <Tab title="JavaScript">
    ```javascript theme={null}
    const { Foil, ContentBlock, content } = require('@getfoil/foil-js');

    const foil = new Foil({
      apiKey: 'sk_live_...',
      agentName: 'document-analyzer'
    });

    await foil.trace(async (ctx) => {
      const upload = await foil.uploadMedia('/path/to/report.pdf');

      const response = await ctx.llmCall('gpt-4o', async () => {
        return await openai.chat.completions.create({
          model: 'gpt-4o',
          messages: [{ role: 'user', content: 'Summarize this document' }]
        });
      }, {
        input: content(
          'Summarize this document:',
          ContentBlock.media(upload.mediaId, {
            category: upload.category,
            filename: upload.filename
          })
        )
      });
    });
    ```

    Or use `ContentBlock.file()` to skip manual upload:

    ```javascript theme={null}
    await foil.trace(async (ctx) => {
      const response = await ctx.llmCall('gpt-4o', async () => {
        return await openai.chat.completions.create({
          model: 'gpt-4o',
          messages: [{ role: 'user', content: 'Summarize this document' }]
        });
      }, {
        input: content(
          'Summarize this document:',
          ContentBlock.file('/path/to/report.pdf')
        )
      });
    });
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    from foil import Foil, ContentBlock, content

    foil = Foil(api_key="sk_live_...")

    upload = foil.upload_media("/path/to/report.pdf")

    foil.start_span({
        "spanId": span_id,
        "name": "gpt-4o",
        "agentName": "document-analyzer",
        "input": content(
            "Summarize this document:",
            ContentBlock.media(upload["mediaId"],
                category=upload["category"],
                filename=upload["filename"]
            )
        )
    })
    ```
  </Tab>
</Tabs>

## Retrieving Media

```javascript theme={null}
const media = await foil.getMedia('media-abc-123');
console.log(media.category);            // 'document'
console.log(media.processing.status);   // 'completed'

// Include extracted content
const media = await foil.getMedia('media-abc-123', {
  content: 'extracted'
});
console.log(media.extracted.text.preview); // First 1000 chars
```

When evaluations run on spans containing media content blocks, Foil automatically includes extracted text (up to 15,000 characters per file) in the analysis. No manual configuration needed.

## Limitations

<Warning>
  Current limitations of multimodal support:

  * **No vision model analysis** for images — dimensions are extracted but image content is not analyzed by a vision model
  * **No audio/video transcription** — audio and video files are stored but not processed yet
  * **No archive or notebook processing** — ZIP files and Jupyter notebooks are stored as-is
  * **20 files per span** maximum
  * **Files over 100 MB** require presigned upload (contact support)
  * **Extracted text is capped** at 100 KB per file; evaluation prompts use up to 15,000 characters per file
</Warning>

## Best Practices

<AccordionGroup>
  <Accordion title="Upload media before creating spans">
    Upload files first and use the returned `mediaId` in your content blocks. This ensures media is available when the span is processed. Alternatively, use `ContentBlock.file()` (JavaScript) to handle this automatically.
  </Accordion>

  <Accordion title="Use content blocks for structured input/output">
    Instead of embedding file contents as plain text, use content blocks. This enables Foil to track media associations, provide download links, and include extracted content in evaluations.
  </Accordion>

  <Accordion title="Check processing status for time-sensitive workflows">
    Text extraction is asynchronous. If you need extracted content immediately, poll `getMedia()` until `processing.status` is `'completed'`.
  </Accordion>

  <Accordion title="Prefer file paths over Buffers">
    When possible, pass file paths to `uploadMedia()` or `ContentBlock.file()`. This lets the SDK auto-detect the filename and MIME type. When using Buffer/bytes, always provide a `filename`.
  </Accordion>
</AccordionGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="Traces & Spans" icon="route" href="/concepts/traces-and-spans">
    Learn about traces and span hierarchy
  </Card>

  <Card title="Alerting" icon="bell" href="/features/alerting">
    Get notified on quality issues
  </Card>
</CardGroup>