Building an Autonomous Web Scraping Agent with Firecrawl

Web scraping is one of the most common tasks for AI agents. This tutorial shows how to build a scraping agent that pays per-page via x402 — no Firecrawl API key required.

What We're Building

An autonomous agent that:

Setup

import Mithril from "@mithril/sdk"

const m = new Mithril({
  apiKey: process.env.MITHRIL_API_KEY,
})

Scraping a Single Page

async function scrapePage(url: string) {
  const result = await m.pay({
    url: "https://api.firecrawl.dev/v1/scrape",
    method: "POST",
    body: JSON.stringify({
      url,
      formats: ["markdown", "html"],
      onlyMainContent: true,
    }),
  })
  return result.data
}

Cost: ~$0.01 per page. The x402 payment is handled automatically by the Mithril SDK.

Scraping Multiple Pages

async function scrapeMany(urls: string[]) {
  const results = await Promise.all(
    urls.map(url => scrapePage(url))
  )
  return results
}

For 100 pages, total cost: ~$1.00. No subscription needed.

Extracting Structured Data

Combine Firecrawl scraping with LLM extraction:

async function extractProducts(url: string) {
  const page = await scrapePage(url)

  const extraction = await m.pay({
    url: "https://openrouter.ai/api/v1/chat/completions",
    method: "POST",
    body: JSON.stringify({
      model: "anthropic/claude-sonnet-4-6",
      messages: [{
        role: "user",
        content: `Extract all products from this page as JSON:
        ${page.markdown}`,
      }],
    }),
  })

  return JSON.parse(extraction.data.choices[0].message.content)
}

Crawling an Entire Site

Firecrawl also supports site-wide crawling:

async function crawlSite(url: string, maxPages: number = 50) {
  const result = await m.pay({
    url: "https://api.firecrawl.dev/v1/crawl",
    method: "POST",
    body: JSON.stringify({
      url,
      limit: maxPages,
      scrapeOptions: {
        formats: ["markdown"],
        onlyMainContent: true,
      },
    }),
  })
  return result.data
}

Use Cases

Cost Control

Set daily limits on the scraping agent's wallet:

Monitor via the Mithril dashboard to ensure costs align with expectations.