Self-Hosting Firecrawl with Fire-engine and LLM-Backed Extraction

The default self-hosted Firecrawl stack gives you scrape, crawl, map, and search over two engines: plain HTTP fetch and a bundled Playwright service. That covers a surprising amount of ground, and nothing else in this guide matters until that baseline returns Markdown.

What the default stack does not give you is the two things most teams actually want in production:

• LLM-backed formats — the json format on /v2/scrape and the /v2/extract route. These need a model provider. No provider, no structured extraction.
• Fire-engine — Firecrawl's anti-bot and browser-automation layer. Screenshots and page actions both route through it, and both report no support on fetch and Playwright. Fire-engine is proprietary and is not shipped in the open-source repository, so "installing it" means pointing your API at an endpoint you have access to, not pulling an image from a public registry.

Read that second point twice before you plan a migration. The self-hosted API contains the client for Fire-engine; it does not contain Fire-engine. If you have no endpoint, the honest options are Firecrawl Cloud, a commercial proxy/unblocker behind Playwright, or a different tool. This guide covers wiring the endpoint in when you do have one, and the proxy fallback when you don't.

This walkthrough pins Firecrawl v2.11.162. A different release can change the Compose contract, so treat the tag as part of the configuration.

What you get, and what you have to bring

Three separate systems, three separate verification steps. Do not bring them up at once — you will not know which one broke.

Step 0 — Host sizing and prerequisites

Firecrawl does not publish a verified minimum host size. In practice, Playwright is the memory hog, and 4 GB collapses under sustained crawl concurrency. Plan for 8 GB and 4 vCPU if you intend to do real work; add more if you also run a local model.

Install the toolchain:

# Debian / Ubuntu
sudo apt-get update
sudo apt-get install -y git curl ca-certificates

# Docker Engine + Compose v2 (convenience script)
curl -fsSL https://get.docker.com | sudo sh
sudo usermod -aG docker "$USER"
newgrp docker

Verify the versions and that nothing already owns port 3002:

docker --version
docker compose version          # must be v2.x, invoked as "docker compose"
git --version
curl --version

ss -ltnp | grep -E ':3002|:11434' || echo "3002 and 11434 are free"

nproc and free -h are worth a glance too. If Docker has 2 GB allocated on a Mac or Windows host, the build will fail in a way that looks like a code problem and is not.

nproc
free -h
df -h /var/lib/docker

Step 1 — Clone the pinned release

git clone https://github.com/firecrawl/firecrawl.git
cd firecrawl
git checkout v2.11.162

Confirm you are actually on the tag, not on main:

git describe --tags --exact-match HEAD

Before you change anything, read the Compose contract for this revision. This is the single most useful five minutes in the whole process, because it tells you exactly which environment variables the API container will actually receive:

sed -n '1,60p' docker-compose.yaml

At v2.11.162 the shared x-common-env block passes through REDIS_URL, PLAYWRIGHT_MICROSERVICE_URL, the POSTGRES_* set, USE_DB_AUTHENTICATION, NUM_WORKERS_PER_QUEUE, CRAWL_CONCURRENT_REQUESTS, MAX_CONCURRENT_JOBS, BROWSER_POOL_SIZE, the proxy and SearXNG variables, and — importantly for us — OPENAI_API_KEY, OPENAI_BASE_URL, MODEL_NAME, MODEL_EMBEDDING_NAME, and OLLAMA_BASE_URL.

Note what is not in that list: any Fire-engine variable. That is why Step 5 needs a Compose override rather than just an .env line.

One more thing that trips people up: apps/api/.env.example is a development file for running the API on the host. It is not a drop-in Compose contract. Read it for reference; do not copy it to the repository root and expect Compose to honour every line.

Step 2 — Minimal working .env

Start boring. Authentication off, Postgres queue, no providers:

cat > .env <<'EOF'
USE_DB_AUTHENTICATION=false
POSTGRES_USER=postgres
POSTGRES_PASSWORD=replace-with-at-least-32-random-characters
POSTGRES_DB=postgres
EOF

Generate a real password rather than typing one:

sed -i "s|replace-with-at-least-32-random-characters|$(openssl rand -hex 24)|" .env
grep POSTGRES_PASSWORD .env

Two constraints worth respecting:

• Keep POSTGRES_DB=postgres. At this revision the bundled pg_cron configuration targets that database name. Renaming it produces initialisation failures that read like permission errors.
• Leave NUQ_BACKEND and BULL_AUTH_KEY unset. That keeps you on the Postgres queue with the admin UI off — fewer moving parts for the first scrape. FoundationDB is an intentional choice, not a default.

And make sure it never reaches your git history:

grep -qxF '.env' .gitignore || echo '.env' >> .gitignore

USE_DB_AUTHENTICATION=false means the API accepts unauthenticated requests. That is fine on a trusted network for an evaluation, and unacceptable on anything reachable from outside. Step 7 fixes it.

Step 3 — Build and start

docker compose up --build -d

The first build compiles the API and the Playwright service from source and will take a while. Watch the service table until the long-running services are up and the one-shot initialisers have exited cleanly:

docker compose ps --all

Warnings about unset optional variables are expected here. Follow the logs if something looks stuck:

docker compose logs -f --tail=100 api

Step 4 — Verify the baseline before adding anything

4a. Reachability

curl --fail --silent --show-error --max-time 5 \
  http://localhost:3002/v0/health/readiness

Expected:

{"status":"ok"}

This is a heartbeat and nothing more. It does not touch Redis, Postgres, RabbitMQ, Playwright, the workers, or outbound network access. Do not treat a green readiness check as a working deployment.

4b. A real scrape

This is the test that matters. Note that the request timeout is in milliseconds while curl's --max-time is in seconds — keep curl's window longer so the API gets a chance to return its own timeout response instead of you killing the connection:

curl --fail-with-body --silent --show-error --max-time 75 \
  -X POST http://localhost:3002/v2/scrape \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown"],
    "timeout": 60000
  }'

A healthy response has this shape:

{
  "success": true,
  "data": {
    "markdown": "...",
    "metadata": { "statusCode": 200 }
  }
}

That single call exercises the API, the scraping pipeline, one engine path, and outbound access together. If it succeeds, Firecrawl works end to end on your infrastructure. Snapshot this state mentally — everything from here on is additive, and if a later step breaks, this is what you roll back to.

Worth confirming that crawl and map also work before you move on:

curl -s -X POST http://localhost:3002/v2/map \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com"}' | head -c 400

Step 5 — LLM-backed extraction

The json format on /v2/scrape and the /v2/extract route both need a model provider. The Compose file already forwards the relevant variables, so this step is genuinely just .env plus a restart. Pick one of three paths.

Path A — OpenAI

cat >> .env <<'EOF'

# --- LLM provider: OpenAI ---
OPENAI_API_KEY=sk-your-key-here
MODEL_NAME=gpt-4o-mini
MODEL_EMBEDDING_NAME=text-embedding-3-small
EOF

Path B — Any OpenAI-compatible endpoint (vLLM, LiteLLM, OpenRouter, TGI)

This is the path I'd default to for self-hosting, because it keeps one client contract and lets you swap the model behind it. Set the base URL to the /v1 root — not to /v1/chat/completions:

cat >> .env <<'EOF'

# --- LLM provider: OpenAI-compatible gateway ---
OPENAI_BASE_URL=http://host.docker.internal:4000/v1
OPENAI_API_KEY=sk-anything-your-gateway-accepts
MODEL_NAME=qwen2.5-32b-instruct
MODEL_EMBEDDING_NAME=bge-m3
EOF

The Compose services set extra_hosts: host.docker.internal:host-gateway, so host.docker.internal resolves to your host from inside the containers. If your gateway runs as another Compose service on the backend network, use its service name instead.

Most gateways still require some value in OPENAI_API_KEY, even a dummy one — an empty key often fails validation before any request is made.

Path C — Ollama (local models, experimental)

# On the host
curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5:14b-instruct
ollama pull nomic-embed-text

# Bind Ollama to all interfaces so containers can reach it
sudo systemctl edit --force ollama   # add: Environment="OLLAMA_HOST=0.0.0.0:11434"
sudo systemctl restart ollama
curl -s http://localhost:11434/api/tags | head -c 200

cat >> .env <<'EOF'

# --- LLM provider: Ollama (experimental) ---
OLLAMA_BASE_URL=http://host.docker.internal:11434/api
MODEL_NAME=qwen2.5:14b-instruct
MODEL_EMBEDDING_NAME=nomic-embed-text
EOF

Note the /api suffix — Ollama's native path, not the /v1 compatibility shim. If you'd rather use the shim, treat it as Path B with OPENAI_BASE_URL=http://host.docker.internal:11434/v1.

Two practical warnings. Small quantised models fail schema-constrained extraction far more often than their benchmark scores suggest; if json output comes back malformed, try a larger model before you debug Firecrawl. And a local model on the same 8 GB host as Playwright will contend for RAM — give the model its own machine or accept the OOM risk.

Apply and test the LLM path in isolation

docker compose up -d
docker compose ps --all

Now test only extraction, with a schema simple enough that a failure means plumbing rather than model capability:

curl --fail-with-body --silent --show-error --max-time 120 \
  -X POST http://localhost:3002/v2/scrape \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "formats": [{
      "type": "json",
      "schema": {
        "type": "object",
        "properties": {
          "page_title":       { "type": "string" },
          "primary_link_url": { "type": "string" }
        },
        "required": ["page_title"]
      }
    }],
    "timeout": 90000
  }'

You want a populated data.json object. If you get a scrape success with an empty or absent json field, the provider is not being reached — check that the variables landed inside the container rather than only in your shell:

docker compose exec api env | grep -E 'OPENAI|OLLAMA|MODEL_' || echo "no provider vars in container"
docker compose logs --tail=200 api | grep -iE 'openai|ollama|model|extract'

env | grep returning nothing while .env clearly has the values almost always means the variable isn't referenced in docker-compose.yaml for this revision, or the containers were never recreated. docker compose up -d recreates on env change; docker compose restart does not.

Then the dedicated route:

curl --fail-with-body --silent --show-error --max-time 180 \
  -X POST http://localhost:3002/v2/extract \
  -H 'Content-Type: application/json' \
  -d '{
    "urls": ["https://example.com"],
    "prompt": "Return the page title and a one-sentence summary.",
    "schema": {
      "type": "object",
      "properties": {
        "title":   { "type": "string" },
        "summary": { "type": "string" }
      }
    }
  }'

/v2/extract is asynchronous in most releases — expect a job id and poll it:

JOB=$(curl -s -X POST http://localhost:3002/v2/extract \
  -H 'Content-Type: application/json' \
  -d '{"urls":["https://example.com"],"prompt":"Title and summary."}' \
  | python3 -c 'import sys,json; print(json.load(sys.stdin).get("id",""))')

echo "job: $JOB"
curl -s "http://localhost:3002/v2/extract/$JOB" | head -c 600

Extraction runs in the extract-worker role, which needs RabbitMQ. If extract jobs queue forever while plain scrapes work, look there first:

docker compose logs --tail=200 rabbitmq
docker compose ps rabbitmq

Step 6 — Attaching Fire-engine

Now the honest part.

Fire-engine is Firecrawl's proprietary scraping engine — IP-block handling, bot-detection evasion, and the browser layer that screenshots and page actions depend on. It is not in the AGPL-3.0 repository and there is no public image to pull. The open-source API ships a client that will call a Fire-engine endpoint if you configure one; obtaining that endpoint is a commercial or beta-access question, not an installation step.

So there are exactly three real situations:

1. You have a Fire-engine endpoint (Firecrawl beta access, an enterprise arrangement, or a compatible internal service). Wire it in as below.
2. You don't, but you need block resistance. Use the proxy path in Step 6c. It won't match Fire-engine, but it's the honest open-source ceiling.
3. You don't, and you need screenshots or actions. Those route through Fire-engine. Use Firecrawl Cloud for those calls, or drive Playwright directly outside Firecrawl.

Anyone selling you a docker run fire-engine command is selling you something else.

6a. The configuration hook

The API reads a Fire-engine base URL from the environment. Historically and at the time of writing this is:

FIRE_ENGINE_BETA_URL=https://your-fire-engine-endpoint.example.com

Verify the exact variable name against the revision you checked out rather than trusting any blog post, including this one:

grep -rn "FIRE_ENGINE" apps/api/src --include='*.ts' | head -20
grep -rn "fire-engine\|fireEngine" apps/api/src/scraper --include='*.ts' | head -20

That grep tells you the variable name, whether an auth header or key is also read, and which engines the URL activates. If the code at your revision expects something different from the name above, the code wins.

6b. Pass it through Compose

Here's the gotcha: at v2.11.162 the x-common-env block does not include any Fire-engine variable. Adding it to .env alone does nothing — Compose only forwards variables it explicitly references. Use an override file so you don't have to patch the tracked Compose file (and so upgrades stay clean):

cat > docker-compose.override.yaml <<'EOF'
services:
  api:
    environment:
      FIRE_ENGINE_BETA_URL: ${FIRE_ENGINE_BETA_URL}
EOF

If your grep in 6a turned up additional variables — an API key, a separate proxy or stealth URL — add each one as another line in the same block.

Add the value to .env:

cat >> .env <<'EOF'

# --- Fire-engine (external, not bundled) ---
FIRE_ENGINE_BETA_URL=https://your-fire-engine-endpoint.example.com
EOF

Confirm the override merged as you expect before restarting — config prints the fully resolved configuration:

docker compose config | grep -A3 -i fire_engine

Then recreate and verify the variable is inside the container:

docker compose up -d
docker compose exec api env | grep FIRE_ENGINE

Now test the capability that only Fire-engine provides. If the screenshot format returns data, the engine is genuinely wired in:

curl --fail-with-body --silent --show-error --max-time 120 \
  -X POST http://localhost:3002/v2/scrape \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown", "screenshot"],
    "timeout": 90000
  }' | head -c 600

And a page-action sequence, which is the other Fire-engine-only path:

curl --fail-with-body --silent --show-error --max-time 150 \
  -X POST http://localhost:3002/v2/scrape \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown"],
    "actions": [
      { "type": "wait", "milliseconds": 2000 },
      { "type": "screenshot" }
    ],
    "timeout": 120000
  }' | head -c 600

An engine-not-supported error here means the URL isn't reaching a working Fire-engine — check the endpoint's own logs, then:

docker compose logs --tail=200 api | grep -iE 'fire.?engine|engine.*(unsupported|not support|error)'

6c. The open-source fallback: proxy the Playwright service

If Fire-engine isn't available to you, this is what's left. The Playwright service accepts upstream proxy credentials, so a rotating residential or mobile proxy provider gets you meaningful (not equivalent) block resistance:

cat >> .env <<'EOF'

# --- Upstream proxy for Playwright ---
PROXY_SERVER=http://gateway.your-proxy-provider.com:7000
PROXY_USERNAME=your-proxy-user
PROXY_PASSWORD=your-proxy-password
BLOCK_MEDIA=true
EOF

docker compose up -d

BLOCK_MEDIA=true is not cosmetic — it drops image, font, and video requests, which on metered proxy bandwidth is often the difference between viable and absurd. PROXY_SERVER, PROXY_USERNAME, and PROXY_PASSWORD are already referenced by both the API and the Playwright service in the tracked Compose file, so no override is needed.

Confirm your egress IP actually changed:

curl -s -X POST http://localhost:3002/v2/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://api.ipify.org","formats":["markdown"]}' | head -c 300

This still gets you nothing on screenshots or actions. Those are Fire-engine paths regardless of how good your proxy is.

Step 7 — Before you expose this

Compose gets you to first success. It does not get you to production, and no single .env switch closes the gap.

Authentication. The baseline API is unauthenticated and accepts any bearer token. Flipping USE_DB_AUTHENTICATION=true is not a complete design — it expects a provisioned database schema and matching application configuration. The pragmatic pattern is to leave the API unauthenticated on a private network and put a gateway in front of it that owns the only public hostname and checks a key you control:

cat > Caddyfile <<'EOF'
firecrawl.example.com {
  @authorized header Authorization "Bearer {env.FIRECRAWL_API_KEY}"
  handle @authorized {
    reverse_proxy api:3002
  }
  respond 401
}
EOF

Run it on the backend network, publish only the gateway, and stop publishing 3002 to the host at all.

Persistence. The tracked Compose file defines no durable volumes for Postgres, Redis, or RabbitMQ. Replace a container and the state goes with it. If job history matters, add volumes and then test restore — an untested backup is a rumour.

Secrets. Move POSTGRES_PASSWORD and every provider key out of .env into your platform's secret store.

Capacity. The cpus and mem_limit values in the Compose file are a starting point, not verified minimums. Tune NUM_WORKERS_PER_QUEUE, CRAWL_CONCURRENT_REQUESTS, MAX_CONCURRENT_JOBS, and BROWSER_POOL_SIZE together, and watch memory while you do.

Data flow. Scraping sends outbound requests to third-party sites. A hosted model provider adds a second flow, and Fire-engine adds a third. If data residency is a compliance question for you, map all three before enabling them — self-hosting the API doesn't make the other two local.

Licensing. Firecrawl's core is AGPL-3.0. If you deploy a modified version as a network service, the copyleft terms apply to your changes. Worth a conversation with whoever owns that decision before you fork.

Useful operational commands:

docker compose logs --tail=200 api playwright-service
docker compose stats --no-stream 2>/dev/null || docker stats --no-stream
docker compose down            # stop
docker compose down -v         # stop and discard volumes

Troubleshooting

"You're bypassing authentication." Expected with USE_DB_AUTHENTICATION=false. Only a problem if the API is reachable from an untrusted network — in which case stop and do Step 7.

A container exits immediately.

docker compose ps --all
docker compose logs --tail=200

Check the revision matches the configuration, and give Docker more CPU, memory, or disk if a build was killed.

Postgres won't initialise. Check .env syntax, keep POSTGRES_DB=postgres, and make sure the user and password match everywhere they appear.

Redis connection refused inside a container. Keep the Compose service address redis://redis:6379. localhost inside a container points at that container, not at Redis. If you added a REDIS_URL override, remove it.

Port 3002 doesn't answer.

docker compose ps api
docker compose logs --tail=200 api
ss -ltnp | grep 3002

Stop whatever owns the port, or change PORT and INTERNAL_PORT consistently.

Readiness passes but /v2/scrape fails. The heartbeat doesn't validate dependencies. Look at the API and Playwright logs together:

docker compose logs --tail=200 api playwright-service

Scrapes time out. Confirm the host can reach the target, and keep curl's --max-time longer than the request body's timeout so the API can return its own error.

json format returns nothing. Provider variables aren't reaching the container, or the model is too small for the schema. Check docker compose exec api env | grep -E 'OPENAI|OLLAMA|MODEL_' first, model size second.

Screenshots or actions report no support. That's the expected answer without a working Fire-engine endpoint. Re-check docker compose config | grep -i fire_engine and the endpoint's own health.

Verification checklist

Run these in order. Each one isolates a different subsystem, and a failure tells you exactly where to look:

# 1. Reachability
curl -sf --max-time 5 http://localhost:3002/v0/health/readiness

# 2. Core pipeline + outbound
curl -sf --max-time 75 -X POST http://localhost:3002/v2/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","formats":["markdown"],"timeout":60000}' \
  | head -c 200

# 3. LLM provider reachable from the API container
docker compose exec api env | grep -E 'OPENAI|OLLAMA|MODEL_'

# 4. LLM-backed extraction
curl -sf --max-time 120 -X POST http://localhost:3002/v2/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","formats":[{"type":"json","schema":{"type":"object","properties":{"page_title":{"type":"string"}}}}],"timeout":90000}' \
  | head -c 300

# 5. Fire-engine wired through Compose
docker compose config | grep -i fire_engine
docker compose exec api env | grep FIRE_ENGINE

# 6. Fire-engine-only capability
curl -sf --max-time 120 -X POST http://localhost:3002/v2/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","formats":["markdown","screenshot"],"timeout":90000}' \
  | head -c 200

Steps 1 and 2 green means you have a working Firecrawl. Steps 3 and 4 green means structured extraction is live. Steps 5 and 6 green means Fire-engine is genuinely attached rather than merely configured — and until step 6 returns image data, it isn't.

References

• Firecrawl self-hosting guide — https://docs.firecrawl.dev/contributing/self-host
• SELF_HOST.md at the revision you checked out — https://github.com/firecrawl/firecrawl/blob/main/SELF_HOST.md
• Pinned Compose contract — https://github.com/firecrawl/firecrawl/blob/v2.11.162/docker-compose.yaml
• Open Source vs Cloud capability comparison — https://docs.firecrawl.dev/contributing/open-source-or-cloud
• Kubernetes and Helm examples — examples/kubernetes/ in the repository

The Compose file for your exact tag is the authoritative source for which variables exist. When this guide and that file disagree, the file is right.