HookRelay Documentation

Architecture

HookRelay accepts events from producers via a REST API, buffers them in Redis, and guarantees delivery to endpoints through BullMQ workers. The architecture supports horizontal scaling and automatic recovery.

Data Flow

1. Producers submit events via POST /events with an X-Api-Key header.
2. The API authenticates (Redis-cached tenant lookup, 60s TTL), applies the Lua fixed-window rate limiter, checks idempotency via Redis GET, then pushes the event to the Redis buffer (LPUSH) and immediately returns 202.
3. The Ingest Worker runs a poll loop with BRPOP (blocking pop). When data arrives it uses an atomic Lua script (LRANGE + LTRIM) to pop up to 100 events in a single round-trip, batch-inserts them into PostgreSQL, bulk-enqueues fanout jobs via BullMQ, and sets Redis idempotency keys for successful inserts.
4. The Fanout Worker (concurrency=10) reads fanout jobs from BullMQ, queries PostgreSQL for matching active endpoints (respecting eventTypeFilter), batch-inserts delivery rows, bulk-enqueues delivery jobs into the per-tenant delivery queue, and publishes a notification to Redis Pub/Sub so the Delivery Worker Manager can spin up a new worker for new tenants.
5. Delivery Workers (concurrency=50 per tenant) process jobs from per-tenant BullMQ queues. They fetch endpoint config and event payloads from Redis cache (falling back to Postgres), check the circuit breaker state, sign the payload with HMAC-SHA256, and POST to the endpoint URL via undici.
6. On success: delivery attempt is recorded and circuit breaker is reset. On failure: attempt is recorded, circuit breaker failure count is incremented, and a retry is scheduled with exponential backoff (10s → 30s → 1m → 5m → ... → 24h). After max attempts the delivery is marked dead.

Database Schema

Performance Benchmarks

HookRelay is benchmarked using k6 on a local Docker framework. Ingestion throughput processes over **1,259 requests per second** concurrently.

Throughput Profile

Optimization History

Project Structure

HookRelay is organized as a pnpm workspace monorepo divided into applications (`apps`) and packages (`packages`).

hookrelay/
├── apps/
│   ├── api/                    # Fastify HTTP server ingestion gateway
│   │   └── src/
│   │       ├── main.ts         # Server entry point
│   │       ├── routes/         # REST API routes (events, tenants, endpoints)
│   │       └── middleware/     # Rate limiter & auth filters
│   ├── worker/                 # Background job worker orchestrators
│   │   └── src/
│   │       └── workers/        # BullMQ worker processors (ingest, fanout, delivery, dlq)
│   ├── mock-server/            # Webhook consumer mock server
│   └── web/                    # Next.js dashboard for tenants and endpoints
|
├── packages/
|   ├──config/                    # Environment configuration
│   ├── db/                     # Drizzle ORM schema, push migrations, queries
│   ├── lib/                    # Redis client, logger (pino), HTTP client (undici)
│   ├── queue/                  # BullMQ definitions, Redis connection clients
│   ├── services/               # Ingest, fanout, circuit-breakers, signature validators
│   └── typescript-config/      # Workspace shared typescript settings
├── k6/                         # Load test scripts
├── docker/
│   ├── Dockerfile.api
│   └── Dockerfile.worker
└── docker-compose.yml          # Container configuration

Prerequisites

• Node.js: version 18 or higher.
• pnpm: version 9.0 or higher.
• Docker / Docker Compose: for running database containers.
• PostgreSQL 16 and Redis 7.

Getting Started

Running with Docker (Recommended)

# 1. Clone repository
git clone https://github.com/SoloDevAbu/HookRelay.git
cd HookRelay

# 2. Copy env configs
cp .env.example .env

# 3. Spin up environment containers
docker compose up -d

Running without Docker

# Install workspace dependencies
pnpm install

# Run database push migrations
pnpm --filter @hookrelay/db db:push

# Launch API and Worker tasks
pnpm --filter @hookrelay/api dev
pnpm --filter @hookrelay/worker dev

Configuration Settings

Settings are validated at startup using Zod schemas. Configure these using `.env` file variables.

API Reference

All client endpoints require client authorization header key: X-Api-Key. Admin tasks require: X-Admin-Key.

curl -X POST http://localhost:3000/tenants \
  -H "X-Admin-Key: <secret>" \
  -d '{"name": "production-app"}'

{
  "success": true,
  "data": {
    "tenant": { "id": "uuid", "name": "production-app" },
    "apiKey": "hr_prod_key_..."
  }
}

curl -X POST http://localhost:3000/events \
  -H "X-Api-Key: wh_live_..." \
  -d '{
    "eventType": "order.placed",
    "payload": {"id": 12},
    "idempotencyKey": "order-12"
  }'

{
  "success": true,
  "data": {
    "eventId": "uuid-event",
    "duplicate": false,
    "status": "accepted"
  }
}

Delivery Pipeline

Webhooks pass through a fast and reliable pipeline to isolate failures and guarantee payload delivery.

• Redis Buffering: The API responds in 202 Accepted after a single LPUSH — no SQL on the hot path. Your producers never wait on DB latency.
• Atomic Lua Batch Pop: The Ingest Worker uses a custom Lua script (LRANGE + LTRIM) to atomically pop up to 100 events in a single Redis round-trip, then performs a single bulk INSERT and addBulk to BullMQ.
• Fanout & Pub/Sub: The Fanout Worker resolves matching endpoints from Postgres and enqueues delivery jobs into a per-tenant BullMQ queue. It then publishes a Redis Pub/Sub message so the Delivery Worker Manager can spin up a new worker process for new tenants on-the-fly.
• Tenant Isolation: Every tenant gets its own BullMQ delivery queue and dedicated worker (concurrency=50). One slow or unreachable destination never delays deliveries for other tenants.
• Caching in Delivery: Delivery Workers cache endpoint config (5 min TTL) and event payloads (1 hr TTL) in Redis to eliminate N+1 Postgres queries when the same event fans out to many endpoints.

Retry Strategy

Failed delivery calls (non-2xx response, timeout, or network error) automatically retry using a fixed backoff schedule defined in retry.service.ts. After MAX_DELIVERY_ATTEMPTS (default 10) the delivery is marked dead.

Circuit Breaker

Each destination endpoint operates under an isolated Redis-backed circuit breaker state machine:

• CLOSED: Normal operation. Delivery jobs execute immediately.
• OPEN: Triggered after CIRCUIT_BREAKER_THRESHOLD (default 5) consecutive failures. Incoming delivery jobs for this endpoint are requeued into the per-tenant BullMQ queue with a 30-second delay (no attempt is counted), preventing wasted HTTP calls against a clearly-down endpoint.
• HALF-OPEN: After CIRCUIT_BREAKER_COOLDOWN_MS (default 60s), the state transitions to HALF-OPEN. One probe delivery is allowed through. If it returns 2xx the circuit closes and the failure counter resets. If it fails the circuit re-opens.

Rate Limiting

Rate limiting is active per-tenant using Redis counters. If a workspace exceeds its rate limits (e.g. 1,000 req/min), HookRelay returns a 429 response block containing a standard Retry-After header specifying when ingestion re-opens.

Load Testing

k6 test configurations exist inside the project repository root for auditing local deployments:

# Run full ingestion load test (200 VUs, 5 minutes)
k6 run k6/load-test.js

# Validate rate limiting response headers
k6 run k6/rate-limit-test.js

Tech Stack