HookOps

About

HookOps Webhook Workflow

HookOps handles the full lifecycle of outbound webhooks. Your app publishes an event, HookOps fans it out to every subscribed endpoint, signs each payload, delivers it, and deals with failures along the way.

Where it goes further is recovery. When an endpoint crosses the failure threshold and gets disabled, HookOps doesn't just leave it there. It periodically probes the endpoint and re-enables it automatically once it starts responding again. No one has to remember to go flip it back on.

The entire system runs on Go and PostgreSQL. PostgreSQL handles both storage and job queuing via SKIP LOCKED.

Architecture

HookOps Engine Architecture

How Delivery Works

Concurrency Model

Graceful shutdown pipeline in main.go:

1. Stop accepting HTTP requests (5s timeout for in-flight requests)
2. Stop the dispatcher (no more tasks claimed from the database)
3. Close the task channel, drain remaining workers (15s timeout)
4. Close the database connection pool

Retry Strategy

Failed deliveries follow a stepped exponential backoff schedule with ±20% uniform jitter:

After 5 failures (configurable), the attempt moves to the dead-letter queue and can be replayed through the API.

The jitter prevents thundering herd problems when a downstream endpoint recovers.

Security

Every outbound webhook is signed using HMAC-SHA256 with a per-endpoint secret. The signature is sent as X-HookOps-Signature: sha256=<hex>.

Headers sent with every webhook:

Verifying Signatures

// Go HMAC-SHA256 signature verification
func VerifyWebhook(payload []byte, secret, signature string) bool {
    sig := strings.TrimPrefix(signature, "sha256=")
    mac := hmac.New(sha256.New, []byte(secret))
    mac.Write(payload)
    expected := hex.EncodeToString(mac.Sum(nil))
    return hmac.Equal([]byte(sig), []byte(expected))
}

# Python HMAC-SHA256 signature verification
import hmac, hashlib

def verify_webhook(payload: bytes, secret: str, signature: str) -> bool:
    sig = signature.removeprefix("sha256=")
    expected = hmac.new(secret.encode(), payload, hashlib.sha256).hexdigest()
    return hmac.compare_digest(sig, expected)

Both use constant-time comparison to prevent timing side-channel attacks.

API

Interactive Swagger docs at localhost:8080/docs.

Quick Example

# Register a tenant
curl -s -X POST http://localhost:8080/api/v1/applications \
  -H "Content-Type: application/json" \
  -d '{"name": "My App"}' | jq .

# Create an event type
curl -s -X POST http://localhost:8080/api/v1/event-types \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "order.completed", "description": "Order fulfilled"}'

# Register an endpoint
curl -s -X POST http://localhost:8080/api/v1/endpoints \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://webhook.site/your-uuid", "description": "Partner CRM", "rate_limit": 10}'

# Subscribe endpoint to event type
curl -s -X POST http://localhost:8080/api/v1/endpoints/$ENDPOINT_ID/subscriptions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"event_type_id": "'$EVENT_TYPE_ID'"}'

# Publish an event
curl -s -X POST http://localhost:8080/api/v1/events \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "event_type": "order.completed",
    "idempotency_key": "order_12345",
    "payload": {"order_id": "order_12345", "total": 149.99}
  }'
# 202 Accepted. Delivery tasks are queued and processing.

Project Structure

hookops/
├── cmd/api/main.go              # Entrypoint, dependency wiring, graceful shutdown
├── internal/
│   ├── config/                  # Env-based configuration with validation
│   ├── database/                # pgxpool connection factory
│   ├── dto/                     # Request/response data transfer objects
│   ├── engine/                  # Core delivery engine
│   │   ├── deliverer.go         # HTTP POST with timeout and body caps
│   │   ├── dispatcher.go        # Background poller with adaptive drain
│   │   ├── worker_pool.go       # Bounded goroutine pool with backpressure
│   │   ├── recorder.go          # Outcome recording and circuit breaker updates
│   │   ├── retry.go             # Stepped backoff with jitter
│   │   ├── limiter.go           # Per-endpoint token bucket rate limiter
│   │   └── signer.go            # HMAC-SHA256 signing and verification
│   ├── handler/                 # HTTP handlers (Gin)
│   ├── middleware/              # Auth, CORS, logging, panic recovery
│   ├── models/                  # Domain types
│   ├── repository/              # PostgreSQL data access layer
│   ├── response/                # Standardized JSON response helpers
│   ├── router/                  # Route registration and middleware wiring
│   ├── server/                  # HTTP server lifecycle
│   └── service/                 # Business logic layer
├── migrations/                  # 8 sequential SQL migrations (up + down)
├── tests/integration/           # End-to-end tests against real PostgreSQL
├── docs/                        # Auto-generated OpenAPI spec
├── Dockerfile                   # Multi-stage build (golang:alpine > alpine:3.20)
├── docker-compose.yml           # One-command local setup
└── Makefile                     # Build, test, lint, migrate, swagger targets

Running Locally

# Start PostgreSQL and HookOps
docker compose up -d

# Verify
curl http://localhost:8080/health
# {"status":"healthy","service":"hookops","database":"connected"}

# Or run outside Docker
cp .env.example .env
make migrate-up
make run

Configuration

All settings via environment variables:

Testing

~6,000 lines of production code, ~6,000 lines of test code across 29 test files.

make test              # Unit tests
make test-integration  # End-to-end against real PostgreSQL
make test-coverage     # Coverage report with HTML output
make lint              # golangci-lint, zero warnings

Unit tests use hand-written interface mocks (no codegen) for the repository and service layers.

Integration tests run against a real PostgreSQL instance and cover the full lifecycle: event ingestion, fan-out, delivery attempt creation, idempotency enforcement, and retry state transitions.

Database

8 migrations applied in order:

1. applications - Tenant accounts with auto-generated ho_live_... API keys
2. endpoints - Webhook destination URLs with unique signing secrets (whsec_...)
3. event_types - Named event categories (e.g. invoice.paid)
4. subscriptions - Many-to-many link between endpoints and event types
5. events - Ingested payloads with idempotency keys
6. delivery_attempts - The queue. Status lifecycle: pending > processing > delivered | failed | dead_letter
7. rate_limit - Per-endpoint RPS configuration
8. health_tracking - Consecutive failure counter for circuit breaker logic

The delivery_attempts table doubles as a job queue using FOR UPDATE SKIP LOCKED, which lets PostgreSQL handle locking semantics instead of requiring a separate message broker.

Design Decisions