Server Configuration

Everything you can pass to serve(), what each setting actually controls inside the GPU runtime, and complete examples you can copy and run.

The minimal server

Every option has a sensible default — this is a complete, working Symmetra application:

// server.ts — run with: sym server.ts
import { Router, serve } from '@symmetra/http'

const app = new Router()

app.get('/health', ctx => {
  ctx.json({ status: 'ok' })
})

serve(app)  // port 8080, 32 streams, batch 64

All serve() options at a glance

serve(app, {
  // Network
  port:     8080,        // TCP port                          (default: 8080)
  hostname: '0.0.0.0',   // bind address                      (default: '0.0.0.0')

  // GPU dispatch
  streams:  32,          // CUDA stream pool size             (default: 32, max: 64)
  batch:    64,          // requests per GPU dispatch         (default: 64)

  // Runtime
  logLevel:     'info',  // 'debug' | 'info' | 'warn' | 'error'
  maxBodyBytes: 1_048_576, // reject bodies larger than 1 MB

  // Advanced scheduler tuning
  scheduler: {
    streams:          32,                  // same as top-level streams
    ioThreads:        8,                   // CPU threads for TCP accept/recv/send
    requestSlabBytes: 4 * 1024 * 1024,    // 4 MB VRAM arena per request
    heapBytes:        512 * 1024 * 1024,  // 512 MB persistent VRAM heap
  },
})

port & hostname

What they really mean: standard TCP listener settings, handled by the CPU side of the runtime (the GPU never touches raw sockets). '0.0.0.0' accepts connections from any interface — use '127.0.0.1' to restrict to local-only during development.

streams

What it really means: a CUDA stream is an independent command queue on the GPU. Each stream can move data and trigger work concurrently with the others. More streams allow more requests to be in-flight on the GPU simultaneously.

When to change it: rarely. 32 is enough to keep the persistent dispatch kernel fed at the request rates the network layer can deliver. Reducing it (e.g. to 8) lowers VRAM overhead on small GPUs; raising it toward 64 only helps if profiling shows streams saturated.

batch

What it really means: how many requests the runtime groups into a single GPU dispatch. GPUs are most efficient processing many items at once — batching amortizes kernel-launch overhead across requests.

The trade-off: larger batches improve throughput under heavy load but can add latency under light load (the runtime waits briefly to fill a batch). Smaller batches minimize per-request latency but cost more overhead per request.

• Latency-sensitive, light traffic: batch: 16 or 32
• Default — balanced: batch: 64
• Throughput benchmarking, heavy sustained load: batch: 128 or 256

logLevel

'debug' | 'info' | 'warn' | 'error' — default 'info'. At 'debug' the runtime logs the compilation pipeline (esbuild → AST → CUDA C++ → NVRTC → PTX) and per-dispatch details. Use 'debug' when a route doesn't behave as expected — the generated CUDA source in the log is the ground truth for what your handler compiled to.

maxBodyBytes

Requests with bodies larger than this are rejected before any GPU work happens. Set it to the largest payload you genuinely expect — request bodies are staged into pinned memory and VRAM, so unbounded bodies waste GPU resources. If you expect large uploads, also raise scheduler.requestSlabBytes (below) so the body fits in the per-request arena.

scheduler — advanced tuning

ioThreads in practice: on our Windows test rig, the network layer — not the GPU — is the throughput ceiling (handlers ran on 128 of 6,144 CUDA cores and the GPU sat >98% idle at 47–80K req/s). If you tune anything for throughput, tune this: more I/O threads help until you saturate your CPU cores.

requestSlabBytes in practice: if a handler allocates large intermediate arrays (e.g. a 1M-element Float32Array = 4 MB), the default slab is too small — raise it. Symptom: requests failing on allocations that look innocent.

heapBytes in practice: budget = your KV data + compiled kernel storage. If you configure KvStore({ maxMb: 256 }), the heap must be comfortably larger than 256 MB. Keep total (heap + slabs × slots) well under your GPU's VRAM — leave room for the display and other processes.

Complete example: development server

Local-only, verbose logging, low latency:

// dev-server.ts — run with: sym dev-server.ts
import { Router, serve } from '@symmetra/http'

const app = new Router()

app.get('/health', ctx => {
  ctx.json({ status: 'ok', device: ctx.gpu.deviceName })
})

serve(app, {
  port:     3000,
  hostname: '127.0.0.1',  // local connections only
  batch:    16,           // small batches → lowest latency at dev traffic levels
  logLevel: 'debug',      // see the full compile pipeline + generated CUDA
})

Complete example: throughput-tuned API

The configuration we use for bombardier load testing (this exact shape produced the 40–80K req/s homepage numbers — handlers on 128 of 6,144 CUDA cores):

// api-server.ts — run with: sym api-server.ts
import { Router, serve } from '@symmetra/http'
import { KvStore } from '@symmetra/kv'

const cache = new KvStore({ maxMb: 128, namespace: 'api' })
const app = new Router()

app.get('/health', ctx => ctx.json({ ok: true }))

app.get('/api/value', async ctx => {
  const key = ctx.query.key ?? 'default'
  const cached = await cache.get<string>(key)
  if (cached !== null) {
    ctx.json({ value: cached, cached: true })
    return
  }
  const value = `computed-${key}`
  await cache.set(key, value, { ttlMs: 60_000 })
  ctx.json({ value, cached: false })
})

serve(app, {
  port:    8080,
  streams: 32,
  batch:   128,           // large batches for sustained load
  logLevel: 'warn',       // keep the log quiet under load
  scheduler: {
    ioThreads: 16,        // network is the bottleneck — give it threads
    heapBytes: 256 * 1024 * 1024,
  },
})

Complete example: compute-heavy handlers

For handlers that do serious GPU work per request (like the rayk-heavy engine: 1,024 machines × 50K iterations, ~39ms per request measured):

// compute-server.ts — run with: sym compute-server.ts
import { Router, serve } from '@symmetra/http'

const app = new Router()

app.get('/simulate', ctx => {
  // Verified engine pattern: outer machine loop, tape array inside,
  // while-loop steps, accumulate into an outer scalar.
  let total = 0
  for (let id = 0; id < 64; id++) {
    const tape: number[] = []
    for (let i = 0; i < 64; i++) tape.push(i === 32 ? 1 : 0)

    let steps = 0
    while (steps < 500) {
      const pos = steps % 64
      const cell = tape[pos]
      tape[pos] = 1 - cell
      steps = steps + 1
    }
    total = total + steps
  }
  ctx.json({ machines: 64, totalSteps: total })
})

serve(app, {
  port:  8080,
  batch: 32,              // smaller batches — each request already saturates threads
  scheduler: {
    requestSlabBytes: 16 * 1024 * 1024,  // room for larger tapes/arrays
  },
})

Important: the codegen supports a specific subset of JavaScript patterns reliably (the structure above is the verified one). Before writing complex handlers, read the constraints page — it documents the patterns that work and the ones that don't yet.

Complete example: configuration from environment variables

// prod-server.ts — run with: PORT=9000 sym prod-server.ts
import { Router, serve } from '@symmetra/http'
import { env } from '@symmetra/env'

const settings = env.load({
  PORT:       { type: 'port',    default: 8080 },
  HOST:       { type: 'string',  default: '0.0.0.0' },
  LOG_LEVEL:  { type: 'string',  default: 'info', choices: ['debug', 'info', 'warn', 'error'] },
  BATCH_SIZE: { type: 'number',  default: 64, min: 1, max: 512 },
})

const app = new Router()
app.get('/health', ctx => ctx.json({ ok: true }))

serve(app, {
  port:     settings.PORT,
  hostname: settings.HOST,
  batch:    settings.BATCH_SIZE,
  logLevel: settings.LOG_LEVEL as 'debug' | 'info' | 'warn' | 'error',
})

Behind the scenes: how dispatch actually works

Understanding the runtime makes the options meaningful:

• At startup, sym compiles your TypeScript: esbuild strips types → the parser extracts your routes → the code generator emits CUDA C++ → NVRTC JIT-compiles it to PTX → the persistent kernel launches.
• The GPU runs a persistent dispatch kernel — it never exits. In the current build, 256 request slots are spread across 32 GPU blocks of 128 threads each (8 slots per block); each block round-robins through its slots looking for work. A handler executes on its block256 request slots are spread across 32 GPU blocks (8 slots per block); each block round-robins through its slots looking for work.#39;s 128 threads — about 2% of an RTX 5070256 request slots are spread across 32 GPU blocks (8 slots per block); each block round-robins through its slots looking for work.#39;s 6,144 CUDA cores — which is why published benchmarks describe a small fraction of the hardware.
• CPU I/O threads (scheduler.ioThreads) accept TCP connections and write incoming requests into shared pinned-memory slots. The GPU picks them up, runs your handler as compiled machine code, and writes the response back. The CPU sends it.
• Cooperative I/O (KV, fetch, DB) works in reverse: GPU thread 0 posts a sub-request to the slot, the CPU services it, and the kernel resumes. This is why I/O-heavy handlers measure so much slower than compute-only ones — each operation is a GPU↔CPU round trip.

The practical summary: network settings shape your ceiling, batch shapes your latency/throughput trade-off, and slab/heap sizes must fit your data. The GPU itself has enormous headroom at beta-scale request rates.