Benchmark Architecture

Request flows left → right; response returns. The load generator and GPU server sit in the same cloud zone (10 GbE) so the internet is never a variable.

How every number in these reports was produced. Three machines, one request path.

   +------------------+        +----------------------+        +------------------+
   |  LOAD GENERATOR  |        |    GPU SERVER        |        |    NVIDIA L4     |
   |  (Client)        | 10GbE  |  (Server under test) |  PCIe  |    (The GPU)     |
   |                  |------->|                      |<------>|                  |
   |  n2-standard-16  |  ~0.2ms|  g2-standard-32      |        |  58 SMs, 24 GB   |
   |  16 vCPU         |        |  32 vCPU + L4        |        |  72 W TDP        |
   |  runs wrk        |        |  runs sym            |        |  executes every  |
   |                  |        |  (TS -> CUDA)        |        |  handler         |
   +------------------+        +----------------------+        +------------------+

The three CPUs (do not confuse them)

When any report says “CPU-bound,” it means the GPU-server’s own CPU — not the load generator, not your machine.

CPU	Machine	Role	In our findings
Load-generator CPU	16-vCPU VM	Runs `wrk`, sends requests	Never the bottleneck (stays underused)
GPU-server CPU	32-vCPU server	Accepts TCP, parses HTTP, dispatches to GPU	This is "the CPU" in every finding. Maxes out on light endpoints.
(Home CPU)	your local machine	Only used in the earlier home->cloud test	Capped at 988 RPS by internet latency

When a report says "CPU-bound," it means the GPU-server's own CPU — the cores next to the GPU doing network I/O — not the load generator, not your machine.

The request path

wrk on the load generator opens 300 TCP connections and fires HTTP requests over the internal 10 GbE network (~0.2 ms round trip).
sym on the GPU server accepts the connection, parses HTTP on the CPU, and writes the request into a pinned memory slot.
The persistent GPU kernel (always resident) picks up the slot, runs the compiled TypeScript handler as a Turing machine on the GPU, and writes the response back.
sym reads the response from the slot and sends it back over TCP.

GPU grid deployed

32 SMs used (55% coverage)

26 SMs idle

4,096 of 89,088 threads = 4.6% of GPU capacity

Measured live from sym's /gpu-stats:

  Blocks:            32  (of 58 SMs available -> 55% SM coverage)
  Threads/block:    128
  Threads total:  4,096  (of 89,088 GPU max -> 4.6% of thread capacity)
  Warps:            128
  Request slots:    256  (8 per block - concurrent requests in flight)
  IO threads:        30  (CPU-side, feed the GPU)

We deliberately deployed 4.6% of the GPU's thread capacity. A light request activates only ~4-64 of those 4,096 threads (one GPU thread per Turing machine in the handler). This is why light endpoints leave the GPU near-idle.

Endpoints (identical across all 17 engines)

Endpoint	TM scale	Purpose
`/health`	none	Baseline - measures pure request plumbing (CPU limit)
`/light`	1x base	Light per-request compute
`/medium`	8x base	Medium compute
`/heavy`	64x base	Heavy compute - where the GPU does real work

Each endpoint reports its exact workPerReq (machines x inner-steps), so steps/sec = RPS x workPerReq is directly measurable and comparable across every engine.

Tools & parameters

Parameter	Value
Load tool	`wrk` 16 threads x 300 connections
Window	20 s per endpoint (after 5 s warm-up)
GPU telemetry	sym `/gpu-stats` (NVML: power, clock, temp; live slot count)
Network	internal 10 GbE (client and server in same GCP zone)
Errors	zero across all engines and endpoints

<- Back to Master Index