Kafka

A kafka binding is a typed connection to a Kafka topic — the durable, cross-instance sibling of an in-process topic. You produce with .send(v) from any handler; values cross the wire as JSON.

type Order = { id: int, item: string }

kafka orders -> Order {
    brokers env("KAFKA_BROKERS")   # comma-separated host:port list
    topic   "orders"
}

route POST "/orders" (body o: Order) -> string {
    orders.send(o)                 # produce to Kafka
    respond "published"
}

Producing

name.send(v) JSON-encodes v (which must match the binding's message type) and produces one record to the topic. It speaks the Kafka wire protocol directly — no client library or sidecar — using a Produce request with a v2 record batch (CRC32C checked), so any Kafka-compatible broker (Apache Kafka, Redpanda) accepts it.

send is best-effort: broker/connection errors are logged, not thrown, so a Kafka blip never fails the request. For guaranteed delivery, produce from a worker that can retry.

Consuming

Consume a binding with for m in name inside a worker — the durable counterpart of consuming an in-process topic:

worker {
    for o in orders {          # blocks for the next record; ends on shutdown
        ship(o)                # process it — write a DB row, call an upstream, …
    }
}

Each record's JSON is decoded into the binding's message type. Weld consumes with the message's committed group offset (or the earliest record for a new group), and commits progress as it goes — so a restart resumes where it left off (at-least-once). One worker per binding drains the topic; state touched in the loop is locked per message.

Configuration

Field	Meaning
brokers	comma-separated host:port bootstrap list (usually env(...))
topic	the Kafka topic name
group	consumer group id for committed offsets; defaults to the binding name

Any of these can be an expression, so pull them from the environment per deploy.

Configuration

Field	Meaning
brokers	comma-separated host:port bootstrap list (usually env(...))
topic	the Kafka topic name
group	consumer group id (for consuming; defaults to the binding name)

Any of these can be an expression, so pull them from the environment per deploy.

Scope

Weld speaks the Kafka wire protocol against the bootstrap broker — ideal for single-broker and dev/staging clusters (Apache Kafka, Redpanda). Producing sends to partition 0 with acks=1; consuming reads all partitions and commits offsets to the group. Still to come:

• Multi-broker leader/coordinator routing (via Metadata/FindCoordinator) for large clusters.
• Partitioning by key, producer batching/compression, and configurable acks.
• Consumer groups with rebalancing (today one consumer instance owns the topic).