Kafka

yf can keep its queues in Apache Kafka and has an admin page for a Kafka cluster. Both are optional: nothing loads until a site names its brokers, and no library is required. _class('kafka') (yf_kafka, plugin kafka) holds the settings and makes the clients; the queue driver is yf_queue_driver_kafka (plugin queue); the admin page is Tools / Kafka (?object=manage_kafka, engine _class('admin_kafka')).

Setting it up

// the brokers and the security (environment, $CONF or a constant, like REDIS_HOST)
define('KAFKA_BROKERS', 'kafka1:9092,kafka2:9092');   // KAFKA_QUEUE_BROKERS for other brokers for the queue
define('KAFKA_SECURITY_PROTOCOL', 'sasl_ssl');      // plaintext (default), ssl, sasl_plaintext, sasl_ssl
define('KAFKA_SASL_MECHANISM', 'SCRAM-SHA-512');    // PLAIN, SCRAM-SHA-256, SCRAM-SHA-512
define('KAFKA_SASL_USERNAME', 'yf');
define('KAFKA_SASL_PASSWORD', '...');
define('KAFKA_SSL_CA_FILE', '/etc/ssl/kafka-ca.pem');  // KAFKA_SSL_CERT_FILE / KAFKA_SSL_KEY_FILE for a client certificate
// the queue on Kafka
$PROJECT_CONF['wrapper_queue']['driver'] = 'kafka';

KAFKA_SSL_VERIFY (on by default) checks the broker certificate, KAFKA_CLIENT_ID (default yf) names the client in the broker logs, and $CONF['KAFKA_RDKAFKA_CONF'] passes librdkafka settings to the extension (for example compression.type). The Settings page has a Queue & Kafka tab for the switches below; brokers and passwords stay in the site config.

The two clients

  • rdkafka: the php-rdkafka extension on librdkafka (pecl install rdkafka, or the php-rdkafka package of the system; PHP 8.1 to 8.5). The producer is idempotent with acks=all; the consumer joins a consumer group, so several workers share a queue and Kafka rebalances its partitions between them. Use it for the queue in production.
  • wire: the Kafka protocol in plain PHP, no extension. It speaks the request versions of Kafka 2.1 to 4.x, TLS with an optional client certificate, and SASL PLAIN, SCRAM-SHA-256 and SCRAM-SHA-512. It reads uncompressed and gzip batches (snappy, lz4 and zstd when their PHP extension is loaded) and writes uncompressed ones. It has no group membership: as a queue consumer it starts at the group's committed offset, reads the partitions in turn and commits without a member, so run one worker per queue with it. The admin page always uses it, because the extension has no admin API yet (topic configs, groups, create and delete topics).

kafka CLIENT picks the queue's client: auto (default) takes rdkafka when the extension is loaded, else wire.

The queue driver

A queue is a topic (TOPIC_PREFIX + name, default no prefix) read by a consumer group (GROUP_PREFIX + name, default yf_queue.), so every queue has its own offsets and lag. The wrapper_queue methods keep their meaning:

CallOn Kafka
add($queue, $what)Produces one message and waits for the broker (acks=all). Arrays and objects are sent as JSON. The driver's own add($queue, $what, ['key' => .., 'headers' => .., 'partition' => ..]) takes a key (same key, same partition, so their order is kept).
get($queue)The payload of the next message, null when none comes within POLL_TIMEOUT_MS (1000). With AUTO_ACK (default) the offset is committed before it returns. A message whose value is null (a tombstone, written by a compacting producer) also gives null: last() tells it from an empty queue (the message, or null).
del($queue)Skips the next message (reads and commits it).
all($queue)The waiting payloads from the group's committed offsets, ALL_MAX per partition, nothing committed (debugging).
listen($queue, $callback)As before: get() in a loop.

Kafka-only calls, through _class('wrapper_queue')->connect() or the wrapper's __call: ack($queue), nack($queue, $reason), last() (the whole message: topic, partition, offset, timestamp, key, headers) and consume($queue, $callback, ['max' => 100, 'seconds' => 50, 'idle' => 5]), which does it all:

// a cron worker: at least once, retries, dead letters
$q = _class('wrapper_queue');
$r = $q->consume('emails', function ($payload, $params, $message) {
    send_one_email(json_decode($payload, true));   // throw on failure
}, ['seconds' => 55, 'idle' => 10]);
// $r = ['done' => 12, 'failed' => 1]

Delivery, retries, dead letters

  • At most once (AUTO_ACK on, the default): get() commits before you work, like RPOP of the redis driver. A worker that dies mid-way loses that message; nothing is processed twice.
  • At least once (AUTO_ACK off and ack() after the work, or consume()): the offset moves only after the work. A crash, a timeout or a rebalance delivers the message again, so the work must be idempotent (check an id before sending, upsert instead of insert). Offsets are positions: ack() of a message also commits every earlier one of its partition, so get() returns a message that was neither acked nor nacked again rather than the next one.
  • Retries: nack() (an exception in consume()) writes the message again at the end of its topic with the headers yf-attempt and yf-error, then commits the original; after MAX_RETRIES (3) it goes to the dead-letter topic <topic>.dlq (DLQ_SUFFIX) with yf-source (topic/partition@offset). DEAD_LETTER off drops it with a log line instead. When that write fails (broker down), nack() returns 'failed', logs it, commits nothing and the next get() returns the message again. Browse the dead-letter topic on the admin page; to replay, produce the messages again.
  • Order holds inside a partition. A queue topic with one partition is a FIFO queue; more partitions let more rdkafka workers work in parallel, in order per key only. A retried message goes to the end.
  • Producer: rdkafka is idempotent (enable.idempotence), so its own retries add no duplicates; the wire client is not, so a timed-out produce that it does not retry may still have been written. Exactly once is not offered.
  • Topics: add() lets the broker create a missing topic when auto.create.topics.enable is on (the broker's partition count and replication); create them on the admin page otherwise, and create <topic>.dlq too.

The admin page

  • Overview: cards (brokers and the controller, the Kafka version read from metadata.version, topics, partitions with the under-replicated and offline ones, consumer groups) with the ⟳ refresh control (off until you choose), the brokers and how many partitions each leads, and the partition checks.
  • Topics: name filter, sort, pages; partitions, replication and the messages kept (next offset minus first offset). A topic's page lists its partitions (leader, replicas, in-sync replicas, offsets) and configs (values set on the topic marked, secrets masked).
  • Consumer groups: state, members, lag; a group's page shows its members with their partitions and the lag per partition.
  • Browse: N messages of a partition from an offset, the earliest or the latest, read-only (no group, nothing committed): offset, time, key, headers and the value (JSON pretty-printed, binary bytes as \xNN, cut after 64 KB).
  • Changes: produce a test message; create a topic (partitions, replication, a few configs); delete a topic (off until manage_kafka ALLOW_DELETE); reset a group's offsets on a topic to the earliest, the latest or an offset (only while the group has no members, which Kafka requires). Each is a POST with the session token and a confirm; create, delete and reset also need the name typed again. manage_kafka READ_ONLY refuses them all and hides their forms. Every change goes to the admin wall (manage_kafka produce, create_topic, delete_topic, reset_offsets with the offsets before and after).

The page is for super-admins (group 1) only, whatever the block rules say. Its menu item shows only on a site that configures Kafka (manage_kafka MENU_ITEM_NEEDS_CONFIG).

Doctor and admin home

On a site with Kafka the doctor checks that a client can run (a notice when the queue runs on the plain PHP client, without php-rdkafka or with CLIENT wire), that the brokers answer (critical when the queue uses them), offline and under-replicated partitions, and consumer groups behind more than admin_kafka LAG_WARN (10000) messages. The admin home has a Kafka card with the same numbers, read after the page and kept for a minute.

Limits

  • The wire client does not do SASL GSSAPI (Kerberos) or OAUTHBEARER: use php-rdkafka for the queue there (the admin page needs one of the supported logins).
  • The admin page computes the lag of the first admin_kafka LAG_GROUPS_MAX (50) groups, two requests each.
  • Both clients need 64-bit PHP: offsets and timestamps are 64-bit numbers.
  • The wire client does not read message format v0 / v1 (Kafka before 0.11), batches with a wrong CRC, snappy / lz4 / zstd batches without their PHP extension, or a batch that would inflate to more than MAX_INFLATED (32 MB; snappy, lz4 and zstd are sized from their headers before anything is inflated). The browser notes such a batch and reads on; the queue stops before it, logs a warning on every get() (once a minute) and never commits past it. To go on, move the group's offset past the batch on the admin page (reset to an offset), or install the codec's extension.
  • php-rdkafka inflates batches inside librdkafka, where MAX_INFLATED does not apply: librdkafka 2.3 caps zstd at receive.message.max.bytes (100 MB by default; lower it in KAFKA_RDKAFKA_CONF), but not gzip, lz4 or snappy, which grow by their codec's ratio (gzip up to about 1000 times) of a batch the broker accepted (max.message.bytes, 1 MB by default). Let only trusted producers write to queue topics (Kafka ACLs).
  • The wire client gives a whole response RESPONSE_TIMEOUT (30 s) besides the read timeout of each read, refuses a SCRAM iteration count above 1,000,000, and refuses a KAFKA_SECURITY_PROTOCOL other than the four above; so does php-rdkafka.