Kafka
yf can keep its queues in Apache Kafka and has an admin page for a Kafka cluster. Both are optional: nothing loads until a site
names its brokers, and no library is required. _class('kafka') (yf_kafka, plugin kafka) holds
the settings and makes the clients; the queue driver is yf_queue_driver_kafka (plugin queue); the admin page is
Tools / Kafka (?object=manage_kafka, engine _class('admin_kafka')).
Setting it up
// the brokers and the security (environment, $CONF or a constant, like REDIS_HOST)
define('KAFKA_BROKERS', 'kafka1:9092,kafka2:9092'); // KAFKA_QUEUE_BROKERS for other brokers for the queue
define('KAFKA_SECURITY_PROTOCOL', 'sasl_ssl'); // plaintext (default), ssl, sasl_plaintext, sasl_ssl
define('KAFKA_SASL_MECHANISM', 'SCRAM-SHA-512'); // PLAIN, SCRAM-SHA-256, SCRAM-SHA-512
define('KAFKA_SASL_USERNAME', 'yf');
define('KAFKA_SASL_PASSWORD', '...');
define('KAFKA_SSL_CA_FILE', '/etc/ssl/kafka-ca.pem'); // KAFKA_SSL_CERT_FILE / KAFKA_SSL_KEY_FILE for a client certificate
// the queue on Kafka
$PROJECT_CONF['wrapper_queue']['driver'] = 'kafka';
KAFKA_SSL_VERIFY (on by default) checks the broker certificate, KAFKA_CLIENT_ID (default yf)
names the client in the broker logs, and $CONF['KAFKA_RDKAFKA_CONF'] passes librdkafka settings to the extension (for
example compression.type). The Settings page has a Queue & Kafka tab for the switches below; brokers and
passwords stay in the site config.
The two clients
- rdkafka: the php-rdkafka extension on librdkafka
(
pecl install rdkafka, or thephp-rdkafkapackage of the system; PHP 8.1 to 8.5). The producer is idempotent withacks=all; the consumer joins a consumer group, so several workers share a queue and Kafka rebalances its partitions between them. Use it for the queue in production. - wire: the Kafka protocol in plain PHP, no extension. It speaks the request versions of Kafka 2.1 to 4.x, TLS with an optional client certificate, and SASL PLAIN, SCRAM-SHA-256 and SCRAM-SHA-512. It reads uncompressed and gzip batches (snappy, lz4 and zstd when their PHP extension is loaded) and writes uncompressed ones. It has no group membership: as a queue consumer it starts at the group's committed offset, reads the partitions in turn and commits without a member, so run one worker per queue with it. The admin page always uses it, because the extension has no admin API yet (topic configs, groups, create and delete topics).
kafka CLIENT picks the queue's client: auto (default) takes rdkafka when the extension is loaded, else wire.
The queue driver
A queue is a topic (TOPIC_PREFIX + name, default no prefix) read by a consumer group (GROUP_PREFIX + name,
default yf_queue.), so every queue has its own offsets and lag. The wrapper_queue methods keep their meaning:
| Call | On Kafka |
|---|---|
add($queue, $what) | Produces one message and waits for the broker (acks=all). Arrays and objects are sent as JSON. The driver's
own add($queue, $what, ['key' => .., 'headers' => .., 'partition' => ..]) takes a key (same key, same partition, so their
order is kept). |
get($queue) | The payload of the next message, null when none comes within POLL_TIMEOUT_MS (1000).
With AUTO_ACK (default) the offset is committed before it returns. A message whose value is null (a tombstone, written by
a compacting producer) also gives null: last() tells it from an empty queue (the message, or null). |
del($queue) | Skips the next message (reads and commits it). |
all($queue) | The waiting payloads from the group's committed offsets, ALL_MAX per partition, nothing committed (debugging). |
listen($queue, $callback) | As before: get() in a loop. |
Kafka-only calls, through _class('wrapper_queue')->connect() or the wrapper's __call:
ack($queue), nack($queue, $reason), last() (the whole message: topic, partition, offset, timestamp,
key, headers) and consume($queue, $callback, ['max' => 100, 'seconds' => 50, 'idle' => 5]), which does it all:
// a cron worker: at least once, retries, dead letters
$q = _class('wrapper_queue');
$r = $q->consume('emails', function ($payload, $params, $message) {
send_one_email(json_decode($payload, true)); // throw on failure
}, ['seconds' => 55, 'idle' => 10]);
// $r = ['done' => 12, 'failed' => 1]
Delivery, retries, dead letters
- At most once (
AUTO_ACKon, the default):get()commits before you work, likeRPOPof the redis driver. A worker that dies mid-way loses that message; nothing is processed twice. - At least once (
AUTO_ACKoff andack()after the work, orconsume()): the offset moves only after the work. A crash, a timeout or a rebalance delivers the message again, so the work must be idempotent (check an id before sending, upsert instead of insert). Offsets are positions:ack()of a message also commits every earlier one of its partition, soget()returns a message that was neither acked nor nacked again rather than the next one. - Retries:
nack()(an exception inconsume()) writes the message again at the end of its topic with the headersyf-attemptandyf-error, then commits the original; afterMAX_RETRIES(3) it goes to the dead-letter topic<topic>.dlq(DLQ_SUFFIX) withyf-source(topic/partition@offset).DEAD_LETTERoff drops it with a log line instead. When that write fails (broker down),nack()returns'failed', logs it, commits nothing and the nextget()returns the message again. Browse the dead-letter topic on the admin page; to replay, produce the messages again. - Order holds inside a partition. A queue topic with one partition is a FIFO queue; more partitions let more rdkafka workers work in parallel, in order per key only. A retried message goes to the end.
- Producer: rdkafka is idempotent (
enable.idempotence), so its own retries add no duplicates; the wire client is not, so a timed-out produce that it does not retry may still have been written. Exactly once is not offered. - Topics:
add()lets the broker create a missing topic whenauto.create.topics.enableis on (the broker's partition count and replication); create them on the admin page otherwise, and create<topic>.dlqtoo.
The admin page
- Overview: cards (brokers and the controller, the Kafka version read from
metadata.version, topics, partitions with the under-replicated and offline ones, consumer groups) with the ⟳ refresh control (off until you choose), the brokers and how many partitions each leads, and the partition checks. - Topics: name filter, sort, pages; partitions, replication and the messages kept (next offset minus first offset). A topic's page lists its partitions (leader, replicas, in-sync replicas, offsets) and configs (values set on the topic marked, secrets masked).
- Consumer groups: state, members, lag; a group's page shows its members with their partitions and the lag per partition.
- Browse: N messages of a partition from an offset, the earliest or the latest, read-only (no group, nothing committed): offset,
time, key, headers and the value (JSON pretty-printed, binary bytes as
\xNN, cut after 64 KB). - Changes: produce a test message; create a topic (partitions, replication, a few configs); delete a topic (off until
manage_kafka ALLOW_DELETE); reset a group's offsets on a topic to the earliest, the latest or an offset (only while the group has no members, which Kafka requires). Each is a POST with the session token and a confirm; create, delete and reset also need the name typed again.manage_kafka READ_ONLYrefuses them all and hides their forms. Every change goes to the admin wall (manage_kafka produce,create_topic,delete_topic,reset_offsetswith the offsets before and after).
The page is for super-admins (group 1) only, whatever the block rules say. Its menu item shows only on a site that configures Kafka
(manage_kafka MENU_ITEM_NEEDS_CONFIG).
Doctor and admin home
On a site with Kafka the doctor checks that a client can run (a notice when the queue runs on the plain PHP client, without
php-rdkafka or with CLIENT wire), that the brokers
answer (critical when the queue uses them), offline and under-replicated partitions, and consumer groups behind more than
admin_kafka LAG_WARN (10000) messages. The admin home has a Kafka card with the same numbers, read after the page and kept
for a minute.
Limits
- The wire client does not do SASL GSSAPI (Kerberos) or OAUTHBEARER: use php-rdkafka for the queue there (the admin page needs one of the supported logins).
- The admin page computes the lag of the first
admin_kafka LAG_GROUPS_MAX(50) groups, two requests each. - Both clients need 64-bit PHP: offsets and timestamps are 64-bit numbers.
- The wire client does not read message format v0 / v1 (Kafka before 0.11), batches with a wrong CRC, snappy / lz4 / zstd batches
without their PHP extension, or a batch that would inflate to more than
MAX_INFLATED(32 MB; snappy, lz4 and zstd are sized from their headers before anything is inflated). The browser notes such a batch and reads on; the queue stops before it, logs a warning on everyget()(once a minute) and never commits past it. To go on, move the group's offset past the batch on the admin page (reset to an offset), or install the codec's extension. - php-rdkafka inflates batches inside librdkafka, where
MAX_INFLATEDdoes not apply: librdkafka 2.3 caps zstd atreceive.message.max.bytes(100 MB by default; lower it inKAFKA_RDKAFKA_CONF), but not gzip, lz4 or snappy, which grow by their codec's ratio (gzip up to about 1000 times) of a batch the broker accepted (max.message.bytes, 1 MB by default). Let only trusted producers write to queue topics (Kafka ACLs). - The wire client gives a whole response
RESPONSE_TIMEOUT(30 s) besides the read timeout of each read, refuses a SCRAM iteration count above 1,000,000, and refuses aKAFKA_SECURITY_PROTOCOLother than the four above; so does php-rdkafka.