Compatibility

@cookiemonsterdev/kafka-core is a TypeScript Apache Kafka client for Node.js that speaks the Kafka wire protocol directly — no JVM.

The client negotiates protocol versions from ApiVersions. The support floor is Kafka 0.10+. See Apache Kafka compatibility and the protocol guide.

Tested brokers

Integration tests cover 0.10, 0.11, 1.1, 2.4, 3.6, 4.0, and 4.3. Default KAFKA_VERSION remains 4.0. Compose files also exist for 4.1 and 4.2 (apache/kafka:4.1.2 / 4.2.1 / 4.3.1); CI PRs run 4.3 and default-branch pushes run the full matrix.

The client talks to 4.0 and 4.3 via overlap (Produce 3–13, Fetch 4–18, Metadata 0–13). Kafka 4.0 brokers dropped Produce v0–2 and Fetch v0–3 (KIP-896; see Apache Kafka compatibility); the client still encodes those versions for 0.10 clusters and will not send them to 4.0 because the broker does not advertise them. Kafka 4.0 advertises Produce through v12; v13 (topic IDs, KIP-516) is used when the broker advertises it and Cluster metadata includes a topicId. Kafka 4.0 advertises Fetch through v17; v18 (KIP-1166) is used on 4.1+. Fetch v13+ (topic IDs, KIP-516) is used when the broker advertises it and Cluster metadata includes a topicId.

Metadata v10–v13 decode KIP-516 topic IDs (topicId as a 16-byte Buffer on each topic). Produce v13 and Fetch v13+ address topics by those IDs; earlier versions still use topic names. admin.describeTopicPartitions (key 75, Kafka 4.0+) also returns topicId on each described topic.

Constructor defaults

These defaults are product choices.

See producer configs and consumer configs for the equivalent broker-facing property names.

The opt-in sticky partitioner keeps unkeyed records on one partition for each Produce batch formed by this client’s lingerMs / batchSize model, then rotates uniformly to a different available partition. Explicit partitions and keyed murmur2 routing are unchanged.

Throughput preset

throughputPreset() is a named profile for load-oriented clients. Constructor defaults stay as in the table above. Spread the returned fragments:

import { Kafka, throughputPreset } from '@cookiemonsterdev/kafka-core';

const kafka = new Kafka({ clientId: 'load', brokers: ['localhost:9092'] });
const { producer, consumer } = throughputPreset();
const p = kafka.producer({ ...producer });
await kafka.consumer({ groupId: 'load' }).run({
  ...consumer,
  eachBatch: async ({ batch }) => {
    for (const message of batch.messages) {
      void message;
    }
  },
});

producer sets the sticky partitioner and a 32 MiB bufferMemory. Linger, batch size, and in-flight caps are already constructor defaults (lingerMs: 5, batchSize: 16384, maxInFlightRequests: 5). consumer sets partitionsConsumedConcurrently: 4 on run() (not on kafka.consumer()).

eachBatch plus partitionsConsumedConcurrently is the heavy-load consume API. Pass lingerMs: 0 for one Produce per send(). Compression: GZIP and ZSTD already use the zlib threadpool; Snappy and LZ4 are off-thread as well (optional native packages if installed).

Full walkthrough: Throughput.

Implemented surface

Method lists: Producer API, Consumer API, Admin API. The notes below are version floors and protocol keys, not a backlog.

Consumer. Range, RoundRobin, Sticky, and CooperativeSticky are built in (PartitionAssigners). The default assigner is still round-robin. Classic JoinGroup/SyncGroup remains the default membership protocol. Fetch uses incremental fetch sessions (KIP-227) on Kafka 2.3+; older brokers stay sessionless (sessionId = 0). Set groupProtocol: 'consumer' (broker property group.protocol) to opt into KIP-848 ConsumerGroupHeartbeat on Kafka 4.0+; assignment is server-side and incremental. admin.describeConsumerGroups uses ConsumerGroupDescribe (key 69) via each group coordinator on Kafka 4.0+. admin.describeClassicGroups is an alias for admin.describeGroups (DescribeGroups, key 15) for classic JoinGroup groups. fromBeginning is boolean (earliest vs latest). autoOffsetReset: 'none' throws if there is no committed offset. autoOffsetReset: 'by_duration:PT1H' (KIP-1106) starts at the first offset at or after now minus that ISO-8601 duration. Cooperative-sticky uses KIP-429 incremental revoke semantics and performs the follow-up generation needed to settle partitions that move between members. This assignor support applies to the classic group protocol. kafka.shareConsumer() implements KIP-932 share groups (ShareGroupHeartbeat / ShareFetch / ShareAcknowledge, keys 76–79) on Kafka 4.1+. ShareFetch and ShareAcknowledge negotiate v1–v2; v2 adds shareAcquireMode (KIP-1206) and RENEW acknowledgements (KIP-1222) on Kafka 4.2+. Classic consumer() remains the default.

Admin. admin.alterConfigs is kept for older brokers. Prefer admin.incrementalAlterConfigs (key 44). admin.electLeaders is key 43 (historically ElectPreferredLeaders). admin.deleteGroupOffsets is OffsetDelete (key 47). admin.describeUserScramCredentials and admin.alterUserScramCredentials are keys 50–51. admin.describeClientQuotas / admin.alterClientQuotas are keys 48–49. admin.describeLogDirs / admin.alterReplicaLogDirs are keys 34–35. admin.describeReplicaLogDirs filters DescribeLogDirs (35) to specific broker/replica pairs. admin.describeCluster uses DescribeCluster (key 60) when advertised and Metadata otherwise. admin.describeProducers uses key 61 on Kafka 3.0+ and queries partition leaders unless a brokerId is supplied. admin.describeTransactions uses key 65, dynamically discovers transaction coordinators, and requires Kafka 3.0+. admin.listTransactions uses key 66, fans the request out to every broker, unique-merges by transactional ID, and requires Kafka 3.0+; v1 adds durationFilter and v2 adds transactionalIdPattern. admin.fenceProducers uses InitProducerId (22) via transaction coordinators (Kafka 2.5+ / v3+). admin.abortTransaction uses WriteTxnMarkers (27) on partition leaders (Kafka 3.0+ / v1+; v0 removed in 4.0). admin.forceTerminateTransaction fences one transactional ID via InitProducerId. admin.describeFeatures reads ApiVersions (18) v3+ tagged fields (KIP-584) from the active controller. admin.removeMembersFromConsumerGroup uses LeaveGroup (13) v3+ with explicit member identities. admin.describeConsumerGroups uses ConsumerGroupDescribe (key 69) via group coordinators and requires Kafka 4.0+. admin.describeClassicGroups aliases admin.describeGroups (DescribeGroups, key 15) for classic JoinGroup groups. admin.describeShareGroups, listShareGroupOffsets, alterShareGroupOffsets, deleteShareGroupOffsets, and deleteShareGroups implement share-group Admin (keys 77, 90–92, plus DeleteGroups 42) on Kafka 4.1+. admin.describeTopicPartitions uses key 75 on Kafka 4.0+ (KIP-966), sends topic names, and returns one page plus nextCursor for the caller to continue. admin.updateFeatures implements UpdateFeatures (key 57) v0–v2 and targets the active controller; v0 cannot validate-only and rejects unsafe downgrades. admin.describeMetadataQuorum implements DescribeQuorum (key 55) v0–v2 against the active controller and requires KRaft 3.6+. v1 adds replica timestamps (KIP-836); v2 adds directory IDs and node listeners (KIP-853). admin.unregisterBroker implements UnregisterBroker (key 64) v0. admin.assignReplicasToDirs implements AssignReplicasToDirs (key 73) v0 (KIP-858) and assigns replicas on a broker to log directories by 16-byte directory UUID. admin.addRaftVoter and admin.removeRaftVoter implement keys 80–81; v1 of AddRaftVoter adds optional ackWhenCommitted (default true). These controller RPCs require KRaft 3.7+ when the broker advertises the API. admin.listConfigResources implements ListConfigResources (key 74) v0–v1 and targets the active controller. v0 lists client metrics names only (Kafka 4.0); filtering by resourceTypes needs v1 (Kafka 4.1+ / KIP-1142). An empty resourceTypes list is valid: v1 returns the broker’s default supported types, v0 returns all client metrics. admin.createDelegationToken, admin.renewDelegationToken, admin.expireDelegationToken, and admin.describeDelegationToken implement keys 38–41 (Kafka 1.1+).

Security. SASL PLAIN, SCRAM, OAUTHBEARER, and GSSAPI / Kerberos are implemented. GSSAPI is opt-in (mechanism: 'gssapi'): supply gssProvider or install the optional kerberos package; you still need a KDC and a ticket or keytab. CI does not run a Kerberos stack. The aws SASL helper is extra (non-Apache). Admin can create, describe, renew, and expire delegation tokens. SASL login with a delegation token is opt-in: set sasl.mechanism to scram-sha-256 or scram-sha-512 and pass tokenId / tokenHmac (see Security). The broker still needs delegation.token.secret.key and SASL/SCRAM. See SASL authentication and the security guide.

Not implemented

These are not in @cookiemonsterdev/kafka-core. Broker-internal RPCs (controller replication, share-group state keys 83–87, and similar) are not client APIs and are omitted here.

Packages. No Kafka Streams or Kafka Connect packages. See Kafka Streams and Kafka Connect.

Streams groups (KIP-1071). No StreamsGroupHeartbeat or StreamsGroupDescribe (keys 88–89). Admin has no describeStreamsGroups, listStreamsGroupOffsets, alterStreamsGroupOffsets, deleteStreamsGroupOffsets, or deleteStreamsGroups.

Consumer shape. Membership is run() / stream(), not a blocking poll() loop. Offsets as bigint, MessageSet, ZSTD, and KAFKA_* env vars: Breaking changes.