At VMware, we provided a managed Kafka service that resided on our centralized K8s platform. We supported this service in our K8s clusters in both AWS and OnPrem with a home grown controlplane platform that provided a management UI and self service features along with keeping it up to date and compliante with 24/7 support and a 99.9% SLO.
Not only did our managed Kafka provide cost savings, support, upgrades, and self service one click deployment, but we also developed two open source projects as added benefits.
An SLO exporter that provides visibility and metrics for monitoring Kafka clusters and a Metadata Reaper that will delete leftover empty topic metadata.
The open sourced SLO exporter for Kafka is a GoLang service that starts a Kafka producer or consumer that sends generic dummy data and exposes it as prometheus metrics. The repo provides dashboards for Datadog and Wavefront. My fellow Engineer colleague who developed the service wrote about it here.
The open sourced Kafka metadata reaper is another service written in GoLang that creates an admin client that lists all the topics that reside on the cluster and then allows you to list empty topics ready for deletion in dry-run mode or just list and delete the empty topics.
The same great engineer who wrote the SLO exporter also developed this service and wrote about it here.
Empty topics in Kafka can occur in several scenarios:
1. Newly Created Topic — A topic is created but hasn’t received any messages yet.
2. Retention Policy Cleanup — If log.retention.ms or log.retention.bytes removes all segments, the topic remains but has no data.
3. Compacted Topics — In log-compacted topics, all messages with outdated keys might be removed, making the topic appear empty.
4. Manual Deletion of Messages — If an admin truncates a topic (kafka-delete-records.sh), it may become empty.
5. Producer Not Sending Data — A topic exists, but no producer is actively writing to it.
6. Topic with No Active Consumers — Consumers may have processed and deleted all records if using Kafka as a queue with delete retention.
Cleaning metadata from empty topics in Kafka can be beneficial for several reasons:
1. Reduce ZooKeeper and Broker Metadata Overhead — Kafka stores topic metadata in ZooKeeper and brokers, and too many empty topics can increase metadata storage and processing overhead.
2. Improve Metadata Fetch Performance — Kafka clients (producers, consumers, and admin tools) fetch metadata frequently. Removing unused topics reduces metadata size, improving response times.
3. Reduce Broker Memory Usage — Even empty topics consume memory for partition and leader metadata, affecting overall broker performance.
4. Avoid Unnecessary Topic Replication — Kafka replicates metadata across brokers, and retaining empty topics increases unnecessary synchronization efforts.
5. Prevent Configuration Clutter — Keeping a clean set of active topics makes it easier for administrators to manage and configure Kafka.
6. Enhance Monitoring and Troubleshooting — Too many empty topics may clutter dashboards, making it harder to identify active ones.
7. Reduce Storage of Unused Log Files — Even with no messages, Kafka maintains log files for empty topics, taking up disk space.
8. Avoid Zombie Topics — Topics that were created for testing or temporary purposes might be left unused, leading to unnecessary resource consumption.
Do You Still Need to Clean Metadata from Empty Topics in KRaft?
With KRaft (Kafka Raft) mode, Kafka no longer depends on ZooKeeper for metadata management. Instead, metadata is stored internally within the Kafka brokers and replicated across a quorum of controllers using the Raft consensus algorithm.
Metadata Cleanup in KRaft vs. ZooKeeper Mode
• ZooKeeper Mode (Pre-Kafka 2.8)
• Metadata about topics, partitions, brokers, and configurations was stored in ZooKeeper.
• Deleting a topic required cleaning up ZooKeeper metadata manually in some cases.
• KRaft Mode (Kafka 2.8+ and Default in Kafka 3.3+)
• Metadata is managed entirely by Kafka brokers.
• When a topic is deleted, metadata automatically gets removed from the Raft log.
• No manual cleanup is required.
When Might Metadata Still Need Attention?
1. Long-Retained Metadata Logs:
• If your KRaft metadata log grows significantly, consider adjusting retention settings (metadata.log.retention.ms).
2. Zombie Metadata Issues (Rare cases of stale metadata):
• If a topic deletion fails or a broker crashes mid-deletion, you might need to restart affected brokers to trigger cleanup.
3. Cluster Performance Issues:
• Large numbers of empty topics can still increase metadata load. Periodically auditing unused topics can help optimize cluster performance.
With KRaft mode, Kafka handles metadata cleanup automatically, reducing operational overhead compared to ZooKeeper-based clusters. However, keeping an eye on metadata retention settings and topic lifecycle management is still a best practice.