Kafbat Kafka UI Implementation
· Part of my work at Carousell
Abstract: Identified issues with existing Kafka management tools and led the migration to Kafbat for a more secure, modern, and maintained solution.
The Problem
Our engineering teams relied on two separate web UIs, CMAK and AKHQ, to manage and interact with our Kafka clusters. Both of these tools posed significant problems:
- Lack of Maintenance: They were no longer actively maintained by their communities.
- Security & Performance Issues: The lack of updates introduced potential security risks and performance bottlenecks.
- Operational Overhead: Managing two separate tools was inefficient for the SRE team and confusing for developers.
The Solution
I took the initiative to find a better alternative. My research identified Kafbat as a modern, actively maintained, and feature-rich open-source web UI for Kafka.
After proposing the solution, I led the implementation project, which involved:
- Deployment: Deploying the Kafbat service on both our staging and production Kubernetes clusters.
- Configuration: Securely configuring Kafbat to connect to our Kafka clusters.
- Rollout & Decommissioning: Successfully rolling out Kafbat to all engineering teams and subsequently decommissioning the two legacy tools.
The result was a unified, secure, and efficient platform for Kafka management. This improved the developer experience, reduced operational load, and eliminated the security risks associated with the unmaintained tools.