Apache Kafka® is an open-source stream-processing software platform developed by LinkedIn and donated to the Apache Software Foundation, written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Kafka can connect to external systems (for data import/export) via Kafka Connect and provides Kafka Streams, a Java stream processing library.

Apache-Kafka

Table of Contents

  1. Verify Java installation
  2. Install Zookeeper & Kafka
  3. Running Kafka Locally

Verify Java installation

Kafka requires Java version 8 or higher.

Check Java version run java -version, The version should be atleast 1.8.X

If the version is lesser then install the latest java version using below command.

1
2
brew tap caskroom/cask
brew cask install java

Note: To install specific java version like Java 8, use brew cask install java8

Install Zookeeper & Kafka

Install ZooKeeper(service discovery application) along with Kafka.

Run,

1
brew install kafka

Output should look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
❯ brew install kafka
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 2 taps (homebrew/cask and homebrew/bundle).
No changes to formulae.

==> Installing dependencies for kafka: zookeeper
==> Installing kafka dependency: zookeeper
==> Downloading https://homebrew.bintray.com/bottles/zookeeper-3.4.14.catalina.bottle.tar.gz
==> Downloading from https://akamai.bintray.com/85/854225ed94e18cdf9a08b992a658e851d4c4d77d826e8ae243488e65b38af84c
######################################################################## 100.0%
==> Pouring zookeeper-3.4.14.catalina.bottle.tar.gz
==> Caveats
To have launchd start zookeeper now and restart at login:
brew services start zookeeper
Or, if you don't want/need a background service you can just run:
zkServer start
==> Summary
🍺 /usr/local/Cellar/zookeeper/3.4.14: 430 files, 36.5MB
==> Installing kafka
==> Downloading https://homebrew.bintray.com/bottles/kafka-2.3.1.catalina.bottle.tar.gz
==> Downloading from https://akamai.bintray.com/0d/0d3bdac93bd2602f0a7ee444fa77081362da52ea54e3d66488904a67128b7afd
######################################################################## 100.0%
==> Pouring kafka-2.3.1.catalina.bottle.tar.gz
==> Caveats
To have launchd start kafka now and restart at login:
brew services start kafka
Or, if you don't want/need a background service you can just run:
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties
==> Summary
🍺 /usr/local/Cellar/kafka/2.3.1: 171 files, 59.5MB
==> `brew cleanup` has not been run in 30 days, running now...
Removing: /usr/local/Cellar/gettext/0.19.8.1... (1,935 files, 16.9MB)
Pruned 0 symbolic links and 4 directories from /usr/local
==> Caveats
==> zookeeper
To have launchd start zookeeper now and restart at login:
brew services start zookeeper
Or, if you don't want/need a background service you can just run:
zkServer start
==> kafka
To have launchd start kafka now and restart at login:
brew services start kafka
Or, if you don't want/need a background service you can just run:
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties

Running Kafka Locally

Let’s start ZooKeeper first.

1
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties

In another terminal, Let’s start Kafka.

1
kafka-server-start /usr/local/etc/kafka/server.properties

Optionally, To have launchd start zookeeper & kafka now (in background) and restart at login.
run sequentially,
brew services start zookeeper

brew services start kafka