Prerequisites

Before you begin, ensure you have the following:

1. Update Your System

Always start by updating your package lists and upgrading existing packages to ensure your system is up-to-date.

sudo apt update
sudo apt upgrade -y

2. Install Java Development Kit (JDK)

Kafka is written in Scala and runs on the Java Virtual Machine (JVM). Install OpenJDK, a widely used Java implementation.

Install JDK:

sudo apt install default-jdk -y

Verify installation:

java -version

Expected output: openjdk version "11.0.x" (or similar, depending on the current default version).

3. Install Apache ZooKeeper

Kafka relies on ZooKeeper for cluster coordination and metadata management. This section guides you through its setup.

3.1 Download ZooKeeper

Visit the Apache ZooKeeper releases page to find a stable version. Replace the version number in the command below if needed.

wget https://downloads.apache.org/zookeeper/zookeeper-3.8.4/apache-zookeeper-3.8.4-bin.tar.gz

(Note: Example uses version 3.8.4. Adjust as necessary.)

3.2 Extract ZooKeeper

tar -xvzf apache-zookeeper-3.8.4-bin.tar.gz
sudo mv apache-zookeeper-3.8.4-bin /opt/zookeeper

3.3 Create Data Directory

sudo mkdir -p /var/lib/zookeeper
sudo chown -R $(whoami):$(whoami) /var/lib/zookeeper

This command gives your current user ownership of the data directory.

3.4 Configure ZooKeeper

Copy the sample configuration and edit it:

sudo cp /opt/zookeeper/conf/zoo_sample.cfg /opt/zookeeper/conf/zoo.cfg
sudo nano /opt/zookeeper/conf/zoo.cfg

Inside zoo.cfg, find the line dataDir=/tmp/zookeeper and change it to:

dataDir=/var/lib/zookeeper

Save and exit the editor (e.g., Ctrl+X, then Y, then Enter in nano).

3.5 Create Systemd Service for ZooKeeper

This allows ZooKeeper to run as a background service.

sudo nano /etc/systemd/system/zookeeper.service

Paste the following content. Remember to replace your_username with your actual VPS username (e.g., ubuntu, debian, or root if you are operating as root, though not generally recommended for services).

[Unit]
Description=Apache ZooKeeper
After=network.target

[Service]
Type=forking
User=your_username
Group=your_username
ExecStart=/opt/zookeeper/bin/zkServer.sh start
ExecStop=/opt/zookeeper/bin/zkServer.sh stop
ExecReload=/opt/zookeeper/bin/zkServer.sh restart
WorkingDirectory=/opt/zookeeper
Restart=on-failure

[Install]
WantedBy=multi-user.target

3.6 Start and Enable ZooKeeper

sudo systemctl daemon-reload
sudo systemctl start zookeeper
sudo systemctl enable zookeeper

3.7 Verify ZooKeeper Status

sudo systemctl status zookeeper

You should see active (running) in the output.

4. Install Apache Kafka

With ZooKeeper running, you can now install Apache Kafka.

4.1 Download Kafka

Visit the Apache Kafka downloads page. Choose a stable binary release.

wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz

(Note: Example uses version 3.7.0 for Scala 2.13. Adjust as necessary.)

4.2 Extract Kafka

tar -xvzf kafka_2.13-3.7.0.tgz
sudo mv kafka_2.13-3.7.0 /opt/kafka

4.3 Configure Kafka

Edit Kafka's main configuration file: server.properties.

sudo nano /opt/kafka/config/server.properties

Key configurations to check/modify:

Save and exit the editor.

4.4 Create Kafka Log Directory

sudo mkdir -p /var/lib/kafka-logs
sudo chown -R $(whoami):$(whoami) /var/lib/kafka-logs

4.5 Create Systemd Service for Kafka

sudo nano /etc/systemd/system/kafka.service

Paste the following content. Remember to replace your_username.

[Unit]
Description=Apache Kafka
After=zookeeper.service

[Service]
Type=simple
User=your_username
Group=your_username
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
WorkingDirectory=/opt/kafka
Restart=on-failure

[Install]
WantedBy=multi-user.target

4.6 Start and Enable Kafka

sudo systemctl daemon-reload
sudo systemctl start kafka
sudo systemctl enable kafka

4.7 Verify Kafka Status

sudo systemctl status kafka

You should see active (running).

5. Configure Firewall (UFW)

If you are using UFW (Uncomplicated Firewall), you need to allow traffic on ZooKeeper and Kafka ports.

sudo ufw allow 2181/tcp   # ZooKeeper
sudo ufw allow 9092/tcp   # Kafka
sudo ufw enable           # Enable UFW if not already enabled

Run sudo ufw status to check the firewall status and rules.

6. Basic Kafka Operations (Verification)

Test your Kafka installation by creating a topic, producing messages, and consuming them.

6.1 Create a Topic

/opt/kafka/bin/kafka-topics.sh --create --topic my_first_topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

6.2 List Topics

/opt/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092

You should see my_first_topic in the list.

6.3 Produce Messages

Open a new SSH session to your VPS for this producer. Then run:

/opt/kafka/bin/kafka-console-producer.sh --topic my_first_topic --bootstrap-server localhost:9092

Type some messages, pressing Enter after each (e.g., Hello Kafka!, This is a test.). Press Ctrl+C to exit the producer.

6.4 Consume Messages

Open another new SSH session for the consumer. Then run:

/opt/kafka/bin/kafka-console-consumer.sh --topic my_first_topic --bootstrap-server localhost:9092 --from-beginning

You should see the messages you produced. Press Ctrl+C to exit the consumer.

7. ✨ Troubleshooting Assistant

Describe any issue you're encountering during the Kafka installation, and our AI assistant will provide troubleshooting tips.

Summary & Next Steps

Congratulations! You have successfully installed and verified Apache Kafka and ZooKeeper on your VPS. All the steps above should be marked as complete.

From here, you can start building applications that produce to and consume from Kafka, or integrate it with other systems. Remember to review Kafka's documentation for advanced configurations, security, and performance tuning.

Exit Lesson