Apache Kafka is a fast, scalable, durable, and distributed messaging system. It is useful in use-cases where there are a huge number of messages to be processed. This could be user activity tracking, operational metrics, log data, and so on.
Here is how you can use apache Kafka with Node.js,
1. First, Install Apache Kafka on your machine:
- To use Apache Kafka, you need to download it from the official website. After downloading, extract it to a folder.
- Navigate to the folder using terminal and then to the sub-folder kafka\_2.11-0.9.0.0.
- To start Zookeeper Service, use the command `bin/zookeeper-server-start.sh config/zookeeper.properties`
- To Start Kafka Service, use the command `bin/kafka-server-start.sh config/server.properties`
- Create a Kafka topic, use the command `bin/kafka-topics.sh —create —zookeeper localhost:2181 —replication-factor 1 —partitions 1 —topic test`
- To send some messages, use the command `bin/kafka-console-producer.sh —broker-list localhost:9092 —topic test` and type some messages.
- To receive the messages, use the command `bin/kafka-console-consumer.sh —zookeeper localhost:2181 —topic test —from-beginning`
1. Setting Up Node.js:
- Make a new folder and navigate to it via your terminal.
- Initialize a new project using the command `npm init`.
- You need to install a Kafka Node.js module, which can be installed via npm using the command `npm install kafka-node`.
1. Producer:
In order to send messages to Kafka, you have to create a producer. Here’s some sample code:
```
var kafka = require(‘kafka-node’),
Producer = kafka.Producer,
client = new kafka.KafkaClient(),
producer = new Producer(client);
payloads = [
{ topic: ‘test’, messages: ‘hi’, partition: 0 }
];
producer.on(‘ready’, function () {
producer.send(payloads, function (err, data) {
console.log(data);
});
});
producer.on(‘error’, function (err) {})
```
This Node.js app will wait for the producer to be ready. As soon as it’s ready, it will send a ‘hi’ message to Kafka to the ‘test’ topic.
The producer.send() function, sends the payloads (messages) to Kafka. This function, takes two arguments, payloads and a callback function. The callback function, will get two arguments err and data.
1. Consumer:
To receive the messages, you’ll need a consumer.
```
var kafka = require(‘kafka-node’),
Consumer = kafka.Consumer,
client = new kafka.KafkaClient(),
consumer = new Consumer(
client,
[
{ topic: ‘test’, partition: 0 }
],
{
autoCommit: false
}
);
consumer.on(‘message’, function (message) {
console.log(message);
});
consumer.on(‘error’, function (err) {})
```
This Node.js app will listen for new messages and log them to the console.
Please note that, messages that are sent to Kafka, are sent as streams. The consumer.on(‘message’, function (message) {}); will listen for new messages that are posted to the topic.
The Kafka with Node.js setup is now ready to handle real-time data feed and processing for analytical applications, perform extraction and load into data systems, execute distributed processing, etc. Make sure both applications have error handling code to deal with network failures.
