The Kafka log appender is responsible for transferring logs from the Operations server to the Apache Kafka service. The logs are stored in the specified topic.
The easiest way to create a Kafka log appender for your application is by using Admin UI.
It is also possible to create a Kafka log appender for your application by using REST API. The following example illustrates how to provision the Kafka log appender for the Cell Monitor demo application available in Kaa Sandbox.
The Kafka log appender configuration should match the following Avro schema.
|kafkaServers||list of kafka bootstrap servers (hostname and port pairs)|
|topic||logs destination topic|
|useDefaultPartitioner||if false, appender will calculate partition independently|
|partitionCount||amount of event partitions|
|kafkaKeyType||type of generated message key|
|executorThreadPoolSize||number of threads that can simultaneously perform operation with your Kafka|
|bufferMemorySize||message buffer size in bytes|
|kafkaCompression||type of built-in message compression types|
|kafkaAcknowledgement||the number of acknowledgments the producer requires the leader to have received before considering a request complete|
|retries||failover property. Amount of connection retries on failed message delivery|
The following configuration taken from the Cell Monitor demo matches the previous schema.
The following REST API call example illustrates how to create a new Kafka log appender.
Kafka must be installed, running and reachable from Kaa to complete this example. For details about Kafka installation refer to official Apache documentation.
We have next log schema:
The following JSON example matches the previous schema.
"level" : "KAA_INFO",
"tag" : "TEST_TAG",
"message" : "My simple message"
Go to Data collection demos in Sandbox.
Follow Installation instructions.
Next, in the Admin UI follow to Data collection demo application
Go to application's Log appenders configuration and add a new one.
Enter name of the new appender (in this example it is "Kafka")
Select Kafka appender type.
Set up appender Configuration similar to screenshot
In this example, Kafka server installed in the Sandbox VM.
Now click Add button on the top of the screen to create and deploy appender.
Verify that newly created appender has appeared in list.
From Kafka installation directory run next command:
This will bring up Kafka consumer, so we can see logs transferred from Kaa.
Now run Data collection demo application. Verify that logs have been successfully sent to Kaa
Make sure, that Kafka consumer receive logs: