Kaa releases
Skip to end of metadata
Go to start of metadata

Kaa can append logs to Apache Storm by using the Flume log appender and a custom Spout with the ability to receive Flume events.

In this instruction, we use our Apache Storm sample application code as an example.

Flume appender to Storm architecture 

Storm application

In this section, we create a Storm processing application that will consume data from the Flume log appender. Note that only certain code blocks are highlighted; the complete source code of this example is available at the corresponding sample application.

Creating Storm application

Our application receives log data transferred by the Flume log appender as a Flume event from the Kaa clients and then prints that data. 

To learn more about Storm basics, such as a spout, a bolt, etc., please refer to the official Apache Storm tutorial.

The “…” in the listing stands for skipped lines of code, which you can find at GitHub.

Setting up topology

Storm configuration file

Setting up spout

Setting up bolt

Running Storm application

Put the log schema (*.jar file) library (you can download it from the Admin UI) into the Storm classpath (commonly it's ${STORM_HOME}/lib folder).

Then, build and run the topology.

Setting up Flume log appender

Setting up a Flume log appender is covered in detail on the Flume log appender page. In this section, we review some specific steps.

  1. As a tenant admin, go to your application >> Log appenders, then click Add log appender.
  2. In the Add log appender window that opens, fill in the required fields.
    In our example, we use the name StormDemoAppender.

    In the Type drop-down list, select Flume.

    Then, specify the fields Flume event format (we selected Records container), and Hosts balancing (we selected prioritized).

    Finally, specify the Storm cluster parameters: host, port and priority. We host it at localhost:10001.

  3. To finish, click Add at the top of the window.
    In case of success, you will see your new log appender in the log appenders list.


To sum up, all you need to start appending logs to Storm is to configure Flume log appender and use a custom Spout.

Copyright © 2014-2016, CyberVision, Inc.

  • No labels