Why Flume.?

Flume is not limited to collect logs from distributed systems, but it is capable of performing other use cases such as

• Collecting readings from array of sensors
• Collecting impressions from custom apps for an ad network
• Collecting readings from network devices in order to monitor their performance.
• Flume is targeted to preserve the reliability, scalability, manageability and extensibility while it serves maximum number of clients with higher QoS.

Flume was awesome to me because it is very easy to extend. The source and sink architecture is flexible and provides simpler APIs.

What is FLUME?

Flume is a distributed system that gets your logs from their source and aggregates them to where you want to process them . The primary use case is as a logging system that gathers a set of log files on every machine in a cluster and aggregates them to a centralized persistent store such as HDFS.

Apache Flume is a distributed, reliable, and available system for efficiently collecting, aggregating and moving large amounts of log data from many different sources to a centralized data store.

The use of Apache Flume is not only restricted to log data aggregation. Since data sources are customizable, Flume can be used to transport massive quantities of event data including but not limited to network traffic data, social-media-generated data, email messages and pretty much any data source possible.