Stack Overflow
Stack Overflow

Questions tagged [flume-ng]

Apache Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. The Flume-NG is refactoring of the first generation Flume to solve certain known issues and limitations of the original design.

This tag should be used with questions about Flume-NG API and specific features of new-generation versions (e.g. Flume HDFS Sink was introduced only in NG version and cannot be used in previous releases).

397 questions
13
votes
6 answers

failing to load log4j2 while running fatjar

i am working on a project where i utilize log4j2 logging. while developing in intellij, all works fine and the logging is done as expected. the log4j2.xml is linked through java property passed to jvm on startup via intellij settings. but once i…
atarno
  • 329
  • 1
  • 3
  • 14
11
votes
1 answer

How to configure Flume to listen a web api http petitions

I have built an api web application, which is published on IIS Server, I am trying to configure Apache Flume to listen that web api and to save the response of http petitions in HDFS, this is the post method that I need to listen: [HttpPost] …
MelgoV
  • 661
  • 8
  • 21
11
votes
2 answers

Apache Flume vs Apache Flink difference

I need to read a stream of data from some source (in my case it's UDP stream, but it shouldn't matter), transform the each record and write it to the HDFS. Is there any difference between using Flume or Flink for this purpose? I know I can use…
Kateryna Khotkevych
  • 1,248
  • 1
  • 12
  • 22
9
votes
0 answers

org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 30662099 larger than 30662028)

I am trying to push data from Flume channels to Kafka cluster using Kafka sink and i can see related data into associated topic but simultaneously i am observing below mentioned exception trace in Kafka logs too frequently , [2017-03-21…
Ritesh Sharma
  • 163
  • 1
  • 3
  • 10
7
votes
1 answer

Avro Text file generated by Flume Twitter Agent not being read in Java

Not able to read and parse the File created by streaming twitter data using Flume twitter agent, neither using Java nor Avro Tools. My requirement is to convert the avro format into JSON format. When using either of the method, I get the exception :…
Ashu
  • 367
  • 5
  • 14
6
votes
1 answer

Flume HDFS sink: Remove timestamp from filename

I have configured flume agent for my application, where source is Spooldir and sink is HDFS I am able to collect files in hdfs. agent configuration is: agent.sources = src-1 agent.channels = c1 agent.sinks = k1 agent.sources.src-1.type =…
Sandeep Reddy
  • 61
  • 3
6
votes
3 answers

How to insert JSON in HDFS using Flume correctly

I am using the HTTPSource in Flume for receiving POST events in json format as follows: {"username":"xyz","password":"123"} My question is: Do I have to modify the source of the events (I mean the one that is sending the JSON to the Flume) so the…
nanounanue
  • 7,942
  • 7
  • 41
  • 73
5
votes
2 answers

Usable space exhausted in flume using file channel

I’m working on Flume with Spool Directory as the Source,HDFS as sink and File as channel. When executing the flume job. I’m getting below issue. Memory channel is working fine. But we need to implement the same using File channel. Using file…
sarath endluri
  • 53
  • 4
5
votes
2 answers

log4j2- ERROR Appenders contains an invalid element or attribute "Flume"

I am trying to use Flume Appender Properties of log4j2 .But the following errors are obtained when run the program . 2016-01-20 16:36:42,436 main ERROR Appenders contains an invalid element or attribute "Flume" 2016-01-20 16:36:42,436 main…
Rabindra Nath Nandi
  • 1,433
  • 1
  • 15
  • 28
5
votes
1 answer

Flume not processing keywords from Twitter source with flume-ng with Hadoop 2.5 cdh5.3

I am trying to process some twitter keywords with MemChannel and HDFS. But flume-ng is not showing further progress after HDFS started status on the console. Here are /etc/flume-ns/conf/flume-env.sh file contents. # Licensed to the Apache Software…
Hadoop_Guru
  • 53
  • 1
  • 3
5
votes
2 answers

How to Retain file name when tranferring file in Apache Flume?

I am using Flume 1.3.1 ng, I am transfering a file from spoolDir to HDFS Sink and I need outpufile name same as input file. For example if input file name is sample.gz then output also need as sample.gz Could you please provide the procedure how…
kasi
  • 51
  • 2
5
votes
4 answers

flume - flume.root.logger=DEBUG,console only logs INFO level log statements

I installed Flume 1.4.0-cdh4.7.0 in CentOS (cloudera VM) I ran the following command to start the flume flume-ng agent -n agent-name -c conf -f conf/flume.conf -Dflume.root.looger=DEBUG,console but it is only writing the default (INFO) level to the…
scott
  • 235
  • 4
  • 12
4
votes
2 answers

Flume - HiveSink: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/streaming/RecordWriter

I am trying create a hive sink in my flume configuration and when I run flume-ng I get some dependency problems as shown below. Could you tell me how can I overcome this dependency? It looks there is some runtime dependency. I have installed hive…
F. Aydemir
  • 2,665
  • 5
  • 40
  • 60
4
votes
2 answers

Flume agent: add host to message, then publish to a kafka topic

We started to consolidate eventlog data from our applications by publishing messages to a Kafka topic. Although we could write directly from the application to Kafka, we chose to treat it as a generic problem and use the Flume agent. This provides…
Alex Woolford
  • 4,433
  • 11
  • 47
  • 80
4
votes
1 answer

Flume use case: reading from HTTP and push to HDFS via Kafka

I'm new to Flume and thinking to use Flume in the below scenario. Our system receives events as HTTP POST, and we need to store a copy of them in Kafka (for further processing) and another copy in HDFS (as permanent store). Can we configure Flume…
Hemanth Abbina
  • 99
  • 7
1
2 3
…
26 27