Stack Overflow
Stack Overflow

Questions tagged [flume-twitter]

Apache Flume that connects to Twitter firehose.

Requires the consumer and access tokens and secrets of a Twitter developer account to access.

Flume-Twitter

79 questions
7
votes
1 answer

Avro Text file generated by Flume Twitter Agent not being read in Java

Not able to read and parse the File created by streaming twitter data using Flume twitter agent, neither using Java nor Avro Tools. My requirement is to convert the avro format into JSON format. When using either of the method, I get the exception :…
Ashu
  • 367
  • 5
  • 14
5
votes
1 answer

Flume not processing keywords from Twitter source with flume-ng with Hadoop 2.5 cdh5.3

I am trying to process some twitter keywords with MemChannel and HDFS. But flume-ng is not showing further progress after HDFS started status on the console. Here are /etc/flume-ns/conf/flume-env.sh file contents. # Licensed to the Apache Software…
Hadoop_Guru
  • 53
  • 1
  • 3
4
votes
1 answer

Remote debugging Flume's custom source and logging

I have a custom source for my Flume (version 1.5.0) agent and I want to debug it. It's actually custom Twitter source, from Cloudera's example here. I have a number of questions: (1) Is it possible to remote debug the Flume source (written in Java)…
oikonomiyaki
  • 7,691
  • 15
  • 62
  • 101
4
votes
3 answers

Expected timestamp in the Flume event headers, but it was null

I am using below configuration details to push Twitter feeds into HDFS using Flume, but getting Expected timestamp in the Flume event headers, but it was null twitter.conf TwitterAgent.sources = Twitter TwitterAgent.channels =…
Farooque
  • 3,616
  • 2
  • 29
  • 41
3
votes
0 answers

Flume classpath contains multiple SLF4J bindings, fetching twitter data

when fetching twitter data using command: ./bin/flume-ng agent -n TwitterAgent -c conf -f /usr/lib/apache-flume-1.4.0-bin/conf/flume.conf a warning popups in terminal saying: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding…
Anish Arya
  • 518
  • 1
  • 7
  • 24
3
votes
1 answer

Tweets data in Avro format can not be loaded

I am working on HDP (Hortonworks) and trying to collect Tweets through flume and to load stored data from Hive. The problem is select * from tweetsavro limit 1; works but select * from tweetsavro limit 2; does not work because Failed with exception…
H. Shindoh
  • 906
  • 9
  • 23
3
votes
0 answers

Apache Flume stuck with Twitter agent

When I try to stream Twitter data with Apache Flume with this command bin/flume- agent --conf ./conf/ -f conf/twitter.conf Dflume.root.logger=DEBUG,console -n TwitterAgent The console is stuck here and flume didn't stream data in the HDFS system:
yanesof__
  • 144
  • 7
3
votes
1 answer

Cloudera 5.4.2: Avro block size is invalid or too large when using Flume and Twitter streaming

There is tiny problem when I try Cloudera 5.4.2. Base on this article Apache Flume - Fetching Twitter Data http://www.tutorialspoint.com/apache_flume/fetching_twitter_data.htm It tries to fetching tweets using Flume and twitter streaming for data…
dong
  • 51
  • 1
  • 4
3
votes
1 answer

FLUME IllegalStateException: begin() called when transaction is OPEN

I have written custom flume sink, named MySink, whose process method is indicated in the first snippet below. I am getting an IllegalStateException as follows (detailed stack trace is available in the 2nd snippet below): Caused by:…
F. Aydemir
  • 2,665
  • 5
  • 40
  • 60
3
votes
1 answer

Unable to correctly load twitter avro data into hive table

Need your help! I am trying a trivial exercise of getting the data from twitter and then loading it up in Hive for analysis. Though I am able to get data into HDFS using flume (using Twitter 1% firehose Source) and also able to load the data into…
Rakesh Gupta
  • 31
  • 1
  • 3
2
votes
1 answer

How to change configuration file of Apache flume through Java code?

Iam currently working on a big data project for sentiment analysis of twitter's trending topics. I followed the tutorial of cloudera and understood how to get tweets to Hadoop through…
Mohammed Zubair Khan
  • 57
  • 3
2
votes
0 answers

Exception while streaming tweets Received fatal alert: access_denied in Flume

I currently have this configuration in Flume TwitterAgent.sources = Twitter TwitterAgent.channels = MemChannel TwitterAgent.sinks = HDFS TwitterAgent.sources.Twitter.type=…
Mohit.kc
  • 73
  • 1
  • 1
  • 7
2
votes
1 answer

UTF-8 encoding error while connecting Flume twitter stream to spark in python

I am having a trouble while passing the Twitter data collected by the Flume agent to Spark Stream. I can download the twits independently while only using the Flume. But I am getting following error. I feel that it is the issue about the default…
smm
  • 838
  • 1
  • 9
  • 31
2
votes
1 answer

How to configure apache flume for Facebook data Streaming

Could you please provide me the steps to configure Flume for Facebook Data streaming . We have successfully configured flume for extracting data from Twitter . Please have a look in the configuration that we have created for extracting data from…
Akhil PS
  • 21
  • 2
2
votes
2 answers

How to set log filename in flume

I am using Apache flume for log collection. This is my config file httpagent.sources = http-source httpagent.sinks = local-file-sink httpagent.channels = ch3 #Define source properties httpagent.sources.http-source.type =…
Avinash Sahu
  • 249
  • 3
  • 19
1
2 3 4 5 6