5

I’m working on Flume with Spool Directory as the Source,HDFS as sink and File as channel. When executing the flume job. I’m getting below issue. Memory channel is working fine. But we need to implement the same using File channel. Using file channel I’m getting below issue.

I have configured the JVM memory size to 3GB in flume.env.sh file. Please let me know any other settings we need to do.

20 Jan 2016 20:05:27,099 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver event. Exception follows. java.lang.IllegalStateException: Channel closed [channel=Artiva-memory-channel]. Due to java.io.IOException: Usable space exhausted, only 427515904 bytes remaining, required 524288000 bytes

sarath endluri
  • 53
  • 4

2 Answers2

1

File channel has nothing to do with memory but with HDD (disk). Such a channel uses the file system for storing the data. Thus, check how much free space is available in the disks where the checkpoint file and data files are written (please, have a look on the FileChannel parameters).

frb
  • 3,738
  • 2
  • 21
  • 51
  • Hi..Thanks for the reply...i have given minimumRequiredSpace=524288000 ...but still i am getting the same issue... – sarath endluri Jan 25 '16 at 02:15
  • 1
    Because you are wanting to configure 524288000 bytes the configuration is failing. The log you posted said that only 427515904 bytes are available in your disk. Thus, you must configure 427515904 bytes or less in order it may work. – frb Jan 25 '16 at 07:17
  • @frb I get java.io.IOException: Usable space exhausted, only 0 bytes remaining, required 524288000 bytes. error says 0 bytes remaining but ı controlled there is 777 priviledge on folder and has a lot of space for it. How can I fix? – neverwinter Mar 15 '16 at 07:38
  • how to find where checkpoint file and data files are written? And in which configuration file we set the minimumRequiredSpace? – Ajay Mar 21 '18 at 10:24
0

The ERROR message is about the Memory Channel. See "Channel closed [channel=Artiva-memory-channel]".
Check what channel is assigned to the HDFS sink in question. It would be in flume.conf file, a property like: agent_name.sinks.hdfs_sink_name.channel

Viren
  • 170
  • 1
  • 10