Kinesis agent not sending .log files through firehose

Question

I've setup a Kinesis firehose and the installed the Kinesis agent as described in the AWS docs. I can get test data through to the S3 bucket, but the Kinesis agent won't send any .log files through. I suspect a problem connecting the agent to the firehose.

My /etc/aws-kinesis/agent.json file is below. I've also tried with the "firehose.endpoint" without the https:// but I still can't get any data through.

I've verified that the aws-kinesis-agent service is running.

I'm not using the kinesis.endpoint/kinesisStream, but I've left the flow in the agent.json file. Could this be a problem?

What am I missing?

{
  "cloudwatch.emitMetrics": true,
  "kinesis.endpoint": "",
  "firehose.endpoint": "https://firehose.us-west-2.amazonaws.com",

  "flows": [
    {
      "filePattern": "/home/ec2-user/src/Fake-Apache-Log-Generator/*.log*",
      "kinesisStream": "yourkinesisstream",
      "partitionKeyOption": "RANDOM"
    },
    {
      "filePattern": "/home/ec2-user/src/Fake-Apache-Log-Generator/*.log*",
      "deliveryStream": "apachelogfilesdeliverystream"
    }
  ]
}

EDIT:

The log file at /var/log/aws-kinesis-agent/aws-kinesis-agent.log showed 0 records being parsed. The log message led me to this post, and I made the recommended fixes. In addition I had to remove the flow for kinesis from the /etc/aws-kinesis/agent.json file to avoid an Exception that showed up in the log files.

Botton line is that the aws-kinesis-agent can't read files from /home/ec2-user/ or its subdirectories, and you have to fix up the agent.json file.

Answer 1

Kinesis agent is not able to read the logs from a file which is at /home/ec2-user/<any-file> due to some permissions issue. Try changing your logs location to /tmp/logs/<log-file>.

Answer 2

Add the kinesis agent to the sudoers group:

sudo usermod -aG sudo aws-kinesis-agent-user

Another possibility is data flow, see this answer: https://stackoverflow.com/a/64610780/5697992