3

I'm attempting to use structlog in an AWS Lambda function and directly stream log events to our Logstash server. To this end, I'm using the logstash-async library using the beats transport layer.

The code:

from constants import LOGSTASH_HOST, LOGSTASH_PORT
from logstash_async.handler import AsynchronousLogstashHandler
from logstash_async.transport import BeatsTransport
from s4utils.exceptions import CatConfigNotFoundException
import structlog
from structlog.stdlib import LoggerFactory, BoundLogger

def get_log():
    logstash_handler = AsynchronousLogstashHandler(
        LOGSTASH_HOST,
        LOGSTASH_PORT,
        database_path=None,
        transport=BeatsTransport)
    logging.basicConfig(
        level=logging.INFO,
        format='%(message)s',
        handlers=[logstash_handler, logging.StreamHandler(sys.stdout)])
    structlog.configure(
        wrapper_class=BoundLogger,
        logger_factory=LoggerFactory(),
        processors=[structlog.processors.TimeStamper(fmt='iso', utc=True),
                    structlog.processors.JSONRenderer()])
    return structlog.get_logger().new(fields={'type': 'aws-lambda'})

I find that while I can see the stdout logging in CloudWatch, the Logstash server doesn't appear to be receiving events.

Tim Clemons
  • 6,231
  • 4
  • 25
  • 23
  • 1
    That seems to be less of structlog and more of a logging/logstash question? Do yoou have the same problems if you take structlog out of the equation anmd user logging directly? – hynek Dec 12 '19 at 09:22
  • Independently, structlog and logstash-async work. I just need them integrated. – Tim Clemons Dec 13 '19 at 16:24
  • I have been facing this problem as well. I believe you have the security groups fixed for this, but bringing that up if that helps anyone. It's not yet solved for me, did you get it working? – 0xc0de Nov 02 '20 at 12:51
  • Try using the structlog+logging configuration from here: https://gitlab.com/sagbot/structlog-demo/-/blob/main/demo/configure_logging.py It works for me with the console + file handlers of the stdlib logging library, you can try swapping the ```WatchedFileHandler``` handler with your logstash handler. – Sagiv Oulu Sep 03 '22 at 14:28

0 Answers0