5

Help please solve the problem. There are: RabbitMQ - 3.7.2 Erlang - 20.1

  • Connections: 527
  • Channels: 500
  • Exchanges: 49
  • Queues: 4437
  • Consumers: 131
  • Publish rate ~ 200/s
  • Ack rate ~ 200/s

Config:

disk_free_limit.absolute = 5GB
log.default.level = warning
log.file.level = warning

In the logs constantly appear such messages:

11:42:16.000 [warning] <0.32.0> lager_error_logger_h dropped 105 messages in the last second that exceeded the limit of 100 messages/sec
11:42:17.000 [warning] <0.32.0> lager_error_logger_h dropped 101 messages in the last second that exceeded the limit of 100 messages/sec
11:42:18.000 [warning] <0.32.0> lager_error_logger_h dropped 177 messages in the last second that exceeded the limit of 100 messages/sec

How to get rid of them correctly? How to remove this messages from logs?

Alexander Sokolov
  • 51
  • 1
  • 1
  • 2

1 Answers1

7

The RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.

The message means that RabbitMQ is generating a very large number of error messages and that they are being dropped to avoid filling the log rapidly. If "dropped X messages in the last second" is the only message you are seeing in the logs, you need to determine what the messages are that are being dropped to find the root of the problem. You can do this by temporarily raising that limit by running the following command:

rabbitmqctl eval '[lager:set_loghwm(H, 250) || H <- gen_event:which_handlers(lager_event)].'

You should then see a much larger number of messages that will reveal the underlying issue. To revert back to the previous setting, run this command:

rabbitmqctl eval '[lager:set_loghwm(H, 50) || H <- gen_event:which_handlers(lager_event)].'
Luke Bakken
  • 8,993
  • 2
  • 20
  • 33
  • According to https://github.com/erlang-lager/lager/issues/496 the logger counts *all* messages toward this limit, not just the error ones (or ones above the minimum log level for that matter). – dskrvk Apr 29 '19 at 15:19
  • Yep, that's a bug that will be addressed soon. I have to fix the CI build for `erlang-lager`. My original answer is still correct, however. – Luke Bakken Apr 29 '19 at 20:45
  • @LukeBakken I can't raise the limit by `rabbitmqctl eval '[lager:set_loghwm(H, 250) || H <- gen_event:which_handlers(lager_event)].'`. Get `{ok,50}` by `docker exec rabbitmq rabbitmqctl eval 'application:get_env(lager, error_logger_hwm).'`. – Victor Lee Apr 24 '22 at 01:55