2

I want each consumer to listen on their own private queue. To do this I'm using Elixir to spawn up a process for each user and subscribing that process to listen for new messages on their private user queue.

On my macbook pro with 16G of RAM, my rabbitmq admin panel shows me that I have:

File descriptors: 256 available Socket descriptors: 138 available

Apparently I cannot open up more than 138 connections.

Question 1: What is that limit based on, and am I able to raise it? I want to know how potentially how much higher that would be on a production machine (what kind of EC2 instance would be needed) and whether it's a good idea to have a connection per user. I've read that the limit might be related to ulimit, but when I run ulimit on the command line I see 'unlimited'.

I need to serve 4000 users now and will grow maybe 500-1000 users per year.

Another approach would be to make 1 tcp connection from Elixir app to Rabbitmq and use channels for each process. That works, but it requires a slightly more complicated setup. Instead of each process initializing their own connection, they would need to re-use an existing connection and open a new channel on it instead.

If the connection dies, I need to come up with a strategy for restarting a common connection, and then cascading to restart the channel for each process. I haven't figured out how to do that yet.

Q2: Is channels a better idea? Putting all the channels into one connection seems like the other extreme. Does the bandwidth of messages suffer if they are all multiplexed on the same connection?

What other strategies might be available?

Homan
  • 25,618
  • 22
  • 70
  • 107
  • 1
    See this post: http://stackoverflow.com/questions/18418936/rabbitmq-and-relationship-between-channel-and-connection#18419417 Generally, you will only want one connection for an application and maybe one channel per thread. Given that Elixir handles concurrency differently I can't really answer how you should handle configuring the number of channels. Someone else may have more experience in that area! – Paul Oct 31 '16 at 17:53
  • I recommend reviewing how the Spring AMQP code uses pools of connections and channels. In your environment, start with one connection and benchmark, then try a pool of them, spreading channels among them, and benchmark that. – Luke Bakken Mar 06 '18 at 14:43

1 Answers1

2

Is channels a better idea?

yes.

you should only have one open connection per application instance, and use channels extensively inside of that.

TCP/IP connections are slow and expensive to open and maintain.

channels are ridiculously fast, and cheap to open, do work and close as needed.

Derick Bailey
  • 72,004
  • 22
  • 206
  • 219