19

I have Unicorn, Sidekiq and Postgres setup.

I am trying to understand the right configuration to set up so that I don't hit the maximum db connection limit. In Opsworks, the m1.small Postgres RDS instance can have a max of 121 connections.

I have a db pool size of 5.

Consider this. Sidekiq and Unicorn are its own process. So the db pool size for each process is 5. Correct me if my understanding here is wrong.

If I have 5 unicorn process', that means 5*5=25 database connections

Now this is the part where I am slightly confused, since Sidekiq is multithreaded. If Sidekiq has a concurrency of 5. and the db pool size is also set to 5. Does that mean 25 potential db connections at a given time too?

This means, for one instance, I could have 50 db connections?

Christian Fazzini
  • 19,613
  • 21
  • 110
  • 215

1 Answers1

24

In Unicorn each process establishes its own connection pool, so you if your db pool setting is 5 and you have 5 Unicorn workers then you can have up to 25 connections. However, since each unicorn worker can handle only one connection at a time, then unless your app uses threading internally each worker will only actually use one db connection.

In Sidekiq, the connections in the pool are shared across threads, so you need to have at least one connection available per worker. If you have a concurrency of 5, then your pool needs to be at least 5.

Having a pool size greater than 1 means each Unicorn worker has access to connections it can't use, but it won't actually open the connections, so that doesn't matter.

The total number of actual connections your app requires, unless you're using threads in your application code (and they don't share a db connection), is one per Sidekiq worker plus one per Unicorn worker.

Louis Simoneau
  • 1,781
  • 14
  • 18
  • 5 Unicorn process and a DB pool of 5. Shouldn't it be 25 db connections? As per the answers at http://stackoverflow.com/questions/15661482/unicorn-do-not-close-db-connections – Christian Fazzini Mar 14 '14 at 12:25
  • Wow, sorry! I've just updated my answer in line with that information. Thank you. – Louis Simoneau Mar 14 '14 at 23:18
  • Just thinking. If Unicorn has 5 worker process', which means it can accept 5 web requests at a time. Why does it need 25 db connections? Shouldn't it need only 5? – Christian Fazzini Mar 15 '14 at 03:23
  • Another good point. You'd only need more than one per worker if you were using threads in your application code and they don't share a connection. However, your pool size has to be big enough to handle your Sidekiq concurrency (since the threads share a pool) anyway, and having it higher than Unicorn can make use of doesn't matter. I've updated and clarified my answer, and also removed some superfluous information that wasn't pertinent to the question. – Louis Simoneau Mar 15 '14 at 09:56
  • Yea that analysis seems about right. I have 7 instances running on AWS Opsworks. AWS RDS reports 35-40 (don't know what the other 5 are) db connections. 7*5=35. That explains unicorns part. The db connections from Sidekiq must be idle/sleeping since there are no jobs running at the moment – Christian Fazzini Mar 17 '14 at 03:00
  • @LouisSimoneau thanks for the great explanation. I have problems understanding this sentence: "Having a pool size greater than 1 means each Unicorn worker has access to connections it can't use, but it won't actually open the connections, so that doesn't matter." That would mean I can set the pool to 5 but unicorn will always use the first opened. I think this is not correct but assume you just wrote it a bit unclear. Could you clarify this? Highly appreciated ;-). Thanks – awenkhh Mar 03 '15 at 15:36