1

I'm seeking algorithm to design multithreading application in java where each thread write data to db. I want to control the number of records written to db per second.

the application gets RPS (records per second) param and approximately generate those many load.

Java Spring Coder
  • 1,017
  • 3
  • 12
  • 20
  • 2
    What work have you done? What research? What have you tried? – Gray Feb 26 '13 at 19:28
  • 2
    various ideas here: http://stackoverflow.com/questions/14638349/java-thottling-mechanism – irreputable Feb 26 '13 at 19:28
  • 5
    Jesus, how is this not a real question? Sure the question is not very well defined, but that's the flaw of all questions. It is part of the QA process to discover the question. This particular question is pretty reasonable, readers understand basically what is being sought after, and readers can present helpful ideas. – irreputable Feb 26 '13 at 20:16
  • 1
    strange to see people talking about that the question was not precise. do u guys expect me to add garbage info in it. my question was "a db writer app wants to do rate limiting on it". isn't it simple for u guys to understand? or want me to add couple more paragraphs for the same question? sometimes simple questions are hard to grasp for highly educated PHDs :-) – Java Spring Coder Feb 26 '13 at 21:46
  • No - we/they do not expect you to add garbage to the question - and please don't get angry at "us" about this (it doesn't help anybody, least of all yourself). In the context of your question, think about this like an exam. Your question is an essay question - it calls for a long explanation. S/O answerers prefer short-answer questions... if there's any way that you can *reword* your question to make it a shorter-answer question - that's what is the preferred format, because it makes it easier for us to answer. – Taryn East Feb 26 '13 at 22:43
  • Also, you have not shown us that you have done any work yourself to solve this problem yourself. You probably have done plenty of work, but you haven't *shown* it by putting it into the question... and we can only go by what you've actually written down. and by what you've written, it looks like you haven't done anything... which means you've already put "us" off-side, as it *seems* like your asking us to do your work for you. This is a (small) failing on your part for not telling us what you've done. So I'd recommend you add your work to future questions. – Taryn East Feb 26 '13 at 22:47
  • ok. got it. I did not do anything as I was just starting to design. there were few idea on my mind but I thought those were not worth to mention as they just initial thoughts. btw I know and I was not asking to do the work for me (there is rentacoder for it and SO is not suited for this). I wanted to know some hint, some starting point. thanks for your inputs and will put more details in my future questions. – Java Spring Coder Feb 27 '13 at 00:45
  • http://stackoverflow.com/questions/6271409/limiting-upload-speed-on-java – kervin Aug 09 '15 at 16:23

2 Answers2

4

If you can use external libraries, guava has a RateLimiter class which probably does what you ask for:

Rate limiters are often used to restrict the rate at which some physical or logical resource is accessed. This is in contrast to Semaphore which restricts the number of concurrent accesses instead of the rate.

assylias
  • 321,522
  • 82
  • 660
  • 783
  • this solution may waste opportunities - if I'm allowed to drink 2 beers a day, and I didn't drink any in the first 16 hours, I'll lose 1 beer. it's unclear whether that's good or bad for OP's problem. – irreputable Feb 26 '13 at 19:37
  • @irreputable His requirement is not very precise - this might or might not be a good solution. But it would be one way *to control the number of records written to db per second*. – assylias Feb 26 '13 at 19:38
  • 1
    thanks Assylias. it helps. people who have closed my question saying its not accurately defined, should get away from visiting this thread. – Java Spring Coder Feb 26 '13 at 21:56
  • If what @irreputable describes turns out to be a problem, then users should comment on this at the guava mailing list. There is a google-internal RateLimiter method that solves this; if there is user demand, it is entirely possible that that gets exposed – Dimitris Andreou Jul 06 '13 at 10:39
  • @irreputable RateLimiter.create(permitsPerSecond) resolves the 2-beers-a-day issue, if you can live with the limitation of counting per-second instead of per-day. With a minor change (doesn't seem to be intended for extension) granularity could be made configurable. – Stefan L Aug 02 '13 at 08:45
  • "Token Bucket" implementation https://github.com/bbeck/token-bucket – kervin Aug 09 '15 at 16:50
1

You could create a BlockingQueue of permit objects and have an extra helper thread that adds RPS number of additional permits to the queue once per second. Writing threads block on the queue and do not write until they successfully pop a permit off the top of the queue.

Affe
  • 47,174
  • 11
  • 83
  • 83
  • it's possible to accumulate permits during idle time, then release them all within a second. it does work on average over long time. the problem is often the requirement isn't precise. – irreputable Feb 26 '13 at 19:31
  • There are lots of features you can add, max size, things to spread the load more evenly over a second, lots of ideas. Question wasn't that specific though! – Affe Feb 26 '13 at 19:34