0

I have a spring boot application running on 3 containers. I need to design a service which read from the database every 5 minutes and then perform some calculation over the data.

I can write a function using @schedule annotation to run every 5 minutes. But how can I make sure that all containers are not processing on same data. Is there a way, I can distribute the load to all containers?

Support there are 1800 records in the database on which I need to perform calculation every 5 minutes. So each container can process 600 records every 5 minutes and at any point of time if one container go down then other two can manage the load.

Vikram Jain
  • 376
  • 2
  • 12

1 Answers1

1

One way of doing this is by using SKIP LOCKED. You can find a great explanation on how do to this at The best way to use a DB table as a job queue (a.k.a batch queue or message queue).

Kaj Hejer
  • 955
  • 4
  • 18
  • This may lock the rows when some other operations want to fetch data at same time. So application have various services which are running all together. One of them will be as i explained in problem, another can be simple GET service which return data from DB. So if SKIP LOCKED is used, it might impact other services. – Vikram Jain Mar 31 '21 at 11:30
  • I think the data is available for select for other operations but if the locking is a problem one possible workaround could be to lock as few as possible rows at the time. We do this with locking one and one row. If that not possible for you, would it be an option to duplicate the info you need for the calculations in another table and do the locking in that table? – Kaj Hejer Mar 31 '21 at 11:59
  • If I copy the entire table to another table, then I need to make sure to write to both the tables every time I get a new record and could be another maintenance issue. This may work, but it's more of a hacky solution. – Vikram Jain Mar 31 '21 at 15:37