read tcp stream with java while writing to database

Question

I am not sure what the best solution for following situation is:

My Java-program is permanently reading from a tcp-stream. At the same time it is neccessary to persist these data to a database. The amount of data which should be written to the database could differ

I have read a lot of about message-queueing-systems and so on. In detail, my solution would consider using a LinkedBlockingQueue. Thereby, there are two threads: a) Starting a producer-threat which will perform reading from the tcp-stream b) Starting a consumer-threat which will write the (parsed) data from the stream to the database

The (example-)code looks like following:

Main.java

public static void main(String[] args) {
        LinkedBlockingQueue queue = new LinkedBlockingQueue(50);
        Producer producer = new Producer(queue);
        Consumer consumer = new Consumer(queue, producer);

Produer.java

public class Producer implements Runnable {
  private LinkedBlockingQueue queue;
  private boolean running;

  public Producer(LinkedBlockingQueue queue) {
        this.queue = queue;
        running = true;
    }

 @Override
  public void run() {
    //read TCP-Stream here and save parsed messages to queue
  }
  public boolean isRunning() {
        return running;
  }

Consumer.java

public class Consumer implements Runnable {
    private Producer producer;
    private LinkedBlockingQueue queue;

    public Consumer(LinkedBlockingQueue queue, Producer producer) {
        this.queue = queue;
        this.producer = producer;
    }
   @Override
   public void run() {
       //insert data into database here
       if(producer.isRunning()) {
          //while producer is running, data needs to be inserted to database
       }
   }

Is this a solution you would recommend to use? Or do you know better solutions?

Thank you!

Yes, that's pretty much a standard approach. You might also have a look at the ExecutorService. Duplicate of [Producer/Consumer threads using a Queue](http://stackoverflow.com/questions/2332537/producer-consumer-threads-using-a-queue) (as the code is not specific to tcp/database). — Martin Nyolt, Aug 24 '16 at 14:19
I don't see a reason for two threads. I would do it all in one. There's no gain. — user207421, Aug 25 '16 at 01:22

peterh · Accepted Answer · 2016-08-24T21:08:15.507

2

Your own suggestion is very good.

Ultimately what you are trying to solve is the problem of back pressure, i.e. if you are receiving data faster than you can write them to database. This may happen just because there's a lot of data arriving or simply because your destination is temporarily unavailable. Either way it is a situation you need to deal with.

In your proposed solution this is handled by an in-memory staging area (= your queue). As long as you have enough memory and you are not too concerned about loosing data on power outages then the in-memory strategy will work fine for you. A burst will be absorbed by growing memory in your Java application. Not a problem per se but remember that when your queue is finally drained the JVM GC will kick in and release the memory again from the JVM heap. But from the outside, i.e. from the point of view of the OS, the memory will possibly never be released. The JVM is very, very conservative about releasing memory back to the OS. Again, in most cases this is not a problem.

If you have more strict needs then you need to consider a more "robust" staging area instead of RAM, for example local disk. From my experience your proposed solution fits 95% of use cases.

edited Aug 24 '16 at 21:08

answered Aug 24 '16 at 15:12

peterh

18,404
12
87
115

TCP will handle back pressure automatically by exerting its own on the sender. There's no need for the application to get involved in this. – user207421 Aug 25 '16 at 01:24
@EJP. I just have a wider definition of "back pressure" than you, not solely related to low level network stuff. – peterh Aug 25 '16 at 12:00
No, we have the same definition, I just don't see any need for the application to get involved in it. If you have two threads and a queue, you will have to bound the queue so as not to run out of memory, so you will eventually get TCP back-pressure anyway, and in the meantime you've accomplished nothing useful, you've just read a lot of data you haven't processed yet. What's the point? – user207421 Aug 25 '16 at 22:10
@EJP. OP's design will handle *intermittent* outages on the destination side. That's the point. And that's truly worth the effort. Also: you cannot assume that the TCP sender will be able to handle a slow consumer just by being able to forever grow its own sender-side buffers. It's better to design the recv application in such a way so that it reads from the socket as fast as possible and then have its own buffer (eg a queue). Just my two cents. – peterh Aug 26 '16 at 13:19

read tcp stream with java while writing to database

1 Answers1