stdin blocks after a large amount of rows

Question

I have a python process and a java process.

import subprocess as p 

p.Popen(['java','jar', 'some jar'], stdin = p.PIPE, stdout = p.PIPE) 
#takes in data and repeat...
p.stdin.write('some string that is passed to the java jar \n')
p.stdin.flush() 
output = p.stdout.readline()

Java Program

While (True){
Scanner(System.in);
//do some processing 
System.out.println('processed');}

This works well under 100 rows of data

However, when i get to more than 100 rows. The stdin seems to be waiting for input.

I would have to forcefully end the program by entering ctrl c and the last line of the exception would be at stdin.write(txt). However, i have been flushing the buffer so i do not understand what is the issue. I have about 1000 rows and if i were to split them into 10 sets of 100 running 10 times. It works but if i were to run 1000 rows at once it will not work. This is definitely not a data issue.

This is a follow up on my question

Popen communicate vs Popen stdin.write()

Situation

I have a large number of text documents.

These will be processed by python into individual strings then passed to a java engine for processing.

The workflow is like this.

Read files into python and process. Write the output from python into stdin. Read from stdin in Java. Output to stdout in Java. Read back the input in python.

This works well for the first 100 files and it will hang at around the 120th file.

So the workaround is to run these files in batches of 100.

Like running it from 0 -100 and 100 - 200 and so on.

However, i would like the loop to be from 0 to 1000 instead.