0

So I have a program that logs into websites using different accounts, retrieves data, and sends that data to the database.

When I run the program with one account, it runs fine with no errors and updates the database beautifully. Here is what it looks like with one username and password (roughly):

loginsList.put( "firstUsername", "firstPassword" );

for ( Entry<String, String> nextLogin : logins.entrySet() ) {
    String nextUser = nextLogin.getKey();
    String nextPass = nextLogin.getValue();
//login to the website
    Response authenticateUserResponse = Jsoup.connect( WEBSITE_I_NEED_TO_LOGIN_TO )
            .data( "username", nextUser )
            .data( "password", nextPass )
            .execute();
//then it would retrieve some data

retrieve some data

//then it would update the database

repository.save();

However, as soon as I scale it to 100 accounts (i.e. loginsList contains 100 usernames and passwords), I get a

java.net.SocketTimeoutException: Read timed out

error which is really nerving since the full scale takes 2 hours to run. Any reason why this is happening? I thought that since it works well with only one account, all the timeouts should be coordinated well. By the way, I am using spring-batch if that is of any relevance.

Salman Salman
  • 207
  • 1
  • 11
  • The bottleneck is likely with your database, not your code. What are you using for the DB? – Rome_Leader Jul 19 '16 at 14:38
  • mysql, the program does not even get to the database part when I scale it, it times out while retrieving data from the website (i.e scraping) – Salman Salman Jul 19 '16 at 14:40
  • See here then: http://stackoverflow.com/questions/985431/max-parallel-http-connections-in-a-browser There are a maximum number of simultaneous connections you can make. Maybe this is an issue? – Rome_Leader Jul 19 '16 at 14:42

0 Answers0