0

I have a method which sends a Request and gets a Response back from a Web Site.

The response changes every 4 hours.

I have the same Java Code on two Computers.

  • On the first one the Response is still the same as 24 hours ago, even now.
  • Edit : The second one have also an old (cached) response but a day later (when I first started the Programm).

If I try to do the same Request/Response over this website (https://apitester.com/) it gives me each time the expected result. However my program does not.

I'm using the same settings everywhere such as the website url and User Agent.

So how can I force a new, fresh Version of the Response? (Cloudflare maybe caching the site for a once used ip adress ?)

Edited Version (still do not work):

 String url = "WEBSITE_PLACEHOLDER";
        final String USER_AGENT = "AGENT_PLACEHOLDER";
        URL obj = new URL(url);
        HttpURLConnection con = (HttpURLConnection) obj.openConnection();

        // add reuqest header
       con.setRequestProperty("Cache-Control", "no-cache");
       con.setRequestProperty("Cache-Control", "must-revalidate");
       con.setUseCaches(false);    
       con.setRequestMethod("POST");
       con.setRequestProperty("User-Agent", USER_AGENT);
       con.setRequestProperty("Accept-Language", "en-US,en;q=0.5");

        // Send post request

        int responseCode = con.getResponseCode();

            if(responseCode == 200) {    
            BufferedReader in = new BufferedReader(new 
            InputStreamReader(con.getInputStream()));
            String inputLine;
            StringBuffer response = new StringBuffer();

            while ((inputLine = in.readLine()) != null) {
                response.append(inputLine);
            }
            in.close();

            //System.out.println(response.toString());
            return response.toString();

            }
            else {
                throw new Exception();
            }
        }

@Mike - Runned Header and Body Response on Both Systems

1st Computer (getting even now only the old response Body

I will marked the needed token with "This is what i get and This is what i need"

Printing Response Header...

Transfer-Encoding ,Value : [chunked]

null ,Value : [HTTP/1.1 200 OK]

Strict-Transport-Security ,Value : [max-age=5184000]

CF-RAY ,Value : [46bb5ec486df978c-FRA]

Server ,Value : [cloudflare]

Connection ,Value : [keep-alive]

Vary ,Value : [Accept-Encoding]

Set-Cookie ,Value : [__cfduid=d77d33fbc1ca290cd37b93c3eacff74c31539869538; 
expires=Fri, 18-Oct-19 13:32:18 GMT; path=/; domain=."website"; HttpOnly]

Date ,Value : [Thu, 18 Oct 2018 13:32:18 GMT]

Content-Type ,Value : [application/json; charset=utf-8]
Server - cloudflare


Get Response Body: 

{"response":{"token":"26557695w66YornObzClheLDDBTd","ruleset":"guest","nextPing":562312,


//This is what i get 
    "signed":"**eyJkYXRhIjoie1widGltZVwiOjE1Mzk4Njk1MzgwNTAsXCJ2YWxpZFVudGlsXCI6MTUzOTg3MDEzODA1MCxcImlwc1wiOltcIjkwLjE4Ny4zNS45M1wiXSxcInJ1bGVzZXRcIjpcImd1ZXN0XCIsXCJ2ZXJpZmllZFwiOmZhbHNlLFwiZXJyb3JcIjpudWxsLFwiYXBwXCI6e1wicGxhdGZvcm1cIjpudWxsLFwidmVyc2lvblwiOm51bGwsXCJzZXJpdmNlXCI6bnVsbH19Iiwic2lnbmF0dXJlIjoicEhsWllHcHNOQnZFVTZ1N1BWQk5uZmo4RDlpZlV5L2RneWx3TDcrdjE3NDNMQndPSGpRZEFWRkdLNmNqSCt2amtBTVd6eWZtZUh4TXN1ZWttdXJEQ3NPNXVtN0FjM0RKSFNneDFzRmZNN2ZORFJRSkY5TFpHSmJBNnEvSWtyQ0NEM01sdlpjTlNic3lhT3FGbDlZNWVaZXpHTHd1REZoUjVkZ0QyRFFaK3RRPSJ9**"}}

2nd Computer (same Programm, new Response)

Printing Response Header...

Transfer-Encoding ,Value : [chunked]

null ,Value : [HTTP/1.1 200 OK]

Strict-Transport-Security ,Value : [max-age=5184000]

CF-RAY ,Value : [46bb5a3f1427234e-FRA]

Server ,Value : [cloudflare]

Connection ,Value : [keep-alive]

Vary ,Value : [Accept-Encoding]

Set-Cookie ,Value : [__cfduid=d0ba210989762b597b0149f82d524ee371539869352; expires=Fri, 18-Oct-19 13:29:12 GMT; path=/; domain="website"; HttpOnly]

Date ,Value : [Thu, 18 Oct 2018 13:29:12 GMT]

Content-Type ,Value : [application/json; charset=utf-8]

Server - cloudflare


Get Response Body: 

{"response":{"token":"265576292ukGhcUmhtl8rDBrDJtV","ruleset":"guest","nextPing":834688,


//This is what i need

"signed":"eyJkYXRhIjoie1widGltZVwiOjE1Mzk4NjkzNTI4ODAsXCJ2YWxpZFVudGlsXCI6MTUzOTg2OTk1Mjg4MCxcImlwc1wiOltcIjg3LjEyMy4xMi4yNDNcIl0sXCJydWxlc2V0XCI6XCJndWVzdFwiLFwidmVyaWZpZWRcIjpmYWxzZSxcImVycm9yXCI6bnVsbCxcImFwcFwiOntcInBsYXRmb3JtXCI6bnVsbCxcInZlcnNpb25cIjpudWxsLFwic2VyaXZjZVwiOm51bGx9fSIsInNpZ25hdHVyZSI6IkllRSsrN01GblNheE9VWjZDdWtsU2lUb1RCcFI3VTZQZWJyNlE5UU1jR3ZzSExoeDdZWGFmSytreGJqVzNMSnVCb0hMM3hGMWpHQjVuVWtiRzh2Yys3dkMrR3hBckRETnkvMmlpbXhaa3I4Yk1OZE5QQThIamxFR0FON2V5UlQ4VWNGR05ES1QwaUVtQ1dJVXZiZjhJbW5xaStDR0QwUC9LR3RDYnFIVGFVYz0ifQ=="}}

xAllj
  • 1
  • 5
  • Is the program persistent, or does it shut down and restart before trying to get the next response? Is one computer behind a proxy? Have you tried dumping the headers of each response to see if they're different? – Mike Oct 18 '18 at 12:31
  • it should shutdown and restart before the next response. Nope both System runs without a Proxy. Had the same Idea as you. Struggeling to get Headers Response working right. How can i setRequestProperty (User Agent) Because without the USer Agent i'm getting the header Request 403 forbidden – xAllj Oct 18 '18 at 13:11
  • nevermind got the header Request working i will now compare them – xAllj Oct 18 '18 at 13:17
  • Might want to explicitly disable caching on the connection as well: `con.setUseCaches(false)`. – Mike Oct 18 '18 at 13:23
  • seems not to work :( i posted you the headers and bodys of the responses. Important is the key in the response body he is not changing – xAllj Oct 18 '18 at 13:50

2 Answers2

0

The responses are coming from CloudFlare, which is likely caching the response from the origin server. You need to make sure your request instructs the server, and any proxy servers along the way, to not use caching. You should be able to accomplish this using the Cache-Control header.

https://stackoverflow.com/a/14544664/657224

Mike
  • 4,722
  • 1
  • 27
  • 40
  • Mike if you don't mind would you maybe help me with this, i'm getting a bit confused how to setHeader for Cache Control – xAllj Oct 18 '18 at 14:19
  • But how i'm getting everytime a new Response if i'm using this site https://apitester.com/ ? If CloudFlare only caching the response, this site shouldn't work either am i right? – xAllj Oct 18 '18 at 14:28
  • @xAllj My guess is apitester is setting the no-cache header. You should be able to set it the same as the others: `con.setRequestProperty("Cache-Control", "no-cache");` – Mike Oct 18 '18 at 14:41
  • Might need to set `con.setRequestProperty("Cache-Control: no-cache, must-revalidate");` – Mike Oct 18 '18 at 14:52
  • the constructor can handle only two Strings so tried it like this con.setRequestMethod("POST"); con.setRequestProperty("User-Agent", USER_AGENT); con.setRequestProperty("Accept-Language", "en-US,en;q=0.5"); con.setRequestProperty("Cache-Control", "no-cache"); con.setRequestProperty("Cache-Control", "must-revalidate"); – xAllj Oct 18 '18 at 17:23
  • i tried this but it's not working i will update my class above to show you how i set this up – xAllj Oct 18 '18 at 17:40
0

The Solution was this two commands

con.setDefaultUseCaches(false);
con.setUseCaches(false);
xAllj
  • 1
  • 5