0

I'm trying to retrieve an article price from a website. The problem is, that the prices differ if you choose online price or store price. After selecting a store the website creates a cookie called: CP_GEODATA with a specific value. I tried to send the cookie in different ways, but I keep getting the online price.

public class Parser {

    public static void main(String[] args) throws Exception {
        Map<String, String> cookies =  new HashMap<String, String>();
        cookies.put("CP_COUNTRY ", "%7B%22country%22%3A%22DE%22%7D  ");
        cookies.put("CP_GEODATA ", "%7B%22location%22%3A-1%2C%22firstlocation%22%3A11%2C%22name%22%3A%22Hamburg%22%7D");
        String url = "https://www.cyberport.de/?token=7a2d9b195e32082fec015dca45ba3aa4&sSearchId=565eee12d987b&EVENT=itemsearch&view=liste&query=&filterkategorie=";
        Connection.Response res = Jsoup.connect(url).cookies(cookies).data("query", "4B05-525").execute();
        Document doc = res.parse();
        String tester = doc.select("span[id=articlePrice] > span[class=basis fl]").text();
        String tester2 = doc.select("span[id=articlePrice] > span[class=decimal fl]").text();
        System.out.println(tester + tester2 + " €");
    }
}

The value I'm getting back right now is 2,90 € but it should be 4,90 €. I already tried everything and searched the internet a lot but I did not find any solution working for me.

This is the article I'm receiving the price from: https://www.cyberport.de/micro-usb-2-0-kabel-usb-a-stecker-micro-b-stecker-0-5m--4B05-525_9374.html

I'm trying to receive the price for the store in Hamburg, Germany.

You can see the cookies I'm setting at the top.

Thank you for any help!

Davide Pastore
  • 8,678
  • 10
  • 39
  • 53
  • Some of the answers in [this similar question](http://stackoverflow.com/questions/7139178/jsoup-cookies-for-https-scraping) appear potentially helpful. – Hovercraft Full Of Eels Dec 02 '15 at 16:51
  • @HovercraftFullOfEels thanks for the comment, but the answers provided by your link cannot help me. –  Dec 03 '15 at 09:24
  • I think this information is on the server and it is based on the current session that is in another cookie. – Davide Pastore Dec 03 '15 at 11:14
  • @DavidePastore can't find any session cookies which could help. I think I'm never going to be able to do it :( –  Dec 04 '15 at 12:28

1 Answers1

0

It seems that zone info is stored in session and zone code is sent to server in a post when you select it.

Then you need to do the following steps:

  • Do the POST with the desired zone
  • Get the session cookies
  • Using these cookes do your original POST
  • Hopefully get the correct results

Here is the code

public static void main(String[] args) throws Exception {
    Connection.Response res;

    //11 is for Hamburg
    String zoneId = "11";

    //Set the zone and get the session cookies
    res = Jsoup.connect("https://www.cyberport.de/newajaxpass/catalog/itemlist/0/costinfo/" + zoneId)
            .ignoreContentType(true)
            .method(Method.POST).execute();

    final Map<String, String> cookies = res.cookies();
    //print the cookies, we'll see session cookies here
    System.out.println(cookies);

    //If we use that cookies, your code runs Ok
    String url = "https://www.cyberport.de/?token=7a2d9b195e32082fec015dca45ba3aa4&sSearchId=565eee12d987b&EVENT=itemsearch&view=liste&query=&filterkategorie=";
    res = Jsoup.connect(url).cookies(cookies).data("query", "4B05-525").execute();

    Document doc = res.parse();
    String tester = doc.select("span[id=articlePrice] > span[class=basis fl]").text();
    String tester2 = doc.select("span[id=articlePrice] > span[class=decimal fl]").text();
    System.out.println(tester + tester2 + " €");

    //Extra check
    System.out.println(doc.select("div.townName").text());
}

You'll see:

{SERVERID=realmN03, SCS=76fe7473007c80ea2cfa059f180c603d, SID=pphdh7otcefvc5apdh2r9g0go2}
4,90 €
Hamburg

Which, I hope, is the desired result.

fonkap
  • 2,469
  • 1
  • 14
  • 30