1

I'm new to web-crawlers, trying to crawl ridership data of metro from the cellphone maps app(www.amap.com) with Fiddler, but I got this HTTP connect method, which is not viewable. There are icons of locks next to the URL and in 'Response' it says this: 'Encrypted HTTPS traffic flows through this CONNECT tunnel. HTTPS Decryption is enabled in Fiddler, so decrypted sessions running in this tunnel will be shown in the Web Sessions list.'

Captured traffics of my cellphone

I found a solution suggesting that customizing rules in fiddler may help, so I followed and added this to its script:

if (oSession.oRequest[‘User- Agent’].IndexOf("Android") > -1 && oSession.HTTPMethodIs("connect")) {
    oSession.oResponse.headers["Connection"]="Keep-Alive";
}

The changes to Fiddler Script

But of course, it didn't work, I've tried both iPhone and android and changed the header in the script respectively, none of them helped.

So is this app and HTTP connect method crawlable? The data is constructively helpful to my research, instead, it is not provided in website 'amap', so it has to be done through a cellphone.

Robert
  • 39,162
  • 17
  • 99
  • 152
Freddie-Hawking
  • 11
  • 1

1 Answers1

0

If you have HTTPS decryption enabled in Fiddler but you see (mostly) only CONNECT requests this means that the apps on the device try to open a connection but do not trust the Fiddler root certificate.

If you try to use the apps on-device you will notice that there is currently no working network connection available (requests just don't work as the apps don't accept the server certificate created by Fiddler).

On Android devices since Andorid 6 you need root permissions to instal the Fiddler rot certificate or alternatively if you want to monitor a single app you can try to modify and re-sign the app. All details are described in this question and answer: Some androids apps won't connect through fiddler

Robert
  • 39,162
  • 17
  • 99
  • 152
  • Thank you for your answers, yes I guess my device does not trust the certificate indeed, although I had the certificate installed(I am an iPhone user, and I've tried it many times via many means). BUT it seems that the network was not completely unavailable, with warnings saying that 'can not verify the identity of m.hotmail.com' from time to time, but Safari's traffics are able to be captured. I will try to re-sign the app since it's all I need to monitor the one app, but I do curious about that can the trusted certificate be installed on my iPhone so that I can monitor my iPhone freely? – Freddie-Hawking Nov 11 '21 at 03:05
  • @Freddie-Hawking iPhones do trust Fiddle Root-CA certificate if installed correctly (unless the app performs certificate pinning). BTW: If you are an iPhone user why does your Fiddler script contains an check for Android? – Robert Nov 11 '21 at 17:16
  • Thank you very much, I do find guidance about installing certificate on iPhone. That is because I tried with my friend's android after I failed with my iPhone, so I modified the fiddler script correspondingly. – Freddie-Hawking Nov 14 '21 at 08:26