0

So I have a crawler using WinInet (but could be WinHTTP)

If I have to crawl a website that requires user login, I could in past tell people to login in using their IE browser and it would usually work.

That was because cookies were shared across WinInet / Internet Options / IE.

However, trying the same with the new Edge browser does not appear to work.

Searching Google brings up some information about Edge to IE, but not any mentions of WinInet. Maybe I am just looking the wrong place, but has anyone worked around this issue before?

Tom
  • 3,587
  • 9
  • 69
  • 124

1 Answers1

0

IE is WinInet-based application, but Edge is not. Edge doesn't use WinInet so I think that's why it doesn't work in Edge. You can also refer to this thread for more information.

If you still want to use WinInet cookies sharing, you can open and login the website in Edge IE mode. Edge IE mode using IE engine, so I think the method working in IE will also working in Edge IE mode.

If you want to open the website in Edge without IE mode, maybe you need to change your way to crawl the website. I suggest that you can try to use some web crawling tools such as Scrapy, Beautiful Soup and so on.

Yu Zhou
  • 11,532
  • 1
  • 8
  • 22
  • Thank you for the links. Prefereably I would like to grab the cookies from Edge... Or provide a browser window myself in whatever embedded/browser and grab the login cookies from there. – Tom Jul 06 '21 at 08:11
  • (Switching crawler is not an option. I have on/off 15+ years invested into it. I can easily switch underlying libraries used. But it was extremely convenient sharing cookies in WinInet. But if I can grab cookies from Edge storage and duplicate in WinInet or whatever library I decide on that is OK as well) – Tom Jul 06 '21 at 08:15
  • I have no idea about getting login cookies on modern browsers because I think it's unsafe. Cookies used for logins are sensitive, for security reasons browsers do their best to keep those secret. – Yu Zhou Jul 07 '21 at 03:42
  • But convenient in WinAPI that they are shared. I guess one will have to make a crawler option module using Chromium browser for HTTP then. Probably most feasible solution albeit probably also slower. But AJAX compatible so there's always that as bright side. – Tom Jul 07 '21 at 05:42