2

I am working on a spider that crawls a site while authenticated, it uses multiple users and multiple proxies. Is there anyway I could handle cookies, proxies and session so each user uses their specific cookie with a specific proxy so there is no problems crawling things?

At the moment I am able to crawl with one user while authenticated and single IP, but want to do this with more users.

mmmmmm
  • 32,227
  • 27
  • 88
  • 117
jsjc
  • 1,003
  • 2
  • 12
  • 24

1 Answers1

1

It's possible, though it will require some work.

I can think of two ways to accomplish this:

  1. Tweaking every Request and Response manually, which might seem simpler
  2. Writing a middleware, which is more complex, but more universal

In both cases this info should help:

Community
  • 1
  • 1
warvariuc
  • 57,116
  • 41
  • 173
  • 227