0

Is there any way to find out how much data is transferred from the web server to my machine during web scraping? I want to put minimal load on the web server. Also, I can try different techniques and use whichever put minimal load on the web server.

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
yadav
  • 99
  • 3
  • 9
  • You can check the size of server response (for example, in `requests` library), but your 'load' is probably expressed in number of calls to the server per min or per hour too. It would have helped to know the motivation - low bandwidth for internet? Server behaviour on too many calls? – Evgeny Apr 18 '18 at 05:46
  • My internet speed is 100Mbps so no bandwidth issue. I am mainly concerned about server behavior. – yadav Apr 18 '18 at 05:49
  • Why not review some links on the subject, eg https://stackoverflow.com/q/8768439/1758363. Otherwise you question risks downvotes because of poor research effort. – Evgeny Apr 18 '18 at 06:08
  • 1
    As more general resourse: https://github.com/lorien/awesome-web-scraping/blob/master/python.md – Evgeny Apr 18 '18 at 06:12

0 Answers0