0

I am running a small personal python service that uses a paid rotating proxy with limited bandwidth to make requests to websites and scrapes data from websites that have no api.

My question is now how can I reduce the bandwidth that is used when scraping websites? Can i somehow only get pure text or something like that?

I appreciate any help

critiqalfish
  • 33
  • 4

1 Answers1

1

Maybe you can try to add HTTP compression by adding content-encoding: gzip header to you requests. If proxy and target website are supporting this, then you should be able to reduce the bandwidth. You can check this question on how to add such header in requests library.

piotr.gradzinski
  • 873
  • 1
  • 10
  • 22
  • Thank you. Unfortunately does this not work with the website I am scraping from (pastebin.com) as it does not support HTTP compression. At least i did not got it to work – critiqalfish Jan 27 '22 at 23:55