1

So I recently got in a project that essentially scrapes tweets using twint library, saves the tweets in elastic search and on top of that there are some dashboards in Kibana for showing several metrics. This project has been running for the past one year and imagine that we have collected around 5M tweets.

Suddenly, yesterday the same code without any changes throws the above error.

Has anyone managed to solve this ? Or does anyone know if there is the possibility that somehow the twint scraper is banned or even if I have been banned? Are there any recommended scrapers doing the same job without this issues?

Thank you in advance.

  • Python version is 3.8;
  • Installed twint with pip install git+https://github.com/twintproject/twint.git@e7c8a0c764f6879188e5c21e25fb6f1f856a7221
  • I have also tried to upgrade it with pip install --upgrade git+https://github.com/twintproject/twint.git@e7c8a0c764f6879188e5c21e25fb6f1f856a7221

I have searched a lot about the issue, I have seen that the repository for twint is no longer active and I have not figured it out yet. The most relevant issue that I found is #1257, which has not been solved.

Petrina Troulitaki
  • 11
  • 2
  • Welcome to Stackoverflow! If you want your question to be answered more quickly, it's always best to try and make a [minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example). Please edit your question and add such a minimal reproducible example :) – Koedlt Jan 12 '23 at 12:13
  • 1
    Unfortunately, twint is no longer maintained today, and the repo has been archived. [Here](https://github.com/woluxwolu/twint/issues/13) and [here](https://github.com/yihong0618/twint/issues/4) are being the most current discussions on this error. In a nutshell, Twitter now requires users to log in before using the search function, and twint does not yet have a login option. Maybe some fork will implement this in the future, or maybe it will be necessary to switch to an alternative project such as [twint-zero](https://github.com/twintproject/twint-zero) and use a custom nitter instance. – Christoph Thiede May 01 '23 at 15:00

0 Answers0