-3

I have recently posted a question about how to login to twitter using requests library. Finally, I got the solution for that and another problem i am facing is that i am able to scrape only visible content in the page. How to scrape dynamically loaded content in that page?

Note: I am not using selenium. Please provide any other means to do this.

How to load dynamic content and then scrape it?

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
Akhil Reddy
  • 371
  • 1
  • 6
  • 26
  • Check [this](https://stackoverflow.com/questions/30345623/scraping-dynamic-content-using-python-scrapy) solution – Andersson Jan 28 '18 at 08:28
  • actually we can scrape dynamic content simply using requests and session(get and post). Instead of using webdrivers we can do in this way. – Akhil Reddy Jan 28 '18 at 08:47
  • OK. So what is the point of your question? – Andersson Jan 28 '18 at 08:52
  • I am unable to get hidden content using requests. I am unable to understand what parameters have to be used for that. So I want your help in finding the parameters – Akhil Reddy Jan 28 '18 at 08:53
  • If you are _not using selenium_ why is `selenium` tagged in your question? – undetected Selenium Jan 28 '18 at 12:11
  • Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the [How to Ask](https://stackoverflow.com/help/how-to-ask) page for help clarifying this question. – undetected Selenium Jan 28 '18 at 12:12
  • Sorry to post question in this way, i will improve my skills in asking questions. – Akhil Reddy Jan 30 '18 at 01:49

1 Answers1

1

Without using something like Selenium or another browser (headless or otherwise) which will actually run the JavaScript in a normal-ish manner, the only other method would be to manually reverse engineer the JavaScript, see what kind of calls it's making, and make them yourself directly.

There wouldn't be any other kind of "one-size-fits-all" solution.

samanime
  • 25,408
  • 15
  • 90
  • 139
  • Can you help me with this? how to look for parameters causing javascipt loading and how to use them? – Akhil Reddy Jan 28 '18 at 09:01
  • Sorry, but that isn't really what SO is for. It's for help with your own code, not reverse engineering others. If you aren't able to do it, you might look to a freelancer to write some code for you (for pay, of course). – samanime Jan 28 '18 at 15:35
  • Finally I understood how to scrape it. – Akhil Reddy Jan 29 '18 at 17:49