Yahoo finance and YQL returning different values in console vs code

Question

I am trying to get data from 'https://uk.finance.yahoo.com/q/cp?s=%5EFTSE'

I used select * from html where url = 'https://uk.finance.yahoo.com/q/cp?s=%5EFTSE' in YQL console here : YQL console which seems to be retriving proper results.

But then I do exactly the same query through PHP and this is all I get back http://pastebin.com/MDTwh6n7 it is missing ['query']['results']['body']['div'][3] which I use.

I might be doing here something wrong but it seems like this query works and sometimes doesn't even though I keep it all the same.

Answer 1

What is happening?

That usually happens when the website loads dynamically. If you use Chrome Dev Tools and check the Network tab, you will see that the web page does several requests over time until you see the complete result.

Why am I getting the weird result?

This is a problem for YQL. Your query likely works with the console because when you are using it, the page finished loading already.

However, when your PHP server does the request, it still needs to wait for the rest, and it doesn't. Consequently, your YQL query ends up running with an incomplete HTML.

YQL is good, but only for pure static HTML pages, which I believe is not the case here.

How do you know if a web page is static or dynamic?

Well, there is no way to know for sure, but this article from Quora suggest some strategies you can use:

• https://www.quora.com/How-do-I-check-if-a-website-is-static-or-dynamic

I myself recently did the same mistake you did. I took a wikia page for granted, thinking it was static, but in the end it loaded dynamically as well.

What do I do now?

Overall, it is my believe that if you wish to do HTML scraping, you need to use another tool.

I believe that if you can make sure you are getting the full HTML download with your server, parsing it should be easy.

So I suggest two steps:

1. make an http GET request How to send a GET request from PHP?
2. parse the html http://htmlparsing.com/php.html

Admittedly there are many libraries and many ways in which you can achieve these two objectives, but I think this is more than enough to get you started.