Scrapy - why I cannot extract this data?

Question

I'm trying to scrape this piece of code, but I realize that the region I want to extract only appears at Chrome with "Inspect element". Viewing the source code, this piece disappears.

How can I get this data? I tested with scrapy shell and nothing...

Here (inspect element) the tag appears:

<div id="Brinde"></div>
<script>MostraParcelaPagSeguro(158.0,0, 1886481);</script>
<span class="ParcelaPagSeg">em&nbsp;<b>12x</b> de <b>R$&nbsp;14,93</b></span>
</div>

Here (source code) the tag disappears:

<div id="Brinde"></div>
<script>MostraPreco(158.0,158.0,'PM3602001_30',0);</script><br>
<script>MostraParcelaPagSeguro(158.0,0, 1886481);</script>
</div>

I know this is probably because the script MostraParcelaPagSeguro(). But how can I extract this?

Answer 1

Scrapy doesn't parse javascript or AJAX requests natively. You will have to do do another web request to the data which that script got. Essentially you have to simulate the javascript as well. See Can scrapy be used to scrape dynamic content from websites that are using AJAX?

The easier option is to use a browser-engine (ie Webkit) based robot such as Ghost which will load the page like your web browser does. So all the DOM changes from Javascript will happen and you can then extract what you need.