Goutte - Guzzle/DomCrawler - Scraping from HTML tables - Lots of complications

Question

I started using Goutte to get info I need from sites, and its brilliant, its saving my serious amounts of time and hassle. But on the other hand, sometimes anomalies arise, and I have no idea what causes them. So heres a page I'm scraping from right now: http://lifecenter.sgst.cn/hit/search.jsp?key1=%27a%27&key2=%27b%27&key3=%27c%27&key4=%27d%27&key5=C0305

I've been going through this site, following links and every page on the site has its own kind of table. This table seems pretty straightforward, heres the HTML for some of it:

<table align="center" border="1" cellpadding="4" cellspacing="1" width="927">
<tbody><tr bgcolor="#3399FF">
<th align="center">ENTRY</th>
<th align="center">COMPOUND</th>
<th align="center">HERB</th>
<th align="center">TARGET ID</th>
<th align="center">TARGET NAME</th>
<th align="center">TARGET TYPE</th>
</tr>
</tbody><tbody id="table2">


<tr>
<td align="center"><a href="detail.jsp?compoundid=C0305&amp;pid=T0595">HIT001882</a></td>
<td align="center">linalool</td>
<td align="center">Ardisiae japonicae(ai di cha|矮地茶); Fructus Artemiiae argyi(ai shi|艾实); Folium Artemisiae Argyi(ai ye|艾叶); (bai cong|白葱); Dracoccephalum heterophyllum Benth(bai hua xia ku cao|白花夏枯草); Flos Micheliae albae(bai lan hua|白兰花); bai siu zi; Caulis Perillae frutescentis(bai su geng|白苏梗); Folium Perillae frutescentis(bai su ye|白苏叶); Magnolia denudata(bai yu lan|白玉兰); Fructus Litseae(bi chen jia|毕澄茄); Radix Bupleuri chinensis(chai hu|柴胡); Pericarpium Citri Reticulatae(chen pi|陈皮); Radix albiflorae(chou jie cao gen|臭节草根); Ligusticum brachylobum Franch(chuan fang feng|川防风); Radix chuanxiong; Rhizoma Chuanxiong(chuan xiong|川芎); chun sha hua; Abies nephrolepis(cou leng shan|臭冷杉); Basho(da ba jiao|大芭蕉); Herba Elsholtziae penduliflorae(da hei tou cao|大黑头草); Mosla dianthera (Ham.) Maxim(da ye xiang ru|大叶香薷); (di feng|地枫)</td>
<td align="center"><a href="Protein_ID.jsp?pid=T0595&amp;protein=Katanin p60 ATPase-containing subunit A1">T0595</a></td>
<td align="center">Katanin p60 ATPase-containing subunit A1</td>
<td align="center">Direct Target</td>
</tr>

<tr>
<td align="center"><a href="detail.jsp?compoundid=C0305&amp;pid=T0596">HIT001883</a></td>
<td align="center">linalool</td>
<td align="center">Ardisiae japonicae(ai di cha|矮地茶); Fructus Artemiiae argyi(ai shi|艾实); Folium Artemisiae Argyi(ai ye|艾叶); (bai cong|白葱); Dracoccephalum heterophyllum Benth(bai hua xia ku cao|白花夏枯草); Flos Micheliae albae(bai lan hua|白兰花); bai siu zi; Caulis Perillae frutescentis(bai su geng|白苏梗); Folium Perillae frutescentis(bai su ye|白苏叶); Magnolia denudata(bai yu lan|白玉兰); Fructus Litseae(bi chen jia|毕澄茄); Radix Bupleuri chinensis(chai hu|柴胡); Pericarpium Citri Reticulatae(chen pi|陈皮); Radix albiflorae(chou jie cao gen|臭节草根); Ligusticum brachylobum Franch(chuan fang feng|川防风); Radix chuanxiong; Rhizoma Chuanxiong(chuan xiong|川芎); chun sha hua; Abies nephrolepis(cou leng shan|臭冷杉); Basho(da ba jiao|大芭蕉); Herba Elsholtziae penduliflorae(da hei tou cao|大黑头草); Mosla dianthera (Ham.) Maxim(da ye xiang ru|大叶香薷); (di feng|地枫)</td>
<td align="center"><a href="Protein_ID.jsp?pid=T0596&amp;protein=Adenosine receptor A2a">T0596</a></td>
<td align="center">Adenosine receptor A2a</td>
<td align="center">Direct Target</td>
</tr>

<tr>
<td align="center"><a href="detail.jsp?compoundid=C0305&amp;pid=T0040">HIT001885</a></td>
<td align="center">linalool</td>
<td align="center">Ardisiae japonicae(ai di cha|矮地茶); Fructus Artemiiae argyi(ai shi|艾实); Folium Artemisiae Argyi(ai ye|艾叶); (bai cong|白葱); Dracoccephalum heterophyllum Benth(bai hua xia ku cao|白花夏枯草); Flos Micheliae albae(bai lan hua|白兰花); bai siu zi; Caulis Perillae frutescentis(bai su geng|白苏梗); Folium Perillae frutescentis(bai su ye|白苏叶); Magnolia denudata(bai yu lan|白玉兰); Fructus Litseae(bi chen jia|毕澄茄); Radix Bupleuri chinensis(chai hu|柴胡); Pericarpium Citri Reticulatae(chen pi|陈皮); Radix albiflorae(chou jie cao gen|臭节草根); Ligusticum brachylobum Franch(chuan fang feng|川防风); Radix chuanxiong; Rhizoma Chuanxiong(chuan xiong|川芎); chun sha hua; Abies nephrolepis(cou leng shan|臭冷杉); Basho(da ba jiao|大芭蕉); Herba Elsholtziae penduliflorae(da hei tou cao|大黑头草); Mosla dianthera (Ham.) Maxim(da ye xiang ru|大叶香薷); (di feng|地枫)</td>
<td align="center"><a href="Protein_ID.jsp?pid=T0040&amp;protein=Nitric oxide synthase, inducible">T0040</a></td>
<td align="center">Nitric oxide synthase, inducible</td>
<td align="center">Direct Target</td>
</tr>

<tr>
<td align="center"><a href="detail.jsp?compoundid=C0305&amp;pid=T0054">HIT001884</a></td>
<td align="center">linalool</td>
<td align="center">Ardisiae japonicae(ai di cha|矮地茶); Fructus Artemiiae argyi(ai shi|艾实); Folium Artemisiae Argyi(ai ye|艾叶); (bai cong|白葱); Dracoccephalum heterophyllum Benth(bai hua xia ku cao|白花夏枯草); Flos Micheliae albae(bai lan hua|白兰花); bai siu zi; Caulis Perillae frutescentis(bai su geng|白苏梗); Folium Perillae frutescentis(bai su ye|白苏叶); Magnolia denudata(bai yu lan|白玉兰); Fructus Litseae(bi chen jia|毕澄茄); Radix Bupleuri chinensis(chai hu|柴胡); Pericarpium Citri Reticulatae(chen pi|陈皮); Radix albiflorae(chou jie cao gen|臭节草根); Ligusticum brachylobum Franch(chuan fang feng|川防风); Radix chuanxiong; Rhizoma Chuanxiong(chuan xiong|川芎); chun sha hua; Abies nephrolepis(cou leng shan|臭冷杉); Basho(da ba jiao|大芭蕉); Herba Elsholtziae penduliflorae(da hei tou cao|大黑头草); Mosla dianthera (Ham.) Maxim(da ye xiang ru|大叶香薷); (di feng|地枫)</td>
<td align="center"><a href="Protein_ID.jsp?pid=T0054&amp;protein=Prostaglandin G/H synthase 2">T0054</a></td>
<td align="center">Prostaglandin G/H synthase 2</td>
<td align="center">Indirect Target</td>
</tr>

</tbody>
</table>

So the first thing I did is select the <tbody> since all the data I need is in there:

    $tbody = $compoundPage->filter('tbody#table2');

It all looks good, when I run:

> exit(dump($compoundPage->filter('tbody#table2')->html()));

Heres what I get:

<tr>\n
    <td align="center"><a href="detail.jsp?compoundid=C0305&amp;pid=T0595">HIT001882</a></td>\r\n
    <td align="center">linalool</td>\r\n
    <td align="center">Ardisiae japonicae(ai di cha|矮地茶); Fructus Artemiiae argyi(ai shi|艾实); Folium Artemisiae Argyi(ai ye|艾叶); (bai cong|白葱); Dracoccephalum heterophyllum Benth(bai hua xia ku cao|白花夏枯草); Flos Micheliae albae(bai lan hua|白兰花); bai siu zi; Caulis Perillae frutescentis(bai su geng|白苏梗); Folium Perillae frutescentis(bai su ye|白苏叶); Magnolia denudata(bai yu lan|白玉兰); Fructus Litseae(bi chen jia|毕澄茄); Radix Bupleuri chinensis(chai hu|柴胡); Pericarpium Citri Reticulatae(chen pi|陈皮); Radix albiflorae(chou jie cao gen|臭节草根); Ligusticum brachylobum Franch(chuan fang feng|川防风); Radix chuanxiong; Rhizoma Chuanxiong(chuan xiong|川芎); chun sha hua; Abies nephrolepis(cou leng shan|臭冷杉); Basho(da ba jiao|大芭蕉); Herba Elsholtziae penduliflorae(da hei tou cao|大黑头草); Mosla dianthera (Ham.) Maxim(da ye xiang ru|大叶香薷); (di feng|地枫)</td>\r\n
    <td align="center"><a href="Protein_ID.jsp?pid=T0595&amp;protein=Katanin%20p60%20ATPase-containing%20subunit%20A1">T0595</a></td>\r\n
    <td align="center">Katanin p60 ATPase-containing subunit A1</td>\r\n
    <td align="center">Direct Target</td>\r\n
</tr>\n
<tr>\n
    <td align="center"><a href="detail.jsp?compoundid=C0305&amp;pid=T0596">HIT001883</a></td>\r\n
    <td align="center">linalool</td>\r\n
    <td align="center">Ardisiae japonicae(ai di cha|矮地茶); Fructus Artemiiae Maxim(da ye xiang ru|大叶香薷); (di feng|地枫)</td>\r\n
    <td align="center"><a href="Protein_ID.jsp?pid=T0596&amp;protein=Adenosine%20receptor%20A2a">T0596</a></td>\r\n
    <td align="center">Adenosine receptor A2a</td>\r\n
    <td align="center">Direct Target</td>\r\n
</tr>\n
<tr>\n
    <td align="center"><a href="detail.jsp?compoundid=C0305&amp;pid=T0040">HIT001885</a></td>\r\n
    <td align="center">linalool</td>\r\n
    <td align="center">Ardisiae japonicae(ai di cha|矮地茶); Fructus Artemiiae argyi(ai shi|艾实); Folium Artemisiae Argyi(ai ye|艾叶); (bai cong|白葱); Dracoccephalum heterophyllum Radix Bupleuri chinensis(</td>\r\n
    <td align="center"><a href="Protein_ID.jsp?pid=T0040&amp;protein=Nitric%20oxide%20synthase,%20inducible">T0040</a></td>\r\n
    <td align="center">Nitric oxide synthase, inducible</td>\r\n
    <td align="center">Direct Target</td>\r\n
</tr>\n
<tr>\n
    <td align="center"><a href="detail.jsp?compoundid=C0305&amp;pid=T0054">HIT001884</a></td>\r\n
    <td align="center">linalool</td>\r\n
    <td align="center">Ardisiae japonicae</td>\r\n
    <td align="center"><a href="Protein_ID.jsp?pid=T0054&amp;protein=Prostaglandin%20G/H%20synthase%202">T0054</a></td>\r\n
    <td align="center">Prostaglandin G/H synthase 2</td>\r\n
    <td align="center">Indirect Target</td>\r\n
    </tr>

I see 4 rows.

but now I need to iterate through the rows, so I have to go a step further:

$tbody = $compoundPage->filter('tbody#table2 > tr');

when I run: exit(dump($compoundPage->filter('tbody#table2 > tr')->html()));

it outputs only one row:

<td align="center"><a href="detail.jsp?compoundid=C0305&amp;pid=T0595">HIT001882</a></td>\r\n
<td align="center">linalool</td>\r\n
<td align="center">Ardisiae japonicae(ai di cha|矮地茶);</td>\r\n
<td align="center"><a href="Protein_ID.jsp?pid=T0595&amp;protein=Katanin%20p60%20ATPase-containing%20subunit%20A1">T0595</a></td>\r\n
<td align="center">Katanin p60 ATPase-containing subunit A1</td>\r\n
<td align="center">Direct Target</td>\r\n

But when I count the rows like this:

exit(dump($compoundPage->filter('tbody#table2 > tr')->count()));

It says there are 4 rows.

The problem occurs when I iterate through the rows, like this:

    $data = $rows->each(function($row,$i) use ($client) {
      $tds = $row->filter('td');

      $keys = array('entry','compound','herb','target_id','target_name','target_type');

      foreach($keys as $td_i => $key) {
          $data[$key] = $tds->eq($i)->text();
      }

  });

Firstly when I count the td elements, it says theres only 1. And if I try to actually do anything with the td nodes, I get this error: https://gyazo.com/94042ba2e3101ba918a71cc9e8e3be5a

If I do this:

exit(dump($row->filter('td')));

inside the loop, it tells me there are 24 nodes:

Crawler {#1121 ▼
  #uri: "http://lifecenter.sgst.cn/hit/search.jsp?key1='a'&key2='b'&key3='c'&key4='d'&key5=C0305"
   -defaultNamespacePrefix: "default"
   -namespaces: []
   -baseHref: "http://lifecenter.sgst.cn/hit/search.jsp?key1='a'&key2='b'&key3='c'&key4='d'&key5=C0305"
   -document: DOMDocument {#52 ▶}
   -nodes: array:24 [▼
     0 => DOMElement {#1123 ▶}
     1 => DOMElement {#1124 ▶}
     2 => DOMElement {#1125 ▶}
     3 => DOMElement {#1126 ▶}
     4 => DOMElement {#1127 ▶}
     5 => DOMElement {#1128 ▶}
     6 => DOMElement {#1129 ▶}
     7 => DOMElement {#1130 ▶}
     8 => DOMElement {#1131 ▶}
     9 => DOMElement {#1132 ▶}
     ... (omitted the rest cuz I have to do this indentation manually, its so time consuming.

  ]
  -isHtml: true

I got worried for a bit there, everything I typed at the start of the post, there was a message saying: the body cannot contain "I started using", it looks like spam. It took me ages to write this post, I'm glad I bypassed that spam filter in the end — Horse O'Houlihan, Nov 07 '16 at 16:35

score 0 · Answer 1 · answered Nov 07 '16 at 16:44

I got it working, I added this:

$rows = $compoundPage->filter('tbody#table2')->filter('tr');

instead of this:

$rows = $compoundPage->filter('tbody#table2 > tr');

and now its working properly. I still have no idea what to problem was, because these two ways of selecting should be identical, and I'm pretty sure I tried this method before. I have been running into many problems with looping through tables, and I see that there is definitely a difference between:

$rows = $compoundPage->filter('tbody#table2 > tr');

and

$rows = $compoundPage->filter('tbody#table2')->filter('tr');

I've seen the CSS selection method break the script multiple times.

Goutte - Guzzle/DomCrawler - Scraping from HTML tables - Lots of complications

1 Answers1