xidel: wrong order of results on hacker news

Question

To scrape hacker news, I use:

xidel -e '//span[@class="titleline"]/a/@href|//span[@class="titleline"]' https://news.ycombinator.com/newest

But the output in not in the expected order, the URL come after the text, so it's very difficult to parse.

Does I miss something to have the good order?

I have:

There Is No Such Thing as a Microservice (youtube.com)
https://www.youtube.com/watch?v=FXCLLsCGY0s

I expect:

https://www.youtube.com/watch?v=FXCLLsCGY0s
There Is No Such Thing as a Microservice (youtube.com)

Or even better

https://www.youtube.com/watch?v=FXCLLsCGY0s There Is No Such Thing as a Microservice (youtube.com)

Why not simply use their [API](https://github.com/HackerNews/API)? — Filburt, Apr 06 '23 at 17:02
Parsing is sequential so nodes will be returned on document order not in xpath order. What is the expected output of `//span[@class="titleline"]`?. Please add it to the question. — LMC, Apr 06 '23 at 17:49

score 2 · Accepted Answer · answered Apr 07 '23 at 10:14

Please see "Using / on sequences rather than on sets" on why this is happening and why you should be using the XPath 3 mapping operator ! in this case:

$ xidel -s "https://news.ycombinator.com/newest" -e '
  //span[@class="titleline"]/a ! (@href,.)
'

(also please specify input first)

For a simple string-concatenation this isn't necessary:

-e '//span[@class="titleline"]/a/join((@href,.))'
-e '//span[@class="titleline"]/a/concat(@href," ",.)'
-e '//span[@class="titleline"]/a/x"{@href} {.}"'

(Bonus) Output to JSON:

$ xidel -s "https://news.ycombinator.com/newest" -e '
  array{
    //span[@class="titleline"]/a/{
      "title":.,
      "url":@href
    }
  }
'

LMC · Answer 2 · 2023-04-06T18:09:28.730

1

Nodes are returned in document order not in XPath order so additional parsing is needed. With xmllint and awk

xmllint --html --recover --xpath '//span[@class="titleline"]/a/@href|//span[@class="titleline"]/a/text()' tmp.html 2>/dev/null|\
gawk 'BEGIN{RS="\n? href="; FS="\n"}{ print $1, $2}' | tr -d '"'

Result

https://github.com/thesephist/ink Ink: Minimal, functional programming language inspired by modern JavaScript, Go
https://controlleddigitallending.org/whitepaper/ A White Paper on Controlled Digital Lending of Library Books
item?id=35471687 Ask HN: Connect Guitar to Tesla?

Note: Xpath in the answer does not need awksince href comes before a/text() in document order as OP expected. Added for reference on how to change ouput order.

edited Apr 06 '23 at 18:09

answered Apr 06 '23 at 18:03

LMC

10,453
2
27
52

Yeah, same idea to parte innerText of link. Thanks – Mévatlavé Kraspek Apr 06 '23 at 18:16
Welcome. Wonder how `xidel` handles this xpath `//span[@class="titleline"]`. Seems to return text content instead of the node as is. – LMC Apr 06 '23 at 18:23

score 0 · Answer 3 · answered Apr 06 '23 at 18:14

0

Found a better way:

$ xidel -e '//span[@class="titleline"]/a/@href|//span[@class="titleline"]/a/text()' https://news.ycombinator.com/newest

answered Apr 06 '23 at 18:14

Mévatlavé Kraspek

552
2
15

xidel: wrong order of results on hacker news

3 Answers3