Site scraping with F# and Canopy

Question

I am trying to write a simple scraper using F# and Canopy (see http://lefthandedgoat.github.io/canopy/). I am trying to extract text from all element with the class ".application-tile". However, in the code below, I get the following build error and I don't understand it.

This expression was expected to have type
    OpenQA.Selenium.IWebElement -> 'a    
but here has type
    OpenQA.Selenium.IWebElement

Any idea why this is happening? Thanks!

open canopy
open runner
open System

[<EntryPoint>]
let main argv = 
    start firefox

    "taking canopy for a spin" &&& fun _ ->
        url "https://abc.com/"

        // Login Page
        "#i0116" << "abc@abc.com"
        "#i0118" << "abc"
        click "#abcButton"

        // Get the Application Tiles -- BUILD ERROR HAPPENS HERE
        elements ".application-tile" |> List.map (fun tile -> (tile |> (element ".application-name breakWordWrap"))) |> ignore

    run()

I am trying to get the text/content from each of the element. — Martin, Apr 09 '14 at 20:48

score 6 · Accepted Answer · edited Apr 09 '14 at 22:26

open canopy
open runner

start firefox

"taking canopy for a spin" &&& fun _ ->
    url "http://lefthandedgoat.github.io/canopy/testpages/"

    // Get the tds in tr
    let results = elements "#value_list td" |> List.map read

    //or print them using iter
    elements "#value_list td" 
        |> List.iter (fun element -> System.Console.WriteLine(read element))

run()

That should do what you want.

canopy has function called 'read' that takes in either a selector or an element. Since you have all of them from 'elements "selector"' you can map read over the list.

List.map takes in a function, runs it, and returns a list of results. (in C# its like elements.Select(x => read(x)) List.iter is the same as .foreach(x => System.Console.Writeline(read(x))

score 0 · Answer 2 · answered Apr 09 '14 at 20:50

I believe that the error is happening in the projection lambda inside your List.map call. From the canopy documentation elements returns all elements that match css selector or text. element gets an element with given css selectors or text.

So here you are obtaining a list of Elements that match the selector ".application-tile". List.map requires a lambda that takes an IElement (the type contained in elements) that will project it into a new form (the generic 'a).

I don't know much about this framework but I'm not sure why you're taking an element and then piping it into another call to element.

Looking further through the documentation we find the read function: "Read the text (or value or selected option) of an element." Is this what you want?

"Read the text (or value or selected option) of an element." Is this what you want? -- Yes. This is exactly what I want. My understanding is that "elements" will return me all element with the class ".application-tile" and I want a collection with the content/text of each of the element found. — Martin, Apr 09 '14 at 21:02

Site scraping with F# and Canopy

2 Answers2