1

I'm trying to program my first website scraper, and my first step is to save the HTML to a string. However, from what I can tell, the data that I need to get is not in the HTML code per se, but rather is added after JavaScript executes some stuff.

My current code is this:

let myURLString = "Example URL"
let myURL = URL(string: myURLString)
var myHTMLString = ""

do {
            myHTMLString = try String(contentsOf: myURL!)
} catch let error {
            print("Error: \(error)")
}

But this doesn't seem to execute the javascript and instead just gives me the 'unprocessed' HTMl.

I read this answer here, but it's written in Swift 2.0 and since I, to be honest, didn't really understand what was going on ( I don't have much programming experience ): I couldn't get to work in Swift 3.

So, Is there a way to take the HTML from a website, run the JavaScript and then save that as a String in Swift 3? And if so, how do you do it?

Thanks!

Community
  • 1
  • 1
Giulio Crisanti
  • 370
  • 2
  • 12

1 Answers1

2

After some digging I got something that worked:

import Cocoa
import WebKit

    class ViewController: NSViewController, WebFrameLoadDelegate {

        @IBOutlet var myWebView: WebView!


        override func viewDidLoad() {
            super.viewDidLoad()
            // Do any additional setup after loading the view.

            self.myWebView.frameLoadDelegate = self

            let urlString = "YOUR HTTPS URL"
            self.myWebView.mainFrame.load(NSURLRequest(url: NSURL(string: urlString)! as URL) as URLRequest!)
        }

        override var representedObject: Any? {
            didSet {
                // Update the view, if already loaded.
            }
        }

        func webView(_ sender: WebView!, didFinishLoadFor frame: WebFrame!) {
            let doc = myWebView.stringByEvaluatingJavaScript(from: "document.documentElement.outerHTML")! //get it as html
            //doc now has the 'processed HTML'
        }

}
Giulio Crisanti
  • 370
  • 2
  • 12