2

Is it possible to load a web page in c++ and get the rendered DOM? Not just the HTTP response, but the rendered DOM that occurs after java-script runs (maybe after letting it run for some amount of time). Specifically the dynamic HTML that may have changed over time? Is there a library for this?

Or if not c++, do you know of any other language which this can be done in?

Edit here's an example to illustrate better why one might want to do this:

Imagine you want to crawl a website written in angular. You can't just make an http request and use the HTTP response, because most of the DOM is rendered after javascript/dynamic html manipulates the DOM. The initial http response for an angular site probably doesn't have all the contents, its requested and rendered later through javascript/AJAX/dyanmic html.

Alex Amato
  • 1,591
  • 4
  • 19
  • 32

2 Answers2

2

Since DOM is something implemented differently by each browser, how you use that from C++ will be different with each browser.

I'll give an example for IE. You can use the WebBrowser ActiveX control which exposes the IWebBrowser2 interface. From there you can call IWebBrowser2::get_Document to get an IHTMLDocument2 object, which is the root of the DOM.

#include "StdAfx.h"

using namespace ATL;
using namespace std;

void ThrowIfFailed(HRESULT hr)
{
    if (FAILED(hr))
        throw CAtlException(hr);
}

int main()
{
    ::CoInitialize(nullptr);

    try
    {
        CComPtr<IWebBrowser2> pWebBrowser;
        HRESULT hr = ::CoCreateInstance(CLSID_InternetExplorer, nullptr, CLSCTX_LOCAL_SERVER, IID_PPV_ARGS(&pWebBrowser));
        ThrowIfFailed(hr);

        hr = pWebBrowser->put_Visible(VARIANT_TRUE);
        ThrowIfFailed(hr);

        hr = pWebBrowser->GoHome();
        ThrowIfFailed(hr);

        CComPtr<IDispatch> pDispatch;
        hr = pWebBrowser->get_Document(&pDispatch);
        ThrowIfFailed(hr);

        CComPtr<IHTMLDocument2> pDocument;
        hr = pDispatch->QueryInterface(&pDocument);
        ThrowIfFailed(hr);

        CComBSTR bstrTitle;
        hr = pDocument->get_title(&bstrTitle);
        ThrowIfFailed(hr);

        wcout << bstrTitle.m_str << endl;
    }
    catch (const CAtlException& e)
    {
        wcout << L"Error (" << hex << e.m_hr << L")" << endl;
    }

    ::CoUninitialize();
    return 0;
}

This code just opens an IE window, navigates to the home page, and writes the title of the page to the console. You can also control whether the IE window becomes visible by removing the call to IWebBrowser2::put_Visible.

user1610015
  • 6,561
  • 2
  • 15
  • 18
0

As I am understanding, you are asking: "How to manipulate DOM of already rendered HTML Page through C++?"

If that's what you wanted to ask, here is my answer:

  • Technically, you can do it through C++. However, you need a right tool/lib/framework/ ... for doing this.

  • Normally, we manipulate DOM by Javascript.

  • In my experience, mobile developer have built-in control for load the page, usually called "webview". Android (Java) and iOS (Objective-C) have it. Then they manipulate DOM like this manner: "webview.evaluteScript("your javascript").

  • If you want to do it with C++. I think you can read these link:

How to embed WebKit into my C/C++/Win32 application?

How do I embed WebKit in a window?

Community
  • 1
  • 1
Khoa
  • 2,632
  • 18
  • 13
  • I don't want to manipulate it. As much as I just want the dom string for what a browser would render, after it ran the javascript. Imagine you want to crawl a website written in angular. You can't just make an http request and use the HTTP response, because most of the DOM is rendered after javascript/dynamic html is manipulated. The initial http response for an angular site probably doesn't have all the contents. – Alex Amato Sep 06 '16 at 04:56
  • As I said: "you can manipulate DOM". This means you can do this: "var myHTML = document.getElementsByTagName("html")[0]; myHTML.textContent" – Khoa Sep 06 '16 at 05:01
  • By using "webview" like stuff, you can retrieve the output of "myHTML.textConent". – Khoa Sep 06 '16 at 05:05
  • One more thing, you should recheck using: innerHTML, innerText, to replace textContent. I don't remember the differences, but one of them will give you the full HTML string of rendered page. – Khoa Sep 06 '16 at 05:08
  • You're missing the point. The issue isn't getting the HTML, the issue is getting the HTML that is generated by letting the JavaScript run for a few seconds. – 4castle Sep 06 '16 at 05:19
  • Just wait until the page is full loaded and get what you want. I did not say: "must get right after call loading page". – Khoa Sep 06 '16 at 05:28
  • The issue is that "wait until the page is full loaded" will not run the JavaScript – 4castle Sep 06 '16 at 05:44
  • I did not say: "wait the page by javascript". He's working with C++. He need a way to get a page with full content. I suggested him use something like "webview" to get his page because it works like a browser and it helps him execute the page's script to generate full content. When the time come, I suggest him evaluate javascript through webview to get the string he wanted. Is it clear for you @4castle? – Khoa Sep 06 '16 at 05:53
  • That's great, now make that clear in your answer, instead of answering a different question. – 4castle Sep 06 '16 at 05:56
  • 2
    Umm, thank you for your answer anyways. Lets all be friends XD. – Alex Amato Sep 13 '16 at 06:33
  • I think Khoa is true, because when you embed a webkit in your app, you are actually have a built-in browser in your application, so ? you can wait for it to do its DOM (javascript and other stuff) be ok then you can use the generated html script. – SdSaati Sep 05 '18 at 00:48