Copying the entire dropdown menu from one site to another

Question

We are using a third part web app which does not allow or have an API yet, this third party app is basically a membership registration website and each member belongs to a specific category.

I need to use these category in our internal system and so far I have been manually adding the category in a drop down menu of a form as soon as a new category in created in third party app.

Since there is no access to an api so I am wondering if it is possible to crawl the third party app where the dropdown menu is and copy the entire dropdown menu over to our internal site.

I wish i can show you the efforts I have made so far but I am stuck on how to even begin this. I did however search online but all I could find is how to copy a dropdown on a same page.

Any push to the right direction will really be helpfull, the technologies I am working with is PHP and JS

that I am afraid I don’t know because the third party app is not hosted on our server — Saadia, Sep 01 '15 at 09:49
Check if there is `Access-Control-Allow-Origin: *` in the response of your request (https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS#line1). Also, you can do it from your server, but I don't know this site so I don't know if it's legal. — Mosh Feu, Sep 01 '15 at 09:56
Do the other website allow request from one to another in php using curl for example ? — dvhh, Sep 01 '15 at 10:03
@GuruprasadRao - CORS _may be_ a showstopper for pure client side solution - but there's a **PHP** tag in there, so a server side "scrape" is trivially easy — Jaromanda X, Sep 01 '15 at 10:03
Having less/no knowledge on `php` is my disadvantage @JaromandaX.. :) — Guruprasad J Rao, Sep 01 '15 at 10:07
@JaromandaX easy really depends on the HTML complexity to scrape from, but that is pushing in the right direction — dvhh, Sep 01 '15 at 10:07
@GuruprasadRao - the php code doesn't have to do any more than simply act as a conduit to get the foreign site's html to the client, bypassing CORS requirements altogether - it's less than 10 lines of code any PHP programmer could cobble together easily — Jaromanda X, Sep 01 '15 at 10:11
The dropdown is javascript one it would not be easy to copy in any way — dvhh, Sep 01 '15 at 10:29
@Saadia - I have added an example script to my answer. You may find it useful. — Raad, Sep 01 '15 at 10:57

score 1 · Accepted Answer · edited May 23 '17 at 12:29

I don't think CORS is going to help you here, as it's function is to provide a legal/safe way of sharing web resources across different domains (i.e. images/css files/web fonts), not data.

If there is no API for the data you need, you are almost certainly limited to scraping the data out of the web page. You can do this by first issuing a request for the page to obtain the html, then searching/parsing the html to find the drop-down menu, then finally parsing the menu items to obtain a list that you can use for your own drop-down.

So, some pointers:

Obtain page html - See PHP: how can I load the content of a web page into a variable?

Parse html - See PHP Parse HTML code

Of course how easy this ends up being depends on many factors, e.g.

Can you just request the page containing the drop-down, or does the web app need authentication? You may need to refine the curl request as appropriate.
Can you easily identify the html drop-down, e.g. using a unique id tag. If so, you could use DOMDocument::getElementById, otherwise you may need more complex logic to parse the page html and find the menu.

Either way, it should be possible to achieve - just remember that the third-party app is not under your control, and as such may be subject to changes that break your program.

LATEST UPDATE:

Added in retrieval of value, and we hide parse warnings using internal_errors.

Here's a simple PHP script that will print out the text and value of each of the drop-down options:

    <?php
    libxml_use_internal_errors(true);

    $html = file_get_contents('http://example.com/');
    $domdoc = new DomDocument;
    $domdoc->loadHTML($html);
    libxml_clear_errors();
    $menu = $domdoc->getElementById('tid');
    $options = $menu->childNodes;

    foreach ($options as $option) {
        echo($option->nodeValue)." - ".$option->getAttribute('value')."<br>";
    }

    ?>

wow this is great...... would it at all be possible to get the value as well, for example the above code does give me the name of dropdown but not its value. I am looking for value 24 and name Business Consulting` ` — Saadia, Sep 01 '15 at 11:24

Copying the entire dropdown menu from one site to another

1 Answers1

LATEST UPDATE: