How to get mobile page address from web page address in crawling?

Question

I have made 'crawling tool by python' for personal project and I want to get mobile page address from each web page address.

like transform 'www.youtube.com' to 'm.youtube.com'

I want to know only mobile page address.

Is it possible to get mobile page address without using 'selenium'?

What are you calling the "mobile page" ? + what have you already tried to perform ? — Arount, May 17 '17 at 09:03
Ok, is this for youtube specifically or this is just an example ? — Arount, May 17 '17 at 09:08
i just try using the browser on mobile phone to find mobile page address. — alexmoon, May 17 '17 at 09:09
Ok i will make you an answer, it will not give you _the_ solution, but I believe there is no _one_ solution, you will have to be creative anyway — Arount, May 17 '17 at 09:10

score 0 · Accepted Answer · edited May 23 '17 at 12:26

There is two way to serve a mobile page.

You can first just check URL. As the example you gave in comments m.youtube.com explicit the fact you are browsing in mobile mode.

But you can check the HTTP header User-Agent too. This header show what browser was used to send query. The server can extract this info and according to given value can determine if browser is mobile or desktop one.

This will be your main issue here, all web servers does not use the same method to handle mobile templates. Lot of websites use both methods I explained and it will be hard to create a generic scraper that find alone how to access to mobile template.

Si what you can do:

Use a mobile browser with selenium: https://github.com/SeleniumHQ/selenium/wiki/WebDriver-For-Mobile-Browsers

Or

If you want to dynamically switch from desktop to mobile, play with user-agent: Change user agent for selenium driver

How to get mobile page address from web page address in crawling?

1 Answers1