Get a list of URLs from a site

Question

I'm deploying a replacement site for a client but they don't want all their old pages to end in 404s. Keeping the old URL structure wasn't possible because it was hideous.

So I'm writing a 404 handler that should look for an old page being requested and do a permanent redirect to the new page. Problem is, I need a list of all the old page URLs.

I could do this manually, but I'd be interested if there are any apps that would provide me a list of relative (eg: /page/path, not http:/.../page/path) URLs just given the home page. Like a spider but one that doesn't care about the content other than to find deeper pages.

http://superuser.com/questions/329736/wget-recursively-retrieve-urls-from-specific-website — Ciro Santilli OurBigBook.com, Apr 10 '16 at 12:02

score 92 · Accepted Answer · answered May 13 '09 at 12:26

92

I didn't mean to answer my own question but I just thought about running a sitemap generator. First one I found http://www.xml-sitemaps.com has a nice text output. Perfect for my needs.

answered May 13 '09 at 12:26

Oli

235,628
64
220
299

1

But there is a limit of 5000 links!..:( I'm looking for any free php sitemap generator script. – Jenson M John Aug 03 '13 at 10:25
15

Current limit is **500** — getting smaller… – Oli Studholme Jan 11 '14 at 07:34
1

It errored out for me: ::::::: An error occured There was an error while accessing the URL specified: http://159.121.ssss/ Please make sure to specify the correct website URL and resubmit your request. – JustJohn Jul 26 '16 at 21:29
1

FYI: If you are using frontend routing you will not get those routes from this method. – jasonleonhard Jun 29 '19 at 22:39
1

FYI: if the website uses authentication and/or authorization you will not get all the routes either. – jasonleonhard Jun 29 '19 at 22:54
it doesnt work for dynamic url , example my own react website – Rohan Devaki Feb 15 '22 at 16:06

alamar · Answer 2 · 2018-08-22T08:59:54.687

59

do wget -r -l0 www.oldsite.com

Then just find www.oldsite.com would reveal all urls, I believe.

Alternatively, just serve that custom not-found page on every 404 request! I.e. if someone used the wrong link, he would get the page telling that page wasn't found, and making some hints about site's content.

edited Aug 22 '18 at 08:59

answered May 13 '09 at 12:28

alamar

18,729
4
64
97

21

Notably, since this returns a list of *files*, not URLs, this would only really work for sites that are collections of static HTML files. If the site has URL query parameters, server-side rewritten URLs, or any kind of `include`/`require`/etc. assembling of pages, this won't really work. – T.J. Schuck Jun 24 '11 at 19:41
1

I might be misunderstanding wget. I thought 'wget' was for downloading the contents of the site? – Cosmic Hawk Oct 21 '16 at 16:06
1

@Doomsy yes, but when you've downloaded all the content you surely know all the URLs to that content, and without downloading there's no way to find out URLs. – alamar Oct 24 '16 at 09:21
4

Consider the default depth. https://www.gnu.org/software/wget/manual/html_node/Recursive-Retrieval-Options.html#Recursive-Retrieval-Options – PJ Brunet Aug 22 '18 at 02:36
1

@PJBrunet thx, fixed! – alamar Aug 22 '18 at 09:00
2

@alamar Yes there's "-r -l inf" for infinite recursion, but I recommend people check out the documentation--so many cool options! The "-m" option will mirror and I'm going to try "-R.jpg,.jpeg,.gif,.png" which I think skips images. – PJ Brunet Aug 22 '18 at 23:00
1

can we export this list into a pdf file or excel file? – Pooya Panahandeh Jun 05 '20 at 12:08
1

I don't see why not? `find -type f > urls.csv` – alamar Jun 05 '20 at 12:32

Franck Dernoncourt · Answer 3 · 2016-02-10T22:46:53.520

Here is a list of sitemap generators (from which obviously you can get the list of URLs from a site): http://code.google.com/p/sitemap-generators/wiki/SitemapGenerators

Web Sitemap Generators

The following are links to tools that generate or maintain files in the XML Sitemaps format, an open standard defined on sitemaps.org and supported by the search engines such as Ask, Google, Microsoft Live Search and Yahoo!. Sitemap files generally contain a collection of URLs on a website along with some meta-data for these URLs. The following tools generally generate "web-type" XML Sitemap and URL-list files (some may also support other formats).

Please Note: Google has not tested or verified the features or security of the third party software listed on this site. Please direct any questions regarding the software to the software's author. We hope you enjoy these tools!

Server-side Programs

Enarion phpSitemapsNG (PHP)

Google Sitemap Generator (Linux/Windows, 32/64bit, open-source)

Outil en PHP (French, PHP)

Perl Sitemap Generator (Perl)

Python Sitemap Generator (Python)

Simple Sitemaps (PHP)

SiteMap XML Dynamic Sitemap Generator (PHP) $

Sitemap generator for OS/2 (REXX-script)

XML Sitemap Generator (PHP) $

CMS and Other Plugins:

ASP.NET - Sitemaps.Net

DotClear (Spanish)

DotClear (2)

Drupal

ECommerce Templates (PHP) $

Ecommerce Templates (PHP or ASP) $

LifeType

MediaWiki Sitemap generator

mnoGoSearch

OS Commerce

phpWebSite

Plone

RapidWeaver

Textpattern

vBulletin

Wikka Wiki (PHP)

WordPress

Downloadable Tools

GSiteCrawler (Windows)

GWebCrawler & Sitemap Creator (Windows)

G-Mapper (Windows)

Inspyder Sitemap Creator (Windows) $

IntelliMapper (Windows) $

Microsys A1 Sitemap Generator (Windows) $

Rage Google Sitemap Automator $ (OS-X)

Screaming Frog SEO Spider and Sitemap generator (Windows/Mac) $

Site Map Pro (Windows) $

Sitemap Writer (Windows) $

Sitemap Generator by DevIntelligence (Windows)

Sorrowmans Sitemap Tools (Windows)

TheSiteMapper (Windows) $

Vigos Gsitemap (Windows)

Visual SEO Studio (Windows)

WebDesignPros Sitemap Generator (Java Webstart Application)

Weblight (Windows/Mac) $

WonderWebWare Sitemap Generator (Windows)

Online Generators/Services

AuditMyPc.com Sitemap Generator

AutoMapIt

Autositemap $

Enarion phpSitemapsNG

Free Sitemap Generator

Neuroticweb.com Sitemap Generator

ROR Sitemap Generator

ScriptSocket Sitemap Generator

SeoUtility Sitemap Generator (Italian)

SitemapDoc

Sitemapspal

SitemapSubmit

Smart-IT-Consulting Google Sitemaps XML Validator

XML Sitemap Generator

XML-Sitemaps Generator

CMS with integrated Sitemap generators

Concrete5

Google News Sitemap Generators The following plugins allow publishers to update Google News Sitemap files, a variant of the sitemaps.org protocol that we describe in our Help Center. In addition to the normal properties of Sitemap files, Google News Sitemaps allow publishers to describe the types of content they publish, along with specifying levels of access for individual articles. More information about Google News can be found in our Help Center and Help Forums.

WordPress Google News plugin

Code Snippets / Libraries

ASP script

Emacs Lisp script

Java library

Perl script

PHP class

PHP generator script

If you believe that a tool should be added or removed for a legitimate reason, please leave a comment in the Webmaster Help Forum.

Is there anyone that provide a printscreen from all the url? — ValRob, Apr 06 '18 at 05:56

score 6 · Answer 4 · answered Jun 26 '14 at 13:20

6

The best on I have found is http://www.auditmypc.com/xml-sitemap.asp which uses Java, and has no limit on pages, and even lets you export results as a raw URL list.

It also uses sessions, so if you are using a CMS, make sure you are logged out before you run the crawl.

answered Jun 26 '14 at 13:20

Collins

1,069
14
35

6

sounded good, but it's broken. – NoobishPro Jul 17 '16 at 00:43

score 3 · Answer 5 · answered May 13 '09 at 12:28

So, in an ideal world you'd have a spec for all pages in your site. You would also have a test infrastructure that could hit all your pages to test them.

You're presumably not in an ideal world. Why not do this...?

Create a mapping between the well known old URLs and the new ones. Redirect when you see an old URL. I'd possibly consider presenting a "this page has moved, it's new url is XXX, you'll be redirected shortly".
If you have no mapping, present a "sorry - this page has moved. Here's a link to the home page" message and redirect them if you like.
Log all redirects - especially the ones with no mapping. Over time, add mappings for pages that are important.

score 2 · Answer 6 · edited Aug 07 '15 at 14:23

2

wget from a linux box might also be a good option as there are switches to spider and change it's output.

EDIT: wget is also available on Windows: http://gnuwin32.sourceforge.net/packages/wget.htm

edited Aug 07 '15 at 14:23

jeffshieldsdev

5
5

answered May 13 '09 at 12:28

Thomas Schultz

2,446
3
25
36

score 1 · Answer 7 · answered May 13 '09 at 12:27

Write a spider which reads in every html from disk and outputs every "href" attribute of an "a" element (can be done with a parser). Keep in mind which links belong to a certain page (this is common task for a MultiMap datastructre). After this you can produce a mapping file which acts as the input for the 404 handler.

score 1 · Answer 8 · answered May 13 '09 at 12:29

1

I would look into any number of online sitemap generation tools. Personally, I've used this one (java based)in the past, but if you do a google search for "sitemap builder" I'm sure you'll find lots of different options.

answered May 13 '09 at 12:29

Eric Petroelje

59,820
9
127
177

Get a list of URLs from a site

8 Answers8

Linked