There are lots of reasons you could will need to search out every one of the URLs on a website, but your correct purpose will decide Everything you’re looking for. By way of example, you might want to:
Identify each indexed URL to investigate troubles like cannibalization or index bloat
Gather latest and historic URLs Google has observed, specifically for site migrations
Locate all 404 URLs to Get well from post-migration mistakes
In Just about every scenario, a single Software gained’t Provide you everything you need. Regretably, Google Lookup Console isn’t exhaustive, and a “web-site:example.com” lookup is limited and tough to extract info from.
On this put up, I’ll walk you through some equipment to build your URL list and ahead of deduplicating the information utilizing a spreadsheet or Jupyter Notebook, dependant upon your web site’s measurement.
Old sitemaps and crawl exports
Should you’re searching for URLs that disappeared within the Dwell website recently, there’s a chance an individual on your group could possibly have saved a sitemap file or a crawl export ahead of the variations ended up made. When you haven’t previously, look for these files; they are able to normally supply what you would like. But, in the event you’re examining this, you almost certainly didn't get so Fortunate.
Archive.org
Archive.org
Archive.org is a useful tool for Search engine optimization duties, funded by donations. Should you seek out a domain and choose the “URLs” alternative, you'll be able to obtain around 10,000 mentioned URLs.
On the other hand, There are many limitations:
URL limit: You may only retrieve approximately web designer kuala lumpur 10,000 URLs, which happens to be insufficient for greater web-sites.
High-quality: Many URLs might be malformed or reference resource files (e.g., illustrations or photos or scripts).
No export solution: There isn’t a designed-in technique to export the checklist.
To bypass The dearth of an export button, make use of a browser scraping plugin like Dataminer.io. However, these limitations imply Archive.org may not offer a complete Answer for greater web sites. Also, Archive.org doesn’t show no matter whether Google indexed a URL—but if Archive.org uncovered it, there’s a great prospect Google did, way too.
Moz Professional
While you could ordinarily utilize a hyperlink index to discover external web sites linking to you, these resources also find out URLs on your website in the process.
How you can utilize it:
Export your inbound back links in Moz Professional to obtain a quick and simple listing of target URLs from your web-site. In the event you’re working with a large Web site, consider using the Moz API to export info over and above what’s manageable in Excel or Google Sheets.
It’s imperative that you Take note that Moz Professional doesn’t validate if URLs are indexed or found out by Google. Having said that, since most web pages implement precisely the same robots.txt policies to Moz’s bots since they do to Google’s, this process frequently is effective effectively as a proxy for Googlebot’s discoverability.
Google Research Console
Google Lookup Console presents many precious resources for constructing your list of URLs.
Backlinks stories:
Similar to Moz Pro, the Links part gives exportable lists of concentrate on URLs. However, these exports are capped at one,000 URLs Just about every. You can apply filters for unique pages, but given that filters don’t apply to the export, you may have to depend on browser scraping tools—restricted to 500 filtered URLs at any given time. Not great.
General performance → Search Results:
This export gives you a summary of internet pages receiving search impressions. When the export is restricted, you can use Google Search Console API for more substantial datasets. Additionally, there are free of charge Google Sheets plugins that simplify pulling a lot more substantial facts.
Indexing → Webpages report:
This section presents exports filtered by concern style, even though they are also minimal in scope.
Google Analytics
Google Analytics
The Engagement → Internet pages and Screens default report in GA4 is a wonderful resource for amassing URLs, with a generous Restrict of 100,000 URLs.
Even better, you are able to apply filters to create different URL lists, correctly surpassing the 100k limit. For example, if you wish to export only blog URLs, follow these techniques:
Step 1: Increase a section to the report
Stage 2: Click “Produce a new section.”
Phase 3: Outline the segment having a narrower URL sample, which include URLs that contains /blog/
Take note: URLs present in Google Analytics may not be discoverable by Googlebot or indexed by Google, but they offer important insights.
Server log information
Server or CDN log information are perhaps the last word Software at your disposal. These logs seize an exhaustive record of each URL route queried by people, Googlebot, or other bots during the recorded period.
Concerns:
Information size: Log documents could be massive, lots of internet sites only keep the last two months of knowledge.
Complexity: Analyzing log information can be difficult, but different applications are offered to simplify the procedure.
Incorporate, and great luck
When you finally’ve gathered URLs from every one of these sources, it’s time to combine them. If your site is sufficiently small, use Excel or, for more substantial datasets, equipment like Google Sheets or Jupyter Notebook. Make sure all URLs are consistently formatted, then deduplicate the checklist.
And voilà—you now have an extensive listing of present, outdated, and archived URLs. Good luck!