Make A Crawler Page

Featured: Featured Stories | Picture Gallery

Art/A/Architecture/Greek | Ana-Liofa On Legalizing Guns

Chat11.com Web Bible11.com MyBibleCenter.com

Help Search Engines Discover Your Entire Site

Subjects > ... > Search Engine News (Search for Search Engine News)

Make a Crawler Page

It isn't necessary to submit every page on your site to the search engines. Just make sure they can find all the pages that matter by hopping links from your front door. To do that, make a "crawler page" that contains nothing but a link to every page you want search engines to crawl. Use the page's TITLE info as the link text — this helps improve your site score. For an example, check out Artloop's crawler page.

Basically, the crawler page is a site map that lists all the pages on your site — it may be a bit too big for humans to read through, but it will be no problem for a search engine. Add an obscure link to the crawler page on one of your site's top-level pages, using a small amount of text. MSN used to use 1x1 images for this trick, but the Google geeks warned us to avoid such obviously invisible tags. "Why not just label it 'site map?'" one asked. Search engine spiders will find it as soon as they get to your site, and suck down all the pages it finds on it.

Don't worry, the crawler page won't show up in search results. It does get pulled into the search engine's index, but because it has no text or tags to match a query, it isn't listed as a result. The pages it links to, however, will appear because the search engine's spider found them right after it visited the crawler page. Wired News, for example, uses hierarchical sets of crawler pages to make sure every story ever published is crawlable from the top of the site.

For Artloop, we decided to break the crawler pages down into 100KB pages or smaller, just to be careful — we wanted to prevent search spiders from timing out or deciding the pages were too big to crawl.

For the rest of the article, go to * http://hotwired.lycos.com/webmonkey/01/23/index1a.html

Google in 30 pages or less Getting The Search Engine Ranking Your Website Deserves - Meta tags yield to Google's Page Rank As Search Engine Standard Search Engine Optimization For Dummies Search Engine Optimization and Placement An Internet Marketing Course For Webmasters Internet Marketing and Search Engine Positioning Do It Yourself Guide Step By Step Guide To Successfully Promoting A Web Site Yahoo For Dummies Search and Researching On the Internet and World Wide Web Neal Schuman Authoritative Guide to Kids Search Engines, Search Directories and Kids Portals A Kids Guide To Safe Searching And How To Find Almost Anything On The Internet	Insiders Guide to SEO - How To Get Your Website To The Top of the Search Engines Streetwise Maximize Web Site Traffic Build Website Traffic Fast and FRee by Optimizing Search Engine Placement 10 Amazing Search Engine Secrets For Marketing And Advertising Your Internet Web Site With Top Positioning Strategies That Will Skyrocket Your Traffic Google Hacks - 100 Industrial Strength Tips and Tools Search Engine Visibility Search Engine Optimization With Web Position Gold Google Pocket Guide Search Engines For The World Wide Web Visual Quickstart Extreme Searchers Guide To Web Search Engines Understanding Search Engines Mathematical Modeling and Text Retrieval Sorting Out The Web Approaches To Subject Access Web Developer Guide To Search Engines Ebook: Insiders Guide To Search Engine Marketing	Webmasters Secret Internet Marketing And Search Engine Positioning Strategies Search Engine Advertising, Buying Your Way To The Top To Increase Sales A Beginners Guide To Search Engine Placement And Ranking Web Marketing Achieving Top 10 Ranking On Internet Search Engines Insider Trade Secrets From Positioning Pros Google For Dummies How To Do Everything With Google Cognitive Perspective on Search Engine Technology The Invisible Web - Uncovering Information Sources Search Engines Can't See Google and Other Search Engines - Visual Quick Start Guide Increase Your Web Traffic In A Weekend

Digest Of Search Engine And Website Promotion News Search Engine Visibility Search Engine Advertising http://images.amazon.com/images/P/B0001XQNSE.01-A1KDZ23Y0QWKQ3.MZZZZZZZ.jpg

Bobsgear - Get A Free Enterrpise Wiki Space!
Review: The Bobsgear Project was started to develop a variety of Confluence plugins. This installation of the Confluence Enterprise wiki includes flexible attachments, many Confluence plugins, personal blogs, interesting articles, and more. Bobsgear already has spaces related to politics, art and photography wiki, technical issues wiki, ediscovery wiki, health, Christian theology and Sabbath School wiki, the bible, book reviews, and quotations. Bobsgear allows free signup, and invites anyone to create a free hosted Confluence wiki space.

NEW USERS CLICK HERE! for a quick introduction to Wiki.

Elapsed:1