![[Home]](/wiki.gif)
How Does The Fast Spider WorkHow Does The Fast Spider Work|
Links:
|
Subjects > Computers > Internet > Search Engine News > Search Engine News Lycos
April 2004
The spider goes to a web site and follows hyperlinks throughout the site.
The spider parses (reads in) certain pages and downloads them into a file or database as it moves through a web site. The spider is set to only parse a set number of pages on the same site.
When the spider has reached the limit of URLs determined by the user's subscription service, it stops spidering your web site. The file of spidered URLs are then built into a search catalog. These URL's are displayed as part of search results when a relevant match is made.
Spidering starts at the root URL (ex: www.lycos.com is the root URL vs. www.lycos.com/search) and discovers the total number of URL's in your Web Site, up to a maximum of 500 URL's. Then, the spider collects and submits text from the parsed page into the catalog used to display search results. The spider will gather text from the number of web pages that the user included in the subscription, accessing pages by breadth and then depth. For example, under the root URL, a Web Site has 5 unique paths. Each path has links to 50 web pages. If the user paid for 20 URL's, the spider would collect text from the 5 unique paths first and then drill down into the first unique path and continue this process until the 20 URL's had been spidered.
FAST's spiders can have trouble following links generated by JavaScript?Create. If your account is missing pages and you use JavaScript?Create heavily, you can submit individual pages using our "Advanced Submission".
FAST's spiders do not automatically follow HTML "meta refresh" tags. If your pages use these tags, enter the target destination pages instead.
More informatioon about web spiders:
|
http://images.amazon.com/images/P/B0001XQNSE.01-A1KDZ23Y0QWKQ3.MZZZZZZZ.jpg
|
Search for books about:
|
Interested in Affiliate Programs?
