Contents | (Visit Preferences to set your user name.) | Related To Web Robots And Spiders | RecentChanges | Preferences | Index | Login | Logout
Web Robots And Spiders
Subjects > Computers (Search) > Internet (Search) > Web (Search) > Search Engine News (Search)
Web Spiders (or web robots), are automatic programs used by search engines to visit websites. Some robots are designed to gather content for indexing into search databases, and other robots are designed to do link checking, HTML validation, and other tasks.
Some typically seen robot agents:
Other pages discussing web spiders:
Other sites that talk more about web spiders:
Sites devoted to various spiders:
Other spidering software:
- http://www.ficstar.com/index.htm - Spider customized by company for each extraction job
- No exact prices are quoted. "For a job costing 30 human days, it can finish in only 2-4 hours"
- http://www.newprosoft.com/
- Web spider software - $39 - Trial version 3.0 download
- For keywords specified by user, Visual Web Spider extracts and visits the related URL listed in the top of the search engines like Google and Yahoo, then extracts metadata (title, keywords, description) and new URLs from visited pages. Each URL is ensured to be unique and visited only once.
- Key features:
- Extracts URLs and metadata (title, keywords and description).
- Indexing using specific web page;
- Indexing using specific keywords (all words, exact phrase, any words, without words);
- Indexing using specific language, domain, country, directory category;
- Indexing using specific Urls list;
- Supports multithreaded downloading (up to 50 threads);
- Fully automated, multithreaded web robot.
- Filters out specific keywords;
- Auto-removes duplicate or invalid syntax URLs;
- Limits the search depth, maximum number of pages and bytes;
- Exports the extracted URLs and metadata (title, keywords, description) to a text or CSV format file, directory management system (Gossamer Links2, PowerSeek?Create|Search SQL, LinksCaffe?Create|Search etc) or mySQL database (only registered version);
- Builds tree of the extracted links;
- Very simple to learn and work.
- http://www.trellian.net/sitespider/download.htm - Trellian Site Spider $29.95, version 1.0. Internet Studio $199.
- Features:
- Create a site map
- Detailed resource list
- Imports bookmarks from Internet Explorer
- Integrated windows explorer
- Extract data from website including email addresses, movie, music files, images, photos, applications.
- http://www.velocityscape.com/
- Web Scraper Plus+ puts the power of web data extraction into the hands of decision makers. If you can surf the web, and you can use Excel, then you can mine web data using this intuitive screen scraping product. In just a few minutes, you can create a custom scraping template that will extract an entire website into the an Excel spreadsheet. You need the data, you don’t have time to wait, Web Scraper Plus+ puts the power in your hands.
- 1. Building Contact Lists
- 2. Extracting Product Catalogs
- 3. Aggregating Real Estate Info
- 4. Automating Search Ad Listings
- 5. Clipping News Articles
- 6. Automating Auction Sites
- 7. Extracting Gambling Odds
- 8. Legal Notices (Foreclosures, etc)
- 9. Server Migration (CMS, Commerce)
- 10. Unspecified Military Use
Bobsgear - Get A Free
Enterrpise Wiki Space!
Review: The Bobsgear Project was
started to develop a variety of Confluence
plugins. This installation of
the Confluence Enterprise wiki includes flexible
attachments, many Confluence plugins, personal blogs,
interesting articles, and more. Bobsgear already has spaces related to
politics, art and
photography wiki,
technical issues wiki,
ediscovery wiki, health,
Christian theology and Sabbath
School wiki, the
bible, book reviews,
and quotations. Bobsgear
allows free signup, and invites anyone to create a
free hosted Confluence wiki space.
NEW
USERS CLICK HERE! for a quick introduction to
Wiki.
Interested in Political Science Journals?