![[Home]](/wiki.gif)
Attracting Web TrafficAttracting Web Traffic|
Links:
|
Subjects > ... > Search Engine News
"Search Engine Optimization FREE!" by Paul Boutin, Webmonkey, August 6, 2001 --- http://hotwired.lycos.com/webmonkey/01/23/index1a.html
Before I landed my cushy job as a magazine editor, I spent three years under the hood at Hotbot as an engineer and manager. Between days reading our log files and nights shmoozing with other search engineers, I learned more than I'd ever wanted to know about where search traffic comes from, and where it goes to. I even wrote an article about it for Webmonkey.
But I had put all that behind me ... until my lovely wife, Christina, asked me about search engine optimization for Artloop, her fine art research and location service.
Dozens of companies had pitched their optimization services to her, but Christina, a former MSN manager as smart about database schema as she is about business plans, balked. Why pay someone to set up bogus domains, build huge farms of gateway pages, and cram hundreds of keywords like "britney spears" into Artloop's HTML? The very idea ran contrary to the information architecture and site layout her staff had worked so hard to make as clean and clear as possible for their visitors. Moreover, as a Web user herself, she'd learned to recognize these traffic-grabbing methods and had become wary of sites using tricks to get her to click. Why should she assume her own customers would behave differently?
And she was right: Trying to fool search engine users with keywords and trick tags makes sense only if your goal is to flash a lot of ad banners, return traffic be damned. That used to be the business model for an entire industry. But most sites in business today hope to convert first-time visitors into loyal customers by building long-term relationships. Sure, searchers need to find your site, but the results on Hotbot's Top Ten lists show that the only results people stick with are the ones that don't try to scam them. Trap doors, redirects, keyword spam, and multiple domains that host the same pages are more likely to make people reach for the back button (a move the Direct Hit technology behind Top Ten results can detect), not their credit cards.
So, rather than waste money on consultants, Christina and I decided to create our own search optimization spec. Using data gleaned from representatives of leading search engines, insider data, and old-fashioned trial and error, we came up with our own strategy for getting traffic from search engines and portals without having to fake people out. In the process, we encountered so many dubious "experts" with something for sale — software, books, services — that we decided to raise the bar on them and publish our notes for free.
Imagine our surprise when Google's engineers read this article (when it first published in early June, 2001) and invited us to visit their offices to dig even deeper into the workings of their gigapage Web index. Of course we took them up on the offer, and we've updated this article with our notes from those meetings. We've also included answers to the best questions from the hundreds of emails we've received over the past couple months.
. . .
The Yahoo directory accounts for half the traffic referred to most sites. So get your site listed on Yahoo, and your traffic can literally double overnight. Beyond that, most search engine traffic comes from two places: Google and Inktomi.
Traffic from Google has increased at an astonishing rate over the past year: Jakob Nielsen's search engine referrals to his Useit site confirm this, as do the unpublished reports from retail sites like Stylata. Google, once considered a niche site for nerds, is the Wall Street Journal's pick for best search engine on the Net, and the traffic numbers seem to agree.
Inktomi, the number two traffic generator, doesn't run its own search site. Instead, the company provides the technology behind MSN Search and AOL Search, two top referrers, as well as Hotbot and over a dozen more.
Portal sites like Excite, Lycos, and AltaVista?Create still draw lots of traffic, but together Google and Inktomi outweigh the entire rest of the field. Add it up and it's pretty clear how to maximize your traffic for the least effort:
Get yourself into Yahoo's directory. Make sure your site is thoroughly crawled by Google and Inktomi. Get lots of links to your site from domains that a lot of other sites link to — that's how Google and Inktomi determine relevance when ranking search results. For all other search engines, implement a blanket strategy that gets you reasonable results. By not chasing each one of them separately, you can put your company's time and money to more important uses. All of this can be accomplished with one, three-step process. And it really is as easy as 1-2-3.
There are quite a few things you can do to grab the attention of search engines and directories:
Frames used to be the biggest roadblock to getting crawled, but no more: Both Google and Inktomi now crawl them (the section of Inktomi's support FAQ that claims this isn't so is out of date, according to the company). Instead, the problem with most e-commerce sites today is that their product pages are dynamically generated. While Google will crawl any URL that a browser can read, most of the other search engines balk at links with "?" and "&" characters that separate CGI variables (such as "artloop.com/store?sku=123&uid=456"). As a result, many individual product pages don't show up outside of Google.
One way to circumvent this difficulty is to create static versions of your site's dynamic pages for search engines to crawl. Unfortunately, duplicating your pages is a huge amount of extra work and a constant maintenance chore, plus the resulting pages are never quite up-to-date — all the headaches dynamic pages were designed to eliminate.
A far better strategy is to follow the lead of Amazon and rewrite your dynamic URLs in a syntax that search engines will gladly crawl. So URLs that look like this ...
amazon.com/store?shop=cd&sku= B00004WFIZ&ref=p_ir_m&sessionID= 107-6571839-6268523
... become ...
amazon.com/exec/obidos/ASIN/ref= B00004WFIZ/ref=pd_ir_m/107-6571839-6268523
Amazon's application server knows the fields in the URL are actually CGI parameters in a certain order, and processes them accordingly.
J.K. Bowman's Spider Food site explains how to fix URLs for most popular e-commerce servers. One of Artloop's Web programmers learned Apache rewrite rules that tell Apache how to translate slash-separated URLs into a format used by their Netzyme application server. On the back end, Netzyme is passed something like this:
artloop.com/cgi-bin/CssP?Create.exe?CsspApp?Create= ArtLoopClient1?Create&CssServer?Create=localhost%3A32401&CsspFn?Create =@/details/ArtistDetail?Create.html:@:getForm&ObjectLocation?Create =ART&ArtistID?Create=3918
But users and search engines see the tidier, Apache-served URLs, which look something like this:
artloop.com/artists/profiles/3918.html
Not only are the rewritten URLs crawlable by all search engines, they're also more human-friendly, making them easier to pass around the Net.
Many readers have written in to to ask if the search engines will begin crawling and indexing Flash content soon. The answer, as you might guess, is no. Unlike PDF files, Flash files rarely contain information in text format. Search developers don't want to clutter up their indexes with a million "Skip Intro" pages.
There are a lot of automated search engine submission services that you can use to submit your site to as many search engines as possible. The one most recommended by people I talked to is Submit It, an early player that did so well, Microsoft bought them — Submit It is now part of MSN bCentral, and it charges a minimum fee of US$59 to keep a few URLs submitted for a year.
You can avoid the fees by simply submitting to individual search engines on your own. Start with UseIt?Create's list of top referrers — that's where most of the traffic you can get will come from. And while you'd think submitting your site to one Inktomi-powered site would work for all of them, optimization experts have told us it works better if you hit them all.
Submit It does submit your site to the busiest directory sites, except for the biggies: Yahoo, LookSmart?Create (which MSN serves under its logo), and the Open Directory Project (which powers Lycos, Hotbot, and Netcenter categories). Some of these directories charge for submission, but $400-500 total will get your most important pages into the most trafficked places.
Yahoo still offers free submissions, except for business categories, which cost $199. But even the fee doesn't guarantee they'll accept your site, just that they'll decide on it within a week — with free submissions, you don't even get the promise that they'll ever get around to evaluating it, given the incredible volume of submissions.
Once you've submitted your pages, be ready to wait a month, two, or three before they're crawled and indexed. It's frustrating, but processing a billion Web pages takes time — at a nonstop rate of one hundred per second, it would still take almost four months.
|
http://images.amazon.com/images/P/B0001XQNSE.01-A1KDZ23Y0QWKQ3.MZZZZZZZ.jpg
|
|
Interested in OpenSource?
