A client recently came to Avenue A | Razorfish with a major problem. Their site, containing over 150,000 URLs, had hundreds of non-functioning pages, often leading visitors to a 404 error. With such a large site, the client had difficulty identifying every “broken” page. Even if they had a complete list of broken pages, they couldn’t identify the pages that contained the dynamically generated links to them, making the process of correcting this issue very difficult.

The AA|RF Product Technology team in Philadelphia, tasked with the development and support of two search technologies - SEOsource and SEOdirect - has developed a spider called SiLC (Super Inteligent Link Crawler) that can efficiently crawl large dynamically generated sites. In addition to tracking site errors, SiLC tracks the linking relationship between the links it encounters while performing a crawl, resulting in a hierarchical view of the site’s structure. From this “link map”, AA|RF was able to provide the client with a complete list of broken pages, as well as the URL of the page that contained each broken link.

Since SiLC is designed to crawl and process content that is difficult for standard search engine spiders, it can be used to generate a list of pages that are not visible to search engines. When combined with a link map, we can visually show which sections of a site are effectively invisible to spiders. By providing this information to the client, AA|RF made it much easier for them to identify and fix the broken pages on their site. Future articles will explore additional ways that AA|RF is leveraging SiLC to solve client problems.

Search Engine Optimization will continue to gain acceptance in 2007 as a very important factor to consider when designing or redesigning a Web site. The traffic generated from search engines to sites that have considerable exposure is often greater than 50% of total traffic, and can reach as high as 80-85% in some cases. A recent Point of View from Avenue A | Razorfish explores some of the likely tactics that will increase in importance in 2007. The POV will soon be available at the main Web Site in the Points of View section.

2007 Trends first reminds readers that “Search is Search,” as described (PDF) by Josh Palau in a POV by that name, released late last year. Synergies between all forms of Paid Search with Organic SEO must be capitalized on. Other forecasts include increased focus on content creation, both for the benefit of search engines and for the purpose of driving links to a Web site. The process of making people want to link to a Web site is becoming known as “Link Baiting.”

Other tactics include the use of Web 2.0 “communities” to help further gain exposure, as well as to conduct informal market research. The SEO benefit is increased traffic and the likelihood of links being created to the parent site. SEO analysts such as Andy Powers have already begun to see positive effects from Wikipedia placement, for example, in the form of increased measured traffic and conversions.

The Search Marketing Trends newsletter will now be available online, in a centralized location, at the URL www.SearchMarketingTrends.com. The current newsletter will be posted to the home page, with archives detailing each headline topic for the past issues. Soon, SMTrends will have RSS functionality. All current subscribers will continue to receive the weekly email version of the newsletter. Should a subscriber wish to unsubscribe at any point, that functionality is available in the “Subscribe” section of the Web site.

The SMTrends team is hoping that the Web site becomes a favorite destination for weekly search news seekers across the world. This year, AA|RF offices from the U.S., U.K., Australia, Germany, and Hong Kong will help to contribute content to the newsletter. Additionally, SMTrends will provide coverage of major conferences and presentations attended by AA|RF personnel.

Please feel free to link directly to the newsletter any time you see fit. The individual issues can also be linked-to using the archive section, and each “current” edition provides a “Permalink” at the bottom of the page.

Google Starts Showing Quality Score - More to come next week on this subject, but today Google announced it will be showing advertisers their landing page quality scores. More information available from Shimon Sandler.

Search Engine Strategies London – SES London is currently happening, and AA|RF UK Company DNA’s Nathan Levy will present on February 15 in a session titled “Search Head or Search Tail? Getting the Mix Right.” SMTrends will provide a recap of that presentation next week, although it’s probably not too late to see it live; information here. Some “live” blog coverage of the sessions can be found, including here and here.