Published in

SEO Basics: How Search Engines Index Websites

The ranking of a business’s website is foremost in the mind of many people. This is because we associate a site’s volume of web traffic with its prominence in search results. But how do search engines rank websites? Let’s take a look at the basics of how search engines index websites, so we can better understand the primary factors involved with search engine optimization (SEO).

Indexing the Web

Before search engines can display any results in your browser, they need to scan and index the entire web. This daunting task is performed by robots called “crawlers” or “spiders.” These robots crawl through every openly accessible web page to decipher the content and index it for later retrieval during a search. This indexed content is then ranked by its perceived usefulness.

There are a few best practices to follow when it comes to optimizing your web page’s content for indexing. Since search engines are in the business of providing answers, it helps if your website’s verbiage consists of common terms that people search for. Make sure your page’s url, title and content all use appropriate keywords. These keywords should be targeted to the search term that you want search engines to associate with your website. Overly creative or off-base page titles will confuse search engines about the real content held within your page. Also be aware of language and spelling subtleties as those could significantly impair your page’s SEO.

Use keywords on your site that are targeted to the search term that you want search engines to associate with your website.

Popular Pages Get Noticed

Search engines want to provide the most relevant and important results in search queries. To determine the perceived popularity of a web page, each search engine uses various metrics or algorithms. These algorithms are comprised of hundreds (or thousands) of variables, all of which play a part in determining a web page’s popularity. Two of the most important factors used in determining a page’s popularity are the amount of web traffic it receives and the number of inbound links to the site from related websites.

Crawling and Indexing Problems

Here are some common reasons why a search engine might not crawl a portion of your website:

  • Search engines can not complete online forms to gain access to the content that is curtained off

  • Duplicate versions of a page confuse search engines looking for original content

  • You have orphaned pages with no links pointing to them on your website

  • Your website’s meta robot tag blocks search engines from crawling your site

  • Search engines can’t crawl Flash content, un-parseable javascript, frames, or iframes

SEO is Always Changing

Search engines are notorious for consistently tweaking and enhancing their algorithms to improve search results. With each adjustment, the way in which search engines decipher and rank the popularity of web page also change. By keeping pace with these changes you can ensure that your website will benefit from better search ranking and increased website traffic. For a complete list of search recommendations, take a look at theses SEO resources from Google and Bing.

Follow our blog via RSS or Email.

Privacy Policy We do not sell or share your information.

Ted is a graphic designer and front-end developer who specializes in branding, marketing, and website development.