Saturday, September 17, 2011

How Does a Search Engine Work?

As many of us are only too aware, the internet is an extensively large resource which allows us to tap into information covering every subject under the sun. Since its beginnings, the internet has grown to such a scale that it was recorded in 2007 that the World Wide Web held over 29.7 billion pages. In order to find what we’re looking for amongst that many pages, we need to rely on search engines. But how do search engines work and accurately give us the results we’re looking for?

As the Internet is an extremely popular resource-one of which is being used frequently by businesses all over the world, it is possible that this number has increased by millions more within the past three years. Now, in order to come up in the search rankings, your website must utilize Search Engine Optimization (SEO).

How Do Search Engines Work?

For those who are looking to obtain information, no matter what subject they may be searching for, there is the advantage that there are hundreds of millions pages to help them find what they need. Unfortunately, this adds as a disadvantage that there are hundreds of millions of pages which are mostly titled in regards to the author; this then throws up the issue to the user of which pages they need to read when looking up about a particular subject. This is where Internet search engines come into play.

Search engines are specially designed sites on the World Wide Web of which their main purpose is to help web users find information which is stored on other websites. Each search engine site will work in a way which differs to the other, but they all perform the same three basic tasks which are:
  • Searching through the internet or specified areas within the web, based on important keywords
  • Creating an index of the keywords they are searching for and where they find them within the web; and
  • Allowing web users to search for words or a combination of words that have been found within the index
In its beginnings, search engines were able to hold an index of a few hundred thousand pages and would typically receive around 2,000 queries a day. Today, the most popular search engines can index hundreds of millions of pages and will respond to tens of millions of queries each day.

Before the World Wide Web was truly established as the most visible part of the Internet, search engines were already in place in order to help users find information. Using different programs such as “gopher”, these programs would keep an index of the files which were stored on servers connected to the Internet, and would reduce the amount of time it took to find programs on the web.

In order for a search engine to identify where a file or document is, it must firstly find it for you on the web. In order to do this, a search engine will employ a specific software robot known as spiders, to build lists of the different words which are found on various websites.

When these spiders build these lists, this is known as Web crawling. For a list to be filled with useful words, a search engine’s spider will be required to look at a wide variety of pages in huge quantities, but how exactly does a spider begin its list building on the web?

The beginning of the list building will usually start by listing all heavily used servers and the most popular pages on the web which are related to your chosen words. The spider will start with the most popular site, indexing the words on the pages within this site, following every link which is found within the particular site. Whilst doing this, the spider will travel quickly through the Internet, identifying the most widely visited areas of the web.

This is why a website needs to be built to be navigated easily by both humans and spiders. Utilizing Search Engine Optimization on your website will communicate to the spider if a page is important or contains valuable information.

One of the most widely used search engine sites in the world is Google, which is perceived as being one of the most intelligent search engines on the World Wide Web. Within Google, the site uses multiple spiders, sometimes up to three at a time and each spider has the capability to keep up to 300 connections to different web pages at all times.

At its most capable period, Google is able to use four spiders, with over 100 pages per second being crawled through. This can result in around 600 kilobytes of data being found every second. Being able to keep this system working effectively means that Google’s system needed to be built so that it could feed the required information to the spiders.

When Google’s search engine looks at a web page, it will look at two main characteristics: the words which are on each page, and where on these pages that the words are found. In order for the owner of the site to be able to show up high in search engine results, they can use specific words within their webpage titles, subtitles and meta tags.

Meta tags allow the website owners to choose specific keywords for each indexed page. This is extremely beneficial in the case that words on each page have double or triple meanings as the meta tags can easily guide the search engines in the right direction of selection.

However, it is important that web owners do not rely on these tags as if you include meta tags which have no relevance to the content which appears on the site, this could severely damage your search engine results or not include your site within the results at all.

With these developments in Internet technology, the World Wide Web has opened up new possibilities to businesses that are looking to sell products or services both locally and online by carefully and thoughtfully selecting the correct content, titles and meta tags within each page.

Not only does this allow for businesses to take full advantage of the great benefits which stem from using the internet, thus search engine sites for the growth of their business, but in today’s society, it also helps to keep up with competitors and display yourself as a reputable and professional business.

Gone are the days where people look up local services in the Yellow Pages. Gone are the days where local businesses ranked in order of the size of their advertising budgets. Even locals now use the search engines to look up information on local businesses. Using search engine optimization will lift your website to the top of the first page or search engine results for a fraction of the cost of Yellow Pages advertising.


[Follow me on Twitter]