Search Engine Optimization for Dynamic Websites

What are “Dynamic Websites”?

Dynamic websites are websites whose pages are generated on the fly. Unlike static pages (primarily .htm/.html pages), dynamic pages are generated when an user triggers an action through that particular page.

Here is a sample dynamic URL-

http://www.bbc.co.uk/cgi-bin/search/results.pl?q=%22dynamic+websites%22&tab=news&go=homepage

As per the above example of www.bbc.co.uk, the dynamic part (i.e. the part) of the URL which changes as per surfer request is the part after the question mark (?)

What are the problems that search engines face in indexing Dynamic URLs?

1. Search engines often consider a dynamic URL as an infinite set of links.

2. Since dynamic URLs find maximum application in online shopping carts, there is a possibility of incorporating a session id to a particular page. As session ids of that particular page change, the search engine spider needs to index an infinite number of copies of the same page, which is a Herculean task for them.

3. Proceeding with the same logic presented in point # 2, indexing the same dynamic page might overload the servers of the search engines and therefore prevent the search engines to present with the most relevant information in the fastest possible time.

Here is what Google says about indexing of dynamic websites -

Reasons your site may not be included: Your pages are dynamically generated. We are able to index dynamically generated pages. However, because our web crawler can easily overwhelm and crash sites serving dynamic content, we limit the amount of dynamic pages we index. (Source – http://www.google.com/webmasters/)

What are the options that you have in order to make a search engine spider index your Dynamic URLs?

1. Use of softwares – Exception Digital Enterprise Solutions (http://www.xde.net) offers a software which can change the dynamic URLs to static ones. Named XQASP, it will remove the “?” in the Query String and replace it with “/”, thereby allowing the search engine spiders to index the dynamic content.

Example –
http://www.my-online-store.com/books.asp?id=1190 will change to

http://www.my-online-store.com/books/1190.

The latter being a static URL, it can easily be indexed by the search engine spiders.

2. Use of CGI/Perl scripts – One of the easiest ways to get your dynamic sites indexed by search engines is using CGI/Perl scripts. Path_Info or Script_Name is a variable in a dynamic application that contains the complete URL address (including the query string information). In order to fix this problem, you’ll need to write a script that will pull all the information before the query string and set the rest of the information equal to a variable. You can then use this variable in your URL address.

Example – http://www.my-online-store.com/books.asp?id=1190

When you are using CGI/Perl scripts, the query part of the dynamic URL is assigned a variable.
So, in the above example “?id=1190″ is assigned a variable, say “A”. The dynamuc URL http://www.my-online-store.com/coolpage.asp?id=1190
will change to http://www.my-online-store.com/books/A through CGI/Perl scripts which can easily be indexed by the search engines.

3. Re-configuring your web servers –

(i) Apache Server – Apache has a rewrite module (mod_rewrite) that enables you to turn URLs containing query strings into URLs that search engines can index. This module however, isn’t installed with Apache software by default, so you need to check with your web hosting company for installation.

(ii) ColdFusion – You’ll need to reconfigure ColdFusion on your server so that the “?” in a query string is replaced with a ‘/’ and pass the value to the URL.

4. Creation of a Static Page linked to an array of dynamic Pages –

This approach is very effective, especially if you are the owner of a small online store selling a few products online. Just create a static page linking to all your dynamic pages. Optimize this static page for search engine rankings. Include a link title for all the product categories, place appropriate “alt” tag for the product images along with product description containing highly popular keywords relevant to your business (You can conduct keyword research for your site through http://www.wordtracker.com). Submit this static page along with all the dynamic pages in various search engines, conforming to the search engine submission guidelines.

How Amazon.com, Earth’s Biggest Bookstore, coped with the issue of indexing of dynamic URLs?

A search in Google for internet marketing books, yielded a result that takes you directly to the appropriate dynamic page at Amazon: http://www.amazon.com/exec/obidos/ISBN%3D0395683297/103-0475212-8205437.

Since the above URL does not contain any query strings, all search engines can index Amazon.com’s products. Amazon.com uses this method to get its product selections indexed by search engines. This is very important for Amazon, because being an online bookstore, it is very natural for them to adopt dynamic URLs yet it was equally important for them to make their dynamic URLs search engine index friendly.

Conclusion

Even a few years back, most of the major search engines did not index dynamic URLs, thereby often preventing top search engine rankings for the online stores. With Google starting to index dynamic URLs a few months ago, the picture is going to change in the coming days. This is more so because Google’s numero uno position is currently being threatened by Microsoft’s MSN (developing its own search engine) and Yahoo! who recently acquired Overture, the biggest player in the PPC Search Engine industry.

########################

Changing Domain Name – The Best Way!

Accept the fact that there are really some times that you need to change your domain name. Changing domain names is not in fact a very rare situation, but the fact that is widely common today. Many were now changing domain names for certain purposes, and along with that, many people were also encountering problems.

The process of changing domain names, according to some veterans in the field of domain name, is somewhat very time-consuming and the potential for some mistakes is very huge. Well, every problem has its own solution, right? So a fairly simple solution and a safe alternative for a smooth and seamless switch in changing domain names is made possible through the application of the mod_rewrite and a .htaccess file.

Now, what is a mod_rewrite and .htaccess file?

According to certain studies, a mod_rewrite as it becomes important in changing domain names is an Apache module that grants the mode for the rewriting of the URL. Many considered this tool for changing domain names as transparent to the end-user and thus require no special software on the user’s end.

In an instance of changing domain names, the user who applies the mod_rewrite in changing domain names will still see the old domain name in their browser address bar on the first page that they visit, but then will typically be seeing the content under the new domain name. By using the mod_rewrite in changing domain names will also provide you with a very user friendly search engine.

On the other hand, the .htaccess file plays a role in changing domain names as containing particular Apache directives for specific requests, including security, redirection issues, and it also contains the answers on how to handle certain errors. Such function of the .htaccess file is very necessary in changing domain names for the reason that the Apache that it contains is a very common web server that helps a smooth and easy process of changing domain names.

Knowing the facts about those two important tools, it is therefore important to note that before you do anything, you should consider some issues that surround such the domain name change. These issues must be considered for these will help for a smoother transition. So according to some experts for changing domain names, the first important move is to check the web host to make sure that they support the mod_rewrite module, otherwise the attempt procedure will not work. Then it follows that you must activate your new domain name. It is just necessary that you ensure that the file base construction and naming is accurately similar when it was under the old domain name.

After that, you create an .htaccess file. It is interesting to know that the .htaccess file in changing domain names can be done with a simple text editor like the notepad, and it should be noted that the file has no tail extension; the name should then be just .htaccess. If done, the syntax should then be inserted. Just be aware that when changing domain names, the old domain name will still appear in the browser window but you are viewing the content of the new domain name. Of course there are still some rules behind these steps for changing domain names, but unfortunately, that is beyond the scope of this article.

301 Redirect – The SEO way to rename or move files or folders

In this article I will discuss page redirection techniques, what works and what to avoid.

What is page redirection and why would you want to use it?

Let’s say you rename a page on your website, for whatever reason. Perhaps you decided to revamp your entire naming convention, perhaps you decided to restructure your site and need to move pages into different folders, or you just realized that you are missing valuable keywords.

Let’s elaborate a bit on the keywords issue, since it is part of your search engine ranking success.

Let’s say the page in question is about customized USB drives and you named it page1.htm. Then you read some SEO (Search Engine Optimization) articles and you found out that some search engines use words in the actual file name as search keywords. Next time you do a Google search, take a look at the results, most will have words in the actual file name (in the URL section) bolded, denoting a keyword match. Your USB drives page will definitely benefit if named something like custom_usb_drives.htm instead.

Now that you renamed your page, you just created a symphony of issues for yourself, for your users and for your position in search engine results.

Linking issues:

You will have to point every link on your site to the new page name. If your site is small, it should not be a big deal, but if your site is large, you will inevitably make mistakes, mainly forgetting a link or two. This will result in visitors getting the dreaded “404 page not found” error when clicking on your links, robots (also know as crawlers or spiders) avoiding you, etc. Also, if you are heavily relying on visitors from search engines, then again, people will get a “404 page not found error”.

Let’s use the previous example, for a long time your page1.htm was indexed by major search engines. If someone types “custom usb drives” in a search engine box, your page shows up on the first search results screen. That is fantastic, only if someone clicks on the link, they will be pointed to page1.htm, not to custom_usb_drives.htm, because the first page is the one in the search engine’s index. It will take time, sometimes months, before the search engines update their indexes with your new page name.

Lost Page Rank (PR) issues:

Google developed a proprietary algorithm that assigns a Page Rank (PR) to every page on the web. PR is a number from 1 to 10 (10 being the ideal) and is intended to be a representation of how useful and popular a given page is. PR is influenced by many factors, one of the crucial ones being Link Popularity. Link Popularity is a representation of how many “quality” or “relevant” sites link to your page. Without getting into too much detail, it is increasingly difficult and time consuming to achieve a high PR for your pages, especially if you don’t have a really unique website with exceptional and highly sought after content. If you are merely operating a commercial site, in a competitive market (such as selling custom branded USB drives, as in our example), then it takes a lot of time and hard work to build a good page PR.

When you rename a page and discard the old page, you also discard the PR of the page. Your renamed page will be seen as a totally new page, with 0 PR.

What is the solution?

I will start by enumerating some of the methods used by the non-initiated.

Not recommended solution 1: Duplicate content.

First thing that probably comes in you mind is: well, why can’t you just duplicate the page and let nature take its course. In other words, you will have two identical pages, one named page1.htm and one custom_usb_drives.htm. This gives you time to update all links and the search engines will eventually index the new page.

This solution is not viable because search engines will penalize you quite badly, ‘thinking’ that you are trying to scam them by using the ‘duplicate content’ technique.

Not recommended solution 2: Custom error message.

You could create a custom error page. However, you will lose rankings on the next search engine update as the file will appear to be non-existent. As discussed above, it could be some time before the page with the new name will be indexed and will appear in people’s searches. Also, your web site visitors will be frustrated by the fact that they now have to dig through your site to find the desired information.

Not recommended solution 3: An HTML Meta redirect.

You could implement a so called Meta refresh in a blank or customized page that has the name of the old page (in our example, page1.htm) that points to the new page. The redirect can be instant, or delayed by a predetermine amount of time. The delayed redirect has the advantage that you can place an extra message, such as “please be aware that the page you are looking for changed location….. etc., etc…. you will be redirected automatically to the new location”

In the past, this was probably the most used technique.

Without getting into the mechanics of the Meta redirect, which is basically a META tag statement you ad to your HEADER section, know that there are also Java Script techniques that achieve similar results.

What is bad about this is that this is a technique often used by spammers to trick search engines and it should be avoided, unless the page is in a section of your site that isn’t indexed (also known as spidered or crawled). Search engine spammers create a page that is optimized for certain keywords and phrases – it usually has no real content. The page is then picked up by some search engines, but when a visitor clicks on the search engine entry, they are redirected to another site, often unrelated. Most search engines have filters to detect this. Using this form of search engine deception will see a site eventually banned or penalized by major players such as Google.

The recommended redirect strategy – 301 Redirect

A 301 redirect is the most efficient, visitor friendly, robot (spider, crawler) friendly and search engine friendly solution around for web sites that are hosted on servers running Apache. If you are not sure, check with your hosting provider.

A 301 redirect is just a set of commands you type into your .htaccess file.

When a visitor (whether human or robotic) requests a web page via any means, your web server checks for a .htaccess file. The .htaccess file contains specific instructions for certain requests, including security, redirection issues and how to handle certain errors.

The code “301″ is interpreted as “moved permanently”. After the code, the URL of the missing or renamed page is noted, followed by a space, then followed by the new location or file name.

First of all, you’ll need to find the .htaccess file in the root directory of where all your web pages are stored. If there is no .htaccess file there, you can create one with Notepad or a similar application. Make sure when you name the file that you remember to put the “.” at the beginning of the file name. This file has no tail extension.

Some hosting providers offer redirect services through their “control panels”, so you don’t have to perform low level changes on the .htaccess file itself. Instead, they provide a user friendly interface for this. Check with your hosting provider to see what the optimal way to perform a 301 redirect is in your case. I will continue the article with the barebones solution.

If there is a .htaccess file already in existence with lines of code present, be very careful not to change any existing line unless you are familiar with the functions of the file.

Scroll down past all the existing code, leave a line space, then create a new line that follows this example:

redirect 301 /folder/page1.htm http://www.you.com/folder/custom_usb_drives.htm

It’s as easy as that. Save the file, upload it back into your web and test it out by typing in the old address to the page you’ve changed. You should be instantly and seamlessly transported to the new location.

Notes: Be sure not to add “http://www” to the first part of the statement – just put the path from the top level of your site to the page. Also ensure that you leave a single space between these elements:

redirect 301 (the instruction that the page has moved)

/folder/page1.htm (the original folder path and file name)

http://www.you.com/folder/custom_usb_drives.htm (new path and file name)

The same format applies not only to renamed files, but also to files moved to a different location.

The 301 redirect is the safest way to preserve your rankings. On the next indexing (crawling, spidering), the search engine robot will obey the rule indicated in your .htaccess file and index the new page name every time a link or its internal database tries to access the old page. In the next update (again, this could take months), the old file name and path will be dropped and replaced with the new one. Sometimes you may see alternating old/new file names during the transition period, along with some possible fluctuations in rankings as things settle. Don’t panic, this is normal.

What if your site is hosted on a Microsoft IIS server instead?

If you have access to the server, do this: In internet services manager, right click on the file or folder you wish to redirect. Select the radio titled “a redirection to a URL”. Enter the redirection page, check “The exact url entered above” and the “A permanent redirection for this resource”. Click “Apply”.

If you do not have access to the server, ask your host to point you into the right direction.

In conclusion, the best and the most transparent way (to both human and robotic users) to rename and move files on your web site, while preserving your search engine ranks is the 301 redirect.