Woes of a Big Site: How to Prepare Your Site for Growth

Tips on creating a scaleable site that can deal with growth

Woes of a Big Site: And How to Prepare Your Site for Growth

by Christopher Heng, thesitewizard.com

I often read posts in forums and discussion lists by newcomers asking how their site can become big and well-known. Well, naturally this is every webmaster's dream — who ever creates a site hoping it will become obscure? However, what many newcomers fail to appreciate is that big and well-known sites face a number of problems. Problems which my other site, thefreecountry.com, has begun to face as a result of its rising popularity and size. This article mentions a few of those problems and some things that can be done to pre-empt them (which alas I did not do).

Traffic (Bandwidth) Problems

It's interesting how everyone faces traffic problems. The small site faces traffic problems — few people visit them. When the site increases in popularity, it too faces traffic problems, albeit a different kind: the expensive kind.

When a site is starting out, the typical data transfer (sometimes loosely, and incorrectly, called "bandwidth") allocation of (say) 5GB seems generous and a distant target that looks impossible to hit. But wait till it becomes popular. Suddenly that 5GB looks very skimpy, and you'll be frantically searching your web host's documentation for how they charge for "overages" (the amount by which you exceed your allocation).

One of thefreecountry.com's previous web hosts (no longer mentioned in my list of budget web hosts, or anywhere else on my sites for that matter) had a rather pathetic traffic "overage" policy. If the site exceeded its traffic allocation, it would be shut-down for a month. (Gee, how did I miss that in the fine print?) To prevent that, the owner has to pre-pay for additional blocks of data transfer. The trouble is, the host had no system of warning you in advance when you were nearing your limit, which meant that the webmaster had to monitor the traffic of the site closely. To add insult to injury, the additional data transfer you had to pre-pay for came only in blocks of 10GB. Let's face it: when you initially exceed the "bandwidth" limit, it is unlikely that you'll exceed by 10GB. The initial "overage" is likely to be at most 1 to 2GB (unless of course you had just embarked on an aggressive advertising campaign).

My recommendation: when scouting for web hosts — think ahead. Check out the "bandwidth" or data transfer policy. It should bill you according to the amount you actually exceed and you should not be required to have to pre-pay it. And read the fine print in all their policy statements carefully. Oh yes. For reasons mentioned in my article on The Myth of "Unlimited Bandwidth" and "Unlimited Disk Space" in Web Hosting, it is probably best to stay clear of hosts that advertise "unlimited bandwidth".

Too Many Pages to Update

When I first designed thefreecountry.com, I did not plan for it to be the large site it is today. As such, I made no provision for keeping certain common features found on every page in a single location. Instead, things like the navigation menu, logo, etc, were inserted into every single page.

This was fine when the site was small. But it became a nightmare to maintain as the site grew. Just imagine: each time I wanted to change the design of the site, I had to manually go through the two hundred odd pages to modify them. And it's not just a simple matter of using a search and replace utility to simplify the task — the changes often span multiple lines and are interspersed with page-specific information.

My suggestion is that even if you think your site is small in terms of number of pages, plan for its eventual growth. Put common design elements in a central location, for example, by using Server Side Includes (SSI). Or use a web editor that can automatically update all your pages whenever you modify those elements (for example, the free Microsoft Expression Web editor can do this).

Don't think it will not happen to you. When I started the predecessor of thefreecountry.com in 1996, it had only 3 pages. Today it's hundreds of times its original size. True it is not as big as those sites with thousands of pages — but I can tell you from experience that once you hit a three digit number for the number of pages, updating the design of the site isn't quite as enjoyable any more.


Usually when a site is new, the owners consider it little more than a hobby. As such they tend to simply find a cheap web host to dump it on, since in its infancy, the site is unlikely to generate much income anyway. This is what I did originally with thefreecountry.com.

It would have been fine except that I kept thefreecountry.com on the cheap web host far longer than I should have.

Cheap web hosts seldom have the margin they need to hire good, competent help. Now I don't mean technical support that responds to you fast. Anyone can do that — even that cheap host I was on. Their responses each time however display their ignorance. When I say competent help, I mean the kind that knows the server hardware, the various software that run on it, security issues, and how to fix software and hardware problems. You need that kind of competent help so that they can pre-empt (as far as possible) potential problems as well as handle any true crisis that arise (eg fixing new security holes in the web server, etc, or troubleshooting hardware problems).

When your site grows and gains more visitors, the uptime of the site becomes increasingly important. If the server goes down for a couple of hours when your site is new, you have lost at most a few visitors (if even that). But when a site is well established, every hour that it is down loses you thousands of visitors (and the income associated with them).

If, like me, you placed your site on a cheap host when you started out, you should keep an eye out for good, reputable web hosts so that when it grows, you'll know where to move it to. Such hosts may cost slightly more than a bargain-basement one, but the stability you get is well worth the extra expense, especially when you have lots of traffic.

Unfortunately though, keeping up to date with the web hosting scene is not a one-time affair. If you have followed the changes in my "Which Web Host Do You Recommend?" in its early years, you would have seen that good hosts do turn bad over time. This usually happens when they get too big, and have to hire additional help that do not have the commitment and competence of the original owner.

The verdict: cheap web hosts are fine when your site is small. But always be prepared to move. And you definitely should move when your site grows bigger and becomes a viable commercial venture.

(For the curious, thefreecountry.com is no longer hosted at the cheap web host I mentioned.)

Planning for Growth

This article is really about planning for growth. The best time to do that is when your site is young. The changes that you need to make to prepare for the future will be smaller when it is young. Don't wait till your site is too big. I had to learn the hard way from hindsight. You don't have to.

This article can be found at https://www.thesitewizard.com/archive/bigsitewoes.shtml

Copyright 2001-2018 by Christopher Heng. All rights reserved.
Get more free tips and articles like this, on web design, promotion, revenue and scripting, from https://www.thesitewizard.com/

thesitewizard™ News Feed (RSS Site Feed)  Subscribe to thesitewizard.com newsfeed

Do you find this article useful? You can learn of new articles and scripts that are published on thesitewizard.com by subscribing to the RSS feed. Simply point your RSS feed reader or a browser that supports RSS feeds at https://www.thesitewizard.com/thesitewizard.xml. You can read more about how to subscribe to RSS site feeds from my RSS FAQ.

Please Do Not Reprint This Article

This article is copyrighted. Please do not reproduce or distribute this article in whole or part, in any form.

Related Articles

New Articles

Popular Articles

How to Link to This Page

It will appear on your page as:

Woes of a Big Site: How to Prepare Your Site for Growth

Link to Us
No Spam Policy
Privacy Policy
Site Map

Getting Started
Web Design
Search Engines
Revenue Making
Web Hosting
Perl / CGI
.htaccess / Apache


Free webmasters and programmers resources, scripts and tutorials
HowtoHaven.com: Free How-To Guides
Site Design Tips at thesitewizard.com
Find this site useful?
Please link to us.