Developing Large Web Applications

Kyle Loudon

Mentioned 2

Presents information on designing Web sites that will remain effective as more pages and codes are added.

More on Amazon.com

Mentioned in questions and answers.

Most of the time I have been programming little apps either for myself or for the benifit of learning. Now that my programming skills are considered somewhat intermediate, I was wondering how I would tackle a large programming project.

Lets suppose I wanted to make an application that has a lot of features and is intended for the use of others and not just myself; how do I go about planning for such projects?

Do I just jump in and start coding the thing or is there some sort of recommended process?

Thanks in advance :D

Some of the things your going to want to look at is a tool chain including source code control, unit testing, documentation, automated builds, continuous integration, bug tracking etc. Programs at this level become very complex very quickly. So you definitely don't want to just start coding. you are going to want to develop a list of things the program should do. Then refine this list to must haves, would be nice and in a perfect world. From this list you can start designing a database schema, screen designs, and a class structure (assuming OOP).

This book is based on web applications, but much of it applies equally well no matter what language you are working with:

http://www.amazon.com/Developing-Large-Web-Applications-Producing/dp/0596803028/ref=sr_1_1?ie=UTF8&s=books&qid=1273632445&sr=8-1

You will also want to think about communication, most applications at this level of complexity are not the product of a single person, but rather a team. As a result effective communication and teamwork, become very import considerations.

This is by far not everything you will need to create a successful product, but should start you in the right direction.

I am currently a placement student (web developer) working for a university, and I have been assigned a few big web projects. The projects include a total revamp of the university i.t. help site which draws around 14k hits a month from on the uni campus and around 4k externally. I also have a second project which is a mobile version of the first project. The projects will be sharing some resources.

To generalise this question so the answers could be useful to more people:

  • I have two websites that will share some resources, lets say, index.php, functions.js and style.css, and these scripts will be used on almost all pages on the websites.
  • I have two audiences to cater for (in terms of download speed), the users within the same network that the sites hosted on (100mb/s aprx) and external users.

I would like to know what would be the best way to cache each kind of script (.js, .css, .php) and examples of how this would be done with their pros and cons over other methods if possible. By caching I mean locally, network and server caching.

Note: index.php is a dynamic page which should be refreshed from cache every 2 hours. It would be cool if you start your answer with .js, .css, .php or a combination so I can easily see what type of script you are talking about caching.

Thanks All!

Performance tuning through cachine could be categorized into multi-layers:

Good introduction and practical code examples can be found in Chapter 9 (Performance) - Developing Large Web Applications. It will talk about Caching CSS, Javascript, Modules, Pages, Ajax and Expire headers.

If we need to keep things simple on server-side do the following:

  1. Install APC extension which will make PHP faster for you through the so called opcode caching. No special configuration, it will work silently for you.
  2. Cache the full page for two-hours using this simple Pear library PEAR::Cache_Lite.
  3. For each database SELECT query cache the result in APC with a TTL of 5 Min, md5 hash the SELECT statement and use it as key for APC cache. Docs

In future if you have multiple servers and the performance becomes to be crucial before then you will need to look at:

  1. Shared memory caching between servers. Check Memecache or even Membase
  2. You need a reverse proxy solution: this basically layer between your user and server server so that it will serve the HTTP requests instead of your server. You can use for that Varnish, Squid or Apache Traffic Server.
  3. Mysql innoDB engine is slow, you may need to go for faster engine such as XtraDB
  4. Then maybe you will find that rational databases are stil slow for you. Then you will go for the key-value solution such as MongoDB.

Finally as references in web application performance check:

  1. Front-end Performance: High Performance Web Sites, Even Faster Web Sites and High Performance JavaScript.
  2. Back-end Performance: Pro PHP Application Performance and High Performance MySQL