How Google Tests Software

James A. Whittaker, Jason Arbon, Jeff Carollo

Mentioned 1

Describes the techniques Google uses to test their software, and offers similiar techniques for analyzing risk and planning tests, allowing an Internet company to become more productive.

More on

Mentioned in questions and answers.

We run BDD tests (Cucumber/Selenium) with Jenkins in a Continuous Integration process. The number of tests is increasing day by day and the time to run these tests is getting higher, making the whole CI process not really responsive (if you commit in the afternoon you would risk to see your building results the day after). Is there a way/pattern to keep the CI process quick in spite of increasing number of tests?

You could choose one of the following schemes:

  1. Seperate projects for unit tests and integration tests. The unit tests will return their results faster and the integration project will run once or just a couple of times per day and not after each commit. The drawback is obvious, if the integration tests suite break there is no correlation with the breaking change.
  2. Google approach - sort your tests according to their size: small, medium, large and enormous. Use separate projects for each kind of test and according to the total time it takes to run the specific test suite. You can read more in this book. Also, read this blog to get more wise ideas.
  3. Try to profile your current test suite to eliminate bottlenecks. This might bring it back to give feedback in a timely fasion.

Hope that helps.