Recently I have been diving into optimising performance of selenium cucumber tests, as there’s no single “make the tests faster” method I thought it’d be interesting to document all the different approaches I took (partly for my own future reference, and partly for anybody else who may find it useful.
Some of these are common sense, some of them don’t relate to code at all, some of them may be controversial, but at the end of the day YMMV, these are what worked for this particular project:
1) Reduce the number of tests
Ultimately the biggest performance improvement came from something this simple. An initial pass over the entire test suite revealed there were a number of tests that were either duplicates in some subtle way, redundant as they weren’t doing any more than a unit test, or otherwise unneeded. Pruning out these tests not only had the benefit of reducing total time taken for a test run, but also made reading the test suite as a whole much easier as they were now cleaner and only contained what was required.
2) Reduce redundant asserts in test steps
Again not really related to code, just common sense which often gets lost as a test suite builds up when there hasn’t been the time to go back over and rationalise everything as it grows. In the tests on this project there were many instances where we would assert something in one step and then assert the same thing again in a later step. Of course some of these were valid, but many were not, they were simply left in because the author didn’t realise that particular assert had already been done. There was also an example in this particular project of certain asserts being performed every single time a call was made to a page object; making it so these asserts were only performed the first time the object is called or whenever we were expecting the page to have changed had quite a big impact on performance.
3) Don’t use Thread.sleep()
4) FluentWait/WebDriverWait poll interval
Now we’re getting into the juicy details of WebDriver. We had quite a few instances in our tests of waiting for certain elements to appear on the page or contain certain text before continuing with the tests, nothing complicated:
WebDriverWait wait = new WebDriverWait(driver, 5); wait.until(condition)
Here we are specifying a timeout of 5 seconds in the wait, however we aren’t specifying what the polling interval is (i.e. how often it checks to see whether the condition is true). As it turns out, the default polling interval is 500ms, rather a long time to wait between checking. This 500ms rapidly builds into seconds and minutes if you’re extensively using waits throughout your test suite. By adding a third parameter we can specify our own polling interval:
WebDriverWait wait = new WebDriverWait(driver, 5, 10); wait.until(condition)
Taking all of these waits right down to 10ms polling interval had a noticeable impact on speed of the test run, with no noticeable increase in CPU usage on the machines running the tests.
5) Use more optimised element locators (ID wherever possible, avoid XPath)
A lot has been written on this subject elsewhere so I won’t go into too much detail, but not all element locators perform in the same way. Browsers are optimised to locate elements by ID, so try and use ID wherever possible. Badger your developers to add unique ID’s to all elements, it makes everything so much easier, I can’t stress this enough.
At the other end of the spectrum XPath is often slow and unreadable (though not always of course!), if you absolutely must use this form of locator then we’ve found CSS selectors are at least a little better and perform more reliably across browsers.
6) @BeforeAll instead of @Before hook
There were a few instances in our project where we had a before hook running something before each test that only really needed to be run once before all the tests (for example starting an email server, really didn’t need to stop/start it before every single test). Sadly for whatever reason cucumber-jvm has not implemented a global @BeforeAll hook, however as that post points out, writing your own is relatively simple.
Whether you would want to however is an entirely different matter. Going down this route does give a quite visible performance improvement, but at the cost of risking not triggering certain things if your website is expecting a user to type keys in a certain way, what if not everything is triggered onchange? This is a controversial one, and the purists amongst you will certainly not advocate this approach.
8) Use a different webdriver
In many cases this isn’t possible, you’re using selenium grid to conduct your testing across multiple browsers, but where you have the choice of webdriver to use, remember that Firefox isn’t the only one available!
On our own project we noticed for example that ChromeDriver performed our tests about 10% faster than the FirefoxDriver. Going beyond that you could even look into a headless browser driver such as GhostDriver which uses PhantomJS. Unfortunately for whatever reason some of our tests just refused to run with GhostDriver and there wasn’t much time to investigate. However as an aside we are using PhantomJS in conjunction with Jasmine for some of our unit tests, and just over 500 tests run around 5 seconds there.
For those that really want to stick with the FirefoxDriver then there is always the following snippet of code that may speed up things:
FirefoxProfile firefoxProfile = new FirefoxProfile(); firefoxProfile.setPreference("webdriver.load.strategy", "unstable"); driver = new FirefoxDriver(firefoxProfile);
While this does speed things up by having the driver not wait for pages to finish loading before proceeding, it can introduce instability and unpredictable behaviour in tests. Use with care!
9) Use selenium grid
One final simple way to speed up things is to parallelise tests via Selenium Grid. There is a vast array of documentation on this elsewhere, but essentially you can have your tests running across multiple machines simultaneously.
In order to make this work of course you need to be following one of the most fundamental principles of writing automated tests, that your tests are independent. If there is any coupling between scenarios then clearly splitting up your scenarios to run across multiple machines isn’t going to work at all.
After finishing going through the code and implementing the above we had ended up reducing time spent running the entire test suite by 60%. The best thing about it all, none of this is complex! While there’s no catch-all “make the tests run faster”, a few simple steps can make all the difference.