Automated acceptance testing for customisable off-the-shelf products is useful, but the approach should be different to bespoke-built products in order to get the best bang for buck. This is particulary true for OTS products which provide most of the external interface (e.g. a web browser interface).

TL;DR: Compliment rather than duplicate the testing done by the vendor. Use Specification by Example to avoid vendor lock-in where possible. Jump to the Conclusion for more details.

Example: automated testing on a greenfields product

The product we were building was a web application to sell business insurance online. It was basically designed as a wizard with many input fields requiring validation. At the mid point, the user was given a quote and at the end, the user was invited to sign up for the insurance (and hence become a customer).

The architecture of the application was as follows:

The validation library was written in javascript and was run on both the browser and the server in order to:

• on the browser to display validation messages to the user before submitting data to the server
• on the server (via Rhino) to make sure that no invalid data was passed down to the backend service. This was basically done for security reasons just in case someone bypassed the browser application and used HTTP directly.

To test the validation, we used a combination of Selenium to test through the browser and JS Test Driver to test at the unit level. Our Test Pyramid (for just validation) looked like:

We decided that there was little value in testing all validation cases through the browser as:

• browser tests are slow to run
• unit tests are much more precise and hence any error can be found and fixed more quickly

The other areas of functionality (e.g. referrals, denials) followed the same pattern as for validation. Hence, the pyramid above was representative of the whole test suite.

Example: automated testing on an OTS product

Borrowing again from the insurance space, the product this time was a claims processing system which handled things like:

• claims lodgement
• communications with suppliers / repairers
• payments
• general ledger updates

This time, the company decided to go for the “buy” option and selected a product called Guidewire ClaimCenter. ClaimCenter provides a default confuration which covers the above activities (claims lodgement etc) using a browser interface for human interactions.

ClaimCenter can be customised in 3 ways:

• via XML to tweak the browser interfaces
• via GOSU scripts (a bit like Groovy)
• via plugins written in Java for integration to backend systems (not supported out-of-the-box)

To support the target claims process (across all insurance brands), a significant amount of customisation was necessary.

About 3 months into the project, I sat in on a meeting to discuss automated testing. The conversation went something like this:

• Guidewire consultant
• Automated testers

At the time, the conversation ended in a stalemate, and automated testing continued to be done through the browser. I’ll be honest at this point and admit I was in the “automated tester” camp, though I regret taking that point of view now.

In retrospect, the Guidewire consultant had a good point. For most of the acceptance tests, we could have tested just the customisation code (XML, GOSU, Java) in isolation. By testing the UI, we were duplicating the testing already done internally by Guidewire. Had we tested more pragmatically, our tests plus the internal Guidewire tests would have yielded a pyramid very similar to the bespoke example above:

As it turned out, we ended up with a test suite which took many hours to run and hence couldn’t be run before a developer commited their code.

There are two cases I can think of where a project may want to test at a higher level (eg via the browser):

• as a deployment test - e.g. to make sure no one has deleted a vital piece of config
• when defects are found in the OTS product itself.

How can we avoid vendor lockin?

One advantage of testing at a higher level (e.g. via a browser) is that the implementation beneath can change without affecting the test code. One might suggest that this is a good way of avoiding vendor lock-in. I am of the opinion that isn’t for two reasons:

• the trade-off in time generally isn’t worth it.
• the interface design is often heavily influenced by the defaults of the OTS product. I.e. when the OTS product changes, so does the interface

What is less likely to change (when a vendor changes) are the business rules and processes. What we can do is specify these business rules and then link the automated tests to them. A great technique for doing this is Specification by Example. For the OTS case above, the automated testers were using a tool called Concordion which is capable of this very thing.

Of course if business processes are changed to fit an OTS product, then it is almost impossible to avoid vendor lock-in.

Conclusion

I recommend using Specification By Example when writing acceptance tests in general, regardless of whether a product is built bespoke or purchased off-the-shelf and then customised.

I also recommend using the ideas in Test Pyramid to write tests at the appropriate level. This will help keep the run-time of the automated test suite down, and allow for bugs to be pin-pointed quickly. There is no point in duplicating testing done by a vendor unless there are holes in that test suite - e.g. defects in the OTS product.

I was recently part of a small team which built a dynamic web site for a customer in 4 weeks. I would like to share what I think were the factors in making the project successful. But first some context…

Our customer

The customer was a non profit organisation in the education sector. Part of their remit is to give presentations in high school classrooms, including those in remote parts of Australia. These presentations are typically 5 minutes or less in duration, as part of a longer 1 hour educational session.

Their goal

The goal was to provide the means to train upwards of 40 classroom presenters, who were geographically dispersed, in any one of about 25 different presentations. This varied from the current situation where a small number of presenters were coached in person on a one-to-one basis.

The end product

The end product was an online portal which provided a central place from which classroom presenters could view training materials and upload candidate presentations of their own for feedback and coaching. In addition, it allowed management to endorse presenters for particular classroom sessions, making it easier to staff those sessions.

The team

We were a 4 person team, which included a product owner (from the customer side), a tech lead, a tester (who could write automated tests) and myself as an all-rounder (developer, BA, scrum master).

Our process

We gave ourselves 4 weeks to build a useful product. Ie, we fixed time and cost and left scope variable. The project then broke down as follows:

• 1 day for project inception
• 1 day for technical setup (ie iteration 0)
• 4 one week iterations (sprints) of build and deploy

What made us successful?

Co-location

We all sat around a small 6 person table with power outlets in the center. This gave us enough space for guests/stakeholders and also other equipment such as monitors.

The table was in a small room, bounded by windows and whiteboards. This obviated the need for an electronic planning system as we could communicate verbally and use the wall space to articulate (via index cards, whiteboard diagrams etc) the plan and the work in progress.

Also crucial to this was that our product owner spent half his time co-located with us in the project room.

Single goal

Goals tend to have a one to many relationship with both people and features, so it’s crucial not to focus on too many goals at once. Thankfully, our product owner was happy to focus on one goal: that of being able to train 40+ presenters in the classroom sessions.

A focused inception :: impact mapping

For the inception process, we used a technique called Impact Mapping, albeit a simplified form. For those unaware, Impact Mapping is a structured mind-mapping technique with a goal (the “why”) in the center, people (the “who”) and impacts (the “how”) surrounding the goal, and features/actions (the “what”) on the edge. It was the first time I tried the technique, and after some initial scepticism, I was pleasently surprised at the positive feedback I received (the product owner used the words “awesome” and “so impressive”).

After discovering the people and impacts, we did MoSCoW analysis on the impacts - ie before even considering the software features required. At this point, the product owner marked several impacts as “shoulds” and “coulds”, meaning that we didn’t need to consider them further during the inception.

We then analysed our “must have” impacts and discovered the software features required to achieve each impact. We then did another round of MoSCoW analysis, this time on the software features.

By this stage, the impact map had provided a lot of value in discovering the project’s scope. In essence, it had allowed us to discover the Minimal Marketable Product or Mininimum Viable Product.

The impact map also provided a lot of value during the project, allowing us to explain the essence of the project to stakeholders and to onboard new people who gave specialist advice (eg our user experience design designer who worked with us for a few days).

At the end of the project, I digitised the impact map (although I could have just taken a photo of it) and handed it over to the customer for validation - ie they can now validate whether what we have built actually achieved the impacts and the goal.

Just enough planning and management

The first step was to do an initial assessment of what we could do in 4 weeks. We had as an input our MMF from the impact map. We then went through the following process:

• t-shirt sized each of the “must have” features (XS, S, M, L)
• decided on an relative scale for the t-shirt sizes: 1, 2, 4, 8
• did a shopping-cart style velocity estimation to work out roughly how much we could do as a team in 4 weeks.

At this point, we found that the MMF fitted into the 4 weeks pretty much exactly. We then wrote out the features in user story format (which was quite easy as the role and impact was visible on the impact map) and then scheduled the stories into the 4 one-week iterations. Of course, as it was just a best guess estimate, we advised the product owner to schedule the least important user stories in iteration 4. This all took place on the inception day.

As the release horizon was short, we didn’t see the need for burn-up charts. We simply counted cards (both user stories and tasks) completed in each iteration and used that to plan the next iteration and to groom the release scope. We did find that we completed more cards in iterations 3 and 4 - a ramp up that you might expect in any project. Iteration planning and grooming typically took about 30-40 minutes each week and was done just prior to the start of the next iteration - though we did replan to a lessor extent on an add-hoc basis also.

Weekly showcases were held at the customer’s premises, so as to get feedback from other customers and users.

Our card wall looked something like this during iteration 2:

Triage:	a place where we put ideas and candidate stories when the product owner wasn’t there (for discussion at the next opportunity)
Zoo:	column for bugs which hadn’t been scheduled yet
White cards:	user stories
Yellow cards:	tasks
Red cards:	bugs

Anyone in the team could talk to the product owner to flesh out a story or prioritise something new in. As we were co-located, it was easy for everyone to keep abrest of what was going on.

Frequent customer interaction

Having our product owner co-located with us allowed much quicker turnaround on our questions in comparrison to asychronous communication such as email. It also meant that he could view the state of the project by reading the walls, rather than receiving a report or viewing an electronic tool which saved us time also.

The IT manager also co-located with us in order to get handover of the deployment process and maintenance tools. This was much more effective than relying on documentation alone.

Weekly showcases allowed us to get feedback from presenters and led to a few features, previously de-prioritised by the MoSCoW process, getting planned in (at the expense of another features where necessary).

Everyting was big and visible

In addition to the impact map and card walls, every decision we made was written up on the whiteboards. Examples include:

• the domain model
• the architecure diagram
• what browsers we had to support
• the agreed naming structure for a classroom session

This made it easier to remember everything as well as to onboard new people.

Technology choices

An often underestimated factor in delivering quickly, our technology choices we vital in allowing us to build and deploy a product within 4 weeks.

Our architecture was as follows:

The choice of Heroku and Rails allowed us to setup our development environments in a day despite only one of us having used Rails before. All the tech choices satisfied the following criteria:

• easy to setup multuple accounts for use in development, testing and production. This made it easy to create a “production like” environment for testing and showcasing.
• ammenable to automated testing in that they were:
  • easy to stub-out
  • processed requests quickly, meaning that the automated test suite did not take a long time to run.

The only non-free components were the production Vimeo account, for which we opted for a Vimeo Plus account to meet our speed and storage requirements and RubyMine (our Rails IDE).

Automated tests

Another often underestimated factor in delivering quickly, our automated tests provided a lot of value in iterations 3 and 4. They allowed us to deliver stories right up until the end of the 4th iteration - ie no need for a code freeze or mannual testing period. We did allow about half a day for the customer to test out the production site, however no bugs were uncovered during that time.

To ensure that our tests were adding value, we focused upon writing cucumber specs to ellaborate each story - ala Specification by Example. Specifications which we felt we not suitable for automation were marked as @manual and excluded from the rake build.

Conclusion

This article illustrates some important factors which, in combination, allowed us to deliver a high quality, marketable product in quick time with a small team (ie at low cost). There are of course alternatives to all the examples shown above. For example it is posible to achieve the same thing with a Java stack rather than Rails and other inception techniques (other than impact mapping) can work well also.

However, I believe that as we start to sacrafice these factors (e.g. co-location), we start to inhibit the ability of a team to deliver value quickly. If too much is sacraficed, we tend to find ourselves in an unenviable position of costly and/or slow development.