Death of the White Board - A Parable

In recent weeks I've raised two significant bugs that shared a common characteristic. These bugs did not appear in our bug tracking system. In fact they did not relate to a line of code that had even been written. These bugs were raised against models for potential solutions that had been proposed by the programmers in my company as we were discussing these models around a white board. As part of every story that we work on, the testers and programmers will elaborate the details of the story and produce Acceptance Criteria as well as risks and assumptions that we've identified. Generally as part of these discussions, or soon after, we'll discuss the developers' models and ideas for solutions to meet the requirement. This presents the team with a great opportunity for early testing and potential to raise issues before any time has been wasted in coding these issues into the software.

Whilst I feel that this is a very effective approach, saving the company both time and money, the benefits are very hard to measure. I can see this being an issue in the types of organisations who define their testing productivity based purely on metrics around the "artifacts" of software testing such as test case counts, bug statistics and the like. Reading this excellent post by Elisabeth Hendrickson the quote that really stood out for me was "The best way to screw up the usefulness of any process metric is to use it to judge people.". This got me thinking and the result, in true "weird episode of House" fashion, was a short story.

I hope you enjoy it, if not then let me know and I'll promise never to do it again.

Death of the WhiteBoard

Jen sat in uncomfortable silence as the man aross the desk from her examined the papers on his desk. It was clear to Jen that this was the first time that the man, her new senior manager Paul, had looked at the information in front of him. From what Jen could make out the sheet contained her name at the top and a series of figures in tables below. Finally Paul spoke.

"Hmmm. As you know Jennifer we've put in place some new performance measures in the last few months to measure tester productivity. It seems that your bug count is below what we would consider satisfactory"

Jen's worked for a small company who had recently been acquired in a takeover by a multinational organisation called Zefman inc. Paul worked for Zefman and had been brought in as a senior manager to implement their "Industry Standard Best Practices" around the software development processes in Jen's team. This was her first meeting with Paul, a 6 month performance review.

"But I've not had any issues before and we have a great reputation for quality with our customers"

replied Jen,

"anyway I don't always raise bugs in the tracking system as I've got an excellent working relationships with the programmers in my team. Sometimes I raise issues based on their solutions in discussions around a whiteboard before they've even written any code, and those issues don't get counted in my detection rates."

Paul continued, patronisingly,

"Well that approach may have been fine for a small company such as yours, however you're part of a bigger more professional organisation now and we need to ensure that you are adhering to our processes and meeting our minumum standards in your KPIs. The guys on Omega team have a test to live bug detection rate to be proud of, we expect you to achieve the same, as an encouragement we'll give you a performance related bonus based on your Defect Detection Efficiency"

Jen was tempted to argue further about the false incentives and potential for gaming the system with test to live ratios, but she could see Paul was not interested. The other teams that were part of Zefman were playing the game and she could see that Paul was not going to accept any other approach from her. Walking away with a sinking feeling Jen returned to her desk and forlornly fired up the bug tracking system.


Time passed and six months later a somewhat more jaded Jen found herself sat opposite Paul again.

"The thing is Jen"

, he said in a manner inappropriately familiar given that this was the second time that they'd met,

"your bug count is looking better but your team velicity is down, you guys are now not even achieving half of the story points per sprint than the Omega guys"

Jen bit her lip. Zefman had embraced "Agile" based on the prospects of achieving higher productivity and had latched onto the idea of velocity as a means to measure and compare team productivity. Instinct told Jen that there was a time and audience for pointing out that Story points only had relevance in relation to work previously done by that team and comparing them across teams was meaningless. This wasn't it.

"You were achieving a good velocity when I took over. Now you are making me look bad, what happened"

Resisting the urge to scream, Jen gave a measured response,

"Well, if you remember I stated that I used to raise many issues directly with the programmers in discussions around the whiteboard. These discussions were very effective at reducing errors making it into code. Now I'm focussing on getting bugs into the tracker like you asked and so many more issues are not found until code has been written costing time, velocity and therefore money"

At the mention of money Paul's eyes lit up.

"Do you mean to say that by raising issues in whiteboard discussions we can save the company money and increase velocity? In that case it should be encouraged. I'll discuss with senior management how we can incentivise this, and of course we'll have to introduce some means of formal measurement"

Jen came away ambivalent. She was pleased that the value of the way that she had worked previously but the idea of attempting to measure and place incentives around this left a bitter taste.

As a result of their discussion Paul convinced management to implement the "White Board Bug" incentive whereby bugs raised by testers against developer ideas around white boards were actively encouraged. New white boards were installed in every Zefman office. In order to tackle the difficult task of measuring Tester productivity in this area it was decided that in order for an issue to qualify, all attendees in the discussion should agree that it was a valid issue, and that some change was required in the proposed solution as a result. Initially the project appeared successful. Testers and programmers across Zefman engaged in discussions earlier and issues were being found prior to coding that increased velocity around the company, making senior management very happy. News of the scheme even started to spread outside of the organisation as videos of the annual conference talks with titles such as "The White Board Way - a Zefman success" made it into wider circulation.

Jen's fears did not take long to manifest themselves, however. Jen's team continued to work happily much the same way as before, and Jen enjoyed the bonuses she received for the contributions made in the meetings, even if it did feel a little artificial. Elsewhere in the larger organisation things were not going as well. The Omega team had cottoned on to the fact that the programmers had nothing to lose from the incentive so were arranging with the testers to introduce faulty solution models deliberately and then share the bonuses that the tester received for raising the inevitable issues. In an attempt to counter this the senior management imposed an incentive for the developers whereby there bonus allocation would be diminished each time a "White Board Bug" was raised against their solution. This caused even more issues. The programmers were up in arms about the impact of white board issues on their bonuses and actively avoided white board discussions until they had spent hours working through every facet of their solution idea to make it bullet proof. The testers meanwhile would spend hours trying to think of valid problems in the programmer models and refuse to share these unless in a formally recognised white board session.

When whiteboard discussions did take place they could run for hours while the validity of issues was debated, and Jen heard stories of fist fights in the Gamma department. Management reacted by installing video cameras around the white boards and implementing a specialist team "The White Board Panel" to review the videos and arbitrate any team disagreements.

A few months later, inevitably, the incentive was scrapped. Paul left the company under a cloud and cheers rang out in the Omega and Gamma teams when the white van men ripped out the new whiteboards and dumped them in skips in the car park at Zefman HQ. When the vans eventually got round to removing the whiteboards from her office, Jen was not there. Her and her team were in a small room in the basement discussing possible solutions to a new customer requirement ...

... around a flipchart.