Testing is Tough
Photo by diggersf on Flickr, used under Creative Commons
We’re currently working on a redesign for Connect, and like any project, waterfall, agile or otherwise, we have a testing phase.
I’ve been designing and building software for a long time, assuming you consider mid-90s a long time, and testing has always been the toughest part of the process.
I’m talking about system testing, which means banging away at a piece of software to make sure it works for as many scenarios as possible. The problem is that no matter how hard you try, you can’t foresee every way that your software will be used. It’s impossible, believe me. Try it sometime with a product that does more than a couple functions.
This is why developers and product managers alike hate testing.
Why?
When you write a system test plan, you have to account for as many use cases as possible. That’s really hard for one person. Then, take into account data.
Large, public data sets are great because they provide you with massive amounts of data, which allow you to test both function and performance. However, if you’re testing something proprietary in nature, e.g. accounting data, how can you account for all the variations?
You can’t.
So, you make a best guess, which is probably influenced by your own design, making the test cases less valid and more likely to succeed. That, in turn, affects the testing.
Plus, even if you wanted, you couldn’t get data sets for testing because mostly, those are proprietary and unique to each user.
Data aside, you’re still constrained by your own ability to think of all the test cases.
Enter beta-testing. This is a time-honored tradition, dating back to way before agile programming and Web 2.0 made it popular.
Beta-testing melts down to crowdsourcing, i.e. spread the test cases across as many people as possible to capture all the cases you can. Solid in principle.
Beta-testing is good, at least better. It allows you to reach an unbiased audience, not influenced by your design.
Still, until you can reach more users, you can’t vet all the use cases. This is why products like GMail stay in perpetual beta, i.e. they never go production. I suppose this is a good way to indicate to your users that you’re still under active development.
Still, at some point, you have to leave beta.
I don’t have the answer. Building software is hard. Design and build are comparitively easy. It’s the testing that gets you.
But, we do our best.
What do you think? Find the comments.
Pingback: PM Should Know How to Code, Part 2 | Oracle