There are three things (two features and a datasheet) that, if Microsoft added to Test Manager, would push me past the hesitation I have to move my defect and test case management from our existing tool to TFS and TM.
Currently, we are using Rally across the board: requirements, defect management, test case management. Having all of our data in one unified tool has some definite advantages, and makes traceability easy. But inadequate customization, poor support for required fields (they are either required or not, regardless of the state a defect is in) and an inability to fail a specific test step are just three examples of why I am open to enduring the pain of transitioning to another tool.
But I want three things from Microsoft.
First, the ability to version test cases. I was surprised, maybe even shocked that this was missing. After all, TFS is the base upon which TM stands. And TFS is all about versioning. Like most commercial software companies that deliver software (i.e. not hosted), we have multiple parallel development streams being worked on. As requirements change, as defects are discovered and as gaps are found, our test cases change. I do not want to have to duplicate my test cases for each stream. But I need to be able to modify a test case for one streams and still execute the previous “version” of that test case for another streams . Test case versioning is the obvious way to handle this well and TM doesn’t have it.
Second, and this is not a feature, I want hard data on the system impact of executing test cases within TM. It has some impressive features. It will capture a video of the application execution which can be automatically submitted with defects. It will keep track of what code is executed when a test case (manual or automatic) is run so later, when a change set is introduced it can recommend the test cases which are affected and therefore need to be rerun. It will collect IntelliTrace data to allow a developer to step through the code as it executed when a defect was discovered. This is great stuff. It also sounds like the Heisenberg Principle on steroids. What is impact to the system? I cannot get any data when I ask Microsoft this, only vague responses that “I’ve never seen it be an issue.” That’s not good enough.
The third want is related to the feature I mention above. I can execute manual and automated tests and tell TM to keep track of the code that gets executed. Later, when development checks in a change set, I can know which test cases exercised code that was changed. This is a great feature. So I asked the following question at a Microsoft presentation:
“Since you keep track of what code gets executed when I run a test case, can I run through all of my test cases and then have TM tell me what code never got executed?”
They have all the data to have a very powerful code coverage feature, but the answer, sadly is no, the system will not do that. But it makes too much sense not to add in the future.
Microsoft’s focus on testing is exciting and long overdue. The team has delivered some excellent features and the defect handling in particular is very powerful. But changing tools is a sometimes painful, and usually a hard sell internally. The addition of these three things would make me pull the trigger on the change.
No comments:
Post a Comment