In this episode Federico and Matthew talk about test heuristics. Starting with what are they and why they are useful, they give a run down several heuristics that the ASP.NET QA team uses while exploring software.

News

• MVC 2 Released

Test Heuristics

• What is a heuristic?
• Why are they useful?
• How does the ASP.NET QA team uses them?
• Some useful test heuristics
  • Configuration: Change variables related to configuration (of program as well of machine)
  • Interruptions: Log Off, Hibernate, Kill Process.
  • Undermining: Change the state of the program to an invalid state while functions are running.
  • Starvation: Progressively lower resources until product collapses or gracefully degrades.
  • Dog Piling: Get more processes going concurrently.
  • Click Frenzy / Street Fighter: Run through all the keys and click everywhere.
  • Multi User: Simultaneous users performing actions on shared resources.
  • Multi Instance: Run multiple instances of the application.
  • Flood: Multiple simultaneous transactions or requests flooding the queue.
  • Dependencies: Identify "has a" relationships and apply CRUD to each.
  • Constraints: Violate every constraint.
  • Input Methods: Typing, copy/pate, cut/paste, drag/drop
  • Sequences: Vary order of operations, undo/redo.
  • Position: Begin, middle, end
  • Boundaries: Approaching the boundary, at the boundary
  • Selection/Count: Some, None, All.
• Resources: Most of our heuristics were collected from others across the web. Including, but not limited to James Bach, Michael Bolton and Cem Kaner.

In this episode Matthew and Federico sit to talk about some of the changes that are happening in the ASP.NET QA team. We have talked about the evolution of our team, and today we go over the idea of dividing the test team into functional disciplines and how it has worked so far.

News

• ASP.NET QA team is hiring
• MVP Summit

Organization of the QA team

• What was the problem that we identified?
  • Lack of specialization.
  • Lack of resources to deal with test debt.
  • Lack of resources to improve our efficacy.
  • 100% of tester's time was dictated by a feature crew.
• The 3 pillars of the team
  • Product Presence - test presence during product design and prototypes.
  • Product Development - core testing of new features as they are incorporated into the product.
  • Engineering Services - automation strategy, tools/infrastructure and regression testing.
• Work flow of a feature crew
  • As a new feature is designed PP is engaged during early prototypes, verifying scenarios with customer expectations, first exploratory sessions.
  • When the feature is mature enough, PP hands of testing to PD group. These group of testers are normally assigned to cross discipline feature crews and perform the bulk of exploratory testing and any other test activities.
  • As the feature stabilized, ES comes in to help automate regression tests, making sure tests are robust and maintainable. At the end this group continues to run regression tests of the whole product.
• Take away
  • Still an experimental model.
  • So far the gains seem to indicate that the overall capacity of the team is increased.
  • A potential next step is to recognize the value of "on-demand" testing and "constant" testing, and to focus a group of testers on each.