Test Case Management in Lean Testing has Changed

Posted in: Product news, Quality assurance testing by: Simon on:

If you’ve been around long enough, you’ll remember that our first foray in test case management at Lean Testing was through the acquisition of Overlook.io, which we operated as a separate entity for a little while. Well, we quickly realized that managing two separate tools was not optimal for us or for our users, so we merged what was then our “bug tracking tool” known as Damn Bugs with Overlook.io to create Lean Testing: a comprehensive test management solution.

The initial version of our test case management system was fairly limited in scope and functionality. To be perfectly honest, it was an experiment. We had no idea if and how it would be used by our users. However, we liked the idea of a bug tracker and a test case manager in the same tool.

Fast forward to a few months from now, we realized that over 10,000 test suites and over 130,000 test cases had been created in our test case management tool. That was validation enough for us to start working on a new and improved version of our test case management system.

Last night, after many months of hard work, we released a completely new version of our test case management system. It’s MUCH better than the previous version, although still a little bit rough around the edges in terms of design. That’s how we do things – we build fast and polish later. I’ve got exciting news to share about that later!

Let me tell you a little bit about how our test case management system works.

You start by creating a test suite.

A test suite is an ensemble of test cases related to a software under test. A test case is basically a test that you need to do to ensure the conformity or the proper functionality of a product.

For example, a test case can be: “Log-in with a valid username and password and ensure that you are properly redirected to your dashboard.”

A test suite can have an unlimited number of test cases. There can only be one test suite per project. However, you can organize your test cases per component. For example, you could associate the test case above with a “Log-in” component to keep things neatly organized.

A test suite is a living document. It will evolve over time. However, we chose not to have multiple variations of a test suite. It’s difficult enough to keep one test suite organized, and it’s a nightmare to maintain 10 of them. So, 1 project in Lean Testing = 1 test suite.

If you maintain several different products in a single project in Lean Testing (for example, an iOS app and an Android app), you will still have 1 test suite. However, you can create iOS-specific and Android-specific components which you’ll select when executing your test runs.

test case management

Creating and executing test runs.

When you’ve created your test cases and finalized your test suite, the actual testing work can begin.

Click on “Start a new test run” to begin.

You will then be prompted to select what test cases you’d like to include in this test run.

If you’re conducting full tests, you might want to “select all” test cases.

You can also select all of the test cases related to a specific component, or all of the test cases marked as Critical and High priority in your test suite. This is intended to facilitate the execution of build verification tests (i.e., smoke tests.)

Once you’ve selected the test cases to include in your test run, you’ll be taken to the “execute a test run” screen.

From there, you can indicate a status for each test case (Pass / Fail / Could Not Test / Not Applicable). You can also report bugs, add comments and attach files as part of your results.

When you report a bug from a particular test case, this link is indicated in the bug report.

test case management

Once you’ve completed your test run, click on “Save changes” at the bottom of the list. Your results will be visible to all other project members on their dashboard.

That’s the quick overview of how the new test case management system works in Lean Testing.

I’d love to hear your thoughts – and did you notice how I totally just used images from our upcoming UI update as a teaser? 🙂






Simon is the founder of Crowdsourced Testing. After 10 years in interactive software development, he set his sights on building a world-class crowdsourcing platform to facilitate the software testing process for developers.