Sometimes when testing software you will come across the dreaded white whale. In my case this was a crash that only I have reproduced, but our clients were hitting it quiet often. Every time I hit the crash it seemed something different caused it, and was only reproducible 12% of the time but I had clients hitting the crash left and right. So "Call me Ishmael" I had to hunt this white whale down. I spent 3 weeks hunting using all kinds of tools and using resources just like Moby Dick did to hunt down the white whale, to no avail. Then one day when hunting a run of the mill crash that I knew how to reproduce I hit the whale. It turns out I was running all these tools to catch the run of the mill crash that it slowed down the software enough to allow me to hit the white whale. After I finally got all the info in it was a line change of code and the white whale was no longer a bother.
As the picture states there are several white whales out there in the testing world. You will run into them and you will hunt them down. Just don't get to frustrated when you can't, because sometimes you will run across and figure it out without even realizing that is what you where doing. Remember to think outside the box, and use all the tricks you can. As testers get more tricky, so will the bugs.
What does it take to write a good bug? There is a lot of time where a bug comes in from a somewhere else and you have no idea what they are talking about or what was tried. How can you reproduce it if you are unsure about where the bug is?
Here are some must haves for a bug.
Attachments are a huge help. Even the worse bug can be trumped by a great video or screenshot. I have gotten bugs where I had no clue what they were talking about until the video played. Once you get good at putting in usability and functional bugs you can step using some awesome tools. We can go over that in a later blog.
Damn bugs was founded by Crowd Soured Testing as a online tracker for bugs. You start by creating a Organization and then you have projects. The projects can then have all types of bugs. This is mostly a web based one so there are web browser extensions to put in bugs for websites, which is really useful. This can also be used as a stand alone. There are different statuses for project management, different types to tell you if it is design or dev or any other kind of ticket. You can add other team members with different jobs. They are also really good about listening to feedback and getting back to you. There is sorting, searching, and color coding. Also the bug reporting works well and prompts good reporting to anyone who is not use to bug reporting. All in all, for where it is at, this is a great system, and I can't wait to see what it has in store for the future. Also one of my suggestions I made were to add test plan integration. The response I got was they had bought Overlook and were working on it... yes many awesome things for this sites future.
Overlook was currently bought by Crowd Sourced Testing and is a test plan website that will be integrated with Damn Bugs. Overlook is about making a simple test plan. The test plans for this were also more designed around testing web based applications but again you can use it for other plans. First you will create a project within your account. You can than manage a team, and invite new testers and create new test plans for the project. From the first page you are also able to edit, see results, and start the test. When you create the test you are able to do it line by line with no formatting. You can pass, fail, and add notes to failures. When tied to a site you can run the test plan at the top while having the webpage below. It is very simple and could use a lot of growth. It has the same mechanism Damn Bugs has for feedback but currently that is not heavily watched. My guess is it will grow as Crowd Soured Testing takes a bigger look at it.
Trello helps managing projects, it is useful for several things that a tester can use. Trello you can make list, add cards to the list, and move them around like your very own work board. This is great for managing a backlog and assigning bugs. This shows a great workflow that works really well for agile teams. Our team uses it for managing projects, and for managing our back log. You can organize, prioritize, add labels, and filter by the labels. You can also build test plans where you can create checklist for each card. This site is very versatile and allows for many different uses. You can also create organizations and add people to them.
Workflowy is a list website that can be used for test plans and organization. In this site you make list and list within those list. Things can be marked as completed. You can hide the completed. You can add notes to each element in order to kept track of things. So a test plan can be created using the list, within the list mark them as completed as you go. Then add notes to the completed test. Again you could use this very simple tool for a lot of different things, just use your imagination.
After Testing a new build
When running a familiar test and you hit an unexpected crash.
When you hit a major crash that stops release.
When they release a build filled with bugs.
When a bug gets upgraded to a feature.
Over much of my research I found that a lot of people, to this day use excel for their test plans. There are articles posted all over about how to best use excel. For example this link, talks about how to utilize excel when writing your test plan, and I have attached a template I found on that site that utilizes excel.
So why is excel so widely used? Well, it is simple and versatile. Each tester has their own ways of doing things and excel allows everyone to have a different set up. When I looked at other testing software such as Test link which is widely used and open sourced test plan software. In the end it took me over an hour to set everything up without a tutorial. It is overly complicated, that might be why it is not used as widely as excel. Another one, I am excited to see how it grows, is Overlook. However, the main page currently says it best "Really Simple Software Test Plan". It is very basic, but the reason I am excited about it is because it was recently bought by Crowd Soured Testing, and they told me they would be working on integrating it with another new site of theirs, Damn Bugs. Plus they work with the people on their site to help make the best product. I will go over Overlook and Damn bugs at a later time though when I go over free sites that can be of uses to a QA professional.
The problem lies in the software for test plans, it is either not versatile enough, or it is too simple. Excel allows the test plan to be any type. What needs to be made, and you can find within software companies who make their own, is a system that has an integrated test plan, and bug tracking system. This allows for when something is failed, there is a bug, and it is tracked immediately. There are paid systems out there that are good, but again not everyone is going to fork that much money into QA. So in the end, excel just works where others fail.
Also remember that test plans aren't everything and Exploratory Testing is needed in conjunction with your test plans.
QA Hipster is a Quality Assurance Technical Lead for an software company, mostly working with Macs. I have been in the field since September 2013, and have a bachelor's degree in Management Information Services. I started my studies at Iowa State in Software Engineering. I have been working on moving my company forward with the latest QA techniques.