Today, Chris Bryant, QA Lead and stalwart bug forums warrior, has taken some time away from squashing bugs to talk to us about what he does every day. Read on, as Chris takes us through all the inner workings of Mythic's QA department and discusses everything from the different QA teams to how new content is tested and, most importantly, how bugs are fixed!
Hey there, everyone! My name is Chris Bryant (or as most of the staff in QA calls me, Old Man Bryant) and I am the Senior Lead for the Quality Assurance (QA) department here at Mythic. I have been with Mythic going on 6+ years now and I have spent the majority of that time working QA on either Dark Age or WAR. So, if there is a feature that you love (or hate) within either of those games I was probably involved in the testing of said feature at some point. But being in QA, I live by the following motto:
I don’t make it, I don’t break it, and I don’t fix it. I just get to tell other people their stuff is broken.It is really rare that you find a job that you love; one where you don’t wake up in the morning and want to call in sick “just because.” For me, this is one of them. I have actually had, on more than one occasion, my boss come up to me to yell at me to take time off because I can no longer accrue any more paid time off. And trust me, I really love my free time, but coming to work is fun almost every day. When you come to work and still have a great time, why miss a day?
When the Community guys asked me to write up a Dev Diary on the Day in the Life of the QA team, all I could think of was, “This is going to be really short.” How much space would I need to write, “We test stuff”? Seriously.
If you have seen Grandma’s Boy then you pretty much have a clue (albeit a very oversimplified one) about what we do here on a daily basis. But unlike the movie, or even other QA departments throughout the industry, we don’t grind out game after game. We have a focused QA staff here, some that have been on the WAR project since the first art zone was uploaded onto a server. Having dedicated testers on one project for that amount of time is pretty unique in the industry.
How do we test the new content that is added to the game?
We have a couple teams within the QA department. We have Embedded Testers. These are the guys that sit in with their respective teams (e.g. User Interface, Combat & Careers, RVR, Content, Items, Art, International Engineering and Core Tech). These guys are the first line of testing. They spend a great deal of time testing their respective areas on our internal servers. Most of the time, this testing is done before a lot of their particular features are even in game.
For example, the C&C testers will be testing abilities on one of their Dev “sandbox” servers with no spell effects and no animations and nofeedback in the combat log to let them know what is even going on. They have to use all the same tools that their Devs use to sort out the basic functionality of the abilities. The Content embedded guys have it even harder at times. Try testing out entire new zones for new content that have no art—the quest objects and monster art is all placeholder and there are no waypoints. (I applaud those guys because content is the one area I really hate to test.)
Pretty much every new system or content area in the game that makes it to the Live servers are tested by their respective embedded team. After these systems pass their initial testing by the Embedded crews, the issues are sent down to the Internal Core Testers (the Core Testing group is my group by the way), not to be confused with the player group of Core Testers, or Oracles as they’re called on the forums.
The job of the Internal Core Testing team is, essentially, to retest everything that is passed down to them from our Embedded groups. These guys are not specialized testers that spend their Mythic lives testing one particular area. The crew on the Internal Core Testing team gets their hands dirty in every aspect of the game. They are my “Go To” guys. If I have any questions on almost any system in our game they have an answer or can get one to me quickly. They really know their stuff and I lean on these guys everyday because I don’t actually get to get in there and get my hands dirty much anymore. For those of you that know me from my postings on our Bug Forums, they are the ones that test about 90% of the stuff to which I respond (the other 10% of the testing is done by our phenomenal Embedded groups).
Not only do our Embedded and Core testers have to test every new system that is implemented, but they have to test it on multiple servers as they work their way up the chain until they reach the Live servers in either a scheduled patch or a Hotfix. So when a test request comes down to us to test the 1,000 new items that are going into the game, we have to test them on multiple servers (/sarcasm on) and boy is that a lot of fun (/sarcasm off). So, pretty much every new system is tested by multiple people on multiple servers before it reaches the Live servers.
So why do bugs make it up to the Live Servers?
The first answer is just the quick and easy one: we are human and we miss stuff. There are only so many variations of Ability X, when used with Ability Y in Zone A while grouped with Class B that we are able to check before we sign off on a new feature. I mean seriously, who knew that a small change to a User Interface functionality issue would break an ability that was working perfectly fine before the “fix.” And who knew that fixing that ability to work again would break another ability that is totally unrelated to the first ... and so on and so on. We are usually under a very aggressive testing schedule and there are just some things, unfortunately, that we will miss from time to time.
What happens to that bug I reported in-game/on the forums?
The short answer: If you report it, it is tested.
Unless you are the 100th person to report the same bug, consider your particular issue tested. And even if you are the 100th person to submit the same issue, while we may not test the issue you reported, we did test one of the previous 99 that were reported. We test every issue our playerbase reports to us whether it’s from the in-game appeal system or our forums. While you will never get a response from using the in-game Bug Report system, please don’t hesitate to use it. Lots of valuable information is pulled from the game and added to the Bug ticket that is generated by your report. Art issues in a particular area of a zone are one of those issues that if you don’t report it in-game, we probably won’t find it. Those in-game generated reports have items like your location in game automatically appended to the ticket.
Every morning we get a report from our Customer Service team about the most appealed issues over the previous day. This is the first thing our testers work on in the morning when they get in the door. While they are working on those issues from the CS report, I am combing through our bug forums and assigning out every new issue to be investigated. While I am working on the forums, we have our Live team going through all the bugs that have been reported in-game and getting those into our tester’s hands. Once all of the issues that have been reported from the previous day have been assigned out to the testers, their day of testing begins.
So between testing all of the new content that will be going into the game, and all of the reported bugs from the player base, our tester’s days are pretty much full from the time they walk in the door until they leave each night. Hopefully this gives you all a little insight into the workings of our QA department.
Oh, and please don’t troll me on the forums! :)