Building A Better Borderlands - One Student at A Time

The scene: you’re a college student in the Dallas / Fort Worth Metroplex, and someone on your campus approaches you and says “Hey, do you like video games?” You should respond with a “Yes!” immediately, because you end up testing the next game from Borderlands developer Gearbox Studios. Because that’s exactly how they found people to test their latest blockbuster.

Matthew Armstrong, director and lead designer on Borderlands and Stephanie Puri, the director of the Gearbox “Truth Team” (or testing squad) held a panel titled “User Research: Lessons From Borderlands” to talk about the ways they collect test data, and what they do with it. The good news is that they really like user testing, and they care a lot about their fans.

Initially Gearbox formed their Gearbox User Research “Truth Team” to test their games internally, and to occasionally run focus tests that they had to keep affordable. Their budget was basically “You can use all the printer paper you want to create flyers.” They took those to high-traffic areas on college campuses, and before long they had willing subjects. (Cut to a spike in traffic of student transferring to DFW area colleges).

They also went to midnight video game launches, starting with the Gears of War 2 launch at Best Buy, and went around with a laptop signing people up to try out their game. Kotaku ended up running a front page story on the test recruiting, and they instantly had over 1200 applicants. Clearly, someone wanted to test their games.

What do these game testers look like, and what do they get? According to Puri, they range in age from 18 to 62, and they sometimes lure in the students with “Pizza Testing,” where they provide pizza in exchange for the gaming. That’s evolved to include monetary compensation and very cool “Tester Only” swag that you can’t get anywhere else, and they also get to go to launch parties.

What are these testers providing to Gearbox in return? Information and tons of it. Then Gearbox crunches it and decides if they should make changes to the game. For instance, Borderlands testers thought that the sprint speed felt way too slow. But instead of just upping the speed, Gearbox just put more stuff on the ground to make it look like they were running faster. The result? Players thought the spring speed was faster.

But, the feedback can also result in counterintuitive changes. Like with the Skags in the game. Players thought that there were too many Skags when you start the game. They thought it was travel area, and not a combat area. The Gearbox solution? Add even more Skags. The result was that it changes the pacing, and the testers started enjoying combat in that area.

But it really is impressive at how many changes were simply cosmetic. In a test similar to the running, users complained that the reload speed was way too slow. Developers looked at Call of Duty (which they consider the benchmark of how a great FPS should feel) and realized that the CoD reload speed was actually slow. But, there was a lot going on with the player’s animated hands during the reload. So, they added a charging handle and another elements to Borderlands, and it made the reloading more dynamic, while taking the same amount of time. The result? Gamers thought the reload speeds were faster.

After the dust had settled, they had initially scored 4.5 consistently on game tests, but after 173 test sessions with 686 people, they scored 8.5 on average. The testing paid off. They have also been able to take the lessons they used in those tests and apply them to Aliens: Colonial Marines, Brothers in Arms: Furious Four, and of course, Borderlands 2. And they’re still continuing to test.

We’re not guaranteeing that moving to the Dallas area and hanging out on college campuses will get you into Gearbox, it couldn’t hurt. At least you might score some free pizza.