An­a­lyse This

In­side the grow­ing devel­op­ment net­work aimed at iden­ti­fy­ing and re­fin­ing the hits of to­mor­row


In­side the grow­ing devel­op­ment net­work aimed at iden­ti­fy­ing and re­fin­ing the hits of to­mor­row

Ge­of­frey Zatkin’s de­gree in psy­chol­ogy first proved use­ful when he joined the orig­i­nal EverQuest team. It’s a type of game that, per­haps more overtly than any that be­fore it, em­ploys psy­cho­log­i­cal hooks to in­spire peo­ple to take up res­i­dence in its world. Since 2006, Zatkin has taken a less vis­i­ble role in the in­dus­try, but one that’s no less in­flu­en­tial. His com­pany, EEDAR, is an un­known to most play­ers, yet it works with more than 90 per cent of videogame pub­lish­ers, eval­u­at­ing and guid­ing big-bud­get videogames.

The na­ture of its scru­tiny can be hard to de­fine. EEDAR owns nu­mer­ous videogame-re­lated patents bear­ing ar­cane de­scrip­tions such as Char­ac­ter­is­tics Of Play­ers Sys­tems & Meth­ods For Analysing Elec­tron­i­cally Em­bod­ied Games, and Sys­tems And Meth­ods For Eval­u­at­ing, Clas­si­fy­ing And Pre­dict­ing Game Trends Us­ing Clus­tered Pat­tern Recog­ni­tion. Sim­ply put, how­ever, a pub­lisher will of­ten come to EEDAR with a dozen or so videogames it’s in­ter­ested in mak­ing. EEDAR will then an­a­lyse the de­signs, scru­ti­nise the mar­ket and ad­vise which of the hy­po­thet­i­cal games seems likely to se­cure the high­est re­view scores and the great­est prof­its.

“Game devel­op­ment is ex­pen­sive, and pub­lish­ers always have more games they’d like to make than they have the re­sources to,” says Zatkin. “A suc­cess­ful game can be fi­nan­cially re­ward­ing, but re­lease enough un­suc­cess­ful games, or even a sin­gle ex­pen­sive [fail­ure], and you could put your com­pany in fi­nan­cial jeop­ardy. It is our job to sup­ply a pub­lisher with both the quan­ti­ta­tive data and the qual­i­ta­tive anal­y­sis to make those dif­fi­cult de­ci­sions in­tel­li­gently.” EEDAR, in other words, is a mod­ern-day seer, one that looks not to the stars but to cold, hard data, and then ad­vises pub­lish­ers where to place their bets.

This is just the first in a quiver of newly minted data-driven ser­vices de­signed to help de­vel­op­ers and pub­lish­ers make more in­formed choices and more suc­cess­ful games. At al­most ev­ery stage of the mod­ern videogame’s ges­ta­tion, there is now a group of­fer­ing to lend their re­fin­ing ex­per­tise. Story, sys­tems, user in­ter­face, even the colour of box art: all have been tested and re-tested, and it­er­ated upon to of­fer the best pos­si­ble chance of suc­cess. To­day, when a videogame can rep­re­sent a mul­ti­mil­lion-dol­lar in­vest­ment, it will of­ten go through mul­ti­ple it­er­a­tions be­fore it’s an­nounced. One anony­mous in­sider re­vealed the ver­sion of Far Cry 4 shown at this year’s E3 was, in fact, the fourth at­tempt at the game. Who knows how many ver­sions of The Last Guardian ex­ist in the devel­op­ment hell mul­ti­verse?

“Many fac­tors in­flu­ence the suc­cess of a game, and com­pa­nies should be ex­am­in­ing th­ese fac­tors be­fore and dur­ing the game’s devel­op­ment life­cy­cle,” says Zatkin. “Is the game con­cep­tu­ally new? Does it in­no­vate? Do con­sumers like the idea? How does the game mea­sure up against com­peti­tor ti­tles? Will con­sumers have fun play­ing it? Is there a core value propo­si­tion that the game of­fers to the cus­tomer? Can mar­ket­ing com­mu­ni­cate that value propo­si­tion? Will con­sumers buy the game? How is game qual­ity track­ing through­out devel­op­ment?”

For Zatkin, find­ing an­swers to th­ese ques­tions in data and anal­y­sis brings a form of sci­en­tific rigour and care to the cre­ative process that, with so many hands in­volved, can quickly be­come some­thing of a chaotic wa­ger.

Once the vi­a­bil­ity of a game has been es­tab­lished, it’s the user re­search com­pany’s turn to shape it. Player Re­search is an or­gan­i­sa­tion that aims to help de­vel­op­ers make bet­ter games by ap­ply­ing its staff’s knowl­edge of psy­chol­ogy, neu­ro­science and hu­man com­puter in­ter­ac­tion. It boasts that all the iOS ti­tles it has con­trib­uted to have earned a cov­eted po­si­tion on the front page of the App Store, while its con­sole and non-mo­bile games in­clude num­berone-sell­ing and BAFTA-nom­i­nated ti­tles.

“The field of game user re­search is rel­a­tively new,” says founder Gra­ham McAl­lis­ter. “But for a game to stand the best chance of be­com­ing a suc­cess, th­ese tech­niques should be ap­plied at all stages of devel­op­ment. It should be an in­cor­po­rated seg­ment work­ing across an en­tire project, from day one to re­lease, or even be­yond, if it’s a game as a ser­vice.”

This user test­ing work takes mul­ti­ple forms, from ex­pert re­views, in which staff as­sess a game us­ing in­ter­nal frame­works, through to it­er­a­tive playtests, where real play­ers (vol­un­teers drawn from a data­base, who range in age from three to 70) are re­cruited to play the game and pro­vide feed­back. “Find­ing the right play­ers is es­sen­tial,” says McAl­lis­ter. “If you get the wrong play­ers, then you’ll get the wrong find­ings and take your game in po­ten­tially the wrong di­rec­tion. We rec­om­mend de­vel­op­ers never use friends or fam­ily for playtest­ing. It’s wast­ing ev­ery­one’s time.”

For McAl­lis­ter, it’s es­sen­tial that de­vel­op­ers ap­ply sci­en­tific rigour to their de­signs. “In terms of gen­eral mis­takes that de­vel­op­ers make, top of the list is al­low­ing as­sump­tions to re­main untested un­til very late in devel­op­ment. Gath­er­ing ob­jec­tive ev­i­dence as early as pos­si­ble is cru­cial. Devs be­come very close to a game having worked on it for months or years. This means dur­ing playtests they may only see what they want to see, us­ing their pre­vi­ous ex­pe­ri­ence and knowl­edge to bias what they find.”

While the value of a fresh, out­sider’s per­spec­tive on a game may be cru­cial to cor­rect wrong turns in the devel­op­ment process, many of the world’s larger game stu­dios have re­cently formed their own in­ter­nal teams, de­signed to ap­ply this sci­en­tific frame­work to the devel­op­ment process. Au­drey Lau­rent-An­dré and Sébastien Odasso run Ubisoft’s Parisian ed­i­to­rial user re­search group, AKA lab team, a col­lec­tion of al­most 20 de­sign­ers who or­gan­ise and watch live playtests on any of the com­pany’s var­i­ous pro­jects around its global net­work of stu­dios. “For in­stance, some­one on the As­sas­sin’s Creed team might pose the ques­tion: ‘Will the new game be clearly un­der­stood by new­com­ers to the se­ries?’” says Lau­rent-An­dré. “It’s then a case of se­lect­ing a po­ten­tial tester, find­ing out what kind of games they play, the plat­forms they play on most reg­u­larly and so on. We have a di­verse pool for all types of play­ers around the world. It takes us less than two days to or­gan­ise a test with any group of spe­cific types of player.”

While user test­ing was at one point a some­what loose part of the Ubisoft process, Odasso says, the lab team was founded to en­sure it forms part of “the very DNA of a project”. In fact, us­abil­ity tests oc­cur ev­ery two to three weeks. As well as find­ing out whether a player con­nects with a game’s story, set­ting or sys­tems, the lab team also gath­ers more gran­u­lar data. “Let’s say one of the devel­op­ment teams wants to check that a spe­cific boss fight isn’t too hard,” says Lau­rent-An­dré. “We record var­i­ous statis­tics dur­ing a player’s ses­sion, such as the amount of time to com­ple­tion, the num­ber of deaths in­curred and so on. Th­ese things help us quan­tify chal­lenge. For ex­am­ple, the de­signer may have in­tended the player to de­feat the boss in 20 min­utes. If it takes an ex­pe­ri­enced player an hour, dur­ing which he dies 30 times and leaves re­port­ing a feel­ing of frus­tra­tion or mis­un­der­stand­ing, then we know we have a prob­lem.”

To im­prove con­sis­tency across so many tests, Ubisoft now as­signs a tester from the lab to each ti­tle, some­one who fol­lows the game across its ges­ta­tion. “For in­stance, we have one co­or­di­na­tor who runs all of the test­ing for The Di­vi­sion,” Odasso ex­plains. “He is able to fol­low the dif­fer­ences between ver­sions of the game and, through the process, build up a good un­der­stand­ing of the kind of things that the team wants to know. The deeper the knowl­edge on the project, the more ef­fi­cient the method­ol­ogy.”

Odasso him­self has a back­ground in neu­ro­science and neu­ro­phys­i­ol­ogy. He was trained in per­cep­tion-based user tests – or, in other words, ver­sions of the Pepsi chal­lenge. “The food in­dus­try didn’t ex­cite me,” he says. “I saw an op­por­tu­nity

to come to Ubisoft to do the kind of work that I was in­ter­ested in.”

Odasso has been at Ubisoft for six years now and, dur­ing that time, has seen a huge amount of change in the man­ner and rigour with which games are tested. “Us­abil­ity tests were first car­ried out at Ubisoft in 2001,” he ex­plains. “Since then, we’ve been con­stantly im­prov­ing our method­ol­ogy. When I ar­rived, there were five of us in the lab. We’ve quadru­pled in size since then. Our devel­op­ment process has be­come in­creas­ingly ori­en­tated around the player’s re­sponse to a game dur­ing its devel­op­ment.”

Lau­rent-An­dré, by con­trast, trained as a de­signer and pro­gram­mer. But she saw a rare op­por­tu­nity in an­a­lyt­ics. “I don’t think I’m not do­ing de­sign per se,” she says. “I’m not de­sign­ing things as such, but an un­der­stand­ing of game de­sign is cru­cial to the job. You have to un­der­stand why play­ers in­ter­act with game­play loops, why loops are be­hav­ing in the way they should, why play­ers aren’t sat­is­fied, why they don’t un­der­stand some­thing, and why they don’t feel re­warded. It’s im­por­tant to have peo­ple who are ex­pe­ri­enced in user re­search, but we also need peo­ple who have a deeper un­der­stand­ing of all the de­sign stuff. It’s a com­po­nent of game de­sign, even if it’s not game de­sign in the tra­di­tional sense.”

Some of the chal­lenge for any team in­volved in user test­ing is sep­a­rat­ing the ob­jec­tive data from users’ eval­u­a­tions. “If a player re­peat­edly fails one sec­tion of a game, then that is hard data,” says Odasso. “We usu­ally see a great deal of cor­re­la­tion between play­ers’ per­for­mance on this hard data. For ex­am­ple, on As­sas­sin’s Creed IV: Black Flag, 80 per cent of us­abil­ity is­sues were com­mon among all play­ers dur­ing tests. But when we ask play­ers for sub­jec­tive eval­u­a­tion of what they like or dis­like about a game, there can be a great deal of vari­a­tion.”

Such vary­ing feed­back can be con­fus­ing for a team. For this rea­son, an­other type of free­lance ser­vice in the game-im­prov­ing econ­omy has sprung up dur­ing the past decade: the con­sul­tant critic. It’s a pro­fes­sion that at­tracts many ex-jour­nal­ists. They may not bring pure ob­jec­tiv­ity, but they do of­fer ex­per­tise in the area of pro­fes­sional re­view­ing. For­mer Edge colum­nist N’Gai Croal left his jour­nal­ism ca­reer to found Hit De­tec­tion in 2009, where he of­fers a critic’s eye on pro­jects dur­ing devel­op­ment. Zatkin’s EEDAR also of­fers a ‘mock re­view’ ser­vice.

“Mock re­views, at a base level, give you a heads-up on how the prod­uct will be re­viewed when it fi­nally hits the press,” says Zatkin. “They in­de­pen­dently point out spe­cific strengths and weak­nesses in the ti­tle, which might be dif­fer­ent from what is in­ter­nally per­ceived to be the game’s strengths and weak­nesses.”

As well as giv­ing the game’s cre­ators a dif­fer­ent per­spec­tive, larger pub­lish­ers use this in­for­ma­tion to high­light the game’s strengths in mar­ket­ing ma­te­ri­als. Zatkin: “Mock re­views can also pro­vide last-minute pol­ish sug­ges­tions, can also give an ex­tra heads-up on any tech­ni­cal is­sues go­ing into launch, and can point out any ar­eas of the ti­tle that might be fo­cused on for DLC or even a pos­si­ble se­quel. There’s a lot you can learn by let­ting an ex­pe­ri­enced third­party take a look at the game be­fore it launches.”

Some be­lieve the ‘mock re­view’ comes much too late in the process to be of gen­uine use. “Usu­ally, the re­viewer is brought in when it’s too late to make mean­ing­ful changes, and of­ten re­view­ers can be quite lit­er­ate about their likes and dis­likes and how things feel, but don’t pro­vide de­sign crit­i­cism,” says Leigh Alexan­der, an­other erst­while Edge colum­nist who, in 2014, founded Agency with one-time Edge staffer Ste Cur­ran. “They can tell a de­vel­oper some­thing’s not work­ing, but it’s unusual for a mock re­view to shed light on why.”

Agency was formed to close the gap between the game a team wants to make and the game that’s ac­tu­ally be­ing made, and, to date, has pri­mar­ily worked with smaller stu­dios, such as Tale Of Tales.

“Tra­di­tional com­mer­cial devel­op­ment has a ton of sys­temic prob­lems,” says Alexan­der. “Most com­monly, a game’s dis­parate com­po­nents don’t co­here in

that they don’t serve one an­other as well as they could. This tends to be be­cause ev­ery­one is work­ing closely on their in­di­vid­ual area, and the per­son in charge mainly wants to make sure ev­ery­one fin­ishes their bit on time and within bud­get. There is no all-see­ing eye to see this lack of co­he­sive­ness from a good dis­tance. Like­wise, of­ten peo­ple work­ing on a game start to feel un­sure about whether it will work, but they don’t have the abil­ity to raise their con­cerns, ei­ther be­cause they have mile­stones to meet or be­cause devel­op­ment in­her­ently in­volves com­pro­mises. Get­ting ev­ery­body on the same page, help­ing en­sure there’s a vi­sion in place that ev­ery­one can see clearly and feels pas­sion­ately about, means that pro­jects will be well-scoped and goals clearly iden­ti­fied be­fore the in­vest­ment in full devel­op­ment is un­der­way.”

Like Player Re­search, Agency’s work is broadly sys­tem­atic and log­i­cal. The com­pany de­liv­ers di­ag­noses that can be kept along­side other de­sign doc­u­ments. But Agency tai­lors the na­ture of its con­sul­tancy to the needs of the de­vel­oper. “With Tale Of Tales, we were hired to help them meet their goal of mak­ing a game that could reach a big­ger au­di­ence,” says Alexan­der. Agency then worked with the team to cre­ate a rel­a­tively main­stream de­sign vo­cab­u­lary for the forth­com­ing Sun­set, as well as to de­fine the lan­guage used when dis­cussing the game in public.

It can be dif­fi­cult to test quan­ti­ta­tively if a game’s goal is sim­ply to pro­vide ‘fun’, though. “Games are unique as an in­dus­try in terms of soft­ware devel­op­ment in this re­spect,” says Ben Wib­ber­ley, a direc­tor at VMC, a com­pany that of­fers qual­ity as­sur­ance test­ing both be­fore and af­ter re­lease. “It’s ex­tremely dif­fi­cult to quan­tify the fun fac­tor in a game. You can­not au­to­mate ex­pe­ri­ence. That’s where QA (qual­ity as­sur­ance) can step in with cal­cu­la­ble feed­back on things such as bal­ance, flow and pro­gres­sion.”

The busi­ness of QA, the fi­nal com­po­nent in the mod­ern test­ing ma­chin­ery, is cru­cial to en­sure that a game func­tions as it should, aside from its artis­tic merit or in­tent. Dur­ing the past few years, the job has ex­panded. In many cases, par­tic­u­larly with games that ex­ist as an on­go­ing propo­si­tion, there is work to be done post-re­lease. Games as di­verse as Brink, Ti­tan­fall, Grand Theft Auto V and Fi­nal Fan­tasy XIV: A Realm Re­born have all fallen at the fi­nal hur­dle when it comes to on­line func­tion­al­ity, with is­sues such as dis­con­nec­tions and, in some cases, de­bil­i­tat­ing crashes.

“The in­dus­try has seen some huge ti­tles fail on public launch,” Wib­ber­ley says. “Sim­ply put, this is be­cause there are is­sues that can­not be iden­ti­fied in a QA lab. The only way they can be seen is by test­ing in a live en­vi­ron­ment. Many game com­pa­nies try to do this with public beta tests, but un­for­tu­nately that does not hit to the root of what will cause games to fail at launch, since pri­mar­ily th­ese public be­tas are mar­ket­ing ex­er­cises and are not geared to test­ing.”

To help com­pa­nies stress-test their big­bud­get on­line ti­tles in a way that pro­vides use­ful fore­sight, VMC has built a pri­vate global beta test net­work, a com­mu­nity of thou­sands of beta testers that en­ables it to test in a live sit­u­a­tion, in­clud­ing checks on stress, us­abil­ity and match­mak­ing.

To­day, the work of th­ese videogame sci­en­tists is per­haps bet­ter de­scribed as mid­wifery, a se­ries of roles and sys­tems built to rein in the chaos of bal­loon­ing devel­op­ment teams and safely de­liver a game in its health­i­est form. Videogames are artis­tic pro­jects, but they’re also func­tional prod­ucts that need to work.

Still, there is a dan­ger that end­less mar­ket anal­y­sis and player test­ing can lead to ho­mo­gene­ity in the big-bud­get space. It’s a prob­lem that’s echoed in Hol­ly­wood, where the fi­nan­cial risks in­volved to those fund­ing the most ex­pen­sive work also en­cour­ages cre­ative con­ser­vatism. And, of course, there can be no pre­ex­ist­ing mar­ket data for games that pi­o­neer brand new spa­ces. When it comes to run­ning the next Minecraft or Dwarf Fortress un­der the mi­cro­scope, the videogame sci­en­tists can be­gin to look closer to for­tune tell­ers, and their guess might be as good as any­one’s.






1.3 Devel­op­ment costs can spi­ral when a game stalls, a sce­nario that early ad­vice is in part meant to pre­vent

1.2 Far Cry 4’ s de­sign, like many of its peers, has been it­er­a­tive. Given the costs in­volved, it has to ap­peal

1.1 MMOG de­sign taps into hu­man psy­chol­ogy to en­sure player re­ten­tion and thus sub­scrip­tions or item sales

2.4 Sébastien Odasso, Ubisoft Ed­i­to­rial lab team

2.3 Au­drey Lau­ren­tAn­dré, Ubisoft Ed­i­to­rial lab team

2.2 Player Re­search founder Gra­ham McAl­lis­ter

2.1 Ge­of­frey Zatkin, CPO of EEDAR

3.2 Player Re­search claims that 100 per cent of the iOS games for which the com­pany has pro­vided user test­ing data have made it to the front page of the App Store

3.1 EEDAR has worked with the likes of Rock­star, Mi­crosoft, SOE, 2K, Crys­tal Dy­nam­ics and Ac­tivi­sion

3.3 Agency bills it­self as of­fer­ing an ex­ter­nal eye on pro­jects, its in­sights born from ex­pe­ri­ence

3.4 VMC of­fers qual­ity as­sur­ance test­ing, en­sur­ing that games and soft­ware per­form as they should

4.3 On­line fea­tures are tough to test in a way that re­flects real-world con­di­tions, and thus present unique is­sues

4.2 Agency is work­ing with Tale Of Tales on Sun­set, a first­per­son nar­ra­tive about a maid and a revo­lu­tion

4.1 As­sas­sin’s Creed IV in­cluded feed­back sys­tems within the game to gather play­ers’ sub­jec­tive opin­ions

5.1 N’Gai Croal, founder and CEO of Hit De­tec­tion

5.4 Ben Wib­ber­ley, games direc­tor, VMC

5.3 Ste Cur­ran, Agency co-founder

5.2 Leigh Alexan­der, Agency co-founder

Both DmC and Re­mem­ber Me were tested by Player Re­search, prov­ing that as­sess­ment is no guar­an­tee of com­mer­cial suc­cess

Newspapers in English

Newspapers from Australia

© PressReader. All rights reserved.