Har­ness the power of WEBRTC

Dis­cover real-time com­mu­ni­ca­tion through the browser with ac­cess to cam­era and mi­cro­phone hard­ware.

Web Designer - - Contents -

WEBRTC – short for Web Real-time Com­mu­ni­ca­tion – is a by-prod­uct of Google’s ac­qui­si­tion of GIPS. Google open-sourced most of the in­tel­lec­tual prop­erty held by the video con­fer­enc­ing ven­dor, thereby mo­ti­vat­ing stan­dard­i­s­a­tion com­mit­tees to ac­cept the tech­nol­ogy.

WEBRTC is a bit of a beast due to the breadth of is­sues han­dled by the spec­i­fi­ca­tion. The stan­dard does not limit it­self to the pro­vi­sion­ing of a high-speed low-la­tency con­nec­tion sys­tem, but also pro­vides meth­ods to ac­cess cam­era and mi­cro­phone hard­ware in a con­ve­nient fash­ion. The dif­fi­cul­ties in­volved in the real-time pro­cess­ing of au­dio and video data led to the cre­ation of Rtcpeer­con­nec­tion – a class ded­i­cated to com­bin­ing the two above-men­tioned APIS.

Due to the com­plex­ity of real-time com­mu­ni­ca­tion, cre­at­ing one from scratch is in­fea­si­ble for all but the most ex­pe­ri­enced of web de­vel­op­ers. The fol­low­ing story uses a set of open-source ex­am­ples, which we will dis­cuss in a piece­meal fash­ion. This knowl­edge lets you cre­ate an ac­tual sys­tem by com­bin­ing one or more of the sys­tems.

As for tech­nol­ogy, an AMD Oc­ta­core work­sta­tion is be­ing used. It runs Ubuntu 14.04 and a re­cent ver­sion of Fire­fox, and the AMD ma­chine uses a Black­berry PRIV as a we­b­cam sub­sti­tute.

1. Col­lect pic­tures

Our first ex­am­ple, hosted at https://webrtc.github.io/ sam­ples/src/con­tent/ge­tuser­me­dia/can­vas, con­tains a live We­b­cam pre­view and a can­vas tag in­tended to hold static im­age in­for­ma­tion. In HTML markup, the three con­trols shown next to this step are rel­e­vant.

<video playsin­line=”” au­to­play=””></video> <but­ton>take snap­shot</but­ton>

<can­vas width=”480” height=”360”></can­vas>

2. ‘ge­tuser­me­dia’ gets me­dia in­for­ma­tion

‘ge­tuser­me­dia’ is one of the three core meth­ods of the WEBRTC stan­dard. It queries the me­dia in­put de­vices avail­able to the browser in an asyn­chro­nous fash­ion while keep­ing the con­straints ob­ject in mind – data can only be har­vested from the prom­ise el­e­ment and re­turned af­ter the men­tion. const con­straints = { au­dio: false, video: true

}; nav­i­ga­tor.me­di­ade­vices. ge­tuser­me­dia(con­straints).then(han­dle­suc­cess). catch(han­dleer­ror);

3. Con­nect video and data source

If a me­dia de­vice sat­is­fy­ing the de­mands of the con­straint ob­ject is found, it will be re­turned to the suc­cess event han­dler reg­is­tered in the prom­ise. In the case of our project, we sim­ply pass the stream into the video tags ‘sr­cob­ject’ prop­erty: func­tion han­dle­suc­cess(stream) {

win­dow.stream = stream; // make stream avail­able to browser con­sole

video.sr­cob­ject = stream; }

4. Take out in­for­ma­tion com­fort­ably

Video stream ob­jects change their con­tent per­ma­nently. Our sam­ple stands out from the rest in that it also con­tains a can­vas tag, whose el­e­ments get pop­u­lated with a sin­gle men­tion of ‘draw­im­age’. This bit­map could then be con­vo­luted or post-pro­cessed in any other way you fancy. but­ton.onclick = func­tion() { can­vas.width = video.vide­owidth; can­vas.height = video.video­height; can­vas.get­con­text(‘2d’).draw­im­age(video, 0, 0, can­vas.width, can­vas.height);

};

5. Be­ware of legacy code

Dur­ing the pro­tracted de­vel­op­ment process of WEBRTC, the ‘ge­tuser­me­dia’ method orig­i­nally was not housed in­side of the ‘me­di­ade­vices’ enum. Legacy code is likely to con­tain snip­pets sim­i­lar to the one ac­com­pa­ny­ing this step – they should be re­struc­tured.

nav­i­ga­tor.ge­tuser­me­dia(con­straints, suc­cess­call­back, er­ror­call­back);

6. Use ad­vanced con­straints

While chat ap­pli­ca­tions tend to be able to make do with just about any bar­gain-base­ment we­b­cam, fa­cial iden­ti­fi­ca­tion and sim­i­lar jobs re­quire more. In this case, the ‘con­straints’ ob­ject can be pro­vided with ad­di­tional pa­ram­e­ters lim­it­ing we­b­cam se­lec­tion.

{ au­dio: true, video: { width: { min: 1024, ideal: 1280, max: 1920 }, height: { min: 776, ideal: 720, max: 1080 }

} }

7. Source and sink

One in­ter­est­ing as­pect of the WEBRTC GUI in­te­gra­tion is the ca­pa­bil­ity to make one video tag mir­ror the con­tent of an­other. Sadly, two dif­fer­ent meth­ods ex­ist for this op­er­a­tion at the time of writ­ing. if (left­video.cap­ture­stream) { stream = left­video.cap­ture­stream(); rightvideo.sr­cob­ject = stream; con­sole.log(‘cap­tured stream from left­video with cap­ture­stream’, stream);

} else if (left­video.moz­cap­ture­stream) { stream = left­video.moz­cap­ture­stream(); rightvideo.sr­cob­ject = stream; con­sole.log(‘cap­tured stream from left­video with moz­cap­ture­stream()’, stream); }

8. Trans­mit con­trol data

While it’s ob­vi­ous that send­ing video is the name of the game, let us start out by trans­fer­ring con­trol data via a low la­tency link. This is best han­dled via ‘Rtcpeer­con­nec­tion’ – think of the class as a Web­socket re­place­ment work­ing with Webrtc-de­rived tech­nolo­gies.

9. Cre­ate a con­nec­tion

While Rtcpeer­con­nec­tions can – in the­ory – trans­fer tremen­dously large data streams, they are usu­ally used to trans­mit­ting con­trol in­for­ma­tion. The ex­am­ple at https:// webrtc.github.io/sam­ples/src/con­tent/dat­achan­nel/ ba­sic uses WEBRTC to cre­ate some kind of chat client that trans­mits data be­tween two chat win­dows.

10. Link up

Rtcpeer­con­nec­tions do their magic by tun­nelling through net­work hard­ware. This means that con­nec­tion es­tab­lish­ments can be slightly com­plex. In par­tic­u­lar, one or more in­stances of the ‘Rt­ci­ce­can­di­date’ con­nec­tion pa­ram­e­ter ob­jects spawn dur­ing con­nec­tion es­tab­lish­ment.

win­dow.lo­cal­con­nec­tion = lo­cal­con­nec­tion = new Rtcpeer­con­nec­tion(servers);

send­chan­nel = lo­cal­con­nec­tion.cre­ate­dat­achan nel(‘send­dat­achan­nel’); lo­cal­con­nec­tion.on­ice­can­di­date = e => { on­ice­can­di­date(lo­cal­con­nec­tion, e);

}; send­chan­nel.onopen =

on­send­chan­nel­stat­e­change;

send­chan­nel.on­close = on­send­chan­nel­stat­e­change;

11. Com­mu­ni­cate asyn­chronously

WEBRTC’S com­mu­ni­ca­tion API is mod­elled on the pro­gram­ming model that you might have used in Web­sock­ets. Mes­sages ar­rive in an event han­dler, which you must reg­is­ter be­fore use. Sadly, this can not hap­pen di­rectly – you must in­stead wait un­til the We­brtc­connec­tion es­tab­lishes it­self. func­tion re­ceivechan­nel­call­back(event) { trace(‘re­ceive Chan­nel Call­back’); re­ceivechan­nel = event.chan­nel; re­ceivechan­nel.on­mes­sage = on­re­ceivemes­sage­call­back;

re­ceivechan­nel.onopen = on­re­ceivechan­nel­stat­e­change;

re­ceivechan­nel.on­close = on­re­ceivechan­nel­stat­e­change;

} func­tion on­re­ceivemes­sage­call­back(event) { trace(‘re­ceived Mes­sage’); dat­achan­nel­re­ceive.value = event.data; }

12. Go even deeper

At this point, it is time to an­a­lyse a com­plete com­mu­ni­ca­tion pro­gram. Head over to https://webrtc. github.io/sam­ples/src/con­tent/peer­con­nec­tion/ up­grade and feast your eyes on two win­dows. They can, fur­ther­more, be con­nected to one an­other via a sim­ple group of but­tons.

13. Get au­dio data

The ex­am­ple project starts out by cre­at­ing a stream ob­ject us­ing the ‘ge­tuser­me­dia’ func­tion we dis­cussed pre­vi­ously. This time, the ‘con­straints’ el­e­ment is set up to limit it­self to the pro­vi­sion of au­dio data. ‘got­stream()’ keeps the stream ob­ject in a vari­able called lo­cal­stream. nav­i­ga­tor.me­di­ade­vices

.ge­tuser­me­dia({ au­dio: true, video: false

})

.then(got­stream)

14. Fire up the ‘rtcpeer­con­nec­tion’

As dis­cussed in the in­tro­duc­tion, the ‘Rtcpeer­con­nec­tion’ class is not lim­ited to trans­fer­ring tex­tual data. De­vel­op­ers can add me­dia tracks to an es­tab­lished con­nec­tion; the un­der­ly­ing im­ple­men­ta­tion takes care of mar­shalling and en­cod­ing. const servers = null; pc1 = new Rtcpeer­con­nec­tion(servers); pc2 = new Rtcpeer­con­nec­tion(servers); lo­cal­stream.get­tracks().fore­ach(track => pc1.ad­dtrack(track, lo­cal­stream));

pc1.cre­ate­of­fer(of­fer­op­tions). then(on­cre­ate­of­fer­suc­cess, on­cre­ate­ses­sion­desc rip­tion­error);

15. ac­cept the in­com­ing track

On the side of the data sink, im­ple­ment a stream re­ceiver. The op­er­at­ing sys­tem pro­vides it with a ref­er­ence ob­ject, which con­tains one or more data streams. We pass the first one into the ‘re­mote­v­ideo’ el­e­ment, while set­ting ‘sr­cob­ject’ to null be­fore writ­ing the new con­tent avoids some bugs in Edge. func­tion gotremotestream(e) { re­mote­v­ideo.sr­cob­ject = null; re­mote­v­ideo.sr­cob­ject = e.streams[0]; }

16. Pro­mote con­nec­tion i

If an au­dio con­nec­tion is es­tab­lished, it can be pro­moted to also carry video in­for­ma­tion. This is ac­com­plished by in­vok­ing ‘ge­tuser­me­dia’ once again. The re­sult­ing stream

can then be added to the ex­ist­ing me­dia ob­ject: nav­i­ga­tor.me­di­ade­vices .ge­tuser­me­dia({video: true})

.then(stream => { const video­tracks = stream. getvideo­tracks(); lo­cal­stream.ad­dtrack(video­tracks[0]); . . .

17. Pro­mote con­nec­tion ii

In the next act, the new stream must also be reg­is­tered with the stream that trans­fers con­tent be­tween the two ends. This should also be ac­com­plished in a two-step process, with writ­ing null into ‘sr­cob­ject’ as a bug fix. lo­calvideo.sr­cob­ject = null; lo­calvideo.sr­cob­ject = lo­cal­stream; pc1.ad­dtrack(video­tracks[0], lo­cal­stream);

return pc1.cre­ate­of­fer(); })

18. Clean up when done

Han­dling real-time me­dia is com­pu­ta­tion­ally in­ten­sive. Due to this, stop­ping the process as soon as it is no longer needed is ben­e­fi­cial. The code ac­com­pa­ny­ing this step out­lines the process of clos­ing a con­nec­tion.

const video­tracks = lo­cal­stream. getvideo­tracks(); video­tracks.fore­ach(video­track => { video­track.stop(); lo­cal­stream.re­move­track(video­track);

}); lo­calvideo.sr­cob­ject = null; lo­calvideo.sr­cob­ject = lo­cal­stream;

19. Work around im­ple­men­ta­tion de­tails

In the­ory, WEBRTC should work across all browsers. Sadly, prac­ti­cal browsers tend to bring along idio­syn­cra­sies of their own. https://github.com/webrtc/adapter pro­vides a wrap­per li­brary – in­clude it in a web­site or a NPM pack­age to work around nui­sances.

20. How to han­dle con­nec­tion es­tab­lish­ment

Good com­mu­ni­ca­tion clients dis­tin­guish them­selves from their lesser com­pe­ti­tion by the use of smart con­nec­tion es­tab­lish­ment ser­vices. For a web-based sys­tem, we would rec­om­mend the use of a cen­tral server based on WEBRTC. Clients con­nect to it and ask for peer in­for­ma­tion, which is then sim­ply used for the ac­tual call setup.

21. Col­lect con­nec­tion pro­to­col data

Tra­di­tional net­work snif­fers have not adapted to WEBRTC yet. Browser ven­dors com­bat this trend by adding de­bug­ger and sta­tis­tics util­i­ties to the trap-door pages of their browsers. Sim­ply open them and feast your eyes on the di­a­grams. Fi­nally, be pre­pared to switch browsers when hunt­ing down dif­fi­cult is­sues – not all an­a­lyt­ics suites are ef­fec­tive. chrome://webrtc-in­ter­nals opera://webrtc-in­ter­nals about:webrtc

7

3

4

16

20

8

19

21

Newspapers in English

Newspapers from UK

© PressReader. All rights reserved.