- RTC-Web - example.com

Re: [RTW] Rate control and codec adaption (Re: [dispatch] The charter formerly know as RTC-WEB take 3)
by Matthew Kaufman 27 Jan '11

27 Jan '11

Agreed, but for the purpose of this discussion I believe that rate control of some sort should also be a MUST. Web browsers are extremely prevalent, and we hope that RTC use in browsers will be high, and so it would be good for the Internet for browsers to have sending rate control. Note that this is at the protocol level... so send rate must be controlled whether the codec can have its rate adjusted downward so as to not require the protocol level to enforce or not. For interoperability, it is also required that the feedback mechanism from one end to the other be standardized, even if the way that feedback is used to control send rate and/or codec selection or codec rate selection is proprietary and/or extensions to the feedback are also sent for endpoints that understand the (possibly proprietary) extension(s). Matthew Kaufman On 1/25/2011 7:38 AM, Rosenberg, Jonathan wrote: > > It’s a proprietary algorithm of our own design, supported by some > protocols which exchange feedback in real-time between endpoints. > We’re constantly tweaking it based on user feedback and technical > statistics we collect. > > Indeed – as many folks are aware, rate adaptation has always been an > area of innovation and differentiation. RTP has provided the tools for > feedback but has allowed implementations to do whatever they want. I > think it is important that this continues to be the case in the web > world – that folks designing RTC applications can innovate and define > their own versions of these algorithms. > > Thanks, > > Jonathan R. > > Jonathan D. Rosenberg, Ph.D. SkypeID: jdrosen > > Chief Technology Strategist Mobile: +1 (732) 766-2496 > > Skype SkypeIn: +1 (408) 465-0361 > > jdrosen(a)skype.net http://www.skype.com > > jdrosen(a)jdrosen.net http://www.jdrosen.net > >

3 2

ConnectionPeer API experiments
by Stefan Håkansson LK 26 Jan '11

26 Jan '11

Hi all (only rtc-web as it's API related), as promised some time ago, here are some results from implementing and tesing parts of the ConnectionPeer API. (FYI We just submitted a very similar mail on the WhatWG list). We have done some experimentation with the ConnectionPeer API. We have an initial implementation of a subset of the API, using ICE (RFC 5245) for the peer-to-peer handshaking. More information about our work so far can be found here: https://labs.ericsson.com/developer-community/blog/beyond-html5-peer-peer-c… The short story is that ConnectionPeer seem like a good starting point for API discussion. If you want to get into details, continue reading! We have bumped into some details that we'd like to discuss here right away. The following is our mix of proposals and questions. 1. We propose adding a readyState attribute, to decouple the onconnect() callback from any observers (such as the UI). const unsigned short CONNECTING = 0; const unsigned short CONNECTED = 1; const unsigned short CLOSED = 2; readonly attribute unsigned short readyState; 2. We propose replacing the onstream event with custom events of type RemoteStreamEvent, to distinguish between adding and removing streams. attribute Function onstreamadded; // RemoteStreamEvent attribute Function onstreamremoved; // RemoteStreamEvent ... interface RemoteStreamEvent : Event { readonly attribute Stream stream; }; The 'stream' attribute indicates which stream was added/removed. 3. We propose renaming addRemoteConfiguration to setRemoteConfiguration. Our understanding of the ConnectionPeer is that it provides a single-point-to-single-point connection; hence, only one remote peer configuration is to be set, rather than many to be added. void setRemoteConfiguration(in DOMString configuration, in optional DOMString remoteOrigin); 4. We propose swapping the ConnectionPeerConfigurationCallback callback parameters. The current example seems to use only one (the second one). Swapping them allows clients that care about 'server' to do so, and clients that ignore it (such as the current example) to do so too. [Callback=FunctionOnly, NoInterfaceObject] interface ConnectionPeerConfigurationCallback { void handleEvent(in DOMString configuration, in ConnectionPeer server); }; 5. Should a size limit to text messages be specified? Text messages with UDP-like behavior (unimportant=true) can't really be reliably split into several UDP packets. For such long chunks of data, file transfer seems like a better option anyway. In summary, then, our proposal for a revised ConnectionPeer looks as follows: [Constructor(in DOMString serverConfiguration)] interface ConnectionPeer { void sendText(in DOMString text, in optional boolean unimportant); // if second arg is true, then use unreliable low-latency transport (UDP-like), otherwise guarantee delivery (TCP-like) attribute Function ontext; // receiving void sendBitmap(in HTMLImageElement image); attribute Function onbitmap; // receiving void sendFile(in File file); attribute Function onfile; // receiving void addStream(in Stream stream, in optional DOMString metadata, in optional String mediaFormat); //Start stream, add meta data and encoding parameters void removeStream(in Stream stream); readonly attribute Stream[] localStreams; readonly attribute Stream[] remoteStreams; attribute Function onstreamadded; // receiving new stream attribute Function onstreamremoved; // stream not received any more void getLocalConfiguration(in ConnectionPeerConfigurationCallback callback); // maybe this should be in the constructor, or be an event void setRemoteConfiguration(in DOMString configuration, in optional DOMString remoteOrigin); // remote origin is assumed to be same-origin if not specified. If specified, has to match remote origin (checked in handshake). Should support leading "*." to mean "any subdomain of". void close(); // disconnects and stops listening attribute Function onconnect; attribute Function onerror; attribute Function ondisconnect; const unsigned short CONNECTING = 0; const unsigned short CONNECTED = 1; const unsigned short CLOSED = 2; readonly attribute unsigned short readyState; }; interface RemoteStreamEvent : Event { readonly attribute Stream stream; }; [Callback=FunctionOnly, NoInterfaceObject] interface ConnectionPeerConfigurationCallback { void handleEvent(in DOMString configuration, in ConnectionPeer server); }; In addition to the above there is a need to add support for identifying streams (so that the receiving end can use the right element for rendering) and for negotiating and influencing the media format. Those parts we're still looking into. But all in all we think ConnectionPeer is a very good starting point for an API. Br, Stefan -- Patrik Persson, Ericsson Research mailto:patrik.j.persson@ericsson.com

1 0

Re: [RTW] Rate control and codec adaption (Re: [dispatch] The charter formerly know as RTC-WEB take 3)
by Saverio Mascolo 25 Jan '11

25 Jan '11

> > On Tue, Jan 25, 2011 at 6:36 PM, Soo-Hyun Choi<soohyun.choi(a)cl.cam.ac.uk> > wrote: > >> So, your first and second paper addressed issues around these problem, >> respectively: >> >> (1) limitations of the TCP equation used by TFRC >> > we mathematically derive the dynamic rate-based equation corresponding to tcp > (2) limitations of being rate-based > > we show how is possible to implement rate-based sm > > This was one of my motivations to develop TFWC (a window-based CC, but > still useful for the real-time video streaming applications) - > http://tfwc.hackerslab.eu/. > > Any comments? > > Cheers, > Soo-Hyun > > > > On 1/26/11 2:21 AM, Saverio Mascolo wrote: > > to complete my previous msg, here it is a paper > > > > http://c3lab.poliba.it/images/8/88/ACM04.pdf > > > > that describes "a rate based control" that is the**_*real *_rate-based > > counterpart of TCP window based control. > > > > It is real because it is a dynamic equation and not a steady state > > equation as TFRC. As a consequence, it responds in real-time respond to > > network conditions, as it does TCP, but in real-time. > > > > As any algorithm that computes a rate, it should be implemented using a > > "rate mismatch controller" > > > > http://c3lab.poliba.it/images/d/d9/Icnp09.pdf > > > > Any comment is very welcome! > > > > Thanks for the attention and best regards, > > > > saverio > > > > > > On Tue, Jan 25, 2011 at 4:49 PM, Rosenberg, Jonathan > > <jonathan.rosenberg(a)skype.net <mailto:jonathan.rosenberg@skype.net>> > wrote: > > > > No debate here. The model I like is that there is something built-in > > to the browser (say, TFRC or some variant), but the hooks are > > available to allow an application to customize it. > > > > > > > > -Jonathan R. > > > > > > > > Jonathan D. Rosenberg, Ph.D. SkypeID: jdrosen > > > > Chief Technology Strategist Mobile: +1 (732) 766-2496 > > > > Skype SkypeIn: +1 (408) 465-0361 > > > > jdrosen(a)skype.net <mailto:jdrosen@skype.net> > > http://www.skype.com > > > > jdrosen(a)jdrosen.net > > <mailto:jdrosen@jdrosen.net> > > http://www.jdrosen.net > > > > > > > > *From:*Matthew Kaufman [mailto:matthew.kaufman@skype.net > > <mailto:matthew.kaufman@skype.net>] > > *Sent:* Tuesday, January 25, 2011 7:47 AM > > *To:* Rosenberg, Jonathan > > *Cc:* 'Saverio Mascolo'; 'Stefan Håkansson LK'; 'Cullen Jennings'; > > tom_harper(a)logitech.com <mailto:tom_harper@logitech.com>; 'Justin > > Uberti'; 'Harald Alvestrand'; rtc-web(a)alvestrand.no > > <mailto:rtc-web@alvestrand.no>; 'Peter Musgrave' > > > > *Subject:* Re: [RTW] Rate control and codec adaption (Re: [dispatch] > > The charter formerly know as RTC-WEB take 3) > > > > > > > > Agreed, but for the purpose of this discussion I believe that rate > > control of some sort should also be a MUST. > > > > > > > > Web browsers are extremely prevalent, and we hope that RTC use in > > browsers will be high, and so it would be good for the Internet for > > browsers to have sending rate control. Note that this is at the > > protocol level... so send rate must be controlled whether the codec > > can have its rate adjusted downward so as to not require the > > protocol level to enforce or not. > > > > For interoperability, it is also required that the feedback > > mechanism from one end to the other be standardized, even if the way > > that feedback is used to control send rate and/or codec selection or > > codec rate selection is proprietary and/or extensions to the > > feedback are also sent for endpoints that understand the (possibly > > proprietary) extension(s). > > > > Matthew Kaufman > > > > On 1/25/2011 7:38 AM, Rosenberg, Jonathan wrote: > > > > It’s a proprietary algorithm of our own design, supported by some > > protocols which exchange feedback in real-time between endpoints. > > We’re constantly tweaking it based on user feedback and technical > > statistics we collect. > > > > > > > > Indeed – as many folks are aware, rate adaptation has always been an > > area of innovation and differentiation. RTP has provided the tools > > for feedback but has allowed implementations to do whatever they > > want. I think it is important that this continues to be the case in > > the web world – that folks designing RTC applications can innovate > > and define their own versions of these algorithms. > > > > > > > > Thanks, > > > > Jonathan R. > > > > > > > > Jonathan D. Rosenberg, Ph.D. SkypeID: jdrosen > > > > Chief Technology Strategist Mobile: +1 (732) 766-2496 > > > > Skype SkypeIn: +1 (408) 465-0361 > > > > jdrosen(a)skype.net <mailto:jdrosen@skype.net> > > http://www.skype.com > > > > jdrosen(a)jdrosen.net > > <mailto:jdrosen@jdrosen.net> > > http://www.jdrosen.net > > > > > > > > > > > > > > > > > > -- > > Prof. Saverio Mascolo > > Dipartimento di Elettrotecnica ed Elettronica > > Politecnico di Bari > > Via Orabona 4, 70125 Bari Italy > > Tel. +39 080 5963621 > > Fax. +39 080 5963410 > > email:mascolo@poliba.it <email%3Amascolo(a)poliba.it> <mailto: > email%3Amascolo(a)poliba.it <email%253Amascolo(a)poliba.it>> > > > > http://c3lab.poliba.it > > > > > > ================================= > > This message may contain confidential and/or legally privileged > > information. > > If you are not the intended recipient of the message, please destroy > it. > > Any unauthorized dissemination, distribution, or copying of the material > in > > this message, and any attachments to the message, is strictly forbidden. > > > > > > > > _______________________________________________ > > RTC-Web mailing list > > RTC-Web(a)alvestrand.no > > http://www.alvestrand.no/mailman/listinfo/rtc-web > -- Prof. Saverio Mascolo Dipartimento di Elettrotecnica ed Elettronica Politecnico di Bari Via Orabona 4, 70125 Bari Italy Tel. +39 080 5963621 Fax. +39 080 5963410 email:mascolo@poliba.it <email%3Amascolo(a)poliba.it> http://c3lab.poliba.it ================================= This message may contain confidential and/or legally privileged information. If you are not the intended recipient of the message, please destroy it. Any unauthorized dissemination, distribution, or copying of the material in this message, and any attachments to the message, is strictly forbidden.

1 0

Re: [RTW] Rate control and codec adaption (Re: [dispatch] The charter formerly know as RTC-WEB take 3)
by Saverio Mascolo 25 Jan '11

25 Jan '11

to complete my previous msg, here it is a paper http://c3lab.poliba.it/images/8/88/ACM04.pdf that describes "a rate based control" that is the* **real *rate-based counterpart of TCP window based control. It is real because it is a dynamic equation and not a steady state equation as TFRC. As a consequence, it responds in real-time respond to network conditions, as it does TCP, but in real-time. As any algorithm that computes a rate, it should be implemented using a "rate mismatch controller" http://c3lab.poliba.it/images/d/d9/Icnp09.pdf Any comment is very welcome! Thanks for the attention and best regards, saverio On Tue, Jan 25, 2011 at 4:49 PM, Rosenberg, Jonathan < jonathan.rosenberg(a)skype.net> wrote: > No debate here. The model I like is that there is something built-in to the > browser (say, TFRC or some variant), but the hooks are available to allow an > application to customize it. > > > > -Jonathan R. > > > > Jonathan D. Rosenberg, Ph.D. SkypeID: jdrosen > > Chief Technology Strategist Mobile: +1 (732) 766-2496 > > Skype SkypeIn: +1 (408) 465-0361 > > jdrosen(a)skype.net http://www.skype.com > > jdrosen(a)jdrosen.net http://www.jdrosen.net > > > > *From:* Matthew Kaufman [mailto:matthew.kaufman@skype.net] > *Sent:* Tuesday, January 25, 2011 7:47 AM > *To:* Rosenberg, Jonathan > *Cc:* 'Saverio Mascolo'; 'Stefan Håkansson LK'; 'Cullen Jennings'; > tom_harper(a)logitech.com; 'Justin Uberti'; 'Harald Alvestrand'; > rtc-web(a)alvestrand.no; 'Peter Musgrave' > > *Subject:* Re: [RTW] Rate control and codec adaption (Re: [dispatch] The > charter formerly know as RTC-WEB take 3) > > > > Agreed, but for the purpose of this discussion I believe that rate control > of some sort should also be a MUST. > > > Web browsers are extremely prevalent, and we hope that RTC use in browsers > will be high, and so it would be good for the Internet for browsers to have > sending rate control. Note that this is at the protocol level... so send > rate must be controlled whether the codec can have its rate adjusted > downward so as to not require the protocol level to enforce or not. > > For interoperability, it is also required that the feedback mechanism from > one end to the other be standardized, even if the way that feedback is used > to control send rate and/or codec selection or codec rate selection is > proprietary and/or extensions to the feedback are also sent for endpoints > that understand the (possibly proprietary) extension(s). > > Matthew Kaufman > > On 1/25/2011 7:38 AM, Rosenberg, Jonathan wrote: > > It’s a proprietary algorithm of our own design, supported by some protocols > which exchange feedback in real-time between endpoints. We’re constantly > tweaking it based on user feedback and technical statistics we collect. > > > > Indeed – as many folks are aware, rate adaptation has always been an area > of innovation and differentiation. RTP has provided the tools for feedback > but has allowed implementations to do whatever they want. I think it is > important that this continues to be the case in the web world – that folks > designing RTC applications can innovate and define their own versions of > these algorithms. > > > > Thanks, > > Jonathan R. > > > > Jonathan D. Rosenberg, Ph.D. SkypeID: jdrosen > > Chief Technology Strategist Mobile: +1 (732) 766-2496 > > Skype SkypeIn: +1 (408) 465-0361 > > jdrosen(a)skype.net http://www.skype.com > > jdrosen(a)jdrosen.net http://www.jdrosen.net > > > > > -- Prof. Saverio Mascolo Dipartimento di Elettrotecnica ed Elettronica Politecnico di Bari Via Orabona 4, 70125 Bari Italy Tel. +39 080 5963621 Fax. +39 080 5963410 email:mascolo@poliba.it <email%3Amascolo(a)poliba.it> http://c3lab.poliba.it ================================= This message may contain confidential and/or legally privileged information. If you are not the intended recipient of the message, please destroy it. Any unauthorized dissemination, distribution, or copying of the material in this message, and any attachments to the message, is strictly forbidden.

2 1

Re: [RTW] Rate control and codec adaption (Re: [dispatch] The charter formerly know as RTC-WEB take 3)
by Saverio Mascolo 25 Jan '11

25 Jan '11

regarding rate control, "how to enforce rate control is an issue" since packet scheduling is affected by time delay due to the fact that the OS is shared. we did this work at ICNP09: http://c3lab.poliba.it/images/d/d9/Icnp09.pdf regards, saverio On Tue, Jan 25, 2011 at 4:49 PM, Rosenberg, Jonathan < jonathan.rosenberg(a)skype.net> wrote: > No debate here. The model I like is that there is something built-in to the > browser (say, TFRC or some variant), but the hooks are available to allow an > application to customize it. > > > > -Jonathan R. > > > > Jonathan D. Rosenberg, Ph.D. SkypeID: jdrosen > > Chief Technology Strategist Mobile: +1 (732) 766-2496 > > Skype SkypeIn: +1 (408) 465-0361 > > jdrosen(a)skype.net http://www.skype.com > > jdrosen(a)jdrosen.net http://www.jdrosen.net > > > > *From:* Matthew Kaufman [mailto:matthew.kaufman@skype.net] > *Sent:* Tuesday, January 25, 2011 7:47 AM > *To:* Rosenberg, Jonathan > *Cc:* 'Saverio Mascolo'; 'Stefan Håkansson LK'; 'Cullen Jennings'; > tom_harper(a)logitech.com; 'Justin Uberti'; 'Harald Alvestrand'; > rtc-web(a)alvestrand.no; 'Peter Musgrave' > *Subject:* Re: [RTW] Rate control and codec adaption (Re: [dispatch] The > charter formerly know as RTC-WEB take 3) > > > > Agreed, but for the purpose of this discussion I believe that rate control > of some sort should also be a MUST. > > > Web browsers are extremely prevalent, and we hope that RTC use in browsers > will be high, and so it would be good for the Internet for browsers to have > sending rate control. Note that this is at the protocol level... so send > rate must be controlled whether the codec can have its rate adjusted > downward so as to not require the protocol level to enforce or not. > > For interoperability, it is also required that the feedback mechanism from > one end to the other be standardized, even if the way that feedback is used > to control send rate and/or codec selection or codec rate selection is > proprietary and/or extensions to the feedback are also sent for endpoints > that understand the (possibly proprietary) extension(s). > > Matthew Kaufman > > On 1/25/2011 7:38 AM, Rosenberg, Jonathan wrote: > > It’s a proprietary algorithm of our own design, supported by some protocols > which exchange feedback in real-time between endpoints. We’re constantly > tweaking it based on user feedback and technical statistics we collect. > > > > Indeed – as many folks are aware, rate adaptation has always been an area > of innovation and differentiation. RTP has provided the tools for feedback > but has allowed implementations to do whatever they want. I think it is > important that this continues to be the case in the web world – that folks > designing RTC applications can innovate and define their own versions of > these algorithms. > > > > Thanks, > > Jonathan R. > > > > Jonathan D. Rosenberg, Ph.D. SkypeID: jdrosen > > Chief Technology Strategist Mobile: +1 (732) 766-2496 > > Skype SkypeIn: +1 (408) 465-0361 > > jdrosen(a)skype.net http://www.skype.com > > jdrosen(a)jdrosen.net http://www.jdrosen.net > > > > > > _______________________________________________ > RTC-Web mailing list > RTC-Web(a)alvestrand.no > http://www.alvestrand.no/mailman/listinfo/rtc-web > > -- Prof. Saverio Mascolo Dipartimento di Elettrotecnica ed Elettronica Politecnico di Bari Via Orabona 4, 70125 Bari Italy Tel. +39 080 5963621 Fax. +39 080 5963410 email:mascolo@poliba.it <email%3Amascolo(a)poliba.it> http://c3lab.poliba.it ================================= This message may contain confidential and/or legally privileged information. If you are not the intended recipient of the message, please destroy it. Any unauthorized dissemination, distribution, or copying of the material in this message, and any attachments to the message, is strictly forbidden.

1 0

Re: [RTW] Rate control and codec adaption (Re: [dispatch] The charter formerly know as RTC-WEB take 3)
by Ted Hardie 25 Jan '11

25 Jan '11

Hi Jonathan, Just to clarify, are you expecting the available hooks to allow the kind of tweaking you described up thread or to be a simple selection among a known set (assuming a proprietary method may be among the known set)? I'm a bit concerned if the browser itself is setting rate control policy at a granular level on the fly. The chance that it chooses to be unfriendly to other traffic or that its choices end up being unfriendly seem pretty high. regards, Ted On Tue, Jan 25, 2011 at 7:49 AM, Rosenberg, Jonathan <jonathan.rosenberg(a)skype.net> wrote: > No debate here. The model I like is that there is something built-in to the > browser (say, TFRC or some variant), but the hooks are available to allow an > application to customize it. > > > > -Jonathan R. > > > > Jonathan D. Rosenberg, Ph.D. SkypeID: jdrosen > > Chief Technology Strategist Mobile: +1 (732) 766-2496 > > Skype SkypeIn: +1 (408) 465-0361 > > jdrosen(a)skype.net http://www.skype.com > > jdrosen(a)jdrosen.net http://www.jdrosen.net > > > > From: Matthew Kaufman [mailto:matthew.kaufman@skype.net] > Sent: Tuesday, January 25, 2011 7:47 AM > To: Rosenberg, Jonathan > Cc: 'Saverio Mascolo'; 'Stefan Håkansson LK'; 'Cullen Jennings'; > tom_harper(a)logitech.com; 'Justin Uberti'; 'Harald Alvestrand'; > rtc-web(a)alvestrand.no; 'Peter Musgrave' > Subject: Re: [RTW] Rate control and codec adaption (Re: [dispatch] The > charter formerly know as RTC-WEB take 3) > > > > Agreed, but for the purpose of this discussion I believe that rate control > of some sort should also be a MUST. > > Web browsers are extremely prevalent, and we hope that RTC use in browsers > will be high, and so it would be good for the Internet for browsers to have > sending rate control. Note that this is at the protocol level... so send > rate must be controlled whether the codec can have its rate adjusted > downward so as to not require the protocol level to enforce or not. > > For interoperability, it is also required that the feedback mechanism from > one end to the other be standardized, even if the way that feedback is used > to control send rate and/or codec selection or codec rate selection is > proprietary and/or extensions to the feedback are also sent for endpoints > that understand the (possibly proprietary) extension(s). > > Matthew Kaufman > > On 1/25/2011 7:38 AM, Rosenberg, Jonathan wrote: > > It’s a proprietary algorithm of our own design, supported by some protocols > which exchange feedback in real-time between endpoints. We’re constantly > tweaking it based on user feedback and technical statistics we collect. > > > > Indeed – as many folks are aware, rate adaptation has always been an area of > innovation and differentiation. RTP has provided the tools for feedback but > has allowed implementations to do whatever they want. I think it is > important that this continues to be the case in the web world – that folks > designing RTC applications can innovate and define their own versions of > these algorithms. > > > > Thanks, > > Jonathan R. > > > > Jonathan D. Rosenberg, Ph.D. SkypeID: jdrosen > > Chief Technology Strategist Mobile: +1 (732) 766-2496 > > Skype SkypeIn: +1 (408) 465-0361 > > jdrosen(a)skype.net http://www.skype.com > > jdrosen(a)jdrosen.net http://www.jdrosen.net > > > > > > _______________________________________________ > RTC-Web mailing list > RTC-Web(a)alvestrand.no > http://www.alvestrand.no/mailman/listinfo/rtc-web > >

1 0

RTC-Web Use cases
by Stefan Håkansson LK 20 Jan '11

20 Jan '11

Just to start the discussion, I've put together the simplest use case you can imagine (with an additional error case). Some requirements are drawn from it. The intention is to add more use cases, but before spending any more energy I though it would be good to display this to the community to start the discussion and to get feedback on whether this is a workable model or not. The requirements are split in three categories: 1. User/privacy: the intention is to list what an end user can expect in terms of what an web application can do without user consent and so on 2. API dimension: lists what must be accessible by JS and signalled to JS via APIs 3. Functional dimension: lists functionality that must be supported by the browser (in combination with the underlying operating system, drivers and HW) even though it is not visible from the API. This list is rather short since most of the functionality is visible in the API I can't say that I am extremely satisfied with this division. Perhaps we could come up with something better. Also, some of the requirements in the User/privacy and Functional dimensions are not drawn (e.g. ask for user consent) from this use case per se, but are instead generic requirements. Perhaps they should be handled separately? The term "stream" is used. The thinking (inspired by the WhatWG HTML5 draft <http://www.whatwg.org/specs/web-apps/current-work/multipage/commands.html#d…>) is that some kind of Stream API should represent a stream. A stream in turn is what is generated by a mic or a cam, but also what would be sent between browsers. Obviously these streams would have different formats (the first being raw audio/video samples, the second encoded audio/video in RTP packets). But the Steam API is what the JS code uses to manage the stream. OK, to the use case: ==================== 1. Simple Video Chat ==================== A simple video chat service has been developed. In the service the users are logged on to the same chat web server. The web server publishes information about user login status, pushing updates to the web apps in the browsers. By clicking on an online peer user name, a 1-1 video chat session between the two browsers is initiated. The invited peer is presented with a choice of joining or rejecting the session. The web author developing the application has decided to display a self-view as well as the video from the remote side in rather small windows, but the user can change the display size during the session. The application also supports if a participant (for a longer or shorter time) would like to stop sending audio (but keep video) or video (keep audio) to the other peer ("mute"). Any of the two participants can at any time end the chat by clicking a button. In this specific case two users are using lap-tops in their respective homes. They are connected to the public Internet with a desktop browser using WiFi behind NATs. One of the users has an ADSL connection to the home, and the other fiber access. Most of the time headsets are used, but not always. 1.1 Requirements ================ 1.1.1 User/privacy dimension ---------------------------- -------------------------------------------------------------------------------- !Requirement. The user must: !Comment ! -------------------------------------------------------------------------------- !give explicit consent before a device ! ! !can be used to capture audio or video ! ! -------------------------------------------------------------------------------- !be able to in an intuitive way revoke ! ! !and change capturing permissions ! ! -------------------------------------------------------------------------------- !be able to easily understand that audio! ! !or video is being captured ! ! -------------------------------------------------------------------------------- !be informed about that an invitation to! ! !a peer video chat session has been ! ! !received ! ! -------------------------------------------------------------------------------- !be able to accept or reject an ! ! !invitation to a peer video chat session! ! -------------------------------------------------------------------------------- !be able to stop a media stream from ! ! !being transmitted ! ! -------------------------------------------------------------------------------- 1.1.2 API dimension ------------------- -------------------------------------------------------------------------------- !Requirement. !Comment ! -------------------------------------------------------------------------------- !It must be possible to update presence !Event. Out of scope for RTC-Web? ! !info from web server and make web ! ! !application aware. ! ! -------------------------------------------------------------------------------- !It must be possible to propagate !Out of scope for RTC-Web? ! !intention to start a chat session from ! ! !one web app (via server), and make ! ! !receiving web application aware. ! ! !Likewise, the receiving web application! ! !must be able to propagate its accept/ ! ! !reject to the initiating web app. ! ! -------------------------------------------------------------------------------- !The web application must be able to use!Provided the user has given consent. ! !cams and mics as input devices. ! ! -------------------------------------------------------------------------------- !The web application must be able to !I.e. how they are routed. To e.g. both! !control how streams generated by input !a self-view and a peer ! !devices are used ! ! -------------------------------------------------------------------------------- !The web application must be able to !Use the audio and video elements? ! !control how streams are rendered and ! ! !displayed ! ! -------------------------------------------------------------------------------- !The web application must be able to ! ! !initiate sending of streams to a peer ! ! -------------------------------------------------------------------------------- !The web application must be able to !If the video is going to be displayed ! !define the media format to be used for ! in a large window, use higher bit- ! !the streams sent to a peer. !rate/resolution. Should media settings! ! !be allowed to be changed during a ! ! !session (at e.g. window resize)? ! -------------------------------------------------------------------------------- !The web application must be made aware !Event. ! !of whether set up of stream sending was! ! !successful or not ! ! -------------------------------------------------------------------------------- !The web application must be made aware !Event. To be able to (with or without ! !when a stream from a peer is received !user involvement) accept or reject, ! ! !and to connect the stream to the right! ! !display/rendering element. ! -------------------------------------------------------------------------------- !The web application must be made aware !Event. ! !of when a stream from a peer is not ! ! !received any more ! ! -------------------------------------------------------------------------------- !The web application in a session must ! ! !be able to terminate all incoming and ! ! !outgoing streams ! ! -------------------------------------------------------------------------------- 1.1.3 Functional dimension -------------------------- -------------------------------------------------------------------------------- !Requirement. !Comment ! -------------------------------------------------------------------------------- !The browser must be able to have an !Out of scope for RTC-Web? Use WS or ! !always on connection with the web !S-SE? ! !server to be able to receive presence ! ! !updates and chat initiations ! ! -------------------------------------------------------------------------------- !The browser must be able to use mics ! ! !and cams as input devices ! ! -------------------------------------------------------------------------------- !The browser must be able to send ! ! !streams (includes the associated ! ! !processing like coding, framing, etc.) ! ! !to a peer in presence of NATs. ! ! -------------------------------------------------------------------------------- !The browser must be able to receive ! ! !streams (associated processing) from ! ! !peers and render them ! ! -------------------------------------------------------------------------------- !Streams being transmitted must be !Do not starve other traffic (e.g. on ! !subject to rate control !ADSL link) ! -------------------------------------------------------------------------------- !When there is both incoming and !Headsets not always used ! !outgoing audio streams, echo ! ! !cancellation must be provided to avoid ! ! !disturbing echo during conversation ! ! -------------------------------------------------------------------------------- !Synchronization between audio and video! ! !must be supported ! ! -------------------------------------------------------------------------------- 2 Simple video chat with link loss ================================== The use case is identical to the previous one, but in this case the ADSL link is unreliable and goes down from time to time. 2.1 Requirements ---------------- Only the delta is documented (the other reqs remain). 2.1.1 User/privacy dimension ---------------------------- -------------------------------------------------------------------------------- !Requirement. !Comment ! -------------------------------------------------------------------------------- !The user must be informed that the ! ! !communication has ceased ! ! -------------------------------------------------------------------------------- 2.1.2 API dimension ------------------- -------------------------------------------------------------------------------- !Requirement. !Comment ! -------------------------------------------------------------------------------- !The web application must be made aware !To be able to inform user and take ! !of that the connection with the server !action. Out of scope for RTC-Web? ! !has been dropped ! ! -------------------------------------------------------------------------------- !The web application must be made aware !To be able to inform user and take ! !of when streams from a peer are no !action (one of the peers still has ! !longer received !connection with the server) ! -------------------------------------------------------------------------------- 2.1.3 Functional dimension -------------------------- -------------------------------------------------------------------------------- !Requirement. !Comment ! -------------------------------------------------------------------------------- !The browser must detect when no streams! ! !are received from a peer ! ! -------------------------------------------------------------------------------- //Stefan

2 1

Re: [RTW] [dispatch] Charter proposal: The activity hitherto known as "RTC-WEB at IETF"
by Stefan Håkansson LK 19 Jan '11

19 Jan '11

Markus Isomäki wrote: >Tom Taylor wrote: >> >>My personal advice is to leave the mandatory codec issue alone. You'll >>waste a huge amount of energy without getting satisfactory results. >>Notice that all your references were to specifications generated by >>other bodies. >> > >I agree that at least for the time being it is more fruitful to focus >the energy elsewhere. There is plenty of useful work that can be done >about media transport (the datagram service and the potential bytestream >) and the associated APIs, and I suggest we focus on that. We can try our >luck with the codec thing later on. I agree. Codec discussions seem to go on forever, and we could spend our energy wiser. Stefan PS Sorry for answering late, but I did not follow dispatch. I thought all related messages would go on rtc-web as well. So those of you who do not follow dispatch: perhaps you should look into the dispatch archive.

6 7

Re: [RTW] [dispatch] Charter proposal: The activity hitherto knownas "RTC-WEB at IETF"
by Marshall Eubanks 17 Jan '11

17 Jan '11

On Jan 17, 2011, at 12:49 PM, Ali C. Begen (abegen) wrote: > > >> -----Original Message----- >> From: dispatch-bounces(a)ietf.org [mailto:dispatch-bounces@ietf.org] On Behalf Of Marshall Eubanks >> Sent: Monday, January 17, 2011 12:23 PM >> To: Bernard Aboba >> Cc: dispatch(a)ietf.org; harald(a)alvestrand.no; rtc-web(a)alvestrand.no; tom.taylo(a)huawei.com >> Subject: Re: [dispatch] [RTW] Charter proposal: The activity hitherto knownas "RTC-WEB at IETF" >> >> >> On Jan 17, 2011, at 10:59 AM, Bernard Aboba wrote: >> >>> +1. >>> >>> One place where we could "spend our energy wiser" might be on enabling interoperability >>> of HTTP transported realtime media. Although peer-to-peer traffic is more desirable when >>> possible, "HTTP fallback" is in practice required a significant fraction of the time, due to the >>> prevalence of highly restrictive firewalls. >> >> I would agree, and that raises the issue of the "wrapper" for HTTP streaming. Note that Apple uses MPEG-2 TS for the >> wrapper for its live http video streaming. >> >> ( Each media file MUST >> be formatted as an MPEG-2 Transport Stream, an MPEG-2 Program Stream, >> or an MPEG-2 audio elementary stream - http://tools.ietf.org/html/draft-pantos-http-live-streaming-01 ) >> >> While this is certainly standards based, I do not think it matches or interoperates with anyone else's HTTP streaming. And, of >> course, this is an I-D still. Flash also does http streaming, but I believe it uses its own, proprietary, wrapper. >> >> So, is specifying a media transport protocol for http streaming in scope ? > > What you are referring to is not a transport protocol, rather I guess we could call it encapsulation format. > > And for different vendors' systems to interoperate (for http streaming), there needs to be more than just supporting the same encapsulation format. FWIW, many of the methods do use mp4-based fragments unlike Apple (of course with or without their own special boxes). There is already work undertaken in MPEG in this area. IMO, doing the same work but a few years late is not a much wise move. > I would agree (and I wasn't necessarily calling for protocol development, I would prefer rather protocol selection here), but I don't see a true standard emerging in the wild as yet and so this effort will probably have to spend some time making a selection. Regards Marshall > -acbegen > >> Regards >> Marshall >> >> >>> >>>> From: stefan.lk.hakansson(a)ericsson.com >>>> To: tom.taylo(a)huawei.com; harald(a)alvestrand.no; Markus.Isomaki(a)nokia.com >>>> Date: Sat, 15 Jan 2011 15:17:51 +0100 >>>> CC: rtc-web(a)alvestrand.no; dispatch(a)ietf.org >>>> Subject: Re: [RTW] [dispatch] Charter proposal: The activity hitherto known as "RTC-WEB at IETF" >>>>> >>>>> I agree that at least for the time being it is more fruitful to focus >>>>> the energy elsewhere. There is plenty of useful work that can be done >>>>> about media transport (the datagram service and the potential bytestream >>>>> ) and the associated APIs, and I suggest we focus on that. We can try our >>>>> luck with the codec thing later on. >>>> >>>> I agree. Codec discussions seem to go on forever, and we could spend our >>>> energy wiser. >>>> >>>> Stefan >>>> >>>> PS Sorry for answering late, but I did not follow dispatch. I thought all >>>> related messages would go on rtc-web as well. So those of you who do not >>>> follow dispatch: perhaps you should look into the dispatch archive. >>>> _______________________________________________ >>>> RTC-Web mailing list >>>> RTC-Web(a)alvestrand.no >>>> http://www.alvestrand.no/mailman/listinfo/rtc-web >>> _______________________________________________ >>> dispatch mailing list >>> dispatch(a)ietf.org >>> https://www.ietf.org/mailman/listinfo/dispatch >> >> _______________________________________________ >> dispatch mailing list >> dispatch(a)ietf.org >> https://www.ietf.org/mailman/listinfo/dispatch >

1 0

Some useful thoughts about requirements for audio processing in browsers
by Harald Alvestrand 17 Jan '11

17 Jan '11

Sharing some info from other W3C groups that are working on handling of audio in browsers. Harald -------- Original Message -------- Subject: Feedback to the DAP group on the topic of audio/media capture needed for HTML+Speech Resent-Date: Sat, 15 Jan 2011 05:47:45 +0000 Resent-From: public-device-apis(a)w3.org Date: Sat, 15 Jan 2011 04:54:56 +0000 From: Michael Bodell <mbodell(a)microsoft.com> To: public-device-apis(a)w3.org <public-device-apis(a)w3.org> CC: public-xg-htmlspeech(a)w3.org <public-xg-htmlspeech(a)w3.org> On today's Hypertext Coordination Group Teleconference the issue of "Audio on the Web" was discussed (see minutes: http://www.w3.org/2011/01/14-hcg-minutes.html) and I was given the action item of contacting the DAP group to provide feedback about audio capture. We in the HTML Speech XG (http://www.w3.org/2005/Incubator/htmlspeech/) have been discussing use cases, requirements, and some proposals around speech enabled html pages and the need for the audio to be captured and recognized in real time (I.e., in a streaming fashion, not in a file upload fashion). We recognize that there are interesting security and privacy concerns with supporting this necessary functionality. The HTML Speech XG has currently finished with requirements gathering, and is in the process of prioritizing these requirements. Our requirements document is at http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html. There are a large number (almost half) of our requirements that might be of particular note to the audio capture process. I've tried to pull out and organize the requirements most relevant to the DAP audio capture: ·Requirements about to where the audio is streamed: oFPR12. Speech services that can be specified by web apps must include network speech services [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr12] oFPR32. Speech services that can be specified by web apps must include local speech services. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr32] ·Requirements about the audio streams and the fact that it needs to be streamed: oFPR33. There should be at least one mandatory-to-support codec that isn't encumbered with IP issues and has sufficient fidelity & low bandwidth requirements. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr33] oFPR25. Implementations should be allowed to start processing captured audio before the capture completes. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr25] oFPR26. The API to do recognition should not introduce unneeded latency. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr26] oFPR56. Web applications must be able to request NL interpretation based only on text input (no audio sent). [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr56] ·Requirements about what must be possible while streaming (I.e., getting midstream events in a timely fashion without cutting off the stream; being able to decide to cut off the stream mid request; being able to reuse the stream): oFPR40. Web applications must be able to use barge-in (interrupting audio and TTS output when the user starts speaking). [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr40] oFPR21. The web app should be notified that capture starts. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr21] oFPR22. The web app should be notified that speech is considered to have started for the purposes of recognition. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr22] oFPR23. The web app should be notified that speech is considered to have ended for the purposes of recognition. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr23] oFPR24. The web app should be notified when recognition results are available. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr24] oFPR57. Web applications must be able to request recognition based on previously sent audio. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr57] oFPR59. While capture is happening, there must be a way for the web application to abort the capture and recognition process. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr59] ·Requirements around the UI/API/Usability of speech/audio capture: oFPR42. It should be possible for user agents to allow hands-free speech input. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr42] oFPR54. Web apps should be able to customize all aspects of the user interface for speech recognition, except where such customizations conflict with security and privacy requirements in this document, or where they cause other security or privacy problems. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr54] oFPR13. It should be easy to assign recognition results to a single input field. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr13] oFPR14. It should not be required to fill an input field every time there is a recognition result. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr14] oFPR15. It should be possible to use recognition results to multiple input fields. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr15] ·Requirements around privacy and security concerns: oFPR16. User consent should be informed consent. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr16] oFPR20. The spec should not unnecessarily restrict the UA's choice in privacy policy. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr20] oFPR55. Web application must be able to encrypt communications to remote speech service. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr55] oFPR1. Web applications must not capture audio without the user's consent. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr1] oFPR17. While capture is happening, there must be an obvious way for the user to abort the capture and recognition process. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr17] oFPR18. It must be possible for the user to revoke consent. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr18] oFPR37. Web application should be given captured audio access only after explicit consent from the user. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr37] oFPR49. End users need a clear indication whenever microphone is listening to the user. [http://www.w3.org/2005/Incubator/htmlspeech/live/requirements.html#fpr49] We would be happy to discuss the details and context behind any of these requirements, and we'd also appreciate any feedback on our use cases and requirements. I'm sure many of these are requirements the DAP group is already considering, but the speech use cases may well add some additional requirements that may not have yet been considered as part of the capture work. The HTML Speech XG is also in the process of collecting proposals for our Speech API which we are planning to finish by the end of February. In our discussions to date, we have reviewed and discussed some of the DAP capture API as well as some of the work that has gone on around the <device> tag proposals (We reviewed and discussed at least http://www.w3.org/TR/html-media-capture/ and http://www.w3.org/TR/media-capture-api/ and Robin provided the following links to more in progress work in the htcg call http://dev.w3.org/2009/dap/camera/ and http://dev.w3.org/2009/dap/camera/Overview-API.html) In general I'd characterize our discussions as we would be extremely happy if we could reuse the DAP work, and would be happy to work with you on having proposals that meet this need. To date in our review the large issue has been the streaming issue where the capture API is nearly useless to us if it doesn't support streaming. But happily from today's htcg call it sounds like DAP is actively working on streaming so we strongly support that work direction, think it is extremely important, and will be interesting to see any and all work in that direction. I'm not sure what the most productive next steps for us to take (email discussion back and forth, some HTML Speech XG members come to a DAP audio capture conference call, some DAP members come to a Speech XG teleconference, or something else). In general, the HTML Speech XG tries to do most of our work over the public email alias and we also have a schedule-as-needed Thursday teleconference time for 90 minutes starting at noon New York time. Thanks, and look forward to working on this important topic with you! Michael Bodell (Microsoft) Co-chair HTML Speech XG

1 0