Video and Audio

From RuntimeWiki

Jump to: navigation, search

Contents

Title

Video and Audio

Detailed write-up

Description

HTML5 includes <audio> and <video> elements.

Why Is This Important?

Video is now a big thing on the Web. The web development community will increasing want to host video content within an Ajax application using industry standard markup and industry standard codecs.


What exists today?

WebKit supports video and audio integrated with HTML, and the latest HTML5 spec reflects these features:

SVG Tiny 1.2 includes <video> and <audio> is deployed today as an industry-standard format that is used for mobile video and mobile TV applications.

Technical discussion

  • Compositing - Because web formats support layering (e.g., CSS z-index and SVG), there is a question about what compositing features are needed when video is intermixed with HTML or SVG content. The easy answer to say, but not necessarily to implement in a performant manner, is that a video should be rendered in the same way as an image, except it is an "image" that updates its rendering 30 or 24 times a second (or whatever). In particular, if a different object is higher in Z-order than the video and overlaps with the video, that element should always render "above" the video, usually obscuring the video in the overlap region. Also, if the overlapping object is not fully opaque, it should composite in real-time against the video.
  • Time synchronization - If a document contains multiple videos and audios, the content should be synchronized along a timeline such that audio tracks and video tracks keep in sync. If there is time-based animations, these should also be synchronized against audio and video content.

Discussion

In this section, the contributors should express their opinions about this feature request, such as providing particular technical analysis or describing in prose why this feature is important (or not). It is recommended that each contributor create his own level-3 sub-section (e.g., === Jon Ferraiolo Comments ===).

It is OK for others to insert comments within other people's sections, but as a courtesy please make sure that it is clear which part of the section is original and which section are annotative comments. For example, perhaps annotative comments would be colorized (e.g., enclosed by <span style="color:red">JON: Here is what I think about what you are saying.</span>).

Brad Neuberg's comments

It is a big problem that the web's basic infrastructure doesn't support audio-video communication, especially as more and more video and audio appear. I agree these two tags are very important. The big elephant in the room is what will the codecs be though? Video codecs have become such a patent mine-field. What we need is for a large player to gift the community with their codec + patents, or for someone to step up with a large bundle of money to purchase a codec + patents and then gift it to the community. Any well-heeled philanthropists around? The problem is ogg just isn't good enough - its a great streaming format for 2002, but things have progressed majorly since then, and if we don't want these two tags to bit-rot quickly we need to address this. Jon Ferraiolo: There is lots of controversy over the issue about whether the industry needs a royalty-free codec. Personally, I think an RF codec is low priority. The vast majority of desktop PCs and mobile phones already support MPEG with H.264 content. All we need is a <video> tag within HTML and SVG that can point to the content and then invoke platform services to render the content into a buffer, and then have the browser composite that buffer with the rest of the page.

Personal tools