Methodology

By Dr. Edgar Huang and Clifford Marsiglio

Indiana University-Purdue University Indianapolis
School of Informatics

This study was conducted from May to June in 2006. It compared five streaming technologies developed by

Adobe,
Apple,
Microsoft,
Real Network and
VX30

in terms of their

user accessibility: number of users who have already had the needed players installed and whose players can play the tested videos with no problems
streaming quality: time needed to start the streaming, and the number of times and duration of rebuffering that occur within two minutes
perceived video image quality: sharpness of the images
the cost for setting up streaming: the cost for purchasing the encoding software and server software
time needed for encoding multi-bit-rate versions and
the encoded file sizes.

Each company’s most recent version of proprietary encoder was used to encode the streaming videos. They were

Adobe's Flash 8 On2 VP6 Video Encoder,
Apple's QuickTime Pro 7,
Microsoft's Windows Media Encoder 9 Series,
Real Network's RealProducer Plus 11 and
VX30's Zentu Encoder 1.0.

Each encoder was used to encode three versions of each movie for 128kbps Dual ISDN connections, 384 DSL/cable connections and 768 DSL/cable connections. The ways the videos were encoded and Web-authored can be found in the five tutorials under this same Web site. Two movies were used in the experiment. One lasted 10 minutes and the other 30 seconds. The reason why two movies were used was because we wanted to find out how length affects initial buffering and rebuffering in each technology. To ensure a common hardware platform, all encodings were conducted on a MacBook Pro with a 2.16 GHz Intel Core Duo processor with 1GB of memory1. With the exception of Quicktime, all encodings were done under the Windows operating system (XP Service Pack 2). Quicktime was encoded under Mac OS X 10.4.72.

Both Windows Media Encoder and RealProducer produce one single movie file containing three different bit rates. All the other create three separate video files. Apple’s free MakeRefMovie software was downloaded to make a reference movie that points to three individual QuickTime movies carrying different bit rates.

Video data rate, audio data rate, frame rate and video size were customized for different bandwidths3 when possible. For Windows Media videos to be able to play on a Mac, the same video size and audio data rate must be used. Same or similar values mentioned above were used for all five technologies so as to make apple-to-apple comparisons. Flash did not have a "Deinterlacing" option. The videos were deinterlaced in all the other four technologies to maximize encoded image quality.

Windows Media videos, Real videos and QuickTime videos were streamed off a university Real Helix Universal streaming server. Flash videos were streamed off a school Flash Media Server. And VX30 video was streamed off a university Web server, as it is designed so. The most recent Web authoring methods posted on each company’s Web site were consulted.

A survey Web site was developed to collect the test data regarding user accessibility, streaming quality (buffering and re-buffering) and perceived image quality. The Web site was designed in such a way that users’ operating system (PC or Mac), bandwidth, Web browser and IP address can be automatically detected. An invitation was issued through multiple channels such as BBS, discussion board, listserv, mailing list and so on related to streaming media, video, new technology and visual communication for professionals from different geographic areas to participate in the test. The survey site told the participants that a broadband connection was needed for filling out the questionnaire. Information concerning encoding time and video file sizes was collected either during or after each encoding session. The cost information was collected from each Vendor’s Web site.

Each technology was ranked on each of the seven factors. Since people may disagree which factor is more important than which, we decided not to weigh any factors. As a result, all ranking scores each technology earned were averaged. Each technology had an overall score that represents its place in the overall ranking. The lower the score, the closer to No. 1, the better.

Since this study did not use a systematic random sample4, only descriptive statistics were used. All attempts were made to ensure a randomized blind sample were made5.

Introduction Previous

Next Findings

Footnotes:

Link To Wiki

1. To ensure accurate encoding times, the process was completed with a single computer capable of running both Windows and OS X natively.

The encoding was first posited on a pair of workstations where specific processes were out of balance, as such, we decided to re-run the data on a single MacBook Pro with a 2.16 GHz Intel Core Duo processor with 1GB of memory. This machine could be booted into either Windows or OS X natively and run at full speed. Additionally, the chipset and bulk of the hardware is commonly available between both Microsoft and Apple compatible machines and represents a current mid-range processor that would most likely be used to encode video. ↩

2. Why Encode Mainly in XP?

While it may seem that creative professionals have more of an affinity towards the Macintosh than the average user, this obviously cannot be the case. With the exception of Quicktime, specific Codecs have not been updated towards the new Intel chipset on the OS X side and have generally lagged behind in encoding development for the platform. We had originally run Flash on the OS X platform and it was actually far slower than even an older model G4. Adobe / Macromedia has promised and upgrade to this package by Q4 2006. Several 'third party' codecs showed promise and greater speeds than represented in this study, both on the PC and the Mac, but as we were measuring industry standard software suites instead of what may be “here-today-gone-tommorow” applications, we have decided to note these elsewhere

With the exception of Quicktime and Flash, the development tools for the others are either only available on the PC or have not been updated in years for the Mac. Windows capable hardware is pretty much a commodity item these days and as such, even if you are a diehard MacFan, if the encoder of your choice is not available on your platform, the hardware will most likely cost less than the encoding suite you choose. ↩

3. Bandwidth Estimation

To get an accurate reading of the client browser's bandwidth, we used some common PHP code to get an estimate of the users current speed. Using code similar to Speedtest.php, we can send a block of uncompressed data to the client as fast as both the server can parse the data and the client can read it. We choose to send 512k (0.5MB) as it seemed like a good metric based around the sizes of videos we will be exposing our surveyors to.

A breakdown of this code can be found within our Wiki for further review. ↩

4. Sample Selection

While not a truly representative sample, we tried hard to select a grouping of design oriented individuals that could readily understand some of the more complex ideas such as buffering, yet were spread across the target viewer audience. As such, we contacts X forums and mailing lists and asked these individuals to sign up. These forums and mailing lists included New Media Professionals, videographer forums, musicians forums, technical hubs for the 'Nerd Elite' and a few general access lists. ↩

5. Blind Selection

To minimize fatigue as well as the tendency to de-normalize the ratings process early on, our survey was randomized among the five video groups. A short video would play and then the longer video of the same format. Where possible, menubars and controllers were hidden to further hide the identity of the video. However, as our sample had targeted media professionals, it is likely that they would still indeed be able to identify certain formats ↩