Home
Introduction
Methodology
Findings and Discussions
Conclusions
How to use the technology
About the Researchers

Methodology

The study was conducted in May 2007. Taking advantage of the university summer break, the authors hijacked a computer lab with 20 dual-bootable iMacs and turned it into a streaming lab. Even though every iMac was equipped with an embedded Webcam, an external digital video camera was connected to each broadcasting computer to capture video images. Such a camera is especially useful for DVTS because DVTS could not recognize a Webcam and had to use a DV camera. All iMacs had the same hardware configuration and same Internet connection speed—100Mb/second. Six computers were equipped with Windows 2003 Server and installed with the server-side software for six of the streaming technologies. Another six computers were used as broadcasting machines to stream off their counterpart server machines mentioned earlier. VLC streamed off its residing computer as it is designed. So did DVTS. A Pinnacle TV for Mac HD Stick was hooked to one iMac to stream HD TV signal constantly. The TV signal was aired through an LCD projector to a large screen. All the cameras on the broadcasting iMacs face this large screen. In total, 15 iMacs were used in the test.

The first item tested was the easiness of setting up a stream in each technology. The easiness of setting up streaming was defined as the total amount of time used to set up both server software and broadcasting/encoding software. SnagIt by TechSmith and iShowU by shinywhitebox were used on Windows side and on OSX side respectively to capture screen shot videos during the server software and the broadcasting/encoding software installation and configuration. All the installations and configurations were done without hesitation since the people who did them were trained beforehand. The captured videos from two different computers (if a server was used) were spliced. The time was based on the total video length in each case ( sample video ).

The second item tested was latency. Latency refers to the time difference between the moment when a video/audio signal is sent out from the broadcasting computer and the moment when the signal is received on a user’s computer. For producers who are engaged in video broadcasting, latency may not be a big issue. The audience may not be aware of the signal delay. If a live streaming technology is used for real-time communication, such delay is very annoying. Amount of latency affects the initial buffering time. The longer the latency, the more initial buffering time is needed. An online stopwatch from http://stopwatch.onlineclock.net/ was used to precisely measure the duration between the moment when a signal is sent and that when the signal is received. The stopwatch was used on the receiving computer. The video camera that was hooked to the broadcasting computer pointed at the receiving computer to show both the stopwatch itself and the stopwatch inside the video. A video capturing software program was used to capture the latency performances in each technology. Since the amount of delay varies over time, we sampled five moments in the captured video and averaged the amount of delay in each moment. The average will be the typical delay for each technology ( sample video ). Latency and initial buffering are not the same thing, but, since they are highly positively correlated, amount of latency can be regarded as an indicator of amount of initial buffering a user would normally encounter when watching a live streaming video.

The third factor—easiness of watching—was measured by “can watch without downloading anything” and “rebuffering frequency.” The fourth factor—image quality—was measured with a Likert Scale. Both the third and fourth factors were measured by external reviewers. Conducting a test in a controlled environment can surely yield controlled results. A live streaming, however, is received by users with different operating systems accompanied by different media players and plug-ins, Web browsers if browsers are involved, and Internet connection speeds. A test on live streaming in a controlled environment can hardly tell what the aggregate real-world experience is like. Therefore, the authors of this study asked professionals in the real world to answer the questions related to item 3 and item 4. A survey Web site was set up to collect data. Recruitment was conducted through professional digital media Web sites and listservs. The survey was open 24 hours a day for seven days. Data collected from the testers were analyzed using descriptive statistics since no systematic random sampling was involved. A video was used in the survey sheet to show to the participants different kinds of rebuffering effects.

The fifth factor was cost for setting up a stream. This included the cost of the software, including the server and the broadcasting where applicable, and the hardware. Several applications were based around Open Source technologies, and thus were free of charge. Many had Open Source equivalent and while noted are not considered within this publication. The hardware component was based around a Dell PowerEde 1950 or an Apple XServe for the server components. These were chosen with regard to their equivalency in price and power, both utilizing similar chipsets and form factors. The Dell server cost around $2700, while the Apple came in around $200 more. Additionally, unless the Operating System was required, the price was included in the base cost of the hardware. Several technologies can run on Linux or other Free / Open Source operating systems — or even a 'desktop' operating system with little or no impact upon the performance. QuickTime Streaming Server requires OS X, which was included in the price of the XServe. Windows Streaming Server required Windows 2003 Server, which is an additional cost on top of the hardware.

After we had got data from measuring (1) easiness of setting up a stream, (2) latency, (3) can watch without downloading anything, (4) rebuffering frequency," and (5) cost for setting up a stream, we compared and ranked all technologies for each factor based on the raw data each earned. After all ranking scores were obtained from each factor, we averaged the ranking scores each technology earned across the five factors. The closer this score is to 1 (No. 1), the better the performance. Based on this total score, we ranked all the technologies. Since some people may hold the philosophy that “All is well that ends well,” which is, only what the audience sees should count no matter how much price the media company has to pay, we then took into consideration only "latency," "can watch without downloading anything," and "rebuffering frequency" in the calculation and presented an alternative ranking.

Finally, we found out whether a technology provided operating system compatibility for server, broadcaster and player and whether a technology had live streaming recording capability. Since these are yes or no questions, we did not factor them into the ranking.

The following chart shows the resolution and image size used when each technology was measured in terms of the first four factors.