Addressing the Video Quality and Performance Measurement Gap with Quality of Performance (QoP) Metrics
Ubiquity of Digital Video Means Meeting Viewers’ Quality Expectations
Digital video viewership is on an aggressive growth course. Globally, consumers will spend 84 minutes a day watching online video in 2020, up 25% from 2018[i], and digital video viewership will increase from 35% in 2019 to more than 39% by 2023[ii].
Widespread availability of online video content — instigated by factors like ubiquitous high-speed data connectivity, hyper-connected devices, and living room experiences — are all playing their part in the growth of video content consumption online.
At the same time, increased audience choices and low switching costs can lead to viewers churning if the viewing experience is bad. More than three-quarters (77%) of the survey respondents in the February 2020 edition of the Nielsen Total Audience Report marked playback quality to be a very important video streaming attribute.
Impact of Quality on Engagement — How We React to Video Quality
In order to clearly understand viewer behavior and its correlation to video quality, Akamai and Sensum[iii], a biometric research firm, measured the reactions of more than 1,000 people. Some of the key findings:
- High-quality streams generated 19.8% more emotional engagement according to the Galvanic Skin Response test method
- Rebuffering caused a 16% increase in negative emotions, 9% increase in disgust, 7% increase in sadness, and 8% decrease in focus, according to facial coding techniques
- 76% of participants would stop using a service if a problem like rebuffering occurred several times, according to survey responses
Video Quality and Performance: Metrics, Methodology, and Measurement
Video delivery performance directly impacts a viewer’s quality of experience, which directly impacts the video service provider’s desired business outcome.
However, there is much debate about what metrics to use to define a user’s viewing experience. Each aspect of digital video delivery (components within the online video delivery value chain, from content production to distribution, and internet service providers [ISPs] involved in the last-mile delivery) is measured differently, adding even more complexity to defining the video delivery performance and viewing experience.
Without consistency in measurements across video streaming services, it is difficult to standardize acceptable measurements. There have been efforts underway globally to bring standardization into measuring streaming media quality.
The first step towards standardization is to determine what the metrics are, the methodology around how to define and document those metrics, and what kind of measurement tools should be used to ensure transparency and consistency in measurement.
Creating a Common Language for Measuring Video Quality and Performance
Let’s begin by walking through what defines the viewer experience.
The viewer experience journey starts with identifying two major aspects:
- A viewer’s perception about the video quality determined by their viewing experience on their device — represented by QoE metrics
- The service and performance quality delivered by the components in the video delivery value chain — represented by QoS metrics
Quality of Experience (QoE) Metrics
Quality of experience (QoE) metrics are the parameters that determine the viewer’s experience and perception of the service quality measured at the client device. Refer to the Measuring Video Quality and Performance: Best Practices white paper for some of the common quality of experience metrics. Although these listed metrics are the ones provided by most analytics tools, the methodology used to define them often varies, which can create inconsistencies in measurements and reporting.
Quality of Service (QoS) Metrics
In order to address quality of experience inconsistencies and accurately pinpoint the source of improvement or degradation in performance across the content production infrastructure, the content distribution network (CDN), the viewer’s ISP, and the client’s browser or player app, a separate set of quality of service (QoS) metrics are defined for each of these systems.
For example, when it comes to the CDN component of the video delivery value chain, it’s important to understand the quality of service metrics for a CDN. The white paper provides details about some of the common quality of service metrics used to measure CDN performance.
Introducing Quality of Performance (QoP) Metrics
Quality of Service (QoS) metrics are often characterized as operational metrics, or transactional data points mainly used for real-time decision-making, which differentiates them from quality of experience (QoE) metrics, which are characterized as business metrics mainly used for effective decision-making.
Ensuring a unified view of the viewer experience across business and operations teams requires a unique set of performance metrics (Quality of Performance, or QoP) that provides the quality of service for each component of the video delivery value chain and closely relate to the industry metrics for measuring quality of experience (proxy quality of experience, or proxy QoE, metrics).
Using the Best Methodology to Measure Streaming Quality
Methodology for Measuring End-User or QoE Metrics
While there are multiple ways to monitor the performance of systems, network, and infrastructure, there are two popular methods of monitoring performance from the end-user perspective[iv]:
Need to Measure CDN Edge Performance or CDN Quality of Performance
CDNs today manage a significant portion of the world’s video streaming traffic (72% of internet traffic by 2022, up from 56% in 2017) and are ubiquitous in their presence in mitigating the toughest challenges of delivering video content over the internet[v].
Along with end-to-end performance visibility achieved using the RUM and synthetic tools, it is also imperative to methodically measure the performance of the purpose-built solutions across the video delivery value chain, which includes measuring the CDN platform’s performance. This calls for specialized methods to be defined to measure the CDN performance using quality of performance metrics that provide visibility into the CDN platform performance and at the same time also relate to the end user’s viewing experience.
Choosing the Right Tools for Effective Measurement
Once the metrics to measure the video delivery performance have been established and the methodology to measure these metrics has been identified, the next step is to identify or build the tools that will be used to quantitatively measure the video delivery performance.
Quality of Experience (QoE) Measurement Tools
There are multiple real user monitoring (RUM) and synthetic testing tools available for QoE measurement and monitoring, including Conviva, Nice People At Work, Mux, Bitmovin, Touchstream, Telestream, and Akamai Media Analytics.
Note: Real user monitoring (RUM) and synthetic tools are ideal to understanding an end user’s quality of experience; however, there can be multiple scenarios in which the insights from these tools might not provide a complete picture of video quality improvement or degradation and might make it difficult to pinpoint what is responsible for the issue. It is helpful to have a checklist of questions to evaluate metrics and insights from the tools mentioned above. Refer to the white paper for details.
Quality of Service (QoS) Measurement Tools
Quality of service measurement tools measure the performance of specific components of the video delivery value chain, including origin infrastructure and CDN video delivery performance.
Providing media organizations visibility into platform performance helps organizations make timely decisions. This is a key area of focus for Akamai. Initiatives and enhancements are in progress to provide platform performance visibility using quality of performance metrics; low-latency, raw log data for operational monitoring; and greater visibility into platform capacity and troubleshooting of individual end-user requests.
Understanding video quality and how to measure it is the starting point. Implementing a framework to measure video quality and performance will help provide better visibility into viewer engagement and service quality to optimize monetization strategies to deliver a successful business impact.
We have recently published the Measuring Video Quality and Performance: Best Practices white paper that discusses these concepts in detail and provides a few best practices. Happy reading!
[i] https://www.businessinsider.com/zenith-report-online-video-consumption-2018-7?IR=T
[ii] eMarketer Global Digital Video Report 2019
[iii] https://content.akamai.com/gl-en-pg9246-sensum-whitepaper.html
[iv] A Primer on Real User Monitoring — Monitoring what Matters – eBook Catchpoint
[v] IDC MarketScape: Worldwide Commercial CDN 2019 Vendor Assessment
*** This is a Security Bloggers Network syndicated blog from The Akamai Blog authored by Sandeep Singh. Read the original post at: http://feedproxy.google.com/~r/TheAkamaiBlog/~3/TDvW86anG8A/addressing-the-video-quality-and-performance-measurement-gap-with-quality-of-performance-qop-metrics.html
