Metrics That Matter: Measuring Streaming Performance
The COVID-19 health crisis created a dramatic shift in our lifestyles. With work, school, and entertainment taking place in home environments now, it’s no surprise that screen time is up. Nielsen reports that not only did overall media consumption jump significantly in the first quarter of 2020, the share of video streaming in total time spent online rose from 19% in fourth-quarter 2019 to 25% in second-quarter 2020. Never has there been more pressure to deliver high-quality streaming experiences.
Keeping end users engaged is the key to successful streaming. To measure user engagement, you need insights into the performance of streaming quality. Visibility into video delivery performance requires a common set of metrics for measuring streaming quality.
Viewer Reaction to Video Quality
To better understand the correlation between video quality and viewer behavior, Akamai partnered with biometric research firm Sensum to measure the reactions of more than 1,000 people to a video viewing experience. Respondents were divided into two groups: One viewed a video clip delivered at a high resolution with no buffering, and the other saw the same segment at a lower resolution with a buffering event during a key scene. Reactions were measured using tools such as galvanic skin response monitors and facial coding software. The results:
Higher-quality streams generated 19.8% more emotional engagement
Rebuffering caused a 16% increase in negative emotions, 9% increase in disgust, 7% increase in sadness, and 8% decrease in focus, according to facial coding techniques
More than three-quarters of participants said they would stop using a service if a problem like rebuffering occurred several times
Measurement Blind Spots
Streaming providers use a variety of tools to measure viewing experience quality. Two common sets of metrics are Quality of Experience (QoE) and Quality of Service (QoS). While these are useful measurements, they have some blind spots when it comes to measuring video performance.
QoE determines the viewer experience and perception of the service quality measured at the client device. QoS provides server-side data to help pinpoint the source of improvement or degradation in performance across the infrastructure, such as a content delivery network (CDN). QoS is often characterized as providing operational metrics — transactional data mainly used for real-time operational decisions — and can be difficult to relate to user experience. Neither measurement provides a full picture of video quality and performance.
To ensure unified measurement of user experience across business and operations teams, Akamai uses Quality of Performance (QoP) metrics. QoP metrics are parameters that define operational data points to improve system efficiencies and provide visibility for business decisions by serving as proxy QoE metrics. These measurements provide more visibility into the service quality of the CDN and closely relate to QoE.
An example of a QoP metric is an adaptive bitrate streaming scenario. If you have video segments that are six seconds long, and the CDN takes more than six seconds to deliver that segment, there will be rebuffering on the client player. QoP measures the CDN service quality behavior and serves as a proxy metric to client-side rebuffering. With this insight, you can define effective QoP benchmarks for your video technology service provider that provide visibility into end-user experiences.
Learn more about how QoP metrics measure online video quality.
Where Metrics Matter Most: Live Streaming
With large gatherings on hold, streaming events are a popular alternative to in-person experiences like concerts, theater performances, conventions, and sporting matches. Let’s examine a real-world scenario where video quality metrics are critical: a massive live event.
Delivering a successful streaming event on a global level is built upon assessments and processes that are defined months — or in some cases, years — in advance. How do you ensure that your organization can deliver? Live-streaming success relies heavily on the CDN platform capacity, streaming architecture, and deployment expertise, as well as proper planning preparation and support.
Planning
Getting into planning mode early, armed with historical data and projections, allows you to understand the capacity needed for the estimated audience size, architecture optimizations, and support required for the scale of the event. And Akamai partners with our customers at each step of this stage to help ensure seamless execution.
Proper planning includes an end-to-end architectural assessment to ensure each component of the streaming workflow — from origin to the last mile — is resilient and performs effectively at scale. For example, establishing a strategy to shape last-mile bandwidth pipe size to accommodate COVID-era bandwidth constraints helps content providers and network owners deliver predictable performance.
Preparation
Preparation includes making internal and external arrangements for a successful streaming event. It starts with dry runs to align runbooks internally with support, product, platform, and executive teams. Externally, customer communications, fallbacks, and contingency plans are readied to handle any unexpected issues. Measurement tools, metrics, and thresholds are clearly defined at this stage to properly assess the performance of the event.
Execution
When it’s time to broadcast, streaming quality is reviewed daily. Each event is closely watched as implemented, from start to finish. In response to performance reporting, the actions that were predetermined in the preparation stage are rolled out to fix or optimize experiences.
Retrospective
Thoughtfully assessing the performance of the event before moving on is important for continuous improvement. How did the event perform relative to the target metrics? Do any infrastructure issues need to be addressed? It’s important to work with all of the stakeholders to gather the learnings and apply them to the next event.
Real-World Scenario
One of the largest streaming platforms in Asia partnered with Akamai to live stream one of the biggest sporting leagues in the world where the viewership of the event on mobile devices has been growing exponentially. Akamai has been working with this customer since the launch of their OTT platform to provide scale and low latency so viewers can enjoy a broadcast-quality experience on mobile devices.
To prepare for the tournament, Akamai helped the customer define the appropriate performance metrics — business priorities, available data sets, monitoring telemetry, toolsets, performance thresholds, and runbook plays (mitigation, diagnosis, elevation, and escalation) should an unexpected issue need to be addressed.
We also measured the QoP of the Akamai platform based on throughput, reliability, and server-side rebuffering estimation, a true indicator of CDN performance that closely correlates with QoS and QoE metrics. And finally, there was server-side QoS monitoring.
An important metric given unique importance during this event was client-side errors; this was critical since the majority of the viewers (more than 80%) would be watching the games from their mobile phones, where the impact of user errors is magnified. If an issue arose, we wanted to be ahead of it and mitigate it as fast as possible before users started shifting away.
And we saw that unfold during one of the critical games, where there were widespread 403s reported by a large number of users. These actually impacted the customer’s help desk operations as well, with the number of tickets skyrocketing. With proactive monitoring, we were able to identify the root cause to be ISP-induced header insertion and quickly diverted traffic away from the network regions where this was prevalent. In the longer term, we worked closely with the ISPs and the customer to resolve the issue permanently.
In addition to proactive monitoring and mitigation processes, Akamai deployed tools to monitor performance, created alerting systems, and developed communications channels (both tactical and elevated), as well as trained and practiced for the event. When thresholds were close, we looked at QoP key performance indicator (KPI) data to eliminate any concerns about CDN performance from the Akamai platform — this allowed us to engage directly with the ISP more quickly, saving a lot of time.
The final game of the event handled more than 18 million concurrent streaming viewers, making it one of the largest streaming events by audience size. Setting these records while making sure that viewers are enjoying a live sporting event will always bring the challenge of technology support and delivery capability. Akamai has continued to deliver by managing geographical spikes in traffic every year, while matching streaming quality expectations.
*** This is a Security Bloggers Network syndicated blog from The Akamai Blog authored by Harish Menon. Read the original post at: http://feedproxy.google.com/~r/TheAkamaiBlog/~3/2I7XqXUTK1k/metrics-that-matter-measuring-streaming-performance.html
