Home » Security Bloggers Network » Getting Insights into the Delivery Path of Media with Breadcrumbs

Getting Insights into the Delivery Path of Media with Breadcrumbs

by Pankaj Chaudhari on March 25, 2020

Problem Definition

Media streaming typically involves three main components: a player, the origin, and a CDN. Typically, the content provider owns the player and the origin stack and hence is able to diagnose and troubleshoot issues, but the CDN for the most part is a black box.

The player interacts with the edge network while the origin is only aware of a limited number of machines of the CDN that interact with it to fetch content. Everything else in the middle owned by the CDN is completely opaque to the player and the origin side.

This black box nature of the CDN hides the delivery path taken to serve individual players. Consequently, for any performance or availability issues, the content provider is unaware of the geo, ASN, or specific caching layer of the CDN that might be the source of these issues, resulting in manual escalations and mitigation steps.

Introducing Breadcrumbs

Breadcrumbs is a new feature of Akamai’s media delivery products and provides customers with visibility into how each player request was handled and served from within the CDN.

For every request made by the player, the response will additionally contain a header named “Akamai-Request-BC.” This header provides details about the different IPs involved in handling the request to eventually serve the requested object back to the player.

Improving Visibility into Delivery Performance

Player-Specific View

With Breadcrumbs, a player has visibility into the tier that finally served the data for the response. This can be paired with existing client-side metrics such as TTFB and throughput metrics to get insights, as follows:

Figure 1 — TTFB and throughput to player bucketed into tier serving data

Aggregate View

The player-side data can be aggregated to create a holistic view of overall delivery performance, as shown below:

Figure 2 — Traffic Split View showing percent of requests served from each tier

The Breadcrumbs Format

The format of a single Breadcrumb is a comma-delimited list of strings or fields, where each field represents some aspect of the IP that was involved in handling the current player request. The complete string for this Breadcrumb is wrapped in a pair of square brackets.

Example Syntax of a Single Breadcrumb:

[<field-a1>, <field-a2>, <field-a3>]

where each field is a key/value pair of the form key=value

Given that multiple IPs within a CDN can be involved in handling a request, the Breadcrumb of each IP is concatenated together to create the final Breadcrumbs (plural), making it a comma-delimited list of square bracket encapsulated strings.

Example Syntax of Breadcrumbs:

[<field-a1>, <field-a2>, <field-a3>],[<field-b1>, <field-b2>, <field-b3>],[<field-c1>, <field-c2>, <field-c3>]

Definition of Breadcrumb Fields

The Breadcrumbs response header that’s sent to a player consists of the following fields

Field Key	Field Value	Notes
b	<Request ID>	A unique identifier assigned to the request. (This helps Akamai support teams quickly pull the relevant log line associated with the request.)
a	<Component IP>	The IP address of the node/host currently processing the request. If the node or host is neither an edge server nor an origin, then the IP is excluded.
c	<Component Letter>	This is a single-letter identifier for the Akamai network component that was involved during that phase of the request. (Your Akamai representative can use this value to determine where in the request phase the Breadcrumb was gathered.)
n	<Component Geo>	Geographic details associated with the node/host currently processing the request.
o	<Component ASN>	The autonomous system number associated with the current node/host.

Following are some example values of the Breadcrumbs data returned in the response header:

Example 1 — when served from within Akamai

In this example, the request flow is from Edge → Peer so the Breadcrumbs format will be:

[<component-ip>,<request-id>,<component-letter>,<component-geo>,<component-asn>],[<component-letter>,<component-geo>,<component-asn>]

Example Breadcrumbs Response to Player:

[a=23.61.194.28,b=198488236,c=g,n=US_CA_SANJOSE,o=20940],[c=p,n=US_CA_SANJOSE,o=20940]

Example 2 — when served from origin

In this example, the request flow is from Edge → Cache-H → Origin so the Breadcrumbs format will be:

[<component-ip>,<request-id>,<component-letter>,<component-geo>,<component-asn>],[<component-letter>,<component-geo>,<component-asn>],[<component-ip>,<component-letter>]

Example Breadcrumbs Response to Player:

[a=23.221.239.132,b=36411806,c=g,n=US_CA_SANJOSE,o=20940],[c=c,n=US_CA_SANJOSE,o=20940],[a=199.239.182.16,c=o]

Enabling Breadcrumbs in Property Manager

Breadcrumbs is a self-service feature that can be enabled for a particular property using a new Property Manager behavior, “Breadcrumbs.”

By enabling Breadcrumbs, responses to a player will receive an additional response header with the Breadcrumbs data, as described in the above sections.

Here’s an example of a request/response showing the Breadcrumbs response header:

$ curl -i http://amd-pa-live-1.akamaized.net/hls/live/518979/pankaj/muxed/master.m3u8

HTTP/1.1 200 OK

Last-Modified: Thu, 13 Feb 2020 17:38:17 GMT

Cache-Control: max-age=0, no-cache, no-store

Date: Thu, 27 Feb 2020 15:39:48 GMT

... truncated for brevity ...

Content-Length: 1297

Akamai-Request-BC: [a=23.1.234.47,b=35298223,c=g,n=US_CA_LOSANGELES,o=20940]

Access-Control-Expose-Headers: Server,range,hdntl,hdnts,Akamai-Mon-Iucid-Ing,Akamai-Mon-Iucid-Del,Akamai-Request-BC

... truncated for brevity ...

#EXTM3U

#EXT-X-VERSION:3

#EXT-X-INDEPENDENT-SEGMENTS

#EXT-X-STREAM-INF:BANDWIDTH=1082400,AVERAGE-BANDWIDTH=1170400,CODECS="avc1.4d401e,mp4a.40.2",RESOLUTION=640x360,FRAME-RATE=30.000,CLOSED-CAPTIONS=NONE

master/video_1000k/pl.m3u8

Breadcrumbs Opt Mode — Controlling Delivery of Breadcrumbs Response Header

After enabling Breadcrumbs via Property Manager, the default configuration will send the Breadcrumbs response header to all player requests.

Opt-Out Mode

For a player to opt-out from receiving the Breadcrumbs response header, a player can send the query param ak-bc=off.

Opt-In Mode

If the default behavior of sending Breadcrumbs response header is not desirable for some reason, this can be changed by toggling the “Opt Mode” UI button to the value “Opt-In.”

In the opt-in mode, a player has to explicitly opt into receiving the Breadcrumbs response header by passing the query param ak-bc=on.