ESB3020 - ew-manip

• 1: Introduction
• 2: Pre-roll
• 3: HLS Target Duration
• 4: Installation
• 5: Logging and monitoring
• 6: Configuration
• 7: Releases
• 7.1: Release 0.8.1
• 7.2: Release 0.8.0
• 7.3: Release 0.6.0

1 - Introduction

The service ew-manip is a stateless manifest/playlist manipulator for DASH and HLS. In the following, we use the term manifest to denote either a DASH manifest or an HLS playlist.

ew-manip is meant to be integrated with the nginx front of ew-repackager (ESB3002) and can change DASH and HLS manifests depending on query parameters in the URL.

The manipulations are triggered by a special query ewm=1 which must be at the start of the query string, and will then do different manipulations depending on options in the rest of the query.

The two modifications implemented this far are insertion of pre-roll assets in VoD manifests and change of HLS target duration for “live-like” HLS manifests coming from a live channel.

2 - Pre-roll

The specific query-string trigger for inserting pre-roll assets in VoD is pre=<URL>, but there must first be a general escape query part ewm=1 which will make nginx forward the request to the ew-manip server. The <URL> must be the absolute public URL that the client uses to fetch the preroll video, since this URL will be inserted in the combined manifest delivered to the client.

A complete request, to combine preroll-bumper with program could look like this:

http://repack-host/__cl/s:vod/__c/program/__op/default/__f/index.m3u8?ewm=1&pre=https://public-repack-host/__cl/s:vod/__c/preroll-bumper/__op/default/__f/index.m3u8

Multiple pre= query parts can be added to combine more than two manifests. The service will then fetch all the manifests from the repackager and combine them into the appropriate manifest with multiple periods for DASH or multiple parts separated by discontinuities for HLS.

DASH manifest URLs look similar but must have the file extension .mpd.

Limitations

DASH and HLS URLs cannot be combined with each other. For DASH, any assets can be combined, but for the best user experience it is reasonable to apply similar critera as for HLS. For HLS the following criteria must be met:

HLS Playlist Matching Criteria

In brief, the main and all the pre-roll master playlist(s) must match each other.

In detail, this means that:

• For Renditions identified by #EXT-X-MEDIA: For each rendition in the main playlist, there must be exactly one corresponding rendition in each pre-roll playlist, and the following attributes (when present) must match exactly: TYPE, GROUP-ID and LANGUAGE. Unmatched renditions in the pre-rolls will be discarded.

• For Variant streams identified by #EXT-X-STREAM-INF: For each variant stream in the main playlist, there must be exactly one corresponding variant stream in each pre-roll playlist, and the following attributes (when present) must match exactly: RESOLUTION, FRAME-RATE, CODECS, AUDIO, VIDEO and SUBTITLES. Unmatched variant streams in the pre-rolls will be discarded.

• For I-frame playlists identified by #EXT-X-I-FRAME-STREAM-INF: For each I-frame playlist in the main master playlist , there must be exactly one corresponding I-frame playlist in each pre-roll master playlist, and the following attributes (when present) must match exactly: RESOLUTION and CODECS. Unmatched I-frame playlists in the pre-rolls will be discarded.

Playlist requests where the main and pre-roll master playlists do not meet all of the above criteria, will fail with an HTTP 400 error.

HLS Stream and Playlist Matching by Bandwidth

Typically there will be several variant streams that meet these criteria and and that differ only by BANDWITH. Within such matching groups (for example streams with the same resolution) the main and pre-roll streams are paired together by being sorted by bandwith in decreasing order. The same pairing method is used for I-frame playlists.

3 - HLS Target Duration

A common issue when playing live HLS is the value of target duration (EXT-X-TARGETDURATION). The target duration is an integral number of seconds that correspond to the longest segment in the stream. Depending on HLS version, this value must either be higher than the longest duration, or can be rounded to the closest integer, but many players desire a lower value in order to work best. This is due to that the target-duration value influences how often the client can ask for a playlist update.

Because live ingest node (ESB3003) generates segments with a precise average segment duration, adjusted segmentation boundaries due to SCTE-35 ad markers number may result in a single segment duration being 1.5 times the average duration. These singular values then effect the target-duration to become high.

To be able to fine-tune the target duration, the ew-manip service has an option to change the #EXT-X-TARGET-DURATON value by using the special query-string trigger target-duration= once ew-manip has been triggered with the general ewm=1 trigger.

An example of a request that includes the target-duration=5 query parameter to set #EXT-X-TARGET-DURATON to 5seconds could look like this:

http://repack-host/__cl/cg:live/__c/program/__op/default/__f/index.m3u8?ewm=1&target-duration=5

This will only apply to “live-like” playlists from the repackager. With “live-like” playlists, we mean playlists which come from a live channel including an interval in the past. Such playlists can be detected by searching for the part __cl/cg: in the URL that indicates that the URL is part of a live channel group. Media playlists with the stopTime in the past are technically VoD playlists, but since there is a transition from EVENT to VOD type of playlist as the stopTime is passed, they should be treated as live playlists.

For pure VoD assets, the target-duration can simply be calculated from the longest segment in the playlist, and there is in general no limit on how fast the segments can be requested, so there is no need for target duration modifications.

4 - Installation

ew-manip runs as a systemd service on RedHat 7/8 and CentOS.

It can be installed with:

$ sudo yum install ew-manip-x.y.z-1.el8.x86_64.rpm

The normal systemctl commands apply:

$ systemctl start  ew-manip   # to start the service
$ systemctl status ew-manip   # to check status
$ systemctl enable ew-manip   # to make the service start on reboot
$ systemctl stop   ew-manip   # to stop the service

5 - Logging and monitoring

Structured logging

To provide context to each log message, structured logging is used. Thus instead of free text log messages without any specific format, the log data is to some extent divided into fields of name-value pairs, making the logs more coherent and more suitable for processing by external tools.

Log messages related to a specific HTTP request include the log field named request_id which can be used to filter out specific requests.

The log field named topic is always present and is used to separate different areas of logging. The current topics are access, messages and error, and are used to log HTTP access, debug messages and panic stack traces, respectively.

As a comparison, the topics are similar to having separate files for access logging, debug messages and errors in a conventional file-based logging system.

Log formats

When run from the command line ew-manip can log in one of the formats consolepretty, consolejson or journald depending on the value of the -logformat parameter. Logging can be disabled by using the value discard.

$ ew-manip -upstream http://127.0.0.1 -logformat consolejson

consolepretty logs to standard output (stdout) in a human-readable format. Each log line starts with the date and the log message followed by the context- dependant log fields:

2022-02-07T11:54:42Z INF Incoming request bytes_out=27592 latency_ms=6.18251 method=GET ...

consolejson logs to stdout using a JSON object for each message:

{"level":"info","topic":"access","time":"2022-02-07T12:06:07Z","bytes_out":27592, ...}

When run as a systemd service, ew-manip logs directly to journald using the logjournald configuration parameter. When stored in journald, field names are upper case whereas the values keep their format. Some examples:

To see entries with the topic access:

$ journalctl -u ew-manip TOPIC=access -o verbose

To see entries with a specific request ID:

$ journalctl -u ew-manip REQUEST_ID="host/B1Po0eYz1d-000006" -o verbose

To see the call stack in the event of a crash:

$ journalctl -u ew-manip TOPIC=error -o verbose

Using the verbose output includes all available name-value fields. Without this only the log message is printed. The field named JSON contains the original structured log message as sent by the service. To filter out only this field --output-fields=JSON can be used, however this requires a systemd version of at least 236.

$ journalctl -u ew-manip --output-fields=JSON TOPIC=access

To continuously monitor the service:

$ journalctl -fu ew-manip

Log levels

At startup the log level is set to info. The log level can be changed by posting a form-field named level to the loglevel endpoint on the ew-manip server:

$ curl -F level=debug ew-manip-server:8090/loglevel

The available log levels and how they relate to systemd priority values is listed below:

Log level  Systemd priority
trace      7
debug      7
info       6
warn       4
error      3
fatal      2
panic      0

To e.g. see all error and warning messages using journalctl:

$ journalctl -u ew-manip PRIORITY=3 PRIORITY=4

Use a GET request to see the current log level:

$ curl ew-manip-server:8090/loglevel

Messages with higher level than the configured one will not be logged.

6 - Configuration

Configuring ew-manip

The repackager is normally running as an HTTP server on port 80. The ew-manip service would typically run on the same host with the default port 8094. The port can be changed in the service file at: /usr/lib/systemd/system/ew-manip.service key: ExecStart

If the repackager is not running on port 80, or on a different host from ew-manip, then this should be specified on the same line in the service file by adding: -u <repackager-hostname:port>

In the resulting combined manifest, the pre-roll content URL will point to another asset, and its URL needs to be an absolute URL. This does not have to be configured since the full absolute URL must be sent in the query string.

Repackager configuration (nginx routing)

The nginx routing must be manually configured in the repackager. Two things must be added:

• the forwarding trigger ewm=
• the ew-manip service address

To do that, add the following two location {...} entries to the nginx config file at: /etc/ew-repackager/ew-repackager-custom.conf

location ~ ^/__cl/.*\.(m3u8|mpd)$ {
    error_page 453 = @manip_query;
    if ( $args ~ "^ewm=" ) { return 453; }
    add_header 'Access-Control-Allow-Origin' '*' always;
}

location @manip_query {
    proxy_pass http://localhost:8094;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_connect_timeout 1s;
    proxy_read_timeout 1s;
}

The value HTTP error code 453 is arbitrary, but should be outside the defined value in order to avoid collisions.

This will make requests with URLs ending with .m3u8 or .mpd containing the string ewm= at the start of the query string to be forwarded to the ew-manip server at http://localhost:8094.

7 - Releases

ESB3020 - ew-manip releases

7.1 - Release 0.8.1

Fixes:

• Failing HLS media playlist matching with different leaf pathnames has been fixed

Release information

• Date: 2022-02-16
• Type: Alpha for testing

7.2 - Release 0.8.0

Features:

• Improved DASH manifest handling
• Logging backend has been changed. The logging is now structured and is logged to journald
• Improved HLS variant stream and alternative rendition matching. The names of the variants and the alternatives do not have to be identical anymore.
• Prometheus counters for keeping track of number of requests received for HLS and DASH respectively

Fixes:

• File descriptor leak has been resolved
• For the EXT-X-MEDIA tag, the CHANNELS attribute is now preserved

Release information

• Date: 2022-02-14
• Type: Alpha for testing

7.3 - Release 0.6.0

Features:

• Add pre-roll videos to DASH and HLS assets
• Change HLS target-duration for live manifests

Release information

• Date: 2022-01-12
• Type: Alpha for testing