Wejn s.r.o.

Solving complicated IT problems is our hobby.

Digging Deeper Into the SMIL Playback Problem With Wowza

A few days ago I wrote about a problem with the way Wowza handles some SMIL request over HLS.

After a rather lengthy e-mail exchange with Richard Lanham I decided to dig deeper into this issue because it seemed quite hard to replicate consistently.

What I found is that the problem only crops up when all of the following holds:

  • your SMIL file has the same name (not including extension) as the referenced video file
  • your video/src in that SMIL is either name or mp4:name
  • your request has the smil:name form

My test results follow, including the script you can use to replicate it all.

First the smil file contents:

smil file video/src tag
test0.smil sample
test1.smil mp4:sample
test2.smil sample.mp4
test3.smil mp4:sample.mp4
sample0.smil sample0
sample1.smil mp4:sample1
sample2.smil sample2.mp4
sample3.smil mp4:sample3.mp4

And now the actual results:

call url smil video src result
test0 sample empty chunklist
test0.smil sample OK
smil:test0 sample OK
smil:test0.smil sample OK
test1 mp4:sample empty chunklist
test1.smil mp4:sample OK
smil:test1 mp4:sample OK
smil:test1.smil mp4:sample OK
test2 sample.mp4 empty chunklist
test2.smil sample.mp4 OK
smil:test2 sample.mp4 OK
smil:test2.smil sample.mp4 OK
test3 mp4:sample.mp4 empty chunklist
test3.smil mp4:sample.mp4 OK
smil:test3 mp4:sample.mp4 OK
smil:test3.smil mp4:sample.mp4 OK
sample0 sample0 OK
sample0.smil sample0 OK
smil:sample0 sample0 Error: looped (wowza bug)
smil:sample0.smil sample0 OK
sample1 mp4:sample1 OK
sample1.smil mp4:sample1 OK
smil:sample1 mp4:sample1 Error: looped (wowza bug)
smil:sample1.smil mp4:sample1 OK
sample2 sample2.mp4 OK
sample2.smil sample2.mp4 OK
smil:sample2 sample2.mp4 OK
smil:sample2.smil sample2.mp4 OK
sample3 mp4:sample3.mp4 OK
sample3.smil mp4:sample3.mp4 OK
smil:sample3 mp4:sample3.mp4 OK
smil:sample3.smil mp4:sample3.mp4 OK

By looped (wowza bug) I mean the originally reported problem where playback stalls.

The reason for the stalling is simple: when HLS player requests chunk list from Wowza, it doesn’t include media.ts entries as it should. Instead it contains link to the chunk list itself (same url!). No wonder the stream can’t be played.

Anyway – the results were generated by this test script I whipped up:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
#!/usr/bin/env ruby

require 'open-uri'

def gen_smil(file, video)
  File.open(file, 'w') do |f|
      f.puts <<-EOF
<smil>
<head></head>
<body>
<switch>
 <video src="#{video}" system-bitrate="500000"
     system-screen-size="240x424" width="424" height="240">
 </video>
</switch>
</body>
</smil>
     EOF
  end
end

def gen_smils(content_dir)
  videos = {
      'sample' => 'test0.smil',
      'mp4:sample' => 'test1.smil',
      'sample.mp4' => 'test2.smil',
      'mp4:sample.mp4' => 'test3.smil',
      #
      'sample0' => 'sample0.smil',
      'mp4:sample1' => 'sample1.smil',
      'sample2.mp4' => 'sample2.smil',
      'mp4:sample3.mp4' => 'sample3.smil',
  }

  out = {}
  puts "smil file;video/src tag" unless $VERBOSE
  for video, name in videos
      gen_smil(File.join(content_dir, name), video)
      if $VERBOSE
          puts "Generating #{name} with video/src: #{video}"
      else
          puts "#{name};#{video}"
      end
      out[name] = video
  end
  out
end

def request_chunklist_for(hls_url)
  puts "Requesting PL: #{hls_url}" if $VERBOSE
  pl = open(hls_url).readlines.reject { |x| x =~ /^#/ }
  pl.map!(&:strip)
  case pl.size
  when 0
      puts "Empty playlist, giving up." if $VERBOSE
  when 1
      puts "Single stream." if $VERBOSE
  else
      puts "Multiple streams, duh?" if $VERBOSE
  end
  for chunklist in pl
      cl_url = hls_url.sub(/playlist\.m3u8$/, chunklist)
      puts "Requesting CL: #{cl_url}" if $VERBOSE
      chunks = open(cl_url).readlines.reject { |x| x =~ /^#/ }
      chunks.map!(&:strip)
      media, rest = chunks.partition do |x|
          x =~ /^media[-_].*\.ts\?wowzasessionid=/
      end
      looped, garbage = rest.partition do |x|
          x =~ /^chunklist.*\.m3u8\?wowzasessionid=/
      end

      if media.empty? && rest.empty?
          puts "empty chunklist"
      elsif !media.empty? && rest.empty?
          puts "OK"
      elsif media.empty? && !looped.empty? && garbage.empty?
          puts "Error: looped (wowza bug)"
      else
          m = media.size
          l = looped.size
          g = garbage.size
          puts "Error: something else: m:#{m}, l:#{l}, g:#{g}"
      end
  end
  nil
end

if __FILE__ == $0
  unless ARGV.size == 2
      STDERR.puts "Usage: #{File.basename($0)} <wms_url> <content_dir>"
      exit 1
  end

  wms = ARGV.shift
  content_dir = ARGV.shift

  unless FileTest.directory?(content_dir)
      STDERR.puts "Content dir '#{content_dir}' doesn't exist"
      exit 1
  end

  unless FileTest.exists?(File.join(content_dir, 'sample.mp4'))
      STDERR.puts "sample.mp4 doesn't exist in content dir"
      exit 1
  end

  for i in (0..3)
      src = File.join(content_dir, 'sample.mp4')
      name = File.join(content_dir, 'sample' + i.to_s + '.mp4')
      puts "Generating #{name} (hardlink to #{src})" if $VERBOSE
      File.link(src, name) rescue nil
  end

  smils = gen_smils(content_dir)

  puts
  puts "call url;smil video src;result" unless $VERBOSE

  ext = ".smil"
  tpl = %w[N N.smil smil:N smil:N.smil]

  for smil, video in smils
      for sn in tpl.map { |x| x.sub(/N/, File.basename(smil, ext)) }
          url = [wms, sn, 'playlist.m3u8'].join('/')
          if $VERBOSE
              puts "Request: #{smil} with #{video}, called as: #{sn}"
              request_chunklist_for(url)
              puts
          else
              print "#{sn};#{video};"
              request_chunklist_for(url)
          end
      end
  end
end

which I called like this:

1
$ ./script.rb "http://wowza.local:1935/vod" "/path/to/Wowza/content"

and got the tables outlined above as result (in CSV format).

Calling the script via ruby -v would produce more verbose output.

See for yourself if you can replicate it.

SMIL Playback Not Working in Wowza Media Server 3.5.2

Last night I’ve found strange issue (a bug) when working on a custom StreamNameAlias module.

Wowza 3.X will not properly handle SMIL requests over HLS in some circumstances.

Namely, when you have sample.smil file referencing mp4:sample video file and make the following request:

http://wowza.local:1935/vod/_definst_/smil:sample/playlist.m3u8

the playback won’t start (iOS player hangs on circle of death).

Funny thing is that following request:

http://wowza.local:1935/vod/_definst_/smil:sample.smil/playlist.m3u8

(notice the superfluous .smil extension) will be handled just fine.

So will be the “unqualified” request:

http://wowza.local:1935/vod/_definst_/sample.smil/playlist.m3u8

I’ve tested all this on clean installations of 3.1.0, 3.5.0 and 3.5.2.

I’ve already created ticket for this issue but so far I haven’t been able to clearly explain the issue and/or agree with Wowza support representative about the best plan of action to fix this.

In any case – I’m writing this post hoping it might save someone an hour or two of quality time with debug printouts. Because when I first stepped on this issue I re-checked the entirety of the IMediaStreamNameAliasProvider2 implementation I was working on before I even dared to think Wowza might be at fault.

If you want to test this issue yourself, make a clean installation of Wowza Media Server and install vod application (from examples). Then create content/sample.smil with contents:

1
2
3
4
5
6
7
8
9
10
<smil>
<head></head>
<body>
<switch>
  <video src="mp4:sample" system-bitrate="500000"
      system-screen-size="240x424" width="424" height="240">
</video>
</switch>
</body>
</smil>

Then create test page with contents:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<!DOCTYPE html>
<html>
<head>
<title>Wowza 3.X SMIL issue test page</title>
</head>
<body>
<h1>Wowza 3.X SMIL issue test page</h1>
<h2>smil:sample</h2>
<video controls>
<source src="http://wowza.local:1935/vod/smil:sample/playlist.m3u8" />
</video>

<h2>smil:sample.smil</h2>
<video controls>
<source
  src="http://wowza.local:1935/vod/smil:sample.smil/playlist.m3u8" />
</video>

<h2>sample.smil</h2>
<video controls>
<source src="http://wowza.local:1935/vod/sample.smil/playlist.m3u8" />
</video>
</body>
</html>

(replace wowza.local with hostname of your Wowza instance)

View it on your favorite HLS-playback-capable device and watch in horror how the first embedded video doesn’t work.

Update: The problem definitely goes away when you avoid using mp4:sample filenaming in your smil and use mp4:sample.mp4 instead. Still a bug, though.

Update 2013-03-15: After digging deeper I found out the problem significantly narrowed.

Making Wowza LoadBalancer’s serverInfoXML Pretty

Working with Wowza Media Server was always a breeze for us. We’re geeks. We are friends with Java, we dream in XML and we love commandline tools.

But sometimes it’s nice to have the machines talk nice to us.

If you run Wowza with LoadBalancer on, you’ve no doubt saw the stock serverInfoXML file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
<LoadBalancerServerInfo>
  <LoadBalancerServer>
      <connectCount>13</connectCount>
      <status>RUNNING</status>
      <redirectCount>1</redirectCount>
      <lastMessage>1 seconds 203 milliseconds</lastMessage>
      <redirect>10.0.0.13</redirect>
      <serverId>77bbaa55-1234-5678-9012-34567890abcd</serverId>
  </LoadBalancerServer>
  <LoadBalancerServer>
      <connectCount>14</connectCount>
      <status>RUNNING</status>
      <redirectCount>0</redirectCount>
      <lastMessage>685 milliseconds</lastMessage>
      <redirect>10.0.0.14</redirect>
      <serverId>77bbaa55-1234-5678-9012-34567890abce</serverId>
  </LoadBalancerServer>
  <LoadBalancerServer>
      <connectCount>15</connectCount>
      <status>RUNNING</status>
      <redirectCount>0</redirectCount>
      <lastMessage>313 milliseconds</lastMessage>
      <redirect>10.0.0.15</redirect>
      <serverId>77bbaa55-1234-5678-9012-34567890abcf</serverId>
  </LoadBalancerServer>
</LoadBalancerServerInfo>

and maybe you thought along the lines: “Wouldn’t it be nice if it was a tad more readable?”

We did, too.

That’s why we made a small tweak that makes Wowza LoadBalancer’s serverInfoXML pretty not just to machines (machines love XML) but also to humans.

Contrast the XML above with this screenshot:

And not only it’s more readable; the page will even auto-refresh every 20 seconds for you.

If you’re interested in this module, download binary release of the module that does this. The release file contains *.jar file and documentation how to use it (which is our standard of delivering custom modules). Installation of the module should be a breeze for any seasoned Wowza admin.

You can also get the source package or see the code on github.

And now, how it’s done – the plain XML generated by Wowza’s LoadBalancer is enhanced with embedded XSLT stylesheet.

In effect, the abovementioned XML ends up looking like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet type="text/xsl" href="#stylesheet"?>
<!DOCTYPE LoadBalancerServerInfo [
  <!ATTLIST xsl:stylesheet id ID #REQUIRED>
]>
<LoadBalancerServerInfo>
  <xsl:stylesheet id="stylesheet" version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:template match="/">
    <html><body>
      <h1>Wowza Edge Status</h1>
        <table border="1">
          <tr bgcolor="#bbddff">
            <th>Node</th><th>GUID</th><th>Status</th>
            <th>Connections</th><th>Redirects</th>
            <th>Last Msg</th>
          </tr>
          <xsl:for-each
              select="LoadBalancerServerInfo/LoadBalancerServer">
            <tr>
              <td><xsl:value-of select="redirect"/></td>
              <td><xsl:value-of select="serverId"/></td>
              <td><xsl:value-of select="status"/></td>
              <td><xsl:value-of select="connectCount"/></td>
              <td><xsl:value-of select="redirectCount"/></td>
              <td><xsl:value-of select="lastMessage"/></td>
            </tr>
          </xsl:for-each>
        </table>
      </body></html>
    </xsl:template>
  </xsl:stylesheet>
  <LoadBalancerServer>
    <connectCount>13</connectCount>
    <status>RUNNING</status>
    <redirectCount>1</redirectCount>
    <lastMessage>1 seconds 203 milliseconds</lastMessage>
    <redirect>10.0.0.13</redirect>
    <serverId>77bbaa55-1234-5678-9012-34567890abcd</serverId>
  </LoadBalancerServer>
  <LoadBalancerServer>
    <connectCount>14</connectCount>
    <status>RUNNING</status>
    <redirectCount>0</redirectCount>
    <lastMessage>685 milliseconds</lastMessage>
    <redirect>10.0.0.14</redirect>
    <serverId>77bbaa55-1234-5678-9012-34567890abce</serverId>
  </LoadBalancerServer>
  <LoadBalancerServer>
    <connectCount>15</connectCount>
    <status>RUNNING</status>
    <redirectCount>0</redirectCount>
    <lastMessage>313 milliseconds</lastMessage>
    <redirect>10.0.0.15</redirect>
    <serverId>77bbaa55-1234-5678-9012-34567890abcf</serverId>
  </LoadBalancerServer>
</LoadBalancerServerInfo>

Which is all there is to it. Your web browser does the rest.

How to Pass Query String in RTMP Connect Url in JWPlayer6

Not long ago one of our customers migrated from JWPlayer5 to the brand new JWPlayer6.

To their astonishment, the separate streamer and stream parameters were replaced by a single file parameter.

That works quite well until it doesn’t.

If you happen to need query string in your NetConnection.connect() then you might not be able to properly pass it within the single file param.

That might happen when you’re using token based authentication.

Checking with LongtailVideo’s support, their suggestion was to use the syntax:

rtmp://host/app/?query=string&here=now/streamname

which would inevitably break if either streamname or any part of the query string contained any slashes. Not to mention this way of composing URL parts together is, in our view, a very poor practice.

Fortunately the JWPlayer6’s RTMPMediaProvider class source code reveals undocumented code that makes it possible to work around this file feature.

When you, in your embed code, force the type to be rtmp and set streamer parameter to your connect URL then the file parameter is interpreted to be just a stream name.

In essence, using following embed code:

1
2
3
4
5
6
7
8
9
10
jwplayer("myElement").setup({
  primary: 'flash',
  type: 'rtmp',
  streamer: 'rtmp://example.com/vod/?query=string&here=now',
  file: "mp4:folder/myVideo",
  image: "/assets/myVideo.jpg",
  width: 640,
  height: 360,
  autostart: true
});

will allow you to call the desired connect URL and then play a stream (file) located within some sub-folder.

As explained, this feature is not documented but works great so far.

Last time we checked this workaround works as intended: 2013-03-07, in JWPlayer 6.2.

Custom Wowza Modules Documentation

I’ve received a request to showcase deliverables that constitute our custom modules for Wowza Media Server.

While I can’t publish neither source code nor binaries of any of the modules Wejn s.r.o. created for its customers, I can at least showcase documentation that’s part of each developed module.

I’ve chosen two relatively recent modules to demonstrate our documentation standard:

The abovementioned samples are examples of two modules where client did not have any requirements regarding documentation, therefore bare minimum was supplied.

Any client can, of course, ask for more detailed documentation for any module that’s being developed. This merely shows our minimal documentation standard.