Wejn s.r.o.

Solving complicated IT problems is our hobby.

Hunting Down the CodeIgniter “Disallowed Key Characters” Error

TL;DR: Google Analytics recently started setting __utmt_~1 cookie that breaks CodeIgniter-based websites; fix is mentioned below (adding one character to one of CI’s files).

Intro

Over the weekend I got a strange assignment from my buddy Michal C. at WebHat hosting. One of our customers complained that we broke their website because it keeps showing “Disallowed Key Characters” intermittently.

The customer further asserted that they did not change anything and thus it must be our [hosting’s] fault.

I set to figure out what’s going on.

Path to resolution

Loading the webpage of that customer I couldn’t reproduce the error (in my default, battle hardened browser). I switched browser to vanilla Firefox… and sure enough, the website broke with “Disallowed Key Characters” when I clicked link from their front page. Same story with Chromium.

Whiskey Tango Foxtrot?

Web-searching for “Disallowed Key Characters” did not get me much closer to figuring out what’s going on. Found some posts about the issue, e.g. this SO thread, but nothing that would shed some light why this issue suddenly surfaced on our customer’s website.

I set to test the customer’s assertion that they did not change anything… and ran couple of diffs between their current website and randomly selected older backups. No change, except for some newly uploaded images.

At this point I thought it’s either something in their SQL DB or it’s somehow browser related (remember, my battle hardened browser worked just fine).

Checked cookies in the vanilla Firefox and one thing stood out:

There’s a cookie named __utmt_~1 that I did not encounter before. I know that Google Analytics sets all kinds of cookies with __utm prefix, but I haven’t seen this tilde-one creature before.

Then it hit me – the reason why my default browser did not choke on it: I have many of the online tracking services blocked, thus the GA cookies don’t get set.

OK, it seems CodeIgniter chokes on cookies with ~ in them.

To test, I removed the cookie. Refreshed the page, everything ok. Refreshed again, ka-boom, same problem.

OK, problem found. Now what?

Fix

The fix is actually straightforward, already mentioned in the abovementioned StackOverflow thread.

This diff explains it best:

1
2
3
4
5
6
7
8
9
10
11
12
13
diff --git a/system/libraries/Input.php b/system/libraries/Input.php
index a36420b..5bce0d0 100644
--- a/system/libraries/Input.php
+++ b/system/libraries/Input.php
@@ -214,7 +214,7 @@ class CI_Input {
        */
        function _clean_input_keys($str)
        {
-               if ( ! preg_match("/^[a-z0-9:_\/-]+$/i", $str))
+               if ( ! preg_match("/^[~a-z0-9:_\/-]+$/i", $str))
                {
                        exit('Disallowed Key Characters.');
                }

Basically you add the tilde (~) to the regexp of allowed key characters in the _clean_input_keys method within CodeIgniter’s Input.php file.

It’s that easy. ;)

Of course, you could be more precise than adding tilde to allowed characters. But I’m no CodeIgniter expert and couldn’t see an obvious vulnerability (security or otherwise) in adding the tilde character to allowed chars.

Outro

I hope this post will ease your pain should you trip over the same issue.

Introducing Maya

I’ve spent some time actually using Rhea in production… and I have to say that it’s somewhat limited to my taste.

I happen to manage tens to hundreds of Wowza instances using shellscripts, ssh, and git/svn.

And while it’s actually pretty easy to get things done this way, there are a few (key) operations for which I was forced to interface with JMX. And JMX, in my opinion, is a strange beast of which I’m not exactly fond of.

Which led me to an idea of extending Rhea to expose some of the most frequent operations.

Long story short — I took a short exploratory coding trip and came back with a new module.

Meet Maya, as in “manage your apps.”

The result is still quite simple (and opinionated) module, backward compatible with Rhea. And it adds two nice features:

  • can respond in either plaintext or JSON
  • can start/enable/disable/list applications

That way you can pretty much kiss JMX goodbye for app-administration related tasks.

You can read the full maya documentation or get the maya release file.

Hope you find it useful.

Wowza Streaming Engine Missing conf/Authentication.xml

It’s been a while that I noticed an issue with missing conf/Authentication.xml file in Wowza Streaming Engine (Wowza 4.X).

First I tried to talk to Wowza on the forum, then I redirected the conversation, as instructed, to support email (ticket #89955).

The outcome (official response from Wowza) is still in the making.

And in the meantime I was contacted by couple of people having problems installing my custom Wowza HTTPProviders.

Well, if you’re running Wowza Streaming Engine and need to modify conf/Authentication.xml file, the fix is actually straightforward.

At first I was copying over the file from Wowza Media Server 3.X, but then I actually took a short look at WSE 4.0.3 installation and I found out that they bundle the file in wms-server.jar:

1
2
3
4
5
6
$ pwd
/usr/local/WowzaStreamingEngine-4.0.3/lib
$ grep Authentication.xml *
Binary file wms-server.jar matches
$ jar tvf wms-server.jar | grep Auth.*xml
  2553 Thu Mar 27 01:35:36 CET 2014 com/wowza/wms/conf/Authentication.xml

Which brings me to an quick and easy fix how to source the conf/Authentication.xml file when you need to add custom Authentication methods:

1
2
$ cd ../conf
$ unzip -j ../lib/wms-server.jar com/wowza/wms/conf/Authentication.xml

In other words (or if you’re not on Linux): just extract the file from wms-server.jar, stick it in your conf and modify at will.

If the conf/Authentication.xml is present, Wowza Streaming Engine (at least the 4.0.3 version I happen to have installed at this time) will happily prefer the actual filesystem file (including your modifications).

We’ll see how long it will take to Wowza Media Systems to issue an official guidance on this. But after more than one month of waiting… I’m not exactly holding my breath.

Oh, and btw, this same extraction should also work for any of the following files:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
$ pwd
/usr/local/WowzaStreamingEngine-4.0.3/lib
$ jar tvf wms-server.jar | awk '/wms.conf/ { print substr($8, 15) }'
conf/Application.xml
conf/Authentication.xml
conf/CEACaptionConversion.xml
conf/DVR.xml
conf/HTTPStreamers.xml
conf/HostPortTemplates.xml
conf/LiveStreamPacketizers.xml
conf/LiveStreamTranscoders.xml
conf/MP3Tags.xml
conf/MediaCache.xml
conf/MediaCacheSourceTemplates.xml
conf/MediaCasters.xml
conf/MediaReaders.xml
conf/MediaWriters.xml
conf/PushPublishProfiles.xml
conf/RTP.xml
conf/Server.xml
conf/StartupStreams.xml
conf/Streams.xml
conf/TimedTextProviders.xml
conf/Tune.xml
conf/VHost.xml
conf/VHosts.xml
conf/clientaccesspolicy.xml
conf/crossdomain.xml
conf/live/Application.xml
conf/vod/Application.xml

Setting Up Multi-bitrate Streaming With FMLE and Wowza

Intro

The subject of adaptive (multi-bitrate) streaming through Wowza Media Server is a pretty straightforward one, yet it trips up many people. This post explains how to do it with FMLE.

I assume you’re already familiar with basic live streaming using Flash Media Live Encoder. That is, you can publish single live stream to Wowza and have it play in a flash/html5 player.

Having said all that, let’s set it up.

Part 1: Flash Media Live Encoder

The key to setting up FMLE is two-fold:

First you have to enable multiple bitrates (upto three) in the left pane:

(of course, you can vary the output resolution and bitrate any way you like; I just happen to have only low resolution iSight cam on my 5+ yo MBP)

Then you have to change the stream name to publish each stream under different name using the %i shortcode:

By using the settings depicted in images above, three streams will be published; named output_1, output_2, and output_3.

Part 2: Wowza Media Server

Let’s assume I’m running Wowza Media Server with application live on 10.0.0.9.

To support adaptive streaming of those three streams outlined above under one common name, you’ll have to create an output.smil file under the content directory of your application with following content:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
<smil>
  <head>
  </head>
  <body>
      <switch>
          <video src="output_1" width="320" height="240"
              system-bitrate="512000" />
          <video src="output_2" width="320" height="240"
              system-bitrate="204800" />
          <video src="output_3" width="320" height="240"
              system-bitrate="51200" />
      </switch>
  </body>
</smil>

Please observe that the field values closely match parameters of the streams set up in FMLE (in particular, the system-bitrate reflects video bitrate but should be in 1024 base, therefore 500*1024 = 512000).

Playing the adaptive multi-bitrate stream

Assuming you have setup everything correctly, you can now automagically use the smil:output.smil (or even just smil:output) as a stream name for most of the supported protocols.

url description
http://10.0.0.9:1935/live/smil:output/playlist.m3u8 Apple iOS (HLS)
http://10.0.0.9:1935/live/smil:output/jwplayer.smil JWPlayer6 (playlist)
http://10.0.0.9:1935/live/smil:output/jwplayer.rss JWPlayer5 (old playlist)
http://10.0.0.9:1935/live/smil:output/medialist.smil generic RTMP with SMIL
http://10.0.0.9:1935/live/smil:output/Manifest Silverlight streaming
http://10.0.0.9:1935/live/smil:output/manifest-rtmp.f4m Flash HTTP

If you’re feeling adventurous and want to write your own adaptive bitrate support in your custom player, you can either parse the medialist.smil or create Wowza module that will generate playlist in appropriate format for your player.

Introducing RHEA

I’ve decided to open-source a Wowza Media Server module I originally wrote for a customer of mine and in the end wasn’t compensated for (whatever the reason might be).

Meet rhea, simple module that allows you to restart Wowza Media Server application via HTTP call.

The main difference to the shutdown JMX call is that it will atomically terminate all clients and mediacasters, therefore terminating the application immediately (in most circumstances, anyway). Thus any clients connecting during the shutdown sequence will be connected to a newly started instance, possibly with updated Application.xml.

You can even get rhea release file in case compiling with Ant is not your strongest point.

Severing Official Ties With Wowza Media Systems

As of today we terminated participation in the official independent consultant program run by Wowza Media Systems, LLC.

Therefore going forward we’re truly independent Wowza consulting company as we have absolutely no official ties to Wowza Media Systems, LLC.

Luckily, we still possess is the expert-level knowledge of their product.

Wowza StorageDir Escape Regression

Issue

This issue was reported to Wowza Media Services in early 2009.

Now it surfaced again.

In a nutshell, you can escape Application’s StorageDir using relative path.

Let’s say you have two applications:

  • vod1 with /usr/local/WowzaMediaServer/content1/ as StorageDir
  • vod2 with /usr/local/WowzaMediaServer/content2/ as StorageDir

Requesting to play mp4:../content1/file.mp4 from vod2 will work just fine thus bypassing configured StorageDir.

Possible workarounds

  • Implement custom module that supplies either IMediaStreamNameAliasProvider2 or IMediaStreamFileMapper override which blocks requests falling outside configured StorageDir
  • Use StreamNameAlias module to block requests with relative paths
  • Upgrade to Wowza 3.5.2.06 (patch that hopefully fixes this issue)
  • Don’t use predictable paths

Timeline

  • 2013-04-06 Wowza Media Services contacted about this issue
  • 2013-04-08 Wowza acknowledges this bug, no further info received
  • 2013-04-30 Public release due to vendor’s non-cooperation

Why You Should Keep Properties and Logging Modules Out of Your Application.xml

Issue at hand

Not long ago I was spending some quality time with my client’s Application.xml and for some reason the presence of four default modules (base, properties, logging, flvplayback) bugged me.

So I did a quick recon and almost fell out of my chair when I read the description of properties module in Wowza API documentation.

The properties module exposes the following remotely callable functions:

  • [gs]etAppInstanceProperty
  • [gs]etApplicationProperty
  • [gs]etClientProperty
  • [gs]etStreamProperty

In other words – this innocent-looking little module, that’s included by default in all examples supplied with default WMS installation, allows you to read and write IClient, IMediaStream, IApplicationInstance, and IApplication properties!

Why should that worry me, you might ask?

Well, do you use SecureToken module? Either for securing your player-edge or even origin-edge connections? Or, do you perhaps have some <Properties> in your Application.xml you wouldn’t want anyone to read? (SQL connection parameters, backend passwords, etc)

Then you should be worried.

As anyone can read out (or even re-write) your secureTokenSharedSecret (and other properties) as soon as they get RTMP connection.

I alredy reported this problem to Wowza. And while I’m sure they will solve it in the future releases, I would advise anyone running Wowza Media Server to disable both properties and logging modules in their applications unless they’re actively using them and understand the risk(s).

Demo

It’d be boring post without a demo. I wrote a fairly straightforward patch to JWPlayer 6.3 that will bypass SecureToken protection on any server it’s allowed to connect to.

If you connect with regular JWPlayer (or any other Flash player) to the following RTMP URL:

rtmp://wms-dev.wejn.com/vod/_definst_/mp4:sample

you will be required to reply with valid SecureToken response before the video starts playing.

Well, this JWPlayer (modified with the abovementioned patch) works around that problem:

Player here

by requesting the secureTokenSharedSecret AppInstance property from the server and then decrypting the SecureToken challenge with it.

Note: If the demo doesn’t work, I’ve probably hit the 10 connections limit on my development license. In that case you’ll have to try it against your own wowza instance.

Conclusion

Don’t use properties and logging modules in your Application.xml unless you fully understand the risks and absolutely need the remotely-callable functions these modules expose to Flash clients.

Addendum

Sure, you can try combating this problem by checking referrer and perhaps even by employing SWF hash. But both checks can be easily worked around (check the source of rtmpdump or mplayer if you need convincing).

As for why should the logging module be kept out – I’ll leave this to your darker side to figure out.

Timeline

  • 2013-04-06 Wowza Media Services contacted about this issue
  • 2013-04-08 Wowza rep states that this is a non-issue and accuses me of scaremongering
  • 2013-04-19 After subsequent rounds of communication, Wowza rep threatens to “reevaluate” my “independent” consultant status if this info disclosed
  • 2013-04-27 Contacting Wowza rep with non-public preview of this post including the JWPlayer demo (last shot at responsible disclosure)
  • 2013-04-28 Wowza rep responds with more indirect threats, info that this will be resolved sometime in the future and a plea “won’t you please think of the users”
  • 2013-04-30 Public release due to vendor’s non-cooperation

Wowza Automatic Performance Tuning

If you’re like me and need to install and tune a new Wowza Media Server instance every once in a while, you’ve no doubt found the editing of conf/VHost.xml, as described on the How to do performance tuning page, quite tedious.

Fortunately, there’s a better way – you can let a simple bash+xmlstarlet script do the dirty work of tuning conf/VHost.xml and bin/setenv.sh for you:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
#!/bin/bash

# =====================================================================
# Wowza Media Server auto-tuning script
#
# Allows you to automatically perform basic performance tuning,
# according to guidelines posted by Wowza Media Systems:
# http://www.wowza.com/forums/content.php?46
#
# ---------------------------------------------------------------------
#
# (C) 2003, Michal "Wejn" Jirku (box at wejn dot com)
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# 
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
# 
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
# =====================================================================

# prereqs
(exec >/dev/null 2>&1; type xmlstarlet && type grep && type sed)
if [ $? -ne 0 ]; then
  echo "Prereqs (xmlstarlet, grep, sed) are not satisfied." >&2
  exit 1
fi

# path to WMS
if [ $# -ne 1 ]; then
  WMS_DIR=/usr/local/WowzaMediaServer
else
  WMS_DIR=$1
fi

if [ -d "$WMS_DIR" -a -f "$WMS_DIR/conf/VHost.xml" \
      -a -f "$WMS_DIR/bin/setenv.sh" ]; then
  true
else
  echo "Directory $WMS_DIR is not valid WMS install." >&2
  echo "Please supply valid dir as first parameter." >&2
  exit 1
fi

# Mem & arch
grep -q '\<lm\>' /proc/cpuinfo
if [ $? -eq 0 ]; then
  # 64 bit
  SIXTYFOUR=1
else
  SIXTYFOUR=0
fi

MEMKB=$(sed '/^MemTotal/{s/.*: *//;s/ .*//;p};d' /proc/meminfo)
MB=$[ $MEMKB / 1000 ]
# less than 3GB or 32bit
if [ $MB -lt 3000 -o $SIXTYFOUR -eq 0 ]; then
  HEAP=1500
else
  # over 16GB
  if [ $MB -gt 16000 ]; then
      HEAP=8000
  else
      # 75% of memory, less than 5GB
      HEAP=$[ $MB * 750 / 1000 ]
      if [ $HEAP -gt 5000 ]; then
          HEAP=5000
      fi
  fi
fi

# Core count
CPUS=$(grep ^processor /proc/cpuinfo | wc -l)
if [ $CPUS -gt 12 ]; then
  CPUS=12
fi
CPUS2=$[ $CPUS * 2 ]

# HandlerThreadPool
HTP=$[ $CPUS * 60 ]
if [ $HTP -gt 480 ]; then
  HTP=480
fi

# TransportThreadPool
TTP=$[ $CPUS * 40 ]
if [ $TTP -gt 320 ]; then
  HTP=320
fi

echo -e "Will set:"
echo -e "------------------------"
echo -e "Heap size:\t${HEAP} MB"
echo -e "CPUs:\t\t$CPUS"
echo -e "Threadpools:\t$HTP/$TTP"
echo -e "(enter to confirm)"
read A

# let's do the work
sed -i "s/-Xmx[0-9]*[mMkKgG]/-Xmx${HEAP}M/" $WMS_DIR/bin/setenv.sh
sed -i '/^#.*UseParNewGC/{s/^.//}' $WMS_DIR/bin/setenv.sh
R='/Root/VHost'
f() {
  xmlstarlet ed -O -L -P -u "$1" -v "$2" $WMS_DIR/conf/VHost.xml
}
f $R'/HostPortList/HostPort/Port[text()!=8086]/../ProcessorCount' \
  $CPUS2
f $R'/IdleWorkers/WorkerCount' $CPUS2
f $R'/NetConnections/ProcessorCount' $CPUS2
f $R'/RTP/UnicastIncoming/ProcessorCount' $CPUS
f $R'/RTP/UnicastOutgoing/ProcessorCount' $CPUS2
f $R'/RTP/MulticastIncoming/ProcessorCount' $CPUS
f $R'/RTP/MulticastOutgoing/ProcessorCount' $CPUS
f $R'/HandlerThreadPool/PoolSize' $HTP
f $R'/TransportThreadPool/PoolSize' $TTP

echo "All done."

If you’re interested, you can download the script so you don’t have to copy & paste it.

Btw, to get xmlstarlet on CentOS you’ll have to add EPEL repo (or so I’m told). On Debian Squeeze it’s just simple apt-get install xmlstarlet.

Update 2013-07-08: xmlstarlet 1.3.1 (on Debian Wheeze) formats the conf/VHost.xml file in a strange fashion when you use both -P -S. Updated the script to reflect (avoid) that.

On Multiple Wowza Module Instances

During my short tenure developing custom modules for Wowza Media Server I’ve quite often came across the need for multiple module instances within single Application instance.

Having this feature would make some of my modules more UNIX-y (do one thing and do it well).

Without it, I usually solved this need the hard way; I made the module run everything in single instance and implemented complicated logic just to handle the same task in multiple instances (i.e. publishing multiple Stream instances and filling them with proper Playlist data).

And then it struck me; the only reason I was scratching my left ear with my right hand.

I couldn’t properly configure two Wowza module instances within single Application instance as I was reading the same Property keys.

That was the only thing missing – to be able to pull different WMSProperties out of the Application.xml for each instance.

Well, after this aha moment it took less than one hour (of quality time with ant watch and Wowza API documentation) to arrive at an acceptable solution. Not a beautiful one, mind you; just acceptable.

And it’s this simple:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import com.wowza.wms.module.*;
import com.wowza.wms.application.*;
import com.wowza.wms.logging.*;

// get your own name -- you can use it as property prefix

public class Module extends ModuleBase
{
    public void onAppStart(IApplicationInstance ai)
    {
        for (Object mn: ai.getModuleList().getModuleNames()) {
            if (ai.getModuleList().getModuleItem((String) mn).
                    getInstance().equals(this)) {
                getLogger().info("My name: " + (String) mn);
            }
        }
    }
}

To translate this to English: when you lookup your Module’s name and use that as a prefix for all your application instance Property lookups, you can simply do this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<!-- ... -->
<Modules>
    <!-- ... -->
    <Module>
        <Name>module1</Name>
        <Description>My awesome Wowza Module (1)</Description>
        <Class>cz.wejn.awesome.Module</Class>
    </Module>
    <Module>
        <Name>module2</Name>
        <Description>My awesome Wowza Module (2)</Description>
        <Class>cz.wejn.awesome.Module</Class>
    </Module>
</Modules>
<!-- ... -->
<Properties>
    <Property>
        <Name>module1ImportantProperty</Name>
        <Value>yay</Value>
    </Property>
    <Property>
        <Name>module2ImportantProperty</Name>
        <Value>yay indeed</Value>
    </Property>
</Properties>

and coupled with the following source code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
package cz.wejn.awesome;

import com.wowza.wms.module.*;
import com.wowza.wms.application.*;
import com.wowza.wms.logging.*;

// get your own name -- you can use it as property prefix

public class Module extends ModuleBase
{
    private String name = null;

    public void onAppStart(IApplicationInstance ai)
    {
        for (Object mn: ai.getModuleList().getModuleNames()) {
            if (ai.getModuleList().getModuleItem((String) mn).
                    getInstance().equals(this)) {
                // getLogger().info("Found my name: " + (String) mn);
                this.name = (String) mn;
            }
        }

        if (this.name == null) {
            getLogger().warn("awesome.Module[" + ai.getContextStr() +
                "]: Couldn't quite figure out my own name. :-(");
        } else {
            WMSProperties props = ai.getProperties();
            String prop = props.getPropertyStr(this.name +
                "ImportantProperty");
            if (prop == null) {
                getLogger().warn("awesome.Module<" + this.name + ">[" +
                    ai.getContextStr() + "]: No property for me. :-(");
            } else {
                getLogger().info("awesome.Module<" + this.name + ">[" +
                    ai.getContextStr() + "]: Got it: " + prop);
            }
        }
    }
}

will get you following server comments on Application start:

1
2
INFO awesome.Module<module1>[vod/_definst_]: Got it: yay
INFO awesome.Module<module2>[vod/_definst_]: Got it: yay indeed

Now, if that’s not awesome then I don’t know what is.

Btw, you can get the full source code of the example on GitHub.