Nginx SSL Config Snipet

I came across an nginx config that is commented and has links to references explaining each setting. It has a few extra settings my original config didn’t have, plus some awesome comments and links. You can’t say no to that.

Here’s the blog post it came from

Upgrade your Debian Kernel on Digital Ocean

Digital Ocean doesn’t let you run your own kernels via their control panel, but there is a workaround.

kexec via sudo apt-get install kexec-tools

kexec lets you boot a new kernel from the command line. With a little shell magic (shell magic is what I call my nasty hack), we can interrupt the normal boot part way through, and reboot into our preferred kernel.

First, the magic, don’t uninstall any kernels, but install the kernel you want to boot. At the time of writing, I chose linux-image-3.13-1-amd64, and I have two symlinks

chesty@indifferent:~$ ls -l /vmlinuz /initrd.img
lrwxrwxrwx 1 root root 29 Mar  5 21:24 /initrd.img -> /boot/initrd.img-3.13-1-amd64
lrwxrwxrwx 1 root root 25 Mar  5 21:24 /vmlinuz -> boot/vmlinuz-3.13-1-amd64

create a file in /etc/init.d, call it something like localkexec

chesty@indifferent:~$ cat /etc/init.d/localkexec 
# Provides:          localkexec
# Required-Start:    
# Required-Stop:
# Should-Start:      
# Default-Start:     S
# Default-Stop:
# X-Interactive:     true
# Short-Description: kexec

case "$1" in
        if grep -q kexecDO /proc/cmdline; then
                exit 0
        /sbin/kexec -l /vmlinuz --append="`cat /proc/cmdline` kexecDO" --initrd=/initrd.img
        /sbin/kexec -e
        echo "Error: argument '$1' not supported" >&2
        exit 3
        # No-op
        echo "Usage: $0 [start|stop]" >&2
        exit 3

What happens is, if the word kexecDO doesn’t appear in /proc/cmdline, then reboot into our new kernel. The new kernel will have the same /proc/cmdline as the previous Digital Ocean kernel, with the added word kexecDO. This stops boot loops. Also we need /proc/cmdline, so we have to run this script after /proc is mounted. On my system, /proc is mounted by
/etc/rcS.d/ so /etc/rcS.d/S02localkexec will work OK.

cd /etc/rcS.d
ln -s ../init.d/localkexec S02localkexec

You should test kexec first manually from the command like before installing the init script in /etc/rcS.d, sudo /etc/init.d/localkexec will reboot you into your new kernel. After that, install the symlink in /etc/rcS.d and type reboot. Done.

Nginx Config Snipet to Give You A+ on SSLLabs

This is what I currently use in nginx,  it gives me an A+ on hat tip to kyhwana who linked me to his config. It doesn’t support IE on XP, but I don’t care. Here’s some docs for nginx https.

        listen   443 ssl 

        ssl                  on;
        ssl_certificate ssl.crt;
        ssl_certificate_key ssl.key;
        ssl_session_cache   shared:SSL:10m;
        ssl_session_timeout 10m;
        keepalive_timeout   300;

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        ssl_stapling on;
        add_header Strict-Transport-Security max-age=31536000;
        add_header X-Frame-Options DENY;

I added DHE-RSA-AES256-SHA to the ssl_ciphers so older versions of openssl would work, as a planet I’m on couldn’t connect to my feed.

I think one problem with specifying ciphers like I did above is that over the months/years, it’s going to rot, new more secure ciphers will come along, and they won’t be included until you edit the config. Or worse, some flaws might be found in the ciphers specified, and they won’t be removed. Specifying:

    ssl_ciphers         HIGH:!aNULL:!MD5:!3DES;

Is a little weaker today, but you can set and forget, and use the same config file with newer openssl libraries and nginx software, and benefit from any changes automatically.


Tiny ssh script and bash script organisation

This is one for the beginners.

Firstly, I use ssh tunnels and I’d like them to always be available, unfortunately, the internet and my VPS provider,, aren’t 100% reliable, and they aren’t designed to be. So I’d like my ssh connections to reconnect if they drop out. I use the following

while : ; do
    /usr/bin/ssh $@
    sleep 1

The next step is to have that code easily available to every user so they don’t need to type it out every time. One option is to add a shell shebang, (#!/bin/sh) put it in an executable file called, say, sshretry, and put in /usr/local/bin/

Another way is add a shell function called sshretry to your environment, you could put it in ~/.profile, however this will only work for your user, a better option is to create a file called /etc/profile.d/ and add the following

sshretry () {
    while : ; do
        /usr/bin/ssh $@
        sleep 1

This will make it available to all users once they re-log in. These are the reasons I chose to do it this way are:

  1. shell functions don’t need to run in a new shell (they can, but this one doesn’t), not a big deal, but one less process lying around.
  2. putting a shell function in /etc/profile.d has the advantage of making it available to all users on your system without needing to modify /etc/profile, which makes upgrades a easier. Basically if the package that owns /etc/profile gets updated and it wants to update /etc/profile, it can without stomping all over any changes I may have made to it, because I haven’t made any changes to it.
  3. I called the file localsshretry rather than sshretry to make it clear it was added by me, and also to reduce the likelihood the file will conflict with a package file that might be installed in the future.

Locking Down a Compromised Elastix/Asterisk CentOS System

A small business in Sydney contacted me to look at their Asterisks phone system, it had been compromised and a large amount of overseas phone calls had been made ($28000 worth), you can read more about it here

Dealing with compromised systems 101 says restore from a known clean backup, that option was out, no backups. A reinstall, then, is the only smart thing to do, except we were constrained by money, so I did the best I could with locking down a system in an unknown state. Usually in these type of  breaches they don’t bother gaining root and installing backdoors, they probably didn’t even get a shell on CentOS, it was weak phone account passwords that allowed them to register phones on Asterisk and make calls. You never know though, so I was and still am a little uncomfortable, but we make do with what we have available.

One of the first things I did was yum update, this brought in a lot of updates, including updates to Elastix and Asterisks, and it brought CentOS 5 up to the latest. I also re-installed a bunch of packages, so things like the netstat command would be refreshed, and ran for p in `rpm -qa`; do rpm –verify $p >>/tmp/v 2>&1 ; done. It’s better than nothing. If it was a sophisticated compromise those measures wouldn’t help.

Next job was firewalls, one on the CentOS box itself, restricting ssh as I wasn’t 100% sure there weren’t hidden accounts, there were none in /etc/passwd and /etc/shadow, but you can’t be 100% sure. I configured ssh to only allow root login, generally  you do the opposite and disable root login, but it does one thing, Asterisk, and it’s only going to be looked at once in a blue moon if there is an issue. I restricted 5060 to only allow incoming new connections from the telephone company’s IP address and the LAN, and blocked access to the web ports from the internet. Elastix management is now done through ssh tunnels. I did the same sort of port blocking on the ADSL router.

Next was locking down Elastix/Asterisk, I restricted Asterisk so phones could only register from the LAN, I changed all the phones passwords to random strings, I disabled incoming calls from the internet except those from the phone company.

There are a few layers of security now, so it’s no longer low hanging fruit. One would have to be quite determined to break into it now, and there’s no point because there are 1000s of other Asterisks servers out on the internet in the default convenient but insecure configuration that the criminals will look elsewhere. I hope.


I wonder if this blog still works.

Arpwatch is a useful tool to run on a server on a network with DHCP clients.

It’s good for recording MAC addresses, which can be handy for sending wake-on-lan packets at a later date. It sends you an email whenever it hears of a new MAC address, if an old MAC address has a new IP address, or if an old IP address has a new MAC address.

It’s also good for fault finding. At one workplace I installed arpwatch to collect MAC addresses for wake-on-lan action. The next morning I got an email from arpwatch saying the MAC address of the file server had changed. The old MAC address was a Dell, the new MAC was a Samsung. The next minute it changed back to the original Dell MAC.

What I think was happening was the owner of this company had an android phone,  he has wifi at home and at work. When he was at home, his router gave him an IP address of, he leaves home, arrives at work, and android would automatically connect to the work wifi, his android would recall the last IP address it had, and would test it out real quick to see if it still worked rather than wait for the good old DHCP server to give him a new lease, or something like that, probably nothing like that, but somehow his android phone connected to the network with an IP address that wasn’t even in the DHCP range. talks about how OS X reconnects to a known network so fast.

Unfortunately the IP address it was testing out was also the file servers IP address. Other devices on the network lost contact with the file server as a result. This was causing intermittent issues on the network until arpwatch found it, reported it, and the work wifi was turned off. No more weird intermittent problems.

mod_security False Positives

I’ll go through a false positive example I found on my blog. False positives are inevitable, so it’s not a bad idea to run mod_security for a few weeks on detect only.

Here’s the url that was raising the alarm. The trigger was in the url.

Here’s what modsec_audit.log looked like

[21/Mar/2011:07:54:47 +1100] TYZpl0LcAZkAAHCLBPIAAAAA ::ffff: 34169 81
GET /blog/2007/11/20/blue-tongue-harmonica-talk-cd/ HTTP/1.0
X-Real-IP: ::ffff:
X-Forwarded-For: ::ffff:
Connection: close
Cache-Control: no-cache
Pragma: no-cache
User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.26 (KHTML, like Gecko) Ubuntu/10.10 Chromium/12.0.709.0 Chrome/12.0.709.0 Safari/534.26
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3

HTTP/1.1 403 Forbidden
Content-Length: 370
Connection: close
Content-Type: text/html; charset=iso-8859-1

Message: Access denied with code 403 (phase 2). Pattern match "bcdbW*?[/]" at REQUEST_FILENAME. [file "/etc/apache2/modsecurity_crs/base_rules/modsecurity_crs_40_generic_attacks.conf"] [line "396"] [id "958821"] [rev "2.1.2"] [msg "System Command Injection"] [data "cd/"] [severity "CRITICAL"] [tag "WEB_ATTACK/COMMAND_INJECTION"] [tag "WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5.2"]
Action: Intercepted (phase 2)
Stopwatch: 1300654487217847 3874 (2435 3436 -)
Producer: ModSecurity for Apache/2.5.13 (; core ruleset/2.1.2.
Server: Apache/2.2.14 (Ubuntu) PHP/5.3.2-1ubuntu4.7 with Suhosin-Patch


Section B –f8a03521-B– contains the request headers, and you can see the GET is /blog/2007/11/20/blue-tongue-harmonica-talk-cd/.

Section H is the audit log, why it was blocked in this case.
[msg “System Command Injection”] [data “cd/”] so it didn’t like cd/, I’m not sure that cd/ (without a space between the cd and the directory) would do much on a unix system, but perhaps it does on windows, where apache also runs.

It also tells you the pattern that matched, the file where the rule is located and the rule id.
Pattern match “bcdbW*?[/]” at REQUEST_FILENAME. [file “/etc/apache2/modsecurity_crs/base_rules/modsecurity_crs_40_generic_attacks.conf”] [line “396”] [id “958821”]

What we can do, is when REQUEST_FILENAME = /blog/2007/11/20/blue-tongue-harmonica-talk-cd/ remove rule id 958821.

SecRule REQUEST_FILENAME "^/blog/2007/11/20/blue-tongue-harmonica-talk-cd/(index.php)?$" "nolog,pass,ctl:RuleRemoveById=958821"

Add this rule to your local rules, the reason we don’t modify the base_rules is they get overwritten when there’s an update to the rules.

“pass” above means if the rule matches, keep processing other rules, so it will still catch xss for example:

I’d like to come back to this later and lock it down a little.

mod_security SecAuditLog

In my previous post I didn’t mention how to turn on the audit log, so without explanation, you put the following in your config.

SecAuditEngine RelevantOnly
SecAuditLogType serial
SecAuditLogParts ABCFHZ

We already set SecAuditLog to /var/log/modsecurity/SecAuditLog/modsec_audit.log in the previous post, which has to be writable by the web server user, www-data for ubuntu/debian. So after a reload, you should start seeing entries when mod_security denies a request.

I’m going to modify SecAuditLogType setting in a future post so we can enable the console.


Some time ago, I changed my setup from nginx + spawn-fcgi to nginx + apache so I could play with mod_security. Well, one of my clients got pwned by some sql injection vulnerability, so I finally got to put it in practice.

In a nut shell, mod_security is an apache module that scans http(s) traffic for suspicious strings. It can catch suspicious strings entering apache or leaving apache. It’s a firewall for http(s), it blocks bad traffic, and lets good traffic pass through.

mod_security is one of those applications that doesn’t fit the package model well. I guess it comes in two parts, the apache module and the rules. Packages don’t seem to even attempt to manage the rules, you download them separately and they come with a util to update them. Then you have your local rules that add to, disable, or override the upstream rules to fit your environment.

In ubuntu, the mod_security package is a little outdated, it always will be, especially on LTS, and it’s one of those packages that needs constant updates. The clever and motivated security researchers eventually find weakness in mod_security which are addressed in updates. So you either use the packages, and hope fixes are backported in a timely manner, which hasn’t appeared to of happened in ubuntu, or you install by source.

Installing by source increases your workload and moves the burden of keeping it up to date to yourself. But it’s a pretty straightforward process, and if you have only one or two machines to admin, it’s no issue.

It’s basically a matter of downloading the source, checking the signature, running apt-get build-dep libapache-mod-security, untaring, ./configure; make;make test;make install, then enable the module in apache. It’s pretty well covered in the docs and blogs found in google.

That takes care of the module, but it does nothing without the rules. I downloaded the core rules from OWASP. There some documentation in the package, but there’s a few gotchas, so I’ll try and document step by step the process I went through.

  1. unpack the rules in your home directory
  2. sudo mkdir /etc/apache2/modsecurity_crs/
  3. cd modsecurity-crs_2.1.2
    sudo cp -a *_rules *.example /etc/apache2/modsecurity_crs/
  4. cd /etc/apache2/modsecurity_crs/
    sudo mv modsecurity_crs_10_config.conf{.example,}
  5. You have to edit modesecurity_crs_10_config.conf and uncomment a few lines, I think the main one being SecRuleEngine.
  6. I created two files in /etc/apache2/mods-available
    <IfModule security2_module>
    Include modsecurity_crs/*.conf
    Include modsecurity_crs/base_rules/*.conf

    LoadFile /usr/lib/
    LoadFile /usr/lib/
    LoadModule security2_module /usr/lib/apache2/modules/
  7. sudo a2enmod 0modsecurity
  8. create a file called /etc/apache2/modsecurity_crs/modsecurity_localconfig.conf with the following
    SecUploadDir /var/log/modsecurity/SecUploadDir
    SecAuditLog /var/log/modsecurity/SecAuditLog/modsec_audit.log
    SecAuditLogStorageDir /var/log/modsecurity/SecAuditLogStorageDir
    SecDebugLog /var/log/modsecurity/SecDebugLog/modsec_debug.log
    SecDataDir /var/log/modsecurity/SecDataDir
    SecTmpDir /var/log/modsecurity/SecTmpDir
  9. Make all the directories in /var/log/modsecurity, which have to be owned by www-data
    sudo mkdir -p /var/log/modsecurity/{SecUploadDir,SecAuditLog,SecAuditLogStorageDir,SecDebugLog,SecDataDir,SecTmpDir}
    sudo chown -R www-data: /var/log/modsecurity

That will do for now, I think. To modify the rules I feel it’s best to slow down and buy a book for learning, I’m not sure docs and blogs are the best way to learn this, but it’s up to your learning style. I keep updated by following @ModSecurity on twitter.

Keeping Track of Videos

When watching a TV series on the laptop, I often lose track of what episode I’m up to, especially after a reboot, so I use nautilus emblems to keep track for me.

The idea is when I double click a video file, instead of it launching a video player, I configure it to launch a script, which adds the emblem to the file and then launches the video.

The script is pretty complicated

gvfs-set-attribute -t stringv "$1" metadata::emblems generic &&
vlc "$1"

OK, I lied, it’s pretty simple, gvfs-set-attribute which can be found in the gvfs-bin package sets the emblem then vlc (or totem, or whatever gets your freak on) launches.

So you put the script somewhere sane, like /usr/local/bin/markvideo and click a
video file you want to the script to work on, and then select properties -> open with, and put your script in, it will work with all video files of that extension from then on.

Only one minor pain, you need to refresh nautilus after double clicking a file to see the emblem. I did a similar thing on snow leopard using automator and it didn’t need refreshing. But at least the emblem is there if I need to reboot, or whatever, so the next time I open that folder (or refresh it) I will see it.