linux sysadmin technical

Checking SSL certificate expiry date and issuer: an openssl wrapper in BASH

I manage SSL certificate requests (and renewals) at work and the number of certs in use seems to be growing every year. There’s a nice cozy 3 year expiry on most of them which means to me just enough time for them to be potentially forgotten and cause a mad scurry come renewal time (or worse, have them expire and have a service outage as a result).

Our CA has a recently unveiled service which will scan our public IPS and report on detected certificates close to expiry which is handy, but we have a lot of servers on non-routable and / or firewalled addresses, so an internal scan is the only way to cover them all.

There seem to be at least a couple of other published approaches in the google including ssl certificate expiration check and the neat ssl-cert-check script at prefetch.net which will take a list of servers and express the expiry date. I wanted something slightly different (more minimalist): as we’re using the excellent zabbix for general system monitoring which especially likes system commands or scripts which take a single parameter and spit out a single return value.

In this case that parameter is a server name or IP address,  returning  either the number of days until SSL cert expiry, or the certificate issuer, depending on which version of the script is called. As I said, we’re using this with zabbix but this is just a command line script usable for quick on the spot checks or could easily be incorporated into another monitoring / alerting system.

A nice simple method I would probably use if I didn’t have zabbix would be to incorporate this into a quick and dirty loop script which periodically queries a list of ips or subnets and fires off an email if the ‘days to expiry’  is below a certain value (exactly what zabbix does now). It could be made a bit more elegant perhaps by using a bit of nmap to build a fast list of responding servers to query, but depends what you want, how much time you have, and how much of a stickler you are for efficiency =)

(Command line for all scripts is simply:

./script-name server.name.or.ip.address

)

Script: sslcheck-expiry

#!/bin/bash
# Simple SSL cert days-till-expiry check script
# by Glen Scott, www.glenscott.net

openssl_output=$(echo "
GET / HTTP/1.0
EOT"
\
 | openssl s_client -connect $1:443 2>&1);

if [[ "$openssl_output" = *"-----BEGIN CERTIFICATE-----"* ]]; then

        cert_expiry_date=$(echo "$openssl_output" \
         | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' \
         | openssl x509 -enddate \
         | awk -F= ' /notAfter/ { printf("%s\n",$NF); } ');

        seconds_until_expiry=$(echo "$(date --date="$cert_expiry_date" +%s) - $(date +%s)" |bc);
        days_until_expiry=$(echo "$seconds_until_expiry/(60*60*24)" |bc);

        if [[ $days_until_expiry -ge 0 ]]; then

                echo "$days_until_expiry";
                exit 0

        else

                echo "EXPIRED ($days_until_expiry days)";

                exit 0
        fi

else
    echo "NOT_FOUND";
exit 1

Checking the issuer as well as the expiry

Because we have a bunch of servers setup for dev, test or other purposes with self signed “snake oil” certs and I dont really care about the certs on those, I wanted a method to determine the issuer. (Zabbix then has some logic which only bothers to email us if a cert is about to expire AND is from a real CA.)

Script: sslcheck-issuer-o

Returns the “O” (Organisation) value from the issuer string.

#!/bin/bash
# Simple SSL cert get-issuer-O
# by Glen Scott, www.glenscott.net

openssl_output=$(echo "
GET / HTTP/1.0
EOT"
\
 | openssl s_client -connect $1:443 2>&1);

if [[ "$openssl_output" = *"-----BEGIN CERTIFICATE-----"* ]]; then

        cert_issuer=$(echo "$openssl_output" \
         | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' \
         | openssl x509 -noout -issuer -nameopt sname \
         | tr '/' '\n' | grep O= | cut -c3- );

                echo "$cert_issuer";
                exit 0
else
    echo "NOT_FOUND";
exit 1
fi

Script: sslcheck-issuer-cn

If for some reason you want the CN value from the issuer string, use this instead.

#!/bin/bash
# Simple SSL cert get-issuer-CN
# by Glen Scott, www.glenscott.net

openssl_output=$(echo "
GET / HTTP/1.0
EOT"
\
 | openssl s_client -connect $1:443 2>&1);

if [[ "$openssl_output" = *"-----BEGIN CERTIFICATE-----"* ]]; then

        cert_issuer=$(echo "$openssl_output" \
         | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' \
         | openssl x509 -noout -issuer -nameopt sname \
         | tr '/' '\n' | grep CN= | cut -c4- );

                echo "$cert_issuer";
                exit 0

else
    echo "NOT_FOUND";
exit 1
fi

Additional notes:

All scripts use the openssl s_client function to connect to the first string (assuming IP or server name) on default port 443. It echoes some HTTP GET requests and an EOT, otherwise openssl will sit there until timeout waiting for something to happen. If the remote connection sends a certificate down the pipe (identified by the presence of “”—–BEGIN CERTIFICATE—–“) it will process it, otherwise it returns the value ‘NOT FOUND’, which is a catch all for no certificate, a malformed certificate, a network timeout and so on.

A quick glance at these will reveal a lot of duplication: indeed the last two only differ by one line. This is on purpose; I actually started out building a do-everything script with a switch to determine behavior (like this script I subsequently found). This would be a lot more efficient in terms of network requests, you could retrieve the cert once and analyse it in multiple ways. Unfortunately it turns out zabbix really only likes dealing with the one parameter – a minor issue for which I will forgive it – so I’ve broken this out into three smaller scripts.

OpenSSL will only return the date in a format like “Sat Jan 1 17:15:00 WAST 2010”. I was starting to figure out how to chop it up into a usable format like DDMMYYYY using sed, awk, cut and the rest but discovered  to my surprise that the unix date utility understands the string just fine as is. The date is converted to seconds since epoch and the ‘bc’ utility does some math on it, returning a rounded value in days.

Instead of awk in the last two I’ve used a handy-dandy utility called, simply, ‘text replace’ (tr). I actually don’t use awk, sed and regular expressions much and am correspondingly unfamiliar with the syntax, thus was happy to discover and use this nifty shortcut. It’s not as powerful as sed/awk but is great for a simple character replace like this, especially when the process of trying to manipulate both kinds of slashes in the issuer string and replace them with newlines

'\n'

is frying my brain.

As always, your mileage may vary, particularly regarding any differences in the basic syntax of the various utilities across platforms (I’m using RHEL). ‘date’ for example is very forgiving here, this might not be the same everywhere. I’ve also had issues in the past with the syntax of SED on OSX.

Hope someone finds this helpful, and I will write up a brief post on using zabbix to manage SSL cert expiry at some point as well.

mobile devices technical

Displaying your windows mobile device on a bigger screen via VNC

Some time ago I was tasked with creating some training sessions for staff using smartphones running WM6. So I could both record procedures from the smartphone using a flash-video creation software, and demo things on a big screen live in front of the class, I setup my phone with VNC which allowed me to do both.

I needed to create and deliver several training sessions for PDAs covering the following topics:

  • Connection to the wireless network and config of proxies etc for internet access
  • Synching with the exchange server and related calendaring and email
  • Transferring files back and forth to the device

As well as a few other items which came up on the day.

You’ll need a VNC server (not client) on the windows mobile device. I ended up downloading Pocket VNC Server and running that, with a regular VNC client connection from the laptop.

Once the service is running you have to figure out which IP each device is listening on, then initiate a client vnc connection from the PC to the mobile device. After this you can use the PDA and the PC  (eg projector or big screen) will display what is happening on the PDA in a window, which can be recorded or screenshot as required.

There are two ways to get a VNC session to the PDA – either via an activesync connection (I used a USB cable, possibly you could use ActiveSync via bluetooth if you’re adventurous), or via a local wireless LAN. I wasted a lot of time trying to setup a Bluetooth PAN (Personal Area Network) between my laptop and the PDA so I could connect via a small private subnet, but couldn’t get it working. <rant>I remain convinced that most of bluetooth networking is junk and not yet much good for enterprise use</rant>.

Option 1: Activesync (Cabled)

Activesync is actually the easiest way to get an IP connection. It wasn’t documented anywhere obvious, but once you have an activesync connection established, the two devices (host PC/laptop, PDA) should have the following IPs:

  • 192.168.55.100 – Host PC
  • 192.168.55.101 – PDA

As long as you’re not going to be messing with the activesync connection and don’t want to walk around or move far from the PC, this is fine to use. Its actually preferable in that you’re not at the mercy of a wireless connection for your presentation. Having the wireless drop out or go flaky when you’re trying to demonstrate something to a room full of people is not what you want.

Option 2: Wireless

This method could be used either in AP or ad-hoc config. As there was a good wireless signal from one of our campus access points in the room, I went with the AP option. I also allowed each device to DHCP an address: somewhat risky in case the leases changed for some reason but it worked out ok. A more leak-proof method would have been to hardwire the IP’s to something static while still using the wireless network.

I used a well known freeware tool called VXUtil to determine the ip address of my PDA to use for connection to the VNC server.

Pros: You can walk around the room, you can demonstrate the full range of exchange/activesync settings, you can demonstrate internet connectivity. (I’ve demo’d google earth on the pda on a big screen via this method – it can be a bit of a slideshow over VNC, but it gets the point across).

Cons: Wireless can have random connection issues. You cant demonstrate setup of a wireless connection on the PDA since its already using one. Even if your setup doesn’t involve disabling and re-enabling wireless, You’ll probably end up disconnecting yourself anyway. (Windows mobile networking is fun like that).

Option 3: Both Wireless and Activesync, or: Heres one we prepared earlier.

To get this done, You can use any live screen recording software, or even take screenshots of the steps in your demo. I used an app for windows (camtasia) since we had a license for it, but there are likely suitable programs for linux as well (provided you can get your networking going with or without activesync).

Pros: Its rock solid. Networking is taken out of the picture so you shouldn’t have a problem.

Cons: You have to have it all pre-prepared, if you want to deviate from the prepared material you have to augment your presentation with either option 1 or 2.

That’s about it. Get the VPN server installed, get some form of networking up, connect to it from the PC, and you’re good. Its a little slow but a lot better than trying to get a room full of people to see what’s going on by holding your 320×240 PDA screen above your head…

linux sysadmin technical

How to fix Huawei E620 USB 3G Modem in Ubuntu 9.10 Karmic Koala

I recently upgraded my laptop to Ubuntu 9.10 (Karmic Koala) and among a few other niggles (mostly related to intel video support, or lack thereof) it completely broke support for my Huawei 3G (E620) modem. Fortunately the fix is fairly straightforward: install a new kernel. I went with the latest (v2.6.32 release candidate available over here at kernel.ubuntu.org ) and the problem is solved.

If you want the gory details, check the thread over at bugs.launchpad.net. I’ll distil the useful bits below.

After the upgrade, my huawei 3G modem stopped being detected by NetworkManager. I’d see the fake ‘ZeroCD’ drive try to map itself and occasionally a gnome message box would be thrown up about a failed mount attempt, but no modem.

A look in the logs revealed /var/log/messages filling up with lines like this:

kernel: option 3-1:1.2: GSM modem (1-port) converter detected
kernel: usb 3-1: GSM modem (1-port) converter now attached to ttyUSB0
kernel: option 3-1:1.1: GSM modem (1-port) converter detected
kernel: usb 3-1: GSM modem (1-port) converter now attached to ttyUSB1
kernel: option 3-1:1.0: GSM modem (1-port) converter detected
kernel: usb 3-1: GSM modem (1-port) converter now attached to ttyUSB2
kernel: option1 ttyUSB2: GSM modem (1-port) converter now disconnected from ttyUSB2
kernel: option 3-1:1.0: device disconnected
kernel: option1 ttyUSB1: GSM modem (1-port) converter now disconnected from ttyUSB1
kernel: option 3-1:1.1: device disconnected
kernel: option1 ttyUSB0: GSM modem (1-port) converter now disconnected from ttyUSB0
kernel: option 3-1:1.2: device disconnected

So the modem was being disconnected and reconnected at least a couple of times a second for some reason, and the storage device was not appearing at all.

I tried the usb_modeswitch tool which is supposed to jolt misbehaving HUAWEI (and other brand) devices out of their stupor with some undocumented SCSI/USB commands, but no success this time.

After a bit of googling, it turns out this is (was) a known bug in the way the more recent linux kernel handles the combination USB Modem/Storage device hardware (and was allowed to remain in a major release of Ubuntu which is a bit unfortunate as it seems these types of USB modems are pretty common).

There are a couple of fixes pending an official update: either install a patched version of the kernel, or temporarily disable the USB Storage kernel module which looks pretty easy and apparently worked for a few people:

# rmmod usb-storage

Untested by me: Your mileage may vary.  Be warned that even if this works, by unloading the usb-storage kernel module you will lose support for any USB based storage devices, so this is strictly a temporary workaround. I thought I’d try the more permanent and possibly dangerous (?) kernel solution first, which worked.

Steps to upgrade your kernel to a compatible version:

  1. Check your current version with the uname -a command. My post-9.10-Karmic upgrade version was: 2.6.31-14-generic #48-Ubuntu SMP  x86_64 GNU/Linux
  2. Go to http://kernel.ubuntu.com/~kernel-ppa/mainline/ and download the .deb files for the kernel headers (“all”) and the kernel for your architecture (“amd64” or “i386”). If you don’t have any kind of internet on the affected ubuntu box, grab them via another connected machine and copy them via removable media (windows or mac should be fine for just getting the files). You want the “linux-header” and “linux-image” files from within the folder with the latest (hopefully stable) version number. You can ignore the source file for now.
  3. Go to a command prompt, change to the folder where the downloaded .deb files are located, and execute the following, substituting the .deb file names for the versions you have (make sure you install the headers first).
  4. sudo dpkg -i ./linux-headers-2.6.32-020632rc8_2.6.32-020632rc8_all.deb
  5. sudo dpkg -i ./linux-image-2.6.32-020632rc8-generic_2.6.32-020632rc8_amd64.deb

After this, provided everything worked, you’re a reboot away from your modem working again. After the boot, uname -a should reveal the newly installed kernel version. Mine is: 2.6.32-020632rc8-generic #020632rc8 SMP

Once plugged in, the modem worked instantly and my mobile broadband account connected fine. Hooray!

While lsusb output looked the same as before:

Bus 006 Device 004: ID 12d1:1001 Huawei Technologies Co., Ltd. E620 USB Modem

My /var/log/messages also looked a lot healthier:

kernel: USB Serial support registered for GSM modem (1-port)
kernel: option 6-2:1.0: GSM modem (1-port) converter detected
kernel: usb 6-2: GSM modem (1-port) converter now attached to ttyUSB0
kernel: option 6-2:1.1: GSM modem (1-port) converter detected
kernel: usb 6-2: GSM modem (1-port) converter now attached to ttyUSB1
kernel: option 6-2:1.2: GSM modem (1-port) converter detected
kernel: usb 6-2: GSM modem (1-port) converter now attached to ttyUSB2
kernel: usbcore: registered new interface driver option
kernel: option: v0.7.2:USB Driver for GSM modems
kernel: scsi 8:0:0:0: CD-ROM            HUAWEI   Mass Storage     2.31 PQ: 0 ANSI: 2
kernel: scsi 8:0:0:1: Direct-Access     HUAWEI   SD Storage       2.31 PQ: 0 ANSI: 2
kernel: sr0: scsi-1 drive
kernel: Uniform CD-ROM driver Revision: 3.20
kernel: sr 8:0:0:0: Attached scsi generic sg1 type 5
kernel: sd 8:0:0:1: Attached scsi generic sg2 type 0
kernel: sd 8:0:0:1: [sdb] Attached SCSI removable disk

Additionally, with the new kernel, both the pseudo cdrom, the 3G modem, and presumably the SD storage (though I don’t use it) are working at the same time. So, problem solved.

(Another improvement I noticed with the new version of ubuntu/kernel is I can disconnect the wireless broadband account via networkmanager without nasty gnome freeze-ups. Not sure what the culprit was for this: I worked around by disconnecting the hardware to avoid freezes, but looks like this too is now solved).