Using boto for Amazon EC2

Posted May 20, 2013 By weetabix

Sorry for the month delay. I had things in “the real world” that I needed to deal with. I am back.


Today, I will go over Amazon a bit, and Boto, the Amazon Python SDK.

If you are unfamiliar with either Boto, or Amazon EC2, heres the tl; dr; version:

Boto is a Python2 API for Amazons AWS. I use it specifically to talk to the Elastic Cloud. It also happens to be the normal Amazon supported SDK. It can be ported to Python3.


For a longer version:

For Python2, the library can be installed as normal, from pip, etc. You can do it manually, but this would be the best method.

For Python3, you need to grab the py3kport branch, and run 2to3 after following the README and adjusting to suit your needs.

The documentation is the same for both.


After 2to3, there need to be a few more changes made:

Do vi /usr/local/lib/python3.2/dist-packages/boto/

Find uses of sock.sendall, such as sock.sendall("CONNECT %s HTTP/1.0\r\n" % host),

and change it to read sock.sendall(bytes("CONNECT %s HTTP/1.0\r\n" % host,'utf-8')).

i.e : sock.sendall("payload") becomes sock.sendall(bytes("payload",'utf-8'))

This corrects the last of the issues with 2to3, which happens to be coding strings for transmission over a socket.


Then, you can do things like:


#!/usr/bin/env python3
import boto.ec2
conn = boto.ec2.connect_to_region("us-east-1",
                                  aws_access_key_id='<FILL ME>',
                                  aws_secret_access_key='<FILL ME>')
reservations = conn.get_all_instances()
for x in reservations:
    print(x, x.instances)
    for y in x.instances:
        for key in y.tags:
            print(key, y.tags.get(key))

will give you a printout of all your instances, and the name you gave them, as well as any other tags.

The access keys can be retreived from your AWS account.

Until next week,


Be the first to comment

I was sitting at home reading up on my pets. I have a pair of Sugar Gliders, Icarus and Finn, and they fascinate me. They have an apparent intelligence that intrigues me the most, so I’m trying to figure out effective ways of teaching or measuring them. I want something that is less anthropomorphic than “Ooooh aren’t you smart?”


That brings me to the Arduino. Specifically, I have a FreakLabs HackerMoms FredBoard and an enclusure, and I am very happy with it. I’ve spent the last few days playing around with it more, getting used to collecting, processing, and displaying sensor data. A frozen liquor bottle and a cup of hot tea helped get a thermal sensor working, and the teapot provided a useful reference weight for a pressure sensitive foam.


I’m sure you can see where I’m going here. I figure I want to try and collect data from my pets, and see what interesting things I find. I’ve done some reading and thinking, and I’ve decided on the following:


  • Measuring the average use of the running wheel per night, and an attempt to calculate the distance “run” each night.
  • Measuring the amount of time spent in their nest each night.
  • Recording sounds
  • Recording video
  • Measuring foraging behaviour (I want to build a toy with holes, with sensors, to tell me how frequently they search through it.)


I also want to build a small foraging toy, and “label” the holes with a colour or shape, and see what happens when one shape is always loaded, half loaded, never loaded, etc. I have a few ideas on how they might react, perhaps they will become habituated to ignoring certain shapes and attending to others, or if they always check all their options…

More to come in the future.


See you next week.




Be the first to comment

Fie diddle dee dee, init, and init.d

Posted April 8, 2013 By weetabix


On being lazy:


I love being lazy. In fact, it’s fair to say I take it quite seriously.

One of my favorite teachers taught me something deeply memorable one day. He taught me that being lazy is the best way to approach any job. Not just for you, but for the job itself and everyone else it affects.

Now, on the face of it this might sound like some poor advice.

Follow me along here:


1. The goal of being lazy is to not do more work than you have to.

2. Doing something once is less work than doing it twice.

3. To do something only once, you have to do it right the first time.

So, therefore, if you want to be lazy, you have to do the job right the first time.

– From the unforgettable Chris Venne.


One of the best ways to be lazy is with init scripts. Since I’m a Xubuntu/Debian fanboy, I use both upstart init and (x)init.d style scripts. Now, these can be easy, and sometimes they can be hard, especially when you don’t have useful examples. (and especially more so when half the sites reap off each other and have the same ding dong mistake. More on that later.)

To avoid confusion, when I say upstart script, I mean the newer style scripts run from /etc/init/ by Upstart. When I say init.d or init script, I mean a script run by init from /etc/init.d/. Hopefully this helps avoid confusion. (Though in the future after init is very deprecated, I will likely just call then init scripts.)

Some people reading this likely know what upstart and init/init.d scripts are. If you do not, the short version is, they are scripts run by services (called Upstart and init) that manage other services. This is how you get stuff to start and stop on boot and shutdown.

I keep around some template files, so I never have to worry too much about remembering how to do it.


For init we have: (Remember, you can click on the top blue line to download the file.)

# Provides:          <nameofservice>
# Required-Start:    $local_fs $network $remote_fs
# Required-Stop:     $local_fs $network $remote_fs
# Should-Start:      NetworkManager
# Should-Stop:       NetworkManager
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Durr
# Description:       Herp a Durr
DAEMON=<Binary to run>
#This is optional. It is used to load variables for this file.
SETTINGS=/etc/default/<same as initfilename>
([ -x $DAEMON ] && [ -r $SETTINGS ]) || exit 0
DESC="<What are you starting or stopping>"
DEFOPTS="<options to append to binary call>"
#this script is intended to start python binaries.
PYTHONEXEC="<path to python exec>"
#Set this to true is you decide to require a defaults file.
# these are only accepted from the settings file
. /lib/lsb/init-functions
check_retval() {
	if [ $? -eq 0 ]; then
		log_end_msg 0
		return 0
		log_end_msg 1
		exit 1
is_running() {
	# returns 0 when running, 1 otherwise
    #remove pythonexec if you are calling the binary directly.
    PID="$(pgrep -f -x -u ${USER%:*} "$PYTHONEXEC $DAEMON $DEFOPTS.*")"
	[ $RET -gt 1 ] && exit 1 || return $RET
load_settings() {
	if [ $SETTINGS_LOADED != "TRUE" ]; then
		[ -z "$USER" ] && {
			log_warning_msg "$DESC: not configured, aborting. See $SETTINGS";
			return 1; }
		[ -z "${USER%:*}" ] && exit 1
		[ -n "$CONFIG" ] && OPTIONS="$OPTIONS --config-file $CONFIG"
		[ -n "$HOST" ] && SERVER="$HOST" || SERVER=
		[ -n "$PORT" ] && SERVER="$SERVER:$PORT"
		[ -n "$SERVER" ] && OPTIONS="$OPTIONS --server $SERVER"
	return 0
start_bin() {
	load_settings || exit 0
	if ! is_running; then
		log_daemon_msg "Starting $DESC"
		start-stop-daemon --quiet --chuid $USER --start --exec $DAEMON -- $OPTIONS
		# create a pidfile; we don't use it but some monitoring app likes to have one
		[ -w $(dirname $PIDFILE) ] && \
        #remove pythonexec if you are calling the binary directly.
			pgrep -f -x -n -u ${USER%:*} "$PYTHONEXEC $DAEMON $OPTIONS" > $PIDFILE
		log_success_msg "$DESC: already running (pid $PID)"
stop_bin() {
	load_settings || exit 0
	if is_running; then
		TMPFILE="$(mktemp --tmpdir <binary name>.XXXXXXXXXX)"
		[ $? -eq 0 ] || exit 1
		trap '[ -f "$TMPFILE" ] && rm -f "$TMPFILE"' EXIT
		echo "$PID" > "$TMPFILE"
		log_daemon_msg "Stopping $DESC"
		start-stop-daemon --stop --user ${USER%:*} --pidfile "$TMPFILE" --retry 30
		log_success_msg "$DESC: not running"
	[ -f $PIDFILE ] && rm -f $PIDFILE
case "$1" in
		load_settings || exit 4
		if is_running; then
			log_success_msg "$DESC: running (pid $PID)"
			log_success_msg "$DESC: not running"
			[ -f $PIDFILE ] && exit 1 || exit 3
		log_failure_msg "Usage: $0 {start|stop|restart|force-reload|status}"
		exit 3
exit 0


And for upstart we have two flavours:


The “run a simple binary or script” type:

# <name of daemon>
# <Longer description>
description "Derp"
author "Herp Herp D Derp"
# The below "start on" line controls when it starts up. There are many options available.
start on (local-filesystems and net-device-up)
#Ditto for stop on.
stop on runlevel [016]
respawn limit 5 30
env uid=<user to run as>
env gid=<group to run as>
env umask=000
# This next line gets called on start, and killed on stop. Upstart tracks the running invocation automatically.
# I use the below exec line, but you can use just about anything.
exec start-stop-daemon -S -c $uid:$gid -k $umask -x <path/to/executable/> -- -d -l <log/me/here.log> -L warning
#exec /usr/bin/binary


And the “run some checks and scripting, before running the daemon/binary” type:

# <name of daemon>
# <Longer description>
description "Derp"
author "Herp Herp D Derp"
# The below "start on" line controls when it starts up. There are many options available.
start on (local-filesystems and net-device-up)
#Ditto for stop on.
stop on runlevel [016]
respawn limit 5 30
env uid=<user to run as>
env gid=<group to run as>
env umask=000
#A more insane start condition. This one is based off starting lightdm
start on ((filesystem
           and runlevel [!06]
           and started dbus
           and (drm-device-added card0 PRIMARY_DEVICE_FOR_DISPLAY=1
                or stopped udev-fallback-graphics))
          or runlevel PREVLEVEL=S)
stop on runlevel [016]
#The below lines are run as a shell script on start, and the started binaries killed on stop.
	if [ "$RUNLEVEL" = S -o "$RUNLEVEL" = 1 ]
	    # Single-user mode
	    exit 0
    exec <some binary>
end script
#This runs after stop
post-stop script
    rm /some/tmp/files/
end script


Now, before, I mentioned some common problems with these. Permissions and environmental options are often the place to start. Using start-stop-daemon you have user selection with --chuid and --chgid. This works with both init and Upstart.

Typically, the simplest troubleshooting step is to run each line of the script, or if you suspect your flow control (IF, ELSE, etc), stuff the lines up in a script of their own, with echo output and sleeps to identify and temporally separate steps.

Sometimes, calling an external bash script can be useful as well. The binaries spawned by that script will still be watched by Upstart for the purpose of stopping them. With init, this is more difficult.

One last thing to remember, both init and Upstart scripts are interpreted as sh, or shell, not bash. Look for bashisms in your script, and if you cannot live without them, call an external script, ensuring it is executed by bash. (A bashism is a procedure or command specific to bash, and not found in other shells, such as sh and ksh.)

See you next week.


Be the first to comment

Bash tricks

Posted April 2, 2013 By weetabix

These are a collection of my Bash tricks, and snippets from my .bash_history and .bashrc. I hope they come in handy for you.


There is, of course, ctrl+r that allows you to do a reverse search through your bash history. Type to match, then hit enter to put it on the command line.


The history command allows you to modify your history, or just edit .bash_history.


To add a proxy to a CLI command:

http_proxy='' wget


To find files of the form <filename>(1).flac, such as duplicate music files. (substitute .mp3 for .flac if you need)

find </in/this/path/> -regex .*([0-9]+)\.flac -exec echo {} +


!! reruns the last command issued, !<string> runs the last command that started with <string>, and pwd shows you where you are.


When you are affecting groups of files, you can use a regex, such as vi file[1234].txt to open file1.txt, file2,txt, etc. Of course, in vi, you need to use :n to move to the next file.


A command like rm file[1234].txt or rm file[1-4] will delete all file* that end in 1, 2 ,3 4. You can also do rm file[1-4]+ to catch file12 and file33. The “+” means match one or more “[1234]’s.”

This is similar to how the rename command works, though it uses perl regex instead.


Finally, one brain buster:

If I do more somefile and I intended to (or then need to) do vi somefile, in bash, you can do ^more^vi^ to make a substitution on the previous command only, to run vi somefile/


The command works in the syntax of ^find^replace^

Edited Apr 2nd. Thanks anna-.
Till next week,

Be the first to comment

I’m a git and I love it

Posted March 25, 2013 By weetabix

I use git, and I love it. I use it to hold my personal code and textual files, and I use it to share the code I use on this site, and the code pasted in some of the articles. There is, as always, a lot of contention over svn vs git vs mercurial… but that’s not the point of this article. (Though, I confess, mercurial being written in python tells me I might enjoy fiddling with it.)

The point of this article is, I love git. Most of the time, I don’t even realize I need a repository until … well until I need one. I’m a big fan of bare repos, so thats what I’ll write about. Also, I’ll talk about multiple remote repositories, which is how I manage to keep my pre-published code separate from the github repo I have linked with this blog.

I have three1 github2 plugins3 for WordPress to supply the code to my blog, and I use wp-syntax (or try this for instructions) to colour it up using the fantastic GeSHi highlighting engine and display it.

Git is in all the major distros, and most package managers have a “git” you can install, plus you can find instructions for installing from source, so if you’re a masochist, you can install it on your droid or mini Linux box. There are also a lot of options for Windows. (You can also install git via git…..)


So, once you install git, you want a new repo, right? Well, usually, if you are like me, you want a bucket to hold your stuff, something you can organize, push and pull from multiple locations.

* Most of my notes are from Pro Git by Scott Chacon, the git book freely available on the git-scm website.

Often, online, you see repos being set up ending in .git. This is a convention I do not follow.

For a bare repo, go to or mkdir the empty directory, and run

git init --bare


Then, you can move in the files you need, or at least touch a file.

The usual commands are:


git pushgit commit (or frequently git commit -a),  git pull

man git and git --help are also of some use.


Now, once you get the push-pull working for ya, there’s some additional tricks, like your config file in /.git/config. I keep my repo on a remote server from my home, so I have both that repo, and the one I use on github to share code on here.


Mine looks like:

	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[remote "origin"]
	fetch = +refs/heads/*:refs/remotes/origin/*
	url = ssh://<user>@<myserver>/repos/git/weetabytes
[branch "master"]
	remote = origin
	merge = refs/heads/master
[remote "github"]
	fetch = +refs/heads/*:refs/remotes/origin/*
	url = ssh://
[remote "all"]
	url = ssh://<user>@<myserver>/repos/git/weetabytes
	url = ssh://

This allows me to do fun stuff, like, after a commit, doing git push github, or git push master, if I only want to update one place, I can do git push all to push to all repos, or I can do git push origin github to make sure both remote repos are up to speed with each other, without my copy being included.

If you are sharing to mostly Linux users, SSH is awesome to use. If you have a large amount of Windows users, http/s allows you to get away without using another program like plink.

Talk to you later,



Be the first to comment

Streaming and Transcoding

Posted March 18, 2013 By weetabix

If you are like me, you have all kinds of gadgets in the house…

We have a PS3, an Xbox360, 2 android phones, a tablet, 2 laptops, and a TV. I think I have a PopcornHour and a Western Digital media player.

Like me, you could also have a ton of digital media, and you want to watch or listen to it wherever you are.

What if I want to watch one thing in the kitchen, while Sean watches something else on the TV? There are so many posts on this, on the Internet, and so many of them have lead me down a dead end, that I just hacked (as in with a machete!) a new path.

Right now, I want to talk about playing and transcoding media. In a future post I will talk about acquiring digital media, ripping and downloading legal content.

First of all, you have to decide what you want to use to play your media the most. For me, it was the PS3, but now it’s the Xbox.

Now, managing my media is one of those places I will *definitely* spend a few dollars on good software, but what I describe in here can mostly be accomplished for free. The software I ended up with, after years of using PS3MediaServer (Which is still awesome software if you just want to stream to consoles!) is called Serviio. There are two flavours, free and paid. I chose to buy the additional functionality, but for in-home use only, the free version is just fine.

To be clear, up front, the part I paid for was the ability to transcode and broadcast my music and video to my Android and to browsers *outside* of my home network using a program called MediaBrowser. If you do not pay, and use the free version of Serviio (the software) you can still do all the viewing you want via DLNA/UPNP (That would be your Xbox, PS3, and any DLNA enabled players on your network.)

DLNA/UPNP, of course, allows you to stream media to your consoles. The awesome thing about the MediaBrowser part, is (and this is a true story!) if you find yourself at a houseparty with tepid music, as long as its playing through a computer (and lets face it, that happens often at houseparties!) you can crank up your own playlist of tunes to heat things up.

On the free side, Serviio does thumbnailing of movies, transcodes MKV, AVI, WMV, TS, etc. fast and flawless (though FF/RW on MKV doesn’t always work, natch.) It plays to just about anything, and can keep surround data or downmix to stereo.

In order to get some of the nonDLNA streaming to work (and to improve the DLNA streaming) I recompiled the FFMpeg version they include, to make it my system default. I used their provided FFMpeg 1.1.1 download from this page, and compiled it according to these instructions, substituting my 1.1.1 version for their latest branch. (I did perform the optional lavf portion.)

One bonus to this, is the ability to double-click a .ts file and play it in FFMpeg + VLC, instead of using tsmuxer + VLC.


Be the first to comment

Windows GOG games and Ubuntu

Posted March 11, 2013 By weetabix

I don’t know about you, but I love playing old DOS games… Especially ones like Masters of Orion 2. Usually, they are relatively easy to use. It goes something like:

  • Install DosBox or go to GOG (Thats short for “Good old games!”)
  • Get a game, or download the installer.
  • Run it in windows (or install it with wine or CrossOver in Linux), or the game is installed inside DosBox (a DOS emulator) and boom, instant funtime.


If you want to network a few of these games together, you usually all need a windows machine so the DosBox networking works… When you install the GOG Windows DosBox under wine in Linux however, the networking is being emulated twice, and tends to fail.


Well, I figured out an easy solution, though it can be a bit heavy on disk space if you aren’t already using wine.


1. On your box, install DosBox and wine. (You can use the DosBox PPA like I do, or the Ubuntu repo)


2. Run the GOG installer via wine (I do this mostly for the benefit of not having to “install” the files by hand.)


3. After installation, browse your application menu and select wine => Browse C: Drive
Navigate to the game’s folder (usually something like ~/.wine/dosdevices/c:/ )


4. Locate your appropriate SERVER or CLIENT conf file (for Masters of Orion 2, the config file is dosboxMOO2_server.conf)
Open it up (we want to make a couple changes to help troubleshoot issues. Windows and Linux DosBox are almost, but not quite, the same.)
Change fullscreen=true to fullscreen=false so we can see the warnings scroll up in the terminal.


5. Now, in the terminal in that game folder, do:


dosbox -conf dosboxGAME_server.conf (replace with the appropriate name)


6. Now observe the error messages on the terminal (the game will probably start just fine.)
The changes should be pretty obvious in the errors. You might want to change:


machine=vga to machine=svga_s3
output=ddraw|direct3d to output=surface|overlay|opengl  (“surface” wont allow you to resize or rescale.)


7. Make sure your IPXNET STARTSERVER command uses a port above 1024 to avoid the privileged port denial issue.


8. Return fullscreen=false to fullscreen=true to enjoy that awesome fullscreen gaming.


As a final note, the DosBox .conf files have a ton of options


Happy Gaming!



Be the first to comment

Terminal from anywhere

Posted March 4, 2013 By weetabix

Do you spend a lot of time in a terminal? I do… easily 80% of my “productive” time (e.g. not watching Stargate) is gazing into xfwm4-terminal.

Lets look at some scenarios: (I assume you want to ssh in more than once a week.)

Let’s imagine I’m away from home, at a buddy’s, and all he has is Windows. Simple right? Download PuTTY! (or bring it on a USB key. There is a portable version) Or wait, is there a better option? Yes, which brings me to my next scenario:

Suppose this Windows machine is at… say, your local electronics store, or  you simply cannot download a program for whatever reason?

Well, as long as you have Internet access to your server, you can use ShellInABox! This awesome server side software allows you a mostly functional ANSI/VT100 or Wyse 60 terminal.

The man page is fairly straightforward. You install the product to your webserver, and point a specific port to it, and set it to run as either a background service (I recommend this), or as an on-demand CGI (which uses less resources, but has to spin up. If your box is thrashing, this can take a while.)

Alright, you say… What about on my phone?

Well, if you have an iPhone, I don’t. Sorreh…

As for Android, there is a bevy of awesome software.  I currently use ConnectBot and irssi ConnectBot (since I chat on IRC with Irssi, I find it makes sense to have a terminal client that allows me to use the control keys within Irssi without butchering my usual xterm controls.)

Be the first to comment

First Post!

Posted February 25, 2013 By weetabix



This blog is intended to be a journal of my adventures in hobbytime, from python to Android, Usenet to beef stew. It is my goal to do one post a week, every Monday, for one year. My inspiration for doing this is and the delicious food to be found there.


Since this is my first post, I want to talk about, uh, what i want to talk about. (How’s that for meta?) I will be writing about:

  • Asterisk, FreePBX, and the Xbox360
  • Burgers!
  • Eggdrops! (The bot, not the soup!)
  • BOFH
  • python and vCloud


So, lets start with a teaser:

Heres a nice chunk of my python, for talking to the vCloud API. If you dont know what that means, you can still oogle my python. (Yes, I said that.)

I needed to find a way to list what was running and/or deployed but not running (and still costing us money), so, using “Requests” and “BeautifulSoup4”, two fantastic python modules, and the ingenuity of some co-workers, I came out with:


# Ryan Weetabix Parkyn 2013 
# Free to use with attribution.
import requests
import re
from bs4 import BeautifulSoup
lstfile = open('C:/dir/to/where/you/want/files/', 'w') # Location of where to write data
def main():
    headers = {'Accept': 'application/*+xml;version=1.5', 'Host': 'my fqdn'}
    params = {'pageSize': '150', 'type': 'vApp', 'format': 'idrecords'}
    auth = ('username@vdc', 'password')
    proxies = {"https": "https://proxyURL:80/"} #remove proxies=proxies if no proxy
    s = requests.Session()
    r ='', proxies=proxies, headers=headers, \
               verify=True, auth=auth)
    s.headers.update({'x-vcloud-authorization': r.headers['x-vcloud-authorization']})
    t = s.get('', proxies=proxies, headers=headers, \
              verify=True, params=params)
    for line in t.iter_lines():
        if line:
def parse(text=""):
    soup = BeautifulSoup(text, "xml")
    tags = soup.find_all('VAppRecord')
    for tag in tags:
        toprint = []
        for attr in ['name', 'networkName', 'id', 'taskStatusName', 'memoryAllocationMB']:
            if attr in tag.attrs:
                toprint.append((attr, tag[attr]))
        nsname = re.sub(" ", "_", toprint[0][1]) #remove spaces from names

Click on the link at the top of the code block (“/weetabix/weetabytes/blob/master/”) to download it as a text file.

It logs into the API, saves your token, and lists your running machines. Changing line 23 “parse(line)” to “print line” will show you the raw XML, and allow you to see all the categories of information being filtered out by the parser. Some of this may be more useful to you than me.

EDIT: I moved the code to GitHub, and have added a link at the top of the code blocks.

My repo is :

See you next Monday.


2 Comments so far. Join the Conversation