escapegoat.org

We’re still hard at work on Hyperactive, a content management system for independent news production, and we’ve started to get a lot more people working on the software. So, at this point it seems to make sense to lay out some software design guidelines. Since we’re building software with a non-corporate purpose, let’s think outside of the normal scope of corporate “agile” manifestos (“delivering more value to your client”) and take some design inspiration from a different sphere: Soviet assault rifles.

Despite the rather poor reputation of Soviet technology these days, those guys had some stuff figured out when it came to design. Consider the AK-47 assault rifle designed by Mikhail Kalashnikov. It’s simple, and rugged. It takes a beating, refusing to jam even when it’s been exposed to mud, dirt, or hasn’t been maintained properly. The trigger guard is huge, so it can be fired even by gloved users in Arctic conditions. It can be maintained, and even manufactured by, people with access to simple machine tools.

A news website and an assault rifle might seem like completely unrelated areas of systems design, but let’s think about users for a moment. They’re trying to produce news, sometimes under stressful conditions. Maybe they just got in off the street after getting tear-gassed. Maybe they’re tired. Maybe they’ve got 20 things to do and their boyfriend has just been arrested. These people don’t want to screw around with anything too complicated. They want something that works.

This also goes for another set of users: the people who install and maintain the software when it’s running on production servers. The lower the number of software dependencies, and the less screwing around when it comes to installing, maintaining and upgrading the software, the better.

So, when you’re working on software, and you’re faced with a design choice, ask yourself the simple question:

“What would Kalashnikov do?”

I took a crack at installing Merb today, just to see how it works out. It’s a very minimal Ruby framework which in my (admittedly not very scientific) testing appears to be a lot faster than Rails, especially under conditions of high concurrency.

To get it running, I followed the instructions at the Merb book which is currently a work in progress at http://4ninjas.org. A quick tip: due to a dependency on extlib 0.9.3, the sake edgy technique didn’t work for me at first. I got it all running happily by doing the following.

First, install the Git source-code management tool and the Debian build tools if you don’t already have them:

sudo apt-get install build-essential git-core

sudo gem install rack mongrel json erubis mime-types rspec hpricot mocha rubigen haml markaby mailfactory  english addressable templater

Update: in the last few weeks gem dependencies have changed and you need to ensure you’ve got some specific versions available:

sudo gem install ruby2ruby --version=1.1.8

sudo gem install ParseTree --version=2.1.1

You’ll also need to ensure that you’ve got the MySql headers available in order to build the datamapper MySql libraries.

sudo apt-get install libmysqlclient15off libmysqlclient15-dev

git clone git://github.com/sam/extlib.git  
git clone git://github.com/sam/do.git

cd extlib
rake install ; cd ..
cd do
cd data_objects
rake install ; cd ..
cd do_mysql  # || do_postgres || do_sqlite3
rake install

For whatever reason, DataMapper 0.9.4 has a dependency on Merb-core 0.9.3, so you’ll need to install it and then proceed to build the newest DataMapper:

sudo gem install merb-core

git clone git://github.com/sam/dm-core.git
git clone git://github.com/sam/dm-more.git

cd dm-core ; rake install ; cd ..
cd dm-more
rake install; cd ..

After that stuff, the

sudo gem install sake
sake -i 'http://edgy.4ninjas.org/edgy.sake'
sake edgy:install packages="merb-stack"

commands worked just fine for me.

Bonus: install CouchDb, the wicked distributed database system which is currently an Apache incubator project.

sudo apt-get install build-essential erlang libicu38 libicu-dev libmozjs-dev
wget http://www.apache.org/dist/incubator/couchdb/0.8.0-incubating/apache-couchdb-0.8.0-incubating.tar.gz
tar -xvzf apache-couchdb-0.8.0-incubating.tar.gz 
cd apache-couchdb-0.8.0-incubating/
./configure
make && sudo make install

You can set up the CouchDb datastore as a service, with its own user, like this (thanks to these instructions, slightly modified to avoid the creation of a “couchdb” home directory):

sudo useradd couchdb
sudo mkdir -p /usr/local/var/lib/couchdb
sudo chown -R couchdb /usr/local/var/lib/couchdb
sudo mkdir -p /usr/local/var/log/couchdb
sudo chown -R couchdb /usr/local/var/log/couchdb
sudo mkdir -p /usr/local/var/run
sudo chown -R couchdb /usr/local/var/run

sudo cp /usr/local/etc/init.d/couchdb /etc/init.d/
sudo update-rc.d couchdb defaults
sudo /etc/init.d/couchdb start

After that, go to http://localhost:5984/_utils/index.html and you can administer your new distributed datastore. Of course, you could just ignore CouchDb and use Merb with a more “normal” database server like Mysql or Postgres.

If any of the version dependencies change again please leave a comment and I’ll update these instructions until Merb 0.9.4 is a little easier to get.

We’ve recently moved a couple of Hyperactive sites to run on mod_rails, a recently released Apache module which allows much easier deployment of Rails sites which run on a single box. Using it gives the nice happy glow that you get from using Apache’s PHP module – dump your project code into a directory accessible by Apache and it executes. This contrasts with the more complex process of setting up and maintaining a Mongrel cluster using mod_proxy_balancer (although that option still has a lot to recommend it, especially if you want to run a site across multiple servers in your data centre).

Everything works very well so far, but I did run into one problem. If you’re using full page caching in Rails, mod_rails automatically finds your cache files and tells Apache to render them from disk instead of hitting the Rails stack if a cache for a given URL. However, it only finds the cache files if they’re in the default location (RAILS_ROOT/public). Personally I find the default location a bit messy, since the cache files get mixed in with all the other stuff in your /public directory – so I tend to put cache files at RAILS_ROOT/public/system/cache to keep things clean.

config.action_controller.page_cache_directory = RAILS_ROOT + "/public/system/cache/"

The trouble was that mod_rails didn’t pick up these cache files by default, so I went back to Apache rewrite rules. I dumped in my old rewrite rules from my Mongrel configuration, and everything seemed to be working. Cached pages were served statically, due to this rewrite rule:

# Rewrite to check for Rails cached page
RewriteRule ^([^.]+)$ /system/cache/$1.html [QSA]

I haven’t seen anyone else report this problem, but I had one bit of trouble. My application obviously uses a POST request rather than a GET request to send an update to the server, but the route is otherwise the same. Attempting to edit a piece of content on the site resulted in Apache intercepting the call and just showing the content again, bypassing the “update” method call and making the site’s content impossible to edit. I have no idea why this is the case with mod_rails but didn’t happen with Mongrel, but whatever – I’ve just added the following rewrite condition to my Apache configuration and it seems to work fine:

# Rewrite to check for Rails cached page
RewriteCond %{THE_REQUEST} !^POST
RewriteRule ^([^.]+)$ /system/cache/$1.html [QSA]

This means that Apache won’t attempt to pull the file from the cache for any POST requests (like updates), making it all work again.

Things are moving along a bit more slowly than I’d like but still steadily. The code is now nearly ready for a 0.1 release. This week’s improvements include:

• The use of the SSL Requirement plugin so that the login, admin area, and publishing functions of the site all require SSL. Attempts to use the site in a production situation without an SSL cert will currently fail.
• There is now an “Action Alerts” content type which allows site administrators to publish short messages at the very top of the content on the front page (and exports a list of action alerts as RSS too). This can be used in a “breaking news” situation.

There has been some talk of Twitter integration for the action alerts, I am still thinking about how to acheive that without making the site absolutely require Twitter or any other external commercial service.

When i’ve gone over the code somewhat in a cleanup session, improved the documentation a bit, and packaged a release, I’ll write up a blog post to let people know that a first release is available.

The latest round of development (a.k.a “weekend”) was focused mostly on providing good HTML editing features. This is somewhat trickier than it seems, because it can potentially open up a load of security holes. The site now uses the tiny mce editor, and some special Rails plugins ( white_list and sanitize_params ) to make it safe.

If you know what an XSS attack is, please take a crack at attacking the site and hassle us in #hyperactive on irc.indymedia.org (or just leave a comment on the site) if any of your attacks make it through the filters. I’ve already tried all of the attacks on the XSS Cheat Sheet and none of them succeeded.

Here’s a quick report on the latest code developments. As always, the big need we have is for people who can help out with CSS, as the site still needs somebody knowledgeable to make it look better.

Anway, on the weekend we managed to get the following tasks done:

• Set up the London site on a new server.

• Added a configuration directive allowing sites to use their own local stylesheets and images, so now London and Denmark can look different to each other.

• Comment editing for admins is now working, this was a request by Indy DK as they needed to remove some borderline stuff from a comment.

• Multiple event hiding is now working (bug fix), so all repeating events in a series can be hidden with one click. This should help out with calendar spam in a big way.

• Started work on an integrated system for controlling Bittorrent video downloads from within the site. Torrent creation and control functions work, I just need to make a user interface for it – basically the reason for this one is that torrents didn’t seem to stay running properly, so I added a new process to manage them. Currently it doesn’t look like anything (and isn’t fully working) but when it’s done we should be able to turn torrents on and off for different videos right from within the admin interface, and the torrent downloads should be much more stable.

• Added more documentation on the installation page.

• Added more tickets to the wish-list of features so we don’t forget anybody’s ideas when they get contributed – especially the mobile ones.

I am concentrating on making everything work super-solidly before adding lots of new features, so the next moves will be to get an integrated HTML editor fully working within the site (it’s currently half-done), and probably do more server config (especially SSL for publishing). As far as I know, there are no major bugs currently affecting the site (somebody please let me know if I’ve got this wrong!).

The London indymedia collective recently came up with an interesting idea: “self-administration of comments” . The idea is now working on the development site.

To summarize:

1) Non-admin users can hide comments on their own stuff:

The site allows people to register accounts if they want to (anonymous publishing still works like it always did). If you publish an article, event, or video while you’re logged in, and someone comments on it, you are allowed to hide the comments on your own article, event, or video. If anybody can think of a nice way to encourage people to be responsible for hiding crap comments on their own content, in line with the editorial policy, that’d be cool. If it becomes a problem for some reason I can probably think up some way to bar people from administering their own comments on an individual basis.

2) The site now allows users a choice regarding whether to allow comments:

If you take a look at the publish form for an article, you will see that underneath the body text entry there’s a checkbox with the following explanation:

“You have the option of allowing comments. Untick this box to disable comments.”

So, users can now decide whether they want comments on their stuff (it’s ticked by default).

3) There’s a “latest comments” list in admin.

In other comment-related news, the admin system now allows admins to see a list of the latest comments on the site. Clicking a link on the comment allows the admin to go see the comment in context.

Thanks for the suggestions, keep the ideas coming,

My blood pressure shot through the roof today when I rolled into work to find out that the hep-cats at Indymedia Denmark had switched their Hyperactive installation out of testing and into production. Congratulations to them on the launch of the site, and we can admire them for having the courage to run on such crazy, untested code!

London Indymedia will be going mental with jealousy in short order.

Another round of development on Hyperactive is nearly complete. In this round of features, we’ve concentrated on getting the code ready for production use. This has consisted mostly of the following tasks:

• caching
• localization
• the ability to leave comments

We’ve been hard at work on the Hyperactive codebase for the last few months, it’s probably time to post an update here. This release was code-named “Planet Male Madness” in tribute to one of the greats in the Indymedia movement.

Recent development effort has been spent mostly on video. If you take a look at a video page on the development site you’ll notice that there are lots of new features available.

One of the things that people seem to want out of a new Indymedia CMS is the ability to have an expanded user login system, with multiple levels of permissions and the ability to edit your own content. I’ve been thinking all day about the fact that user logins can visibly tie people to content, and also do so at the database level.

The good things about an expanded login system are obvious – there is a lot more potential for many people to administer the site, less hassle for admins who always end up doing the “can you fix the typo in my article?” requests, etc. However, there are a few problems as well.

A bunch of us have begun working on a new codebase for Indymedia, based on CakePHP. I’m not exactly Captain PHP, so there’s going to be a bit of a learning curve for me, but we’ll see how it goes.

One of the things I am interested in setting up is some kind of FLV video conversion system for the new CMS. There’s a nice pleasant-looking PHP library to deal with the FFmpeg video conversion utility on the server side (is anybody in the Rails world noticing this?), so here’s my experience installing it on Ubuntu.

I did some testing earlier and I actually got a video stream out onto the net and was watching it within about 20 minutes. My head exploded. It was really easy to do.

I was able to view the stream on Linux and Mac OS X using VLC. I was streaming and viewing on the Linux box (a 2.8 Ghz laptop, 512 MB RAM). The CPU sat at about 70% and the machine didn’t look like anything bad was going to happen, I left it running for about 45 minutes and it all seemed cool. The data rates were I think 70 kbps video and 60 kbps audio, which was about 1/4 of my 802.11b wireless card’s bandwidth. So this should work in most wireless situations if you’ve got a decent wifi connection. It was not using very much RAM, it didn’t seem. Hardware requirements should be fairly modest, maybe a 1Ghz machine would do it.

A lot of the time, your web apps need to chop up, resize, or otherwise mangle images. One program that can do this on the server is ImageMagick. Here's how to get ImageMagick to work with Ruby on Rails on a Debian or Ubuntu box.

From 16 – 24 September 2007 the Noborders network set up a camp as a base from which to demonstrate against immigration detention centers in Sussex, England. Throughout the camp, Indymedia UK ran a media centre including a dispatch system for news reporting and a public access point in the middle of a farmer’s field. In case it’s useful to anybody doing something similar, here’s how we set it up.