escapegoat.org - Home

Some thoughts on software design

2008-12-28T16:41:00Z

We’re still hard at work on Hyperactive, a content management system for independent news production, and we’ve started to get a lot more people working on the software. So, at this point it seems to make sense to lay out some software design guidelines. Since we’re building software with a non-corporate purpose, let’s think outside of the normal scope of corporate “agile” manifestos (“delivering more value to your client”) and take some design inspiration from a different sphere: Soviet assault rifles.

Despite the rather poor reputation of Soviet technology these days, those guys had some stuff figured out when it came to design. Consider the AK-47 assault rifle designed by Mikhail Kalashnikov. It’s simple, and rugged. It takes a beating, refusing to jam even when it’s been exposed to mud, dirt, or hasn’t been maintained properly. The trigger guard is huge, so it can be fired even by gloved users in Arctic conditions. It can be maintained, and even manufactured by, people with access to simple machine tools.

A news website and an assault rifle might seem like completely unrelated areas of systems design, but let’s think about users for a moment. They’re trying to produce news, sometimes under stressful conditions. Maybe they just got in off the street after getting tear-gassed. Maybe they’re tired. Maybe they’ve got 20 things to do and their boyfriend has just been arrested. These people don’t want to screw around with anything too complicated. They want something that works.

This also goes for another set of users: the people who install and maintain the software when it’s running on production servers. The lower the number of software dependencies, and the less screwing around when it comes to installing, maintaining and upgrading the software, the better.

So, when you’re working on software, and you’re faced with a design choice, ask yourself the simple question:

“What would Kalashnikov do?”

Installing Merb

2008-07-17T14:21:00Z

I took a crack at installing Merb today, just to see how it works out. It’s a very minimal Ruby framework which in my (admittedly not very scientific) testing appears to be a lot faster than Rails, especially under conditions of high concurrency.

To get it running, I followed the instructions at the Merb book which is currently a work in progress at http://4ninjas.org. A quick tip: due to a dependency on extlib 0.9.3, the sake edgy technique didn’t work for me at first. I got it all running happily by doing the following.

First, install the Git source-code management tool and the Debian build tools if you don’t already have them:

sudo apt-get install build-essential git-core

sudo gem install rack mongrel json erubis mime-types rspec hpricot mocha rubigen haml markaby mailfactory  english addressable templater

Update: in the last few weeks gem dependencies have changed and you need to ensure you’ve got some specific versions available:

sudo gem install ruby2ruby --version=1.1.8

sudo gem install ParseTree --version=2.1.1

You’ll also need to ensure that you’ve got the MySql headers available in order to build the datamapper MySql libraries.

sudo apt-get install libmysqlclient15off libmysqlclient15-dev

git clone git://github.com/sam/extlib.git  
git clone git://github.com/sam/do.git

cd extlib
rake install ; cd ..
cd do
cd data_objects
rake install ; cd ..
cd do_mysql  # || do_postgres || do_sqlite3
rake install

For whatever reason, DataMapper 0.9.4 has a dependency on Merb-core 0.9.3, so you’ll need to install it and then proceed to build the newest DataMapper:

sudo gem install merb-core

git clone git://github.com/sam/dm-core.git
git clone git://github.com/sam/dm-more.git

cd dm-core ; rake install ; cd ..
cd dm-more
rake install; cd ..

After that stuff, the

sudo gem install sake
sake -i 'http://edgy.4ninjas.org/edgy.sake'
sake edgy:install packages="merb-stack"

commands worked just fine for me.

Bonus: install CouchDb, the wicked distributed database system which is currently an Apache incubator project.

sudo apt-get install build-essential erlang libicu38 libicu-dev libmozjs-dev
wget http://www.apache.org/dist/incubator/couchdb/0.8.0-incubating/apache-couchdb-0.8.0-incubating.tar.gz
tar -xvzf apache-couchdb-0.8.0-incubating.tar.gz 
cd apache-couchdb-0.8.0-incubating/
./configure
make && sudo make install

You can set up the CouchDb datastore as a service, with its own user, like this (thanks to these instructions, slightly modified to avoid the creation of a “couchdb” home directory):

sudo useradd couchdb
sudo mkdir -p /usr/local/var/lib/couchdb
sudo chown -R couchdb /usr/local/var/lib/couchdb
sudo mkdir -p /usr/local/var/log/couchdb
sudo chown -R couchdb /usr/local/var/log/couchdb
sudo mkdir -p /usr/local/var/run
sudo chown -R couchdb /usr/local/var/run

sudo cp /usr/local/etc/init.d/couchdb /etc/init.d/
sudo update-rc.d couchdb defaults
sudo /etc/init.d/couchdb start

After that, go to http://localhost:5984/_utils/index.html and you can administer your new distributed datastore. Of course, you could just ignore CouchDb and use Merb with a more “normal” database server like Mysql or Postgres.

If any of the version dependencies change again please leave a comment and I’ll update these instructions until Merb 0.9.4 is a little easier to get.

Mod_rails and full page caching with a custom cache location

2008-07-13T12:25:00Z

We’ve recently moved a couple of Hyperactive sites to run on mod_rails, a recently released Apache module which allows much easier deployment of Rails sites which run on a single box. Using it gives the nice happy glow that you get from using Apache’s PHP module – dump your project code into a directory accessible by Apache and it executes. This contrasts with the more complex process of setting up and maintaining a Mongrel cluster using mod_proxy_balancer (although that option still has a lot to recommend it, especially if you want to run a site across multiple servers in your data centre).

Everything works very well so far, but I did run into one problem. If you’re using full page caching in Rails, mod_rails automatically finds your cache files and tells Apache to render them from disk instead of hitting the Rails stack if a cache for a given URL. However, it only finds the cache files if they’re in the default location (RAILS_ROOT/public). Personally I find the default location a bit messy, since the cache files get mixed in with all the other stuff in your /public directory – so I tend to put cache files at RAILS_ROOT/public/system/cache to keep things clean.

config.action_controller.page_cache_directory = RAILS_ROOT + "/public/system/cache/"

The trouble was that mod_rails didn’t pick up these cache files by default, so I went back to Apache rewrite rules. I dumped in my old rewrite rules from my Mongrel configuration, and everything seemed to be working. Cached pages were served statically, due to this rewrite rule:

# Rewrite to check for Rails cached page
RewriteRule ^([^.]+)$ /system/cache/$1.html [QSA]

I haven’t seen anyone else report this problem, but I had one bit of trouble. My application obviously uses a POST request rather than a GET request to send an update to the server, but the route is otherwise the same. Attempting to edit a piece of content on the site resulted in Apache intercepting the call and just showing the content again, bypassing the “update” method call and making the site’s content impossible to edit. I have no idea why this is the case with mod_rails but didn’t happen with Mongrel, but whatever – I’ve just added the following rewrite condition to my Apache configuration and it seems to work fine:

# Rewrite to check for Rails cached page
RewriteCond %{THE_REQUEST} !^POST
RewriteRule ^([^.]+)$ /system/cache/$1.html [QSA]

This means that Apache won’t attempt to pull the file from the cache for any POST requests (like updates), making it all work again.

Hyperactive::SSL, Action Alerts

2008-06-28T11:20:00Z

Things are moving along a bit more slowly than I’d like but still steadily. The code is now nearly ready for a 0.1 release. This week’s improvements include:

The use of the SSL Requirement plugin so that the login, admin area, and publishing functions of the site all require SSL. Attempts to use the site in a production situation without an SSL cert will currently fail.
There is now an “Action Alerts” content type which allows site administrators to publish short messages at the very top of the content on the front page (and exports a list of action alerts as RSS too). This can be used in a “breaking news” situation.

There has been some talk of Twitter integration for the action alerts, I am still thinking about how to acheive that without making the site absolutely require Twitter or any other external commercial service.

When i’ve gone over the code somewhat in a cleanup session, improved the documentation a bit, and packaged a release, I’ll write up a blog post to let people know that a first release is available.

Hyperactive :: HTML Editing Facilities

2008-06-10T23:50:00Z

The latest round of development (a.k.a “weekend”) was focused mostly on providing good HTML editing features. This is somewhat trickier than it seems, because it can potentially open up a load of security holes. The site now uses the tiny mce editor, and some special Rails plugins ( white_list and sanitize_params ) to make it safe.

If you know what an XSS attack is, please take a crack at attacking the site and hassle us in #hyperactive on irc.indymedia.org (or just leave a comment on the site) if any of your attacks make it through the filters. I’ve already tried all of the attacks on the XSS Cheat Sheet and none of them succeeded.

Hyperactive :: Torrent Features and Minor Fixes

2008-05-27T10:33:00Z

Here’s a quick report on the latest code developments. As always, the big need we have is for people who can help out with CSS, as the site still needs somebody knowledgeable to make it look better.

Anway, on the weekend we managed to get the following tasks done:

Set up the London site on a new server.

Added a configuration directive allowing sites to use their own local stylesheets and images, so now London and Denmark can look different to each other.

Comment editing for admins is now working, this was a request by Indy DK as they needed to remove some borderline stuff from a comment.

Multiple event hiding is now working (bug fix), so all repeating events in a series can be hidden with one click. This should help out with calendar spam in a big way.

Started work on an integrated system for controlling Bittorrent video downloads from within the site. Torrent creation and control functions work, I just need to make a user interface for it – basically the reason for this one is that torrents didn’t seem to stay running properly, so I added a new process to manage them. Currently it doesn’t look like anything (and isn’t fully working) but when it’s done we should be able to turn torrents on and off for different videos right from within the admin interface, and the torrent downloads should be much more stable.

Added more documentation on the installation page.

Added more tickets to the wish-list of features so we don’t forget anybody’s ideas when they get contributed – especially the mobile ones.

I am concentrating on making everything work super-solidly before adding lots of new features, so the next moves will be to get an integrated HTML editor fully working within the site (it’s currently half-done), and probably do more server config (especially SSL for publishing). As far as I know, there are no major bugs currently affecting the site (somebody please let me know if I’ve got this wrong!).

Hyperactive :: Comment Love

2008-05-10T00:18:00Z

The London indymedia collective recently came up with an interesting idea: “self-administration of comments” . The idea is now working on the development site.

To summarize:

1) Non-admin users can hide comments on their own stuff:

The site allows people to register accounts if they want to (anonymous publishing still works like it always did). If you publish an article, event, or video while you’re logged in, and someone comments on it, you are allowed to hide the comments on your own article, event, or video. If anybody can think of a nice way to encourage people to be responsible for hiding crap comments on their own content, in line with the editorial policy, that’d be cool. If it becomes a problem for some reason I can probably think up some way to bar people from administering their own comments on an individual basis.

2) The site now allows users a choice regarding whether to allow comments:

If you take a look at the publish form for an article, you will see that underneath the body text entry there’s a checkbox with the following explanation:

“You have the option of allowing comments. Untick this box to disable comments.”

So, users can now decide whether they want comments on their stuff (it’s ticked by default).

3) There’s a “latest comments” list in admin.

In other comment-related news, the admin system now allows admins to see a list of the latest comments on the site. Clicking a link on the comment allows the admin to go see the comment in context.

Thanks for the suggestions, keep the ideas coming,

Hyperactive :: First Live Installation

2008-04-10T11:13:00Z

My blood pressure shot through the roof today when I rolled into work to find out that the hep-cats at Indymedia Denmark had switched their Hyperactive installation out of testing and into production. Congratulations to them on the launch of the site, and we can admire them for having the courage to run on such crazy, untested code!

London Indymedia will be going mental with jealousy in short order.

Hyperactive :: Caching, Localization and Comments

2008-04-08T10:22:00Z

Another round of development on Hyperactive is nearly complete. In this round of features, we’ve concentrated on getting the code ready for production use. This has consisted mostly of the following tasks:

caching
localization
the ability to leave comments

caching
localization
the ability to leave comments

Caching

Since Ruby is a slow language (it takes a long time to do stuff compared to most other languages) and Rails isn’t a particularly speedy framework, we don’t want to be pulling all the information to construct a page out of the database and building the page every time somebody requests it. The site needs to be fast enough to not fold up under heavy load.

So, the code has been set up to use the fastest possible solution to this set of problems – the full-page caching mechanism in Rails. For any content show page (i.e. show an article, show an event, or show a video), and for the front page of the site, the page is assembled from the database and rendered as HTML the first time it’s requested. This rendered page gets cached on disk. The next time somebody requests the same page, instead of firing up Rails and hitting the database and doing all the calculations necessary to show the page, the Apache server looks in its cache and says “oh, yeah, I’ve got this one, here you go” – which is maybe 100 times faster than assembling the page dynamically. This means that more people can be looking at the site at any given time without the server flipping out.

As with anything, there are a few trade-offs involved. For one thing, the fact that we’re caching the full page rather than just parts of it means that the page needs to be the same for everybody. This might not seem like a big deal at first, but it means we can’t (easily) have stuff like “Now logged in as Noam”, or show different controls on the page based on whether a user is an admin or not. If we did, the user “Marcos” might log into the site, take a look at a page which Noam had previously viewed, see the message “Now logged in as Noam”, and suffer a major identity crisis (“Why am I up here in the mountains? MIT seems so much more comfortable at this time of year.”). The repercussions could be too much to bear, especially if the situation was reversed and Chomsky tried to pick up an AK-47 and head for the mountains. None of us wants to be responsible for this.

The way we’ve solved the problem is to put things like administrative controls into the page as AJAX calls. When viewing an article, if you click on the “inappropriate article?” link, the page asks the server to grab the proper controls for your user role – if you have the ability to hide content, you see the “hide content” controls, if you don’t have the ability to hide content, you see the “report this content” controls. Since this part of the page comes back as the result of a fresh request to the server, we can check the user’s role when the request takes place, which gets around the problem of the cached page.

We still need to do caching for the list pages, and a few other pages, but the front page and the content pages seemed like the best place to start.

Localization

The site can now be fully localized into any UTF-8 language. We’ve used the click-to-globalize plugin together with the globalize plugin to provide translation services. Basically, in order to translate the site’s user interface into your language, all you need to do is log in as a translator, and click directly on the piece of text you want to translate. When you do this, an in-place editor appears on the page, you type in the translated text, and hit a “save” button. This saves the translated text in the database. You can see this in action in the “text area transformation” screencast video on the bottom of the click-to-globalize page.

Currently, any site should only be in one language, but this should make it really easy to translate the site’s user interface into multiple languages and have them all available (so a site could be in French and English, let’s say). Another thing which would be relatively easy to implement would be a translation interface directly in the site, so that site content can also be translated and shown automatically if it exists for a given piece of content. This should make it easy to build multilingual sites.

Comments

Up until now, the site hasn’t had the ability for users to leave comments. Over the past two weeks we’ve built comment functionality into the site, so it’s now possible to leave comments on articles (and soon events and videos).

We’re thinking about having features like the person who created the content having control over comment administration, but we’ll have to ask around and see what people think of that.

Summary

The site is nearly ready for production deployment, there are a bunch of very small things which need to be fixed up so that it can go live at Indymedia Denmark – areas of the interface which still need translation, the ability to un-hide comments, and a short list of other stuff.

Hyperactive :: Video Improvements

2008-03-26T13:57:00Z

We’ve been hard at work on the Hyperactive codebase for the last few months, it’s probably time to post an update here. This release was code-named “Planet Male Madness” in tribute to one of the greats in the Indymedia movement.

Recent development effort has been spent mostly on video. If you take a look at a video page on the development site you’ll notice that there are lots of new features available.

Related content is automatically shown on the right hand side. Related content could be other videos, articles, or events. It’s generated automatically by the search engine software embedded in the site. You can see an example of a vid with related content here.

featured videos are shown on the right hand side

recent videos are shown on the right hand side

if the video was uploaded inside of an article or event, that article or event is displayed.

the flash video player now does full-screen mode if the user’s flash player supports it

all uploaded video is currently retained in its original format, and transcoded to Ogg Theora standard as well. I know that some people have strong feelings about codec choices, this will require more discussion.

clicking on “How to download” reveals instructions regarding how to download the video via Bittorrent peer-to-peer software, and how to subscribe to the site’s video feeds. I would really appreciate it if people could read this text and offer suggestions on how to improve it.

Incidentally, this is the way I’d like to handle help text throughout the site – we can put that little ”?” icon on a page and have any necessary help text get revealed right where the user is.

the hiding controls have now been improved so that the show up as “inappropriate video?”. If you have permission to hide content, this reveals a “hide this” form when clicked. If you don’t have permission to hide content, it reveals a “report this as inappropriate” form when clicked. The contents of the form are automatically emailed to admins, and content is hidden immediately if the user has hiding permission.

the site now implements the Transmission metadata feed standard, which means that people can now subscribe to screening-quality video and the bandwidth is shared among site users (we don’t pay for most of it).

Users vs Groups Smackdown

2007-12-09T18:16:00Z

One of the things that people seem to want out of a new Indymedia CMS is the ability to have an expanded user login system, with multiple levels of permissions and the ability to edit your own content. I’ve been thinking all day about the fact that user logins can visibly tie people to content, and also do so at the database level.

The good things about an expanded login system are obvious – there is a lot more potential for many people to administer the site, less hassle for admins who always end up doing the “can you fix the typo in my article?” requests, etc. However, there are a few problems as well.

First, the act of publishing anonymously on the newswire arguably leads to a collective ethic rather than an individualist one (at least in theory, and on a good day). That is, the newswire is a space where a whole bunch of people can contribute, and it’s not about them as individuals, or even about their online personas. This makes the newswire of a well-maintained and active Indymedia site qualitatively very different from a blog, even a blog contributed to by multiple users. Whereas the Indy wire ideally showcases organized groups of people taking collective action, a blog usually ends up being a “me, me, me, look how great I am” showcase of a bunch of individuals.

Second, there are questions of security. Tying published content (articles, videos, etc) to a user can be considered a good thing if we’re not interested in security – if someone publishes something that’s really of excellent quality, I’d like to be able to conveniently access all of their other stuff. However, if the server gets seized, the foreign key between the content and the user is gonna be there. Although we wouldn’t store email addresses or other identifying info in the system anyway (and there would always be the option of anonymous publishing), it could be a pretty easy job for the cops to figure out who a user is, given a sufficient sample set of articles and enough on-the-street surveillance.

So I was thinking to tie the content to a group instead. Then if there is more than one person operating as part of the group, there is at least some legal deniability (which improves the bigger the group is). I would propose having individual logins, but all the functionality of the site would be keyed on groups rather than individuals. So instead of showing “all articles by UserX” the site would show things like “All articles by london noborders”. Group admins would i guess be in charge of accepting new people into a group. It could be a good way to encourage organization and collaboration between media producers, and it could also help encourage media production by political groups.

Installing ffmpeg-php on ubuntu 7.10

2007-11-11T15:33:00Z

A bunch of us have begun working on a new codebase for Indymedia, based on CakePHP. I’m not exactly Captain PHP, so there’s going to be a bit of a learning curve for me, but we’ll see how it goes.

One of the things I am interested in setting up is some kind of FLV video conversion system for the new CMS. There’s a nice pleasant-looking PHP library to deal with the FFmpeg video conversion utility on the server side (is anybody in the Rails world noticing this?), so here’s my experience installing it on Ubuntu.

First things first. Enable the universe repositories, and then install ffmpeg. Keep in mind that the ffmpeg binaries in Ubuntu are built without the necessary MP3 support (for patent reasons) to actually encode sound in FLV files. In addition, if you install the “normal” FFMpeg binaries from the Ubuntu repositories, you won’t get the development headers necessary to compile the ffmpeg-php code. So, let’s do things the hard way and compile FFMpeg from source, so we have everything working.

I’ll outline the basic steps here. First, you need the mp3 libraries:

apt-get install liblame-dev

Next, you need to get ready to build FFMpeg from source:

sudo apt-get install liblame-dev libfaad2-dev libfaac-dev libxvidcore4-dev liba52-0.7.4 \
liba52-0.7.4-dev libx264-dev libdts-dev libgsm1-dev libvorbis-dev libdc1394-13-dev \
checkinstall build-essential gcc

Get the FFMpeg source:

cd /usr/local/src
sudo apt-get source ffmpeg

cd ffmpeg-*
sudo ./configure --enable-gpl --enable-pp --enable-libvorbis --enable-libogg \
--enable-liba52 --enable-libdts --enable-dc1394 --enable-libgsm --disable-debug \
--enable-libmp3lame --enable-libfaad --enable-libfaac --enable-xvid --enable-pthreads \
--enable-x264 --enable-shared --prefix=/usr

The second-last option,—enable-shared, should allow the php5 script to find the FFMpeg shared libraries so that ffmpeg-php can compile properly.


sudo make
sudo checkinstall

When you do the checkinstall, it’ll ask you a bunch of questions so that it can build you a .deb package and install it. For most of them you can just accept the defaults. I set the package description to FFMpeg video conversion utility, and I set the version to 3:0.cvs20070308-5ubuntu4. This makes the package you’re making slightly newer than the one in the regular Ubuntu repositories, so Ubuntu won’t hassle you all the time, taunting you with a newer version of the package when you log in.

Ok, that was a big hassle, I know. But it should get you video encoding capability.

The next thing I did was download the latest release of ffmpeg-php from http://ffmpeg-php.sourceforge.net/ and extracted it on my machine. The first thing you need to do at that point is to run a utility called “phpize” in order to set up some configuration files for your machine. However, Ubuntu doesn’t include phpize in the main php package, so you’ll need to do this first:

apt-get install php5-dev

Then cd into the ffmpeg-php-0.X.X directory (maybe you extracted it on your desktop?), and run the following command to generate a config script:

phpize5

You can then do this to build the library:

./configure && make

And finally this to install it:

sudo make install

Now you’ve got ffmpeg support for PHP, and an FFMpeg binary installed that will do its stuff with sound encoding capabilities.

Your own live tv feed on the interweb

2007-10-14T15:57:00Z

I did some testing earlier and I actually got a video stream out onto the net and was watching it within about 20 minutes. My head exploded. It was really easy to do.

I was able to view the stream on Linux and Mac OS X using VLC. I was streaming and viewing on the Linux box (a 2.8 Ghz laptop, 512 MB RAM). The CPU sat at about 70% and the machine didn’t look like anything bad was going to happen, I left it running for about 45 minutes and it all seemed cool. The data rates were I think 70 kbps video and 60 kbps audio, which was about 1/4 of my 802.11b wireless card’s bandwidth. So this should work in most wireless situations if you’ve got a decent wifi connection. It was not using very much RAM, it didn’t seem. Hardware requirements should be fairly modest, maybe a 1Ghz machine would do it.

I did some testing earlier and I actually got a video stream out onto the net and was watching it within about 20 minutes. My head exploded. It was really easy to do.

At the same time as I was compressing and uploading the stream to http://radio.indymedia.org, I also tried watching it on a different computer.

I fired up VLC (downloadable for Linux, Mac, and Windows from http://www.videolan.org ) and used the “Transcode Wizard” or something to record the video to disk on a G4 Mac. I could view the stream as it was saved, and playback of the saved file worked perfectly as well, so in theory, we could use VLC on a separate box to record the stream, and upload it to the UK newswire or to http://video.indymedia.org as soon as the streaming event is finished. I am not totally sure but I think this laptop would probably be able simultaneously compress the stream and record it to disk using VLC (but if we can use a separate machine to do the recording it’s probably safer).

Below is what I did to set up the box, there are much more detailed instructions at:

http://24.72.34.35/cgi-bin/twiki/view/Main/LinuxVideoFirewire

The Setup

Basically, I just had to make sure the following packages were installed. I’m using Ubuntu, but this should work pretty much the same on Debian (keep in mind that on Ubuntu you would preface the commands below with “sudo”. On Debian you’d need to run them as root).

apt-get install ffmpeg2theora dvgrab

(Note that I already had ffmpeg installed on my machine, I think that’s also necessary).

The next thing I did was download a program called oggfwd. It’s available from http://www.v2v.cc/~j/ffmpeg2theora/oggfwd/oggfwd.linux.bin

Then I decided I trusted the people who made it, and typed this into the terminal to copy the file to a place where I could run it:

mv oggfwd.linux.bin /usr/local/bin/oggfwd

chmod 755 /usr/local/bin/oggfwd

Then, I plugged the camera in and made sure that the system could capture stuff from the firewire port. To do this, I did:

modprobe raw1394

This made sure the Linux kernel loaded up the proper modules to read Firewire input. Then I did a:

chmod 777 /dev/raw1394

This gives all users read/write/execute permissions on the Firewire port. Not wickedly secure on a server machine but good enough for this situation I think, and it gets reset when the machine is restarted.

At that point, I issued the following command (as root) and it connected to the Indy radio server and I was serving a video stream from my house:

dvgrab --format raw - | ffmpeg2theora -a 0 -v 5 -f dv -x 320 -y 240 -o /dev/stdout  - | oggfwd radio.indymedia.org 8000 thisisnotreallythepassword /rampART.ogg

I am not sure whether it is necessary to be root to do this, we can experiment with permissions a bit.

That’s it, you’ve got your own live video feed, accessible anywhere in the world!

Installing Rmagick with Ruby on Debian

2007-10-05T14:31:00Z

A lot of the time, your web apps need to chop up, resize, or otherwise mangle images. One program that can do this on the server is ImageMagick. Here's how to get ImageMagick to work with Ruby on Rails on a Debian or Ubuntu box. A lot of the time, your web apps need to chop up, resize, or otherwise mangle images. One program that can do this on the server is ImageMagick. Here's how to get ImageMagick to work with Ruby on Rails on a Debian or Ubuntu box. If you haven't already done this when you installed Rails, make sure you install Debian's build-essential package so that you can compile code: sudo apt-get install build-essential Then: sudo apt-get install imagemagick sudo apt-get install librmagick-ruby sudo apt-get install libmagick9-dev Update: I've had no success yet when trying to install the latest version of rmagick on Debian machines, although basically I just haven't looked into it very much as the older versions work just fine for me. So, I just use rmagick 1.15.0 instead of the newest (2.3.0) gem. sudo gem install rmagick -v 1.15.0 Note that libmagick9-dev may have a different version number depending on how old your distribution is (that one is found on Ubuntu 7.04 Feisty). Older version numbers might include libmagick9-dev. You should be able to do something like

apt-cache search libmagick

to find out which version is needed on your system.

Noborders indymedia center tech review

2007-09-22T18:57:00Z

From 16 – 24 September 2007 the Noborders network set up a camp as a base from which to demonstrate against immigration detention centers in Sussex, England. Throughout the camp, Indymedia UK ran a media centre including a dispatch system for news reporting and a public access point in the middle of a farmer’s field. In case it’s useful to anybody doing something similar, here’s how we set it up.

Connectivity

The biggest technical worry was the question of internet connectivity. In previous camps in the UK, we’ve arranged to get the Psand bunch to help us out with their satellite gear and solar-powered laptop setup, but for this one they weren’t available.

Our only realistic option was to use 3G mobile connectivity. We had a T-Mobile web’n’walk card kicking around, and access to an account with a transfer limit of 10GB/month. We were quite worried about 3G network coverage prior to the event, especially as the location wasn’t known until the last minute (police harassment meant that the location changed two days before the camp was due to start).

Initial tests at the site revealed that although GPRS connectivity was great, the much faster UMTS network signal was unstable when using the card with its internal antenna. GPRS is the normal mobile data network, and it’s quite slow, weighing in with about twice the speed of a 56K modem on average. This wasn’t going to be particularly useful for the full-service media centre we envisioned, although we did do some experiments with Squid caching and probably could have made it work OK if we had to do so. As a backup plan, we thought about coding up a website that would work on the local area network. The idea was that people could use this local site to submit content, with submitted content being queued and uploaded to the Indymedia UK site as bandwidth became available. This turned out to be not necessary as the UMTS network worked extremely well once we picked up a UMTS antenna from Panorama antennas.

The antenna plugged into a port on the side of a Novatel Merlin XU870 3G mobile data card. It was necessary to buy an adapter wire to make the Merlin connect to the 3m antenna cable, which then connected to a Client Patch antenna. Signal was still extremely weak and unstable, but by rotating the antenna slowly through 360 degrees and doing a big download, we could watch the transfer speed and figure out which way to point it. We then duct-taped the antenna to a pole outside the media tent. Once this was done, the signal was extremely stable and quite fast. Average download speeds ranged from 50-100 KB / second, with bursts up to 130 KB / second.

We ran the uplink on a Windows XP laptop, using T-Mobile software that came with the data card to do the dial in (basically it’s just a really fancy PPP modem setup). A Linux project called umtsmon provides the same functionality in Linux but we couldn’t figure out how to connect to UMTS rather than GPRS, and didn’t have time to fuck around forever. In theory the Merlin will run perfectly under Linux, but the setup instructions for it aren’t particularly fantastic and we weren’t sure what parameters we needed to enter in order to get the card working with the T-Mobile account.

Bandwidth considerations

The account in question had a download cap of 10 GB, and we weren’t sure now much bandwidth a media center in full operation actually used. As it turned out, the first day’s data transfer total, which was the result of people doing editorial work on the imc-uk site, was only about 45 MB. This made us much more confident that we weren’t going to run the account out of its bandwidth allocation, and over the next few days it was pretty much open season on tcp/ip packets, with people downloading software, uploading video, doing system updates, etc.

Eight computers in heavy use, without any caching system at all, resulted in an average of 500MB / day. This could have been massively reduced if necessary through the use of squid caches, a full set of software installation CDs, apt-caches for software packages that weren’t available locally, etc. However, it was interesting to see what the maximum transfer was, prior to worrying about all the rest of it. Upload speeds were in the 50 KB / second range, which was perfectly acceptable for doing large video uploads.

The network

We had a total of 8 computers available, mostly laptops running Ubuntu 7.04 with a Mac G4 for video editing. Also some nice Dutch people gave us an old laptop running Debian which came in handy. There was nothing too special about most of the network setup, which included a 16-port hub and a 5-port hub as well as two wireless stations (one for each of our media centre tents). The only major trouble came when trying to do the initial network setup. Windows networking setup, with its endless dialog boxes, opaque instructions based on outdated screenshots, and almost total lack of native networking applications in XP Pro, was a nightmare. Hours of screwing around with the Internet Connection Sharing shit in XP resulted in the realization that this great OS doesn’t actually give out IP addresses in any comprehensible way. We finally got pissed off and downloaded a copy of Tftpd, a simple GPL dhcp server for Windows, and set up the network as follows:

a) we disabled the FTP and SNMP servers that run by default.

b) we set up a subnet on the 192.168.0.x IP range, and configured DHCP to start giving out addresses on 192.168.0.2, with 100 addresses in the pool (maybe a bit optimistic considering there were only ever about 500 people on the camp at the most). The router address was 192.168.0.1, and after initially having some problems we smoothed things out by also setting the gateway address and DNS address to 192.168.0.1

c) once the PPP tunnel to T-Mobile was running we had to right-click on the Merlin’s connection in the Network control panel, go to the “advanced” tab, and tick the “share this computer’s internet connection” box. This gets unticked every time the Merlin connects, but luckily the signal was really stable and this didn’t turn into a major hassle.

Closing thoughts

Things worked out incredibly well, and given an appropriate power source (in this case, a 5000w generator) we should be able to use a similar setup to run mobile public access points at any time or place of our choosing as long as a 3G network account is available. The antenna setup cost 60 quid and should work with pretty much any data card as long as the appropriate card adapter lead is available. The opportunities for mobile video and audio streaming are also quite interesting.

T-Mobile accounts come in two packages, either a 3 GB/month account with the stipulation that users don’t stream stuff (29 quid per month) or a 10 GB/month account allowing streaming (44 quid per month). There is also the option of buying pay as you go cards with a transfer limit of 45 MB within a 24-hour time period. This would be enough to run a really lean media centre with text reports, and it might be possible to do video stuff by burning through a bunch of SIMs, changing each one when it hits its 45 MB limit.