escapegoat.org

Hyperactive now sports video embeds, thanks to JimDog. Good work, boyo!

See this video and more at Indymedia London

After a bit of mucking around, we've got a shared git repository going on escapegoat.org with read only access over http and git, and commit access using git (without having to make shell accounts. Oh and there is a nice browsable version of the repository aswell.

Installing Packages

Debian provides most of the packages for you, so start off with:

sudo apt-get install git-core gitweb git-daemon-run gitosis

Debian Etch

Gitosis requires a version of git >= 1.5, however the version in etch is 1.4.x. So we need to enable the backports repository. So add this line to your /etc/apt/sources.list (if it is not there already):

deb http://www.backports.org/debian etch-backports main contrib non-free

Then we need to update, remove a package that doesn't like to be upgraded, and install the required packages from backports:

sudo apt-get update
sudo apt-get remove python-setuptools
sudo apt-get install -t etch-backports python-setuptools git-core gitweb \
      gitosis git-daemon-run

gitosis - allowing commits without shell accounts

Note a lot of this is cribbed from this blog post about gitosis - see that for more detailed discussion.

So let's start by adding a user account to hold the repositories.

sudo adduser \
    --system \
    --shell /bin/sh \
    --gecos 'git version control' \
    --group \
    --disabled-password \
    --home /home/git \
    git

You may change the home path to suit your taste (the git-daemon-run package assumes a home of /var/cache/git, but I ended up changing the git-daemon-run params to fit with gitosis). A successful user creation will look similar to:

Adding system user 'git'...
Adding new group 'git' (211).
Adding new user 'git' (211) with group 'git'.
Creating home directory '/home/git'.

Then you need to copy your ssh public key to your server to add yourself as the first user. (See the above blog post for some help if you don't know how to). So if you copy your ssh key to the /tmp/ directory, you can run the following command to set up gitosis:

sudo -H -u git gitosis-init < /tmp/id_rsa.pub

Success looks like:

Initialized empty Git repository in ./
Initialized empty Git repository in ./

(Yes, two times)

For good measure, let's make sure the post-update hook is set executable. It doesn't always get set (problem with older setuptools):

sudo chmod 755 /home/git/repositories/gitosis-admin.git/hooks/post-update

Here some cool magic happens. Run this on your local machine:

git clone git@YOUR_SERVER_HOSTNAME:gitosis-admin.git
cd gitosis-admin

You will now have a gitosis.conf file and keydir/ directory:

~/dev/gitosis-admin (master) $ ls -l
total 8
-rw-r--r--   1 garry  garry  104 Nov 13 05:43 gitosis.conf
drwxr-xr-x   3 garry  garry  102 Nov 13 05:43 keydir/

This repository that you just cloned contains all the files (right now, only 2) needed to create repositories for your projects, add new users, and defined access rights. Edit the settings as you wish, commit, and push. Once pushed, gitosis will immediately make your changes take effect on the server. So we're using Git to host the configuration file and keys that in turn define how our Git hosting behaves. That's just plain cool.

At this point you may aswell carry on reading the excellent blog post I've copied from a bit already. If you read "Creating new repositories" and "Adding users" there and then come back.

Final gitosis set up

Gitosis can help you work well with gitweb and git-daemon if you add the right bits to the gitosis.conf file. git-daemon can be helped by gitosis automaticaly adding the 'git-daemon-export-ok' file in the repositories. gitweb can also be helped, and you can add an owner and description (which gitweb will show) in the gitosis.conf file. So here is a sample file:

[gitosis]
gitweb = yes
daemon = yes

[group gitosis-admin]
writable = gitosis-admin
members = user1@computer1 user2@computer2

[repo gitosis-admin]
gitweb = no
daemon = no

[group hyper-team]
writable = hyperactive
members = user1@computer1 user2@computer2

[repo hyperactive]
owner = escapegoat
description = A community news and reporting system

So this configuration allows gitweb and git daemon to access all repositories by default, though we have turned it off in the gitosis-admin repository. And we have a put a description and owner into the hyperactive section.

Allowing read only access using git-daemon

git-daemon is installed from the repositories and will run automatically, however it expects the repositories to be in /var/cache/git. I changed this by editing the file /etc/service/git-daemon/run to be

#!/bin/sh
exec 2>&1
echo 'git-daemon starting.'
#exec git-daemon --verbose --base-path=/var/cache /var/cache/git
exec git-daemon --base-path=/home/git/repositories/ --export-all

After this change, and a

sudo /etc/init.d/git-daemon restart

I was able to download the repository by doing:

git clone git://git.escapegoat.org/hyperactive.git

Allowing read only access over http

To allow access over the web, ie

git clone http://git.escapegoat.org/git/hyperactive.git

we did

sudo mkdir /var/www/git.escapegoat.org/git/
sudo ln -s /home/git/repositories/hyperactive.git /var/www/git.escapegoat.org/git/

Now apache needs to be able to read the repository, and so we added the www-data user to the git group. To do this, you need to find out what groups www-data is in already and then update the list of groups. So the commands we used were:

groups www-data
sudo usermod -G www-data,svnowner,git www-data

Make sure you add any other groups www-data is already in to the list, and remove svnowner if you don't have that group. You may also need to do

cd /home/git/repositories/hyperactive.git
sudo git-update-server-info

after each checkin, but this should be done by /home/git/repositories/hyperactive.git/hooks/post-update. If not then make sure it is executable by

sudo chmod 755 /home/git/repositories/hyperactive.git/hooks/post-update

A user should now be able to do

git clone http://git.escapegoat.org/git/hyperactive.git

Browsing the repository using gitweb

gitweb is installed, and may work out of the box with a simple webserver set up, but we have several virtual hosts. So to put the gitweb.cgi file in the path we did

sudo ln -s /usr/lib/cgi-bin /var/www/git.escapegoat.org/gitweb
sudo cp /usr/share/gitweb/* /var/www/git.escapegoat.org/

(The second line copies the css and images required to make gitweb look nice). We also need to link in the repository. (And do the second and third line below if gitosis has not done it for you already).

sudo ln -s /home/git/repositories/hyperactive.git /var/cache/git/hyperactive
sudo touch /home/git/repositories/hyperactive.git/git-daemon-export-ok
sudo chown git:git /home/git/repositories/hyperactive.git/git-daemon-export-ok

And then it all seemed to work like magic. Go see for yourself at http://git.escapegoat.org/gitweb

Extra apache setup

For now I've done a little bit of extra stuff in the apache set up - specifically in /etc/apache2/sites-available/git.escapegoat.org

# redirect the / to /gitweb/ 
    RedirectMatch ^/$ /gitweb/

ScriptAlias /gitweb /usr/lib/cgi-bin/gitweb.cgi
<Location /gitweb >
            Options +ExecCGI
</Location>

So if you go to http://git.escapegoat.org/ you will be redirected to http://git.escapegoat.org/gitweb/ and be able to browse the repository in a nice way.

Sources

The two main articles I used were

• http://scie.nti.st/2007/11/14/hosting-git-repositories-the-easy-and-secure-way - this is a great guide to gitosis set up (for setting up a shared repository that can be committed to over ssh without giving all committers shell accounts).
• http://www.die-welt.net/index.php/blog/199/Notes_on_serving_Git_with_Debian - shows how to set up git, gitweb and git-daemon on Debian.

So I have taken those two guides and made them work together - standing on the shoulders of other bloggers ...

Other useful articles include this more DIY guide by the good folks at riseup.net.

We’re still hard at work on Hyperactive, a content management system for independent news production, and we’ve started to get a lot more people working on the software. So, at this point it seems to make sense to lay out some software design guidelines. Since we’re building software with a non-corporate purpose, let’s think outside of the normal scope of corporate “agile” manifestos (“delivering more value to your client”) and take some design inspiration from a different sphere: Soviet assault rifles.

Despite the rather poor reputation of Soviet technology these days, those guys had some stuff figured out when it came to design. Consider the AK-47 assault rifle designed by Mikhail Kalashnikov. It’s simple, and rugged. It takes a beating, refusing to jam even when it’s been exposed to mud, dirt, or hasn’t been maintained properly. The trigger guard is huge, so it can be fired even by gloved users in Arctic conditions. It can be maintained, and even manufactured by, people with access to simple machine tools.

A news website and an assault rifle might seem like completely unrelated areas of systems design, but let’s think about users for a moment. They’re trying to produce news, sometimes under stressful conditions. Maybe they just got in off the street after getting tear-gassed. Maybe they’re tired. Maybe they’ve got 20 things to do and their boyfriend has just been arrested. These people don’t want to screw around with anything too complicated. They want something that works.

This also goes for another set of users: the people who install and maintain the software when it’s running on production servers. The lower the number of software dependencies, and the less screwing around when it comes to installing, maintaining and upgrading the software, the better.

So, when you’re working on software, and you’re faced with a design choice, ask yourself the simple question:

“What would Kalashnikov do?”

I took a crack at installing Merb today, just to see how it works out. It’s a very minimal Ruby framework which in my (admittedly not very scientific) testing appears to be a lot faster than Rails, especially under conditions of high concurrency.

To get it running, I followed the instructions at the Merb book which is currently a work in progress at http://4ninjas.org. A quick tip: due to a dependency on extlib 0.9.3, the sake edgy technique didn’t work for me at first. I got it all running happily by doing the following.

First, install the Git source-code management tool and the Debian build tools if you don’t already have them:

sudo apt-get install build-essential git-core

sudo gem install rack mongrel json erubis mime-types rspec hpricot mocha rubigen haml markaby mailfactory  english addressable templater

Update: in the last few weeks gem dependencies have changed and you need to ensure you’ve got some specific versions available:

sudo gem install ruby2ruby --version=1.1.8

sudo gem install ParseTree --version=2.1.1

You’ll also need to ensure that you’ve got the MySql headers available in order to build the datamapper MySql libraries.

sudo apt-get install libmysqlclient15off libmysqlclient15-dev

git clone git://github.com/sam/extlib.git  
git clone git://github.com/sam/do.git

cd extlib
rake install ; cd ..
cd do
cd data_objects
rake install ; cd ..
cd do_mysql  # || do_postgres || do_sqlite3
rake install

For whatever reason, DataMapper 0.9.4 has a dependency on Merb-core 0.9.3, so you’ll need to install it and then proceed to build the newest DataMapper:

sudo gem install merb-core

git clone git://github.com/sam/dm-core.git
git clone git://github.com/sam/dm-more.git

cd dm-core ; rake install ; cd ..
cd dm-more
rake install; cd ..

After that stuff, the

sudo gem install sake
sake -i 'http://edgy.4ninjas.org/edgy.sake'
sake edgy:install packages="merb-stack"

commands worked just fine for me.

Bonus: install CouchDb, the wicked distributed database system which is currently an Apache incubator project.

sudo apt-get install build-essential erlang libicu38 libicu-dev libmozjs-dev
wget http://www.apache.org/dist/incubator/couchdb/0.8.0-incubating/apache-couchdb-0.8.0-incubating.tar.gz
tar -xvzf apache-couchdb-0.8.0-incubating.tar.gz 
cd apache-couchdb-0.8.0-incubating/
./configure
make && sudo make install

You can set up the CouchDb datastore as a service, with its own user, like this (thanks to these instructions, slightly modified to avoid the creation of a “couchdb” home directory):

sudo useradd couchdb
sudo mkdir -p /usr/local/var/lib/couchdb
sudo chown -R couchdb /usr/local/var/lib/couchdb
sudo mkdir -p /usr/local/var/log/couchdb
sudo chown -R couchdb /usr/local/var/log/couchdb
sudo mkdir -p /usr/local/var/run
sudo chown -R couchdb /usr/local/var/run

sudo cp /usr/local/etc/init.d/couchdb /etc/init.d/
sudo update-rc.d couchdb defaults
sudo /etc/init.d/couchdb start

After that, go to http://localhost:5984/_utils/index.html and you can administer your new distributed datastore. Of course, you could just ignore CouchDb and use Merb with a more “normal” database server like Mysql or Postgres.

If any of the version dependencies change again please leave a comment and I’ll update these instructions until Merb 0.9.4 is a little easier to get.

We’ve recently moved a couple of Hyperactive sites to run on mod_rails, a recently released Apache module which allows much easier deployment of Rails sites which run on a single box. Using it gives the nice happy glow that you get from using Apache’s PHP module – dump your project code into a directory accessible by Apache and it executes. This contrasts with the more complex process of setting up and maintaining a Mongrel cluster using mod_proxy_balancer (although that option still has a lot to recommend it, especially if you want to run a site across multiple servers in your data centre).

Everything works very well so far, but I did run into one problem. If you’re using full page caching in Rails, mod_rails automatically finds your cache files and tells Apache to render them from disk instead of hitting the Rails stack if a cache for a given URL. However, it only finds the cache files if they’re in the default location (RAILS_ROOT/public). Personally I find the default location a bit messy, since the cache files get mixed in with all the other stuff in your /public directory – so I tend to put cache files at RAILS_ROOT/public/system/cache to keep things clean.

config.action_controller.page_cache_directory = RAILS_ROOT + "/public/system/cache/"

The trouble was that mod_rails didn’t pick up these cache files by default, so I went back to Apache rewrite rules. I dumped in my old rewrite rules from my Mongrel configuration, and everything seemed to be working. Cached pages were served statically, due to this rewrite rule:

# Rewrite to check for Rails cached page
RewriteRule ^([^.]+)$ /system/cache/$1.html [QSA]

I haven’t seen anyone else report this problem, but I had one bit of trouble. My application obviously uses a POST request rather than a GET request to send an update to the server, but the route is otherwise the same. Attempting to edit a piece of content on the site resulted in Apache intercepting the call and just showing the content again, bypassing the “update” method call and making the site’s content impossible to edit. I have no idea why this is the case with mod_rails but didn’t happen with Mongrel, but whatever – I’ve just added the following rewrite condition to my Apache configuration and it seems to work fine:

# Rewrite to check for Rails cached page
RewriteCond %{THE_REQUEST} !^POST
RewriteRule ^([^.]+)$ /system/cache/$1.html [QSA]

This means that Apache won’t attempt to pull the file from the cache for any POST requests (like updates), making it all work again.

Things are moving along a bit more slowly than I’d like but still steadily. The code is now nearly ready for a 0.1 release. This week’s improvements include:

• The use of the SSL Requirement plugin so that the login, admin area, and publishing functions of the site all require SSL. Attempts to use the site in a production situation without an SSL cert will currently fail.
• There is now an “Action Alerts” content type which allows site administrators to publish short messages at the very top of the content on the front page (and exports a list of action alerts as RSS too). This can be used in a “breaking news” situation.

There has been some talk of Twitter integration for the action alerts, I am still thinking about how to acheive that without making the site absolutely require Twitter or any other external commercial service.

When i’ve gone over the code somewhat in a cleanup session, improved the documentation a bit, and packaged a release, I’ll write up a blog post to let people know that a first release is available.

The latest round of development (a.k.a “weekend”) was focused mostly on providing good HTML editing features. This is somewhat trickier than it seems, because it can potentially open up a load of security holes. The site now uses the tiny mce editor, and some special Rails plugins ( white_list and sanitize_params ) to make it safe.

If you know what an XSS attack is, please take a crack at attacking the site and hassle us in #hyperactive on irc.indymedia.org (or just leave a comment on the site) if any of your attacks make it through the filters. I’ve already tried all of the attacks on the XSS Cheat Sheet and none of them succeeded.

Here’s a quick report on the latest code developments. As always, the big need we have is for people who can help out with CSS, as the site still needs somebody knowledgeable to make it look better.

Anway, on the weekend we managed to get the following tasks done:

• Set up the London site on a new server.

• Added a configuration directive allowing sites to use their own local stylesheets and images, so now London and Denmark can look different to each other.

• Comment editing for admins is now working, this was a request by Indy DK as they needed to remove some borderline stuff from a comment.

• Multiple event hiding is now working (bug fix), so all repeating events in a series can be hidden with one click. This should help out with calendar spam in a big way.

• Started work on an integrated system for controlling Bittorrent video downloads from within the site. Torrent creation and control functions work, I just need to make a user interface for it – basically the reason for this one is that torrents didn’t seem to stay running properly, so I added a new process to manage them. Currently it doesn’t look like anything (and isn’t fully working) but when it’s done we should be able to turn torrents on and off for different videos right from within the admin interface, and the torrent downloads should be much more stable.

• Added more documentation on the installation page.

• Added more tickets to the wish-list of features so we don’t forget anybody’s ideas when they get contributed – especially the mobile ones.

I am concentrating on making everything work super-solidly before adding lots of new features, so the next moves will be to get an integrated HTML editor fully working within the site (it’s currently half-done), and probably do more server config (especially SSL for publishing). As far as I know, there are no major bugs currently affecting the site (somebody please let me know if I’ve got this wrong!).

The London indymedia collective recently came up with an interesting idea: “self-administration of comments” . The idea is now working on the development site.

To summarize:

1) Non-admin users can hide comments on their own stuff:

The site allows people to register accounts if they want to (anonymous publishing still works like it always did). If you publish an article, event, or video while you’re logged in, and someone comments on it, you are allowed to hide the comments on your own article, event, or video. If anybody can think of a nice way to encourage people to be responsible for hiding crap comments on their own content, in line with the editorial policy, that’d be cool. If it becomes a problem for some reason I can probably think up some way to bar people from administering their own comments on an individual basis.

2) The site now allows users a choice regarding whether to allow comments:

If you take a look at the publish form for an article, you will see that underneath the body text entry there’s a checkbox with the following explanation:

“You have the option of allowing comments. Untick this box to disable comments.”

So, users can now decide whether they want comments on their stuff (it’s ticked by default).

3) There’s a “latest comments” list in admin.

In other comment-related news, the admin system now allows admins to see a list of the latest comments on the site. Clicking a link on the comment allows the admin to go see the comment in context.

Thanks for the suggestions, keep the ideas coming,

My blood pressure shot through the roof today when I rolled into work to find out that the hep-cats at Indymedia Denmark had switched their Hyperactive installation out of testing and into production. Congratulations to them on the launch of the site, and we can admire them for having the courage to run on such crazy, untested code!

London Indymedia will be going mental with jealousy in short order.

Another round of development on Hyperactive is nearly complete. In this round of features, we’ve concentrated on getting the code ready for production use. This has consisted mostly of the following tasks:

• caching
• localization
• the ability to leave comments

We’ve been hard at work on the Hyperactive codebase for the last few months, it’s probably time to post an update here. This release was code-named “Planet Male Madness” in tribute to one of the greats in the Indymedia movement.

Recent development effort has been spent mostly on video. If you take a look at a video page on the development site you’ll notice that there are lots of new features available.

One of the things that people seem to want out of a new Indymedia CMS is the ability to have an expanded user login system, with multiple levels of permissions and the ability to edit your own content. I’ve been thinking all day about the fact that user logins can visibly tie people to content, and also do so at the database level.

The good things about an expanded login system are obvious – there is a lot more potential for many people to administer the site, less hassle for admins who always end up doing the “can you fix the typo in my article?” requests, etc. However, there are a few problems as well.

A bunch of us have begun working on a new codebase for Indymedia, based on CakePHP. I’m not exactly Captain PHP, so there’s going to be a bit of a learning curve for me, but we’ll see how it goes.

One of the things I am interested in setting up is some kind of FLV video conversion system for the new CMS. There’s a nice pleasant-looking PHP library to deal with the FFmpeg video conversion utility on the server side (is anybody in the Rails world noticing this?), so here’s my experience installing it on Ubuntu.

I did some testing earlier and I actually got a video stream out onto the net and was watching it within about 20 minutes. My head exploded. It was really easy to do.

I was able to view the stream on Linux and Mac OS X using VLC. I was streaming and viewing on the Linux box (a 2.8 Ghz laptop, 512 MB RAM). The CPU sat at about 70% and the machine didn’t look like anything bad was going to happen, I left it running for about 45 minutes and it all seemed cool. The data rates were I think 70 kbps video and 60 kbps audio, which was about 1/4 of my 802.11b wireless card’s bandwidth. So this should work in most wireless situations if you’ve got a decent wifi connection. It was not using very much RAM, it didn’t seem. Hardware requirements should be fairly modest, maybe a 1Ghz machine would do it.