escapegoat.org - Home

Quick video test

2010-12-14T01:00:00Z

Hyperactive now sports video embeds, thanks to JimDog. Good work, boyo!

See this video and more at Indymedia London

Git, gitweb, gitosis and git-daemon in harmony on Debian

2009-05-20T11:15:00Z

After a bit of mucking around, we've got a shared git repository going on escapegoat.org with read only access over http and git, and commit access using git (without having to make shell accounts. Oh and there is a nice browsable version of the repository aswell.

Installing Packages

Debian provides most of the packages for you, so start off with:

sudo apt-get install git-core gitweb git-daemon-run gitosis

Debian Etch

Gitosis requires a version of git >= 1.5, however the version in etch is 1.4.x. So we need to enable the backports repository. So add this line to your /etc/apt/sources.list (if it is not there already):

deb http://www.backports.org/debian etch-backports main contrib non-free

Then we need to update, remove a package that doesn't like to be upgraded, and install the required packages from backports:

sudo apt-get update
sudo apt-get remove python-setuptools
sudo apt-get install -t etch-backports python-setuptools git-core gitweb \
      gitosis git-daemon-run

gitosis - allowing commits without shell accounts

Note a lot of this is cribbed from this blog post about gitosis - see that for more detailed discussion.

So let's start by adding a user account to hold the repositories.

sudo adduser \
    --system \
    --shell /bin/sh \
    --gecos 'git version control' \
    --group \
    --disabled-password \
    --home /home/git \
    git

You may change the home path to suit your taste (the git-daemon-run package assumes a home of /var/cache/git, but I ended up changing the git-daemon-run params to fit with gitosis). A successful user creation will look similar to:

Adding system user 'git'...
Adding new group 'git' (211).
Adding new user 'git' (211) with group 'git'.
Creating home directory '/home/git'.

Then you need to copy your ssh public key to your server to add yourself as the first user. (See the above blog post for some help if you don't know how to). So if you copy your ssh key to the /tmp/ directory, you can run the following command to set up gitosis:

sudo -H -u git gitosis-init < /tmp/id_rsa.pub

Success looks like:

Initialized empty Git repository in ./
Initialized empty Git repository in ./

(Yes, two times)

For good measure, let's make sure the post-update hook is set executable. It doesn't always get set (problem with older setuptools):

sudo chmod 755 /home/git/repositories/gitosis-admin.git/hooks/post-update

Here some cool magic happens. Run this on your local machine:

git clone git@YOUR_SERVER_HOSTNAME:gitosis-admin.git
cd gitosis-admin

You will now have a gitosis.conf file and keydir/ directory:

~/dev/gitosis-admin (master) $ ls -l
total 8
-rw-r--r--   1 garry  garry  104 Nov 13 05:43 gitosis.conf
drwxr-xr-x   3 garry  garry  102 Nov 13 05:43 keydir/

This repository that you just cloned contains all the files (right now, only 2) needed to create repositories for your projects, add new users, and defined access rights. Edit the settings as you wish, commit, and push. Once pushed, gitosis will immediately make your changes take effect on the server. So we're using Git to host the configuration file and keys that in turn define how our Git hosting behaves. That's just plain cool.

At this point you may aswell carry on reading the excellent blog post I've copied from a bit already. If you read "Creating new repositories" and "Adding users" there and then come back.

Final gitosis set up

Gitosis can help you work well with gitweb and git-daemon if you add the right bits to the gitosis.conf file. git-daemon can be helped by gitosis automaticaly adding the 'git-daemon-export-ok' file in the repositories. gitweb can also be helped, and you can add an owner and description (which gitweb will show) in the gitosis.conf file. So here is a sample file:

[gitosis]
gitweb = yes
daemon = yes

[group gitosis-admin]
writable = gitosis-admin
members = user1@computer1 user2@computer2

[repo gitosis-admin]
gitweb = no
daemon = no

[group hyper-team]
writable = hyperactive
members = user1@computer1 user2@computer2

[repo hyperactive]
owner = escapegoat
description = A community news and reporting system

So this configuration allows gitweb and git daemon to access all repositories by default, though we have turned it off in the gitosis-admin repository. And we have a put a description and owner into the hyperactive section.

Allowing read only access using git-daemon

git-daemon is installed from the repositories and will run automatically, however it expects the repositories to be in /var/cache/git. I changed this by editing the file /etc/service/git-daemon/run to be

#!/bin/sh
exec 2>&1
echo 'git-daemon starting.'
#exec git-daemon --verbose --base-path=/var/cache /var/cache/git
exec git-daemon --base-path=/home/git/repositories/ --export-all

After this change, and a

sudo /etc/init.d/git-daemon restart

I was able to download the repository by doing:

git clone git://git.escapegoat.org/hyperactive.git

Allowing read only access over http

To allow access over the web, ie

git clone http://git.escapegoat.org/git/hyperactive.git

we did

sudo mkdir /var/www/git.escapegoat.org/git/
sudo ln -s /home/git/repositories/hyperactive.git /var/www/git.escapegoat.org/git/

Now apache needs to be able to read the repository, and so we added the www-data user to the git group. To do this, you need to find out what groups www-data is in already and then update the list of groups. So the commands we used were:

groups www-data
sudo usermod -G www-data,svnowner,git www-data

Make sure you add any other groups www-data is already in to the list, and remove svnowner if you don't have that group. You may also need to do

cd /home/git/repositories/hyperactive.git
sudo git-update-server-info

after each checkin, but this should be done by /home/git/repositories/hyperactive.git/hooks/post-update. If not then make sure it is executable by

sudo chmod 755 /home/git/repositories/hyperactive.git/hooks/post-update

A user should now be able to do

git clone http://git.escapegoat.org/git/hyperactive.git

Browsing the repository using gitweb

gitweb is installed, and may work out of the box with a simple webserver set up, but we have several virtual hosts. So to put the gitweb.cgi file in the path we did

sudo ln -s /usr/lib/cgi-bin /var/www/git.escapegoat.org/gitweb
sudo cp /usr/share/gitweb/* /var/www/git.escapegoat.org/

(The second line copies the css and images required to make gitweb look nice). We also need to link in the repository. (And do the second and third line below if gitosis has not done it for you already).

sudo ln -s /home/git/repositories/hyperactive.git /var/cache/git/hyperactive
sudo touch /home/git/repositories/hyperactive.git/git-daemon-export-ok
sudo chown git:git /home/git/repositories/hyperactive.git/git-daemon-export-ok

And then it all seemed to work like magic. Go see for yourself at http://git.escapegoat.org/gitweb

Extra apache setup

For now I've done a little bit of extra stuff in the apache set up - specifically in /etc/apache2/sites-available/git.escapegoat.org

# redirect the / to /gitweb/ 
    RedirectMatch ^/$ /gitweb/

ScriptAlias /gitweb /usr/lib/cgi-bin/gitweb.cgi
<Location /gitweb >
            Options +ExecCGI
</Location>

So if you go to http://git.escapegoat.org/ you will be redirected to http://git.escapegoat.org/gitweb/ and be able to browse the repository in a nice way.

Sources

The two main articles I used were

http://scie.nti.st/2007/11/14/hosting-git-repositories-the-easy-and-secure-way - this is a great guide to gitosis set up (for setting up a shared repository that can be committed to over ssh without giving all committers shell accounts).
http://www.die-welt.net/index.php/blog/199/Notes_on_serving_Git_with_Debian - shows how to set up git, gitweb and git-daemon on Debian.

So I have taken those two guides and made them work together - standing on the shoulders of other bloggers ...

Other useful articles include this more DIY guide by the good folks at riseup.net.

Some thoughts on software design

2008-12-28T16:41:00Z

We’re still hard at work on Hyperactive, a content management system for independent news production, and we’ve started to get a lot more people working on the software. So, at this point it seems to make sense to lay out some software design guidelines. Since we’re building software with a non-corporate purpose, let’s think outside of the normal scope of corporate “agile” manifestos (“delivering more value to your client”) and take some design inspiration from a different sphere: Soviet assault rifles.

Despite the rather poor reputation of Soviet technology these days, those guys had some stuff figured out when it came to design. Consider the AK-47 assault rifle designed by Mikhail Kalashnikov. It’s simple, and rugged. It takes a beating, refusing to jam even when it’s been exposed to mud, dirt, or hasn’t been maintained properly. The trigger guard is huge, so it can be fired even by gloved users in Arctic conditions. It can be maintained, and even manufactured by, people with access to simple machine tools.

A news website and an assault rifle might seem like completely unrelated areas of systems design, but let’s think about users for a moment. They’re trying to produce news, sometimes under stressful conditions. Maybe they just got in off the street after getting tear-gassed. Maybe they’re tired. Maybe they’ve got 20 things to do and their boyfriend has just been arrested. These people don’t want to screw around with anything too complicated. They want something that works.

This also goes for another set of users: the people who install and maintain the software when it’s running on production servers. The lower the number of software dependencies, and the less screwing around when it comes to installing, maintaining and upgrading the software, the better.

So, when you’re working on software, and you’re faced with a design choice, ask yourself the simple question:

“What would Kalashnikov do?”

Installing Merb

2008-07-17T14:21:00Z

I took a crack at installing Merb today, just to see how it works out. It’s a very minimal Ruby framework which in my (admittedly not very scientific) testing appears to be a lot faster than Rails, especially under conditions of high concurrency.

To get it running, I followed the instructions at the Merb book which is currently a work in progress at http://4ninjas.org. A quick tip: due to a dependency on extlib 0.9.3, the sake edgy technique didn’t work for me at first. I got it all running happily by doing the following.

First, install the Git source-code management tool and the Debian build tools if you don’t already have them:

sudo apt-get install build-essential git-core

sudo gem install rack mongrel json erubis mime-types rspec hpricot mocha rubigen haml markaby mailfactory  english addressable templater

Update: in the last few weeks gem dependencies have changed and you need to ensure you’ve got some specific versions available:

sudo gem install ruby2ruby --version=1.1.8

sudo gem install ParseTree --version=2.1.1

You’ll also need to ensure that you’ve got the MySql headers available in order to build the datamapper MySql libraries.

sudo apt-get install libmysqlclient15off libmysqlclient15-dev

git clone git://github.com/sam/extlib.git  
git clone git://github.com/sam/do.git

cd extlib
rake install ; cd ..
cd do
cd data_objects
rake install ; cd ..
cd do_mysql  # || do_postgres || do_sqlite3
rake install

For whatever reason, DataMapper 0.9.4 has a dependency on Merb-core 0.9.3, so you’ll need to install it and then proceed to build the newest DataMapper:

sudo gem install merb-core

git clone git://github.com/sam/dm-core.git
git clone git://github.com/sam/dm-more.git

cd dm-core ; rake install ; cd ..
cd dm-more
rake install; cd ..

After that stuff, the

sudo gem install sake
sake -i 'http://edgy.4ninjas.org/edgy.sake'
sake edgy:install packages="merb-stack"

commands worked just fine for me.

Bonus: install CouchDb, the wicked distributed database system which is currently an Apache incubator project.

sudo apt-get install build-essential erlang libicu38 libicu-dev libmozjs-dev
wget http://www.apache.org/dist/incubator/couchdb/0.8.0-incubating/apache-couchdb-0.8.0-incubating.tar.gz
tar -xvzf apache-couchdb-0.8.0-incubating.tar.gz 
cd apache-couchdb-0.8.0-incubating/
./configure
make && sudo make install

You can set up the CouchDb datastore as a service, with its own user, like this (thanks to these instructions, slightly modified to avoid the creation of a “couchdb” home directory):

sudo useradd couchdb
sudo mkdir -p /usr/local/var/lib/couchdb
sudo chown -R couchdb /usr/local/var/lib/couchdb
sudo mkdir -p /usr/local/var/log/couchdb
sudo chown -R couchdb /usr/local/var/log/couchdb
sudo mkdir -p /usr/local/var/run
sudo chown -R couchdb /usr/local/var/run

sudo cp /usr/local/etc/init.d/couchdb /etc/init.d/
sudo update-rc.d couchdb defaults
sudo /etc/init.d/couchdb start

After that, go to http://localhost:5984/_utils/index.html and you can administer your new distributed datastore. Of course, you could just ignore CouchDb and use Merb with a more “normal” database server like Mysql or Postgres.

If any of the version dependencies change again please leave a comment and I’ll update these instructions until Merb 0.9.4 is a little easier to get.

Mod_rails and full page caching with a custom cache location

2008-07-13T12:25:00Z

We’ve recently moved a couple of Hyperactive sites to run on mod_rails, a recently released Apache module which allows much easier deployment of Rails sites which run on a single box. Using it gives the nice happy glow that you get from using Apache’s PHP module – dump your project code into a directory accessible by Apache and it executes. This contrasts with the more complex process of setting up and maintaining a Mongrel cluster using mod_proxy_balancer (although that option still has a lot to recommend it, especially if you want to run a site across multiple servers in your data centre).

Everything works very well so far, but I did run into one problem. If you’re using full page caching in Rails, mod_rails automatically finds your cache files and tells Apache to render them from disk instead of hitting the Rails stack if a cache for a given URL. However, it only finds the cache files if they’re in the default location (RAILS_ROOT/public). Personally I find the default location a bit messy, since the cache files get mixed in with all the other stuff in your /public directory – so I tend to put cache files at RAILS_ROOT/public/system/cache to keep things clean.

config.action_controller.page_cache_directory = RAILS_ROOT + "/public/system/cache/"

The trouble was that mod_rails didn’t pick up these cache files by default, so I went back to Apache rewrite rules. I dumped in my old rewrite rules from my Mongrel configuration, and everything seemed to be working. Cached pages were served statically, due to this rewrite rule:

# Rewrite to check for Rails cached page
RewriteRule ^([^.]+)$ /system/cache/$1.html [QSA]

I haven’t seen anyone else report this problem, but I had one bit of trouble. My application obviously uses a POST request rather than a GET request to send an update to the server, but the route is otherwise the same. Attempting to edit a piece of content on the site resulted in Apache intercepting the call and just showing the content again, bypassing the “update” method call and making the site’s content impossible to edit. I have no idea why this is the case with mod_rails but didn’t happen with Mongrel, but whatever – I’ve just added the following rewrite condition to my Apache configuration and it seems to work fine:

# Rewrite to check for Rails cached page
RewriteCond %{THE_REQUEST} !^POST
RewriteRule ^([^.]+)$ /system/cache/$1.html [QSA]

This means that Apache won’t attempt to pull the file from the cache for any POST requests (like updates), making it all work again.

Hyperactive::SSL, Action Alerts

2008-06-28T11:20:00Z

Things are moving along a bit more slowly than I’d like but still steadily. The code is now nearly ready for a 0.1 release. This week’s improvements include:

The use of the SSL Requirement plugin so that the login, admin area, and publishing functions of the site all require SSL. Attempts to use the site in a production situation without an SSL cert will currently fail.
There is now an “Action Alerts” content type which allows site administrators to publish short messages at the very top of the content on the front page (and exports a list of action alerts as RSS too). This can be used in a “breaking news” situation.

There has been some talk of Twitter integration for the action alerts, I am still thinking about how to acheive that without making the site absolutely require Twitter or any other external commercial service.

When i’ve gone over the code somewhat in a cleanup session, improved the documentation a bit, and packaged a release, I’ll write up a blog post to let people know that a first release is available.

Hyperactive :: HTML Editing Facilities

2008-06-10T23:50:00Z

The latest round of development (a.k.a “weekend”) was focused mostly on providing good HTML editing features. This is somewhat trickier than it seems, because it can potentially open up a load of security holes. The site now uses the tiny mce editor, and some special Rails plugins ( white_list and sanitize_params ) to make it safe.

If you know what an XSS attack is, please take a crack at attacking the site and hassle us in #hyperactive on irc.indymedia.org (or just leave a comment on the site) if any of your attacks make it through the filters. I’ve already tried all of the attacks on the XSS Cheat Sheet and none of them succeeded.

Hyperactive :: Torrent Features and Minor Fixes

2008-05-27T10:33:00Z

Here’s a quick report on the latest code developments. As always, the big need we have is for people who can help out with CSS, as the site still needs somebody knowledgeable to make it look better.

Anway, on the weekend we managed to get the following tasks done:

Set up the London site on a new server.

Added a configuration directive allowing sites to use their own local stylesheets and images, so now London and Denmark can look different to each other.

Comment editing for admins is now working, this was a request by Indy DK as they needed to remove some borderline stuff from a comment.

Multiple event hiding is now working (bug fix), so all repeating events in a series can be hidden with one click. This should help out with calendar spam in a big way.

Started work on an integrated system for controlling Bittorrent video downloads from within the site. Torrent creation and control functions work, I just need to make a user interface for it – basically the reason for this one is that torrents didn’t seem to stay running properly, so I added a new process to manage them. Currently it doesn’t look like anything (and isn’t fully working) but when it’s done we should be able to turn torrents on and off for different videos right from within the admin interface, and the torrent downloads should be much more stable.

Added more documentation on the installation page.

Added more tickets to the wish-list of features so we don’t forget anybody’s ideas when they get contributed – especially the mobile ones.

I am concentrating on making everything work super-solidly before adding lots of new features, so the next moves will be to get an integrated HTML editor fully working within the site (it’s currently half-done), and probably do more server config (especially SSL for publishing). As far as I know, there are no major bugs currently affecting the site (somebody please let me know if I’ve got this wrong!).

Hyperactive :: Comment Love

2008-05-10T00:18:00Z

The London indymedia collective recently came up with an interesting idea: “self-administration of comments” . The idea is now working on the development site.

To summarize:

1) Non-admin users can hide comments on their own stuff:

The site allows people to register accounts if they want to (anonymous publishing still works like it always did). If you publish an article, event, or video while you’re logged in, and someone comments on it, you are allowed to hide the comments on your own article, event, or video. If anybody can think of a nice way to encourage people to be responsible for hiding crap comments on their own content, in line with the editorial policy, that’d be cool. If it becomes a problem for some reason I can probably think up some way to bar people from administering their own comments on an individual basis.

2) The site now allows users a choice regarding whether to allow comments:

If you take a look at the publish form for an article, you will see that underneath the body text entry there’s a checkbox with the following explanation:

“You have the option of allowing comments. Untick this box to disable comments.”

So, users can now decide whether they want comments on their stuff (it’s ticked by default).

3) There’s a “latest comments” list in admin.

In other comment-related news, the admin system now allows admins to see a list of the latest comments on the site. Clicking a link on the comment allows the admin to go see the comment in context.

Thanks for the suggestions, keep the ideas coming,

Hyperactive :: First Live Installation

2008-04-10T11:13:00Z

My blood pressure shot through the roof today when I rolled into work to find out that the hep-cats at Indymedia Denmark had switched their Hyperactive installation out of testing and into production. Congratulations to them on the launch of the site, and we can admire them for having the courage to run on such crazy, untested code!

London Indymedia will be going mental with jealousy in short order.

Hyperactive :: Caching, Localization and Comments

2008-04-08T10:22:00Z

Another round of development on Hyperactive is nearly complete. In this round of features, we’ve concentrated on getting the code ready for production use. This has consisted mostly of the following tasks:

caching
localization
the ability to leave comments

caching
localization
the ability to leave comments

Caching

Since Ruby is a slow language (it takes a long time to do stuff compared to most other languages) and Rails isn’t a particularly speedy framework, we don’t want to be pulling all the information to construct a page out of the database and building the page every time somebody requests it. The site needs to be fast enough to not fold up under heavy load.

So, the code has been set up to use the fastest possible solution to this set of problems – the full-page caching mechanism in Rails. For any content show page (i.e. show an article, show an event, or show a video), and for the front page of the site, the page is assembled from the database and rendered as HTML the first time it’s requested. This rendered page gets cached on disk. The next time somebody requests the same page, instead of firing up Rails and hitting the database and doing all the calculations necessary to show the page, the Apache server looks in its cache and says “oh, yeah, I’ve got this one, here you go” – which is maybe 100 times faster than assembling the page dynamically. This means that more people can be looking at the site at any given time without the server flipping out.

As with anything, there are a few trade-offs involved. For one thing, the fact that we’re caching the full page rather than just parts of it means that the page needs to be the same for everybody. This might not seem like a big deal at first, but it means we can’t (easily) have stuff like “Now logged in as Noam”, or show different controls on the page based on whether a user is an admin or not. If we did, the user “Marcos” might log into the site, take a look at a page which Noam had previously viewed, see the message “Now logged in as Noam”, and suffer a major identity crisis (“Why am I up here in the mountains? MIT seems so much more comfortable at this time of year.”). The repercussions could be too much to bear, especially if the situation was reversed and Chomsky tried to pick up an AK-47 and head for the mountains. None of us wants to be responsible for this.

The way we’ve solved the problem is to put things like administrative controls into the page as AJAX calls. When viewing an article, if you click on the “inappropriate article?” link, the page asks the server to grab the proper controls for your user role – if you have the ability to hide content, you see the “hide content” controls, if you don’t have the ability to hide content, you see the “report this content” controls. Since this part of the page comes back as the result of a fresh request to the server, we can check the user’s role when the request takes place, which gets around the problem of the cached page.

We still need to do caching for the list pages, and a few other pages, but the front page and the content pages seemed like the best place to start.

Localization

The site can now be fully localized into any UTF-8 language. We’ve used the click-to-globalize plugin together with the globalize plugin to provide translation services. Basically, in order to translate the site’s user interface into your language, all you need to do is log in as a translator, and click directly on the piece of text you want to translate. When you do this, an in-place editor appears on the page, you type in the translated text, and hit a “save” button. This saves the translated text in the database. You can see this in action in the “text area transformation” screencast video on the bottom of the click-to-globalize page.

Currently, any site should only be in one language, but this should make it really easy to translate the site’s user interface into multiple languages and have them all available (so a site could be in French and English, let’s say). Another thing which would be relatively easy to implement would be a translation interface directly in the site, so that site content can also be translated and shown automatically if it exists for a given piece of content. This should make it easy to build multilingual sites.

Comments

Up until now, the site hasn’t had the ability for users to leave comments. Over the past two weeks we’ve built comment functionality into the site, so it’s now possible to leave comments on articles (and soon events and videos).

We’re thinking about having features like the person who created the content having control over comment administration, but we’ll have to ask around and see what people think of that.

Summary

The site is nearly ready for production deployment, there are a bunch of very small things which need to be fixed up so that it can go live at Indymedia Denmark – areas of the interface which still need translation, the ability to un-hide comments, and a short list of other stuff.

Hyperactive :: Video Improvements

2008-03-26T13:57:00Z

We’ve been hard at work on the Hyperactive codebase for the last few months, it’s probably time to post an update here. This release was code-named “Planet Male Madness” in tribute to one of the greats in the Indymedia movement.

Recent development effort has been spent mostly on video. If you take a look at a video page on the development site you’ll notice that there are lots of new features available.

Related content is automatically shown on the right hand side. Related content could be other videos, articles, or events. It’s generated automatically by the search engine software embedded in the site. You can see an example of a vid with related content here.

featured videos are shown on the right hand side

recent videos are shown on the right hand side

if the video was uploaded inside of an article or event, that article or event is displayed.

the flash video player now does full-screen mode if the user’s flash player supports it

all uploaded video is currently retained in its original format, and transcoded to Ogg Theora standard as well. I know that some people have strong feelings about codec choices, this will require more discussion.

clicking on “How to download” reveals instructions regarding how to download the video via Bittorrent peer-to-peer software, and how to subscribe to the site’s video feeds. I would really appreciate it if people could read this text and offer suggestions on how to improve it.

Incidentally, this is the way I’d like to handle help text throughout the site – we can put that little ”?” icon on a page and have any necessary help text get revealed right where the user is.

the hiding controls have now been improved so that the show up as “inappropriate video?”. If you have permission to hide content, this reveals a “hide this” form when clicked. If you don’t have permission to hide content, it reveals a “report this as inappropriate” form when clicked. The contents of the form are automatically emailed to admins, and content is hidden immediately if the user has hiding permission.

the site now implements the Transmission metadata feed standard, which means that people can now subscribe to screening-quality video and the bandwidth is shared among site users (we don’t pay for most of it).

Users vs Groups Smackdown

2007-12-09T18:16:00Z

One of the things that people seem to want out of a new Indymedia CMS is the ability to have an expanded user login system, with multiple levels of permissions and the ability to edit your own content. I’ve been thinking all day about the fact that user logins can visibly tie people to content, and also do so at the database level.

The good things about an expanded login system are obvious – there is a lot more potential for many people to administer the site, less hassle for admins who always end up doing the “can you fix the typo in my article?” requests, etc. However, there are a few problems as well.

First, the act of publishing anonymously on the newswire arguably leads to a collective ethic rather than an individualist one (at least in theory, and on a good day). That is, the newswire is a space where a whole bunch of people can contribute, and it’s not about them as individuals, or even about their online personas. This makes the newswire of a well-maintained and active Indymedia site qualitatively very different from a blog, even a blog contributed to by multiple users. Whereas the Indy wire ideally showcases organized groups of people taking collective action, a blog usually ends up being a “me, me, me, look how great I am” showcase of a bunch of individuals.

Second, there are questions of security. Tying published content (articles, videos, etc) to a user can be considered a good thing if we’re not interested in security – if someone publishes something that’s really of excellent quality, I’d like to be able to conveniently access all of their other stuff. However, if the server gets seized, the foreign key between the content and the user is gonna be there. Although we wouldn’t store email addresses or other identifying info in the system anyway (and there would always be the option of anonymous publishing), it could be a pretty easy job for the cops to figure out who a user is, given a sufficient sample set of articles and enough on-the-street surveillance.

So I was thinking to tie the content to a group instead. Then if there is more than one person operating as part of the group, there is at least some legal deniability (which improves the bigger the group is). I would propose having individual logins, but all the functionality of the site would be keyed on groups rather than individuals. So instead of showing “all articles by UserX” the site would show things like “All articles by london noborders”. Group admins would i guess be in charge of accepting new people into a group. It could be a good way to encourage organization and collaboration between media producers, and it could also help encourage media production by political groups.

Installing ffmpeg-php on ubuntu 7.10

2007-11-11T15:33:00Z

A bunch of us have begun working on a new codebase for Indymedia, based on CakePHP. I’m not exactly Captain PHP, so there’s going to be a bit of a learning curve for me, but we’ll see how it goes.

One of the things I am interested in setting up is some kind of FLV video conversion system for the new CMS. There’s a nice pleasant-looking PHP library to deal with the FFmpeg video conversion utility on the server side (is anybody in the Rails world noticing this?), so here’s my experience installing it on Ubuntu.

First things first. Enable the universe repositories, and then install ffmpeg. Keep in mind that the ffmpeg binaries in Ubuntu are built without the necessary MP3 support (for patent reasons) to actually encode sound in FLV files. In addition, if you install the “normal” FFMpeg binaries from the Ubuntu repositories, you won’t get the development headers necessary to compile the ffmpeg-php code. So, let’s do things the hard way and compile FFMpeg from source, so we have everything working.

I’ll outline the basic steps here. First, you need the mp3 libraries:

apt-get install liblame-dev

Next, you need to get ready to build FFMpeg from source:

sudo apt-get install liblame-dev libfaad2-dev libfaac-dev libxvidcore4-dev liba52-0.7.4 \
liba52-0.7.4-dev libx264-dev libdts-dev libgsm1-dev libvorbis-dev libdc1394-13-dev \
checkinstall build-essential gcc

Get the FFMpeg source:

cd /usr/local/src
sudo apt-get source ffmpeg

cd ffmpeg-*
sudo ./configure --enable-gpl --enable-pp --enable-libvorbis --enable-libogg \
--enable-liba52 --enable-libdts --enable-dc1394 --enable-libgsm --disable-debug \
--enable-libmp3lame --enable-libfaad --enable-libfaac --enable-xvid --enable-pthreads \
--enable-x264 --enable-shared --prefix=/usr

The second-last option,—enable-shared, should allow the php5 script to find the FFMpeg shared libraries so that ffmpeg-php can compile properly.


sudo make
sudo checkinstall

When you do the checkinstall, it’ll ask you a bunch of questions so that it can build you a .deb package and install it. For most of them you can just accept the defaults. I set the package description to FFMpeg video conversion utility, and I set the version to 3:0.cvs20070308-5ubuntu4. This makes the package you’re making slightly newer than the one in the regular Ubuntu repositories, so Ubuntu won’t hassle you all the time, taunting you with a newer version of the package when you log in.

Ok, that was a big hassle, I know. But it should get you video encoding capability.

The next thing I did was download the latest release of ffmpeg-php from http://ffmpeg-php.sourceforge.net/ and extracted it on my machine. The first thing you need to do at that point is to run a utility called “phpize” in order to set up some configuration files for your machine. However, Ubuntu doesn’t include phpize in the main php package, so you’ll need to do this first:

apt-get install php5-dev

Then cd into the ffmpeg-php-0.X.X directory (maybe you extracted it on your desktop?), and run the following command to generate a config script:

phpize5

You can then do this to build the library:

./configure && make

And finally this to install it:

sudo make install

Now you’ve got ffmpeg support for PHP, and an FFMpeg binary installed that will do its stuff with sound encoding capabilities.

Your own live tv feed on the interweb

2007-10-14T15:57:00Z

I did some testing earlier and I actually got a video stream out onto the net and was watching it within about 20 minutes. My head exploded. It was really easy to do.

I was able to view the stream on Linux and Mac OS X using VLC. I was streaming and viewing on the Linux box (a 2.8 Ghz laptop, 512 MB RAM). The CPU sat at about 70% and the machine didn’t look like anything bad was going to happen, I left it running for about 45 minutes and it all seemed cool. The data rates were I think 70 kbps video and 60 kbps audio, which was about 1/4 of my 802.11b wireless card’s bandwidth. So this should work in most wireless situations if you’ve got a decent wifi connection. It was not using very much RAM, it didn’t seem. Hardware requirements should be fairly modest, maybe a 1Ghz machine would do it.

I did some testing earlier and I actually got a video stream out onto the net and was watching it within about 20 minutes. My head exploded. It was really easy to do.

At the same time as I was compressing and uploading the stream to http://radio.indymedia.org, I also tried watching it on a different computer.

I fired up VLC (downloadable for Linux, Mac, and Windows from http://www.videolan.org ) and used the “Transcode Wizard” or something to record the video to disk on a G4 Mac. I could view the stream as it was saved, and playback of the saved file worked perfectly as well, so in theory, we could use VLC on a separate box to record the stream, and upload it to the UK newswire or to http://video.indymedia.org as soon as the streaming event is finished. I am not totally sure but I think this laptop would probably be able simultaneously compress the stream and record it to disk using VLC (but if we can use a separate machine to do the recording it’s probably safer).

Below is what I did to set up the box, there are much more detailed instructions at:

http://24.72.34.35/cgi-bin/twiki/view/Main/LinuxVideoFirewire

The Setup

Basically, I just had to make sure the following packages were installed. I’m using Ubuntu, but this should work pretty much the same on Debian (keep in mind that on Ubuntu you would preface the commands below with “sudo”. On Debian you’d need to run them as root).

apt-get install ffmpeg2theora dvgrab

(Note that I already had ffmpeg installed on my machine, I think that’s also necessary).

The next thing I did was download a program called oggfwd. It’s available from http://www.v2v.cc/~j/ffmpeg2theora/oggfwd/oggfwd.linux.bin

Then I decided I trusted the people who made it, and typed this into the terminal to copy the file to a place where I could run it:

mv oggfwd.linux.bin /usr/local/bin/oggfwd

chmod 755 /usr/local/bin/oggfwd

Then, I plugged the camera in and made sure that the system could capture stuff from the firewire port. To do this, I did:

modprobe raw1394

This made sure the Linux kernel loaded up the proper modules to read Firewire input. Then I did a:

chmod 777 /dev/raw1394

This gives all users read/write/execute permissions on the Firewire port. Not wickedly secure on a server machine but good enough for this situation I think, and it gets reset when the machine is restarted.

At that point, I issued the following command (as root) and it connected to the Indy radio server and I was serving a video stream from my house:

dvgrab --format raw - | ffmpeg2theora -a 0 -v 5 -f dv -x 320 -y 240 -o /dev/stdout  - | oggfwd radio.indymedia.org 8000 thisisnotreallythepassword /rampART.ogg

I am not sure whether it is necessary to be root to do this, we can experiment with permissions a bit.

That’s it, you’ve got your own live video feed, accessible anywhere in the world!