Arch Linux (Planet)

My C++ skills are getting a bit rusty as I have not done much programming in that language lately. So I thought what better way to refresh my skills and learn new stuff than to go through the C++11 changes and write something about them. I will not cover all additions and changes to the standard, but instead focus on those I am likely to use, (which means they must be implemented in GCC). And even those I do focus on, will likely miss some subtleties that I find unimportant (or more likely am unaware of…). For those wanting actual detail, a late working paper that is a near final draft of the C++11 standard can be downloaded here.

This first post is going to focus on type deduction and its uses.

auto
In C++98, you have to declare the type of every variable being used. C++11 introduces the auto keyword, which lets the compiler deduce the type of a variable given its initializer. So you could write:

auto x = 5;

and you would get that x is declared to be an int. Like everything in programming, there is times when you should use a feature and times when you should not… This is an example of a time where you should not. The use of auto should be reserved for situations where you really do not know the type of the result or that the type is unwieldy to write. For example:

template<class T, class U>
void dot_product(const vector<T> vt, const vector<U> vu)
{
  //...
  auto tmp = vt[i] * vu[i]
  //...
}

A bit of a contrived example, but it should be clear that the type of the result depends on the template parameters U and V so is unknown. I will get to the return type in the next post…

A more useful example where the type of the variable is known but unwieldy is:

std::vector<std::pair<std::string, int>> v;
//...
for(auto i = v.begin(), i != v.end(), ++i) {
//...

Here the type of i is known (std::vector<std::pair<std::string,int>>::const_iterator), but writing that would quickly become tedious and error prone. I guess most situations like this were previously handled with a typedef statement.

There are other uses of auto that will be covered in later posts when the relevant features are introduced.

decltype
The decltype operator takes an expression and returns it type. A trivial example:

int x = 5;
decltype(x) y = x;

This declares the variable y to have the same type as x. Of course you could use auto here (or preferably neither in this case…), but the use of decltype comes into its own when you actually need a type – for example, a return type, which will be covered in the second post in this series.

So, powertop 2.0 has been released. I given it a try and it found a couple of ‘bad’ tunnables; Powertop allows to ‘fix’ them using its interface, but the fix isn’t permanent.

So, how do I permanently fix them?

First of all I did a search in the web on how to fix them; what I found is a bunch of echos. With initscripts I’d use /etc/rc.local to set the values each time my system starts, but I removed it since I switched to systemd. I did ask in #systemd on IRC and the guys told me to use tmpfiles.d. What’s that? (man tmpfiles.d)

I made this /etc/tmpfiles.d/local.conf file which sets the values of the stuff in /sys

(Sorry I’m having some issue with octopress and I’ve no time to fix them right now, local.conf file is here)

In addition, I used sysctl to set the values of the stuff in /proc

1
2

# Turn OFF NMI watchdog
kernel.nmi_watchdog = 0

I don’t set vm.laptop_mode and vm.dirty_writeback_centisecs because they’re already set by pm-utils ;)

Go and save power! :)

Thomas Bächler wrote:

netcfg 2.8.2 has been moved to [core]. It has been six months since netcfg had an update, so it accumulated quite some changes. Here is a shortlist.

• Configuration has moved out of rc.conf. Please move your configuration to /etc/conf.d/netcfg.
• Connection types ending in -old, -dbus and -iproute are discontinued. Remove the suffix if you still have it: you didn't need it.
• Systemd is now supported for single profile, multiple profile, automatic wired and automatic wireless connections.
• For your basic wifi needs, you can now use the bundled wifi-menu tool. It is a dialog based connection aid inspired by wifi-select.
• Disconnecting an interface is no longer done through -i. The new parameter is -D. Reconnecting an interface is possible through -R.
• There is a new connection type: PPPoE.

The full changelog is at http://projects.archlinux.org/netcfg.git/tree/NEWS?id=2.8.2

The wiki does not reflect all these changes yet, please help updating it!

Ignoring everything I had said in the previous posts in this series about Commander Keen being next (stupid rat things still kill me…), I have instead worked my way though Crystal Caves.

Like many other Apogee software games, this came in three episodes – Trouble with Twibbles, Slugging it Out and Mylo Versus the Supernova – with the first game being free and the other two requiring a purchase. An interesting fact I found out about these games was that a patch was released for the original game 14 years and one day after its original release, fixing a bug when run on Windows XP. That is what I call support!

Meet our hero, Milo Steamwitz. He knows exactly how he is going to make his fortune, but needs to generate some money to invest first. So it is off to some remote planet where large crystals are just lying about in caves waiting to be harvested. You start off in some sort mine shaft that provides access to 16 caves to be explored. In each cave you have to run around, jumping between platforms, flicking switches on and collecting all the crystals while avoiding various obstacles and shooting aliens. Each cave has a bit of a theme to it, with some having continuously falling rocks to avoid and others having “low gravity” (which does not let you jump higher, but does mean you get forced back whenever you shoot – interesting…). Once all the crystals are collected, the exit door unlocks.

I had played the first episode many times when I was younger so I zoomed through to the end quite quickly. For each level there is a key that you can collect which allows you to open all the treasure chests scattered throughout the level, but having no siblings around to eliminate from the high-score board, my motivation to do so was limited… At the end of the first episode, Milo sells up his collected crystals and invests in a Twibble farm. It turns out that Twibbles are prolific at eating and breeding so the planets resources are soon used up. Also, no-one wants to buy Twibbles any more, so I guess he just abandoned them all to die of starvation.

The second and third episodes are very similar to the first. I think there is a slight difficulty increase, but it is hard to judge given how much I had played the first episode previously. The difference I did notice was that a lot of levels required you to do the crystal collecting for different sections in a defined order. There were many places where the only way to go back to collect a crystal you missed was to die and restart the level. What is worse, there is a bug in the third game where in the mine shaft there is an area where you can not escape (pictured). So the two levels there must be left until last, otherwise you have to restart the game.

The second episode ends with Milo buying a slug farm. For some reason, everybody wants slugs and he is in danger of running out. But then the slugs burrowed underground to avoid the heat of the day and ended up in an old salt mine. So Milo’s profits quickly “dried up”. Oh, the hilarity…

The final episode see Milo giving up on farming. Instead he wants to buy a solar system to set up a vacation resort based on some perfectly legitimate sounding scheme he saw on TV. Sure enough, once he signs the contract, the whole solar system gets destroyed in a supernova (I bet you could have never guessed that would happen from the game title…). Luckily, the supernova left a nice looking backdrop for a space burger joint. It is now quite popular and Milo can sell his burgers at a price that looks expensive even accounting for inflation.

I was slightly disappointed at the lack of additional game-play elements in the second and third episodes. The two episodes I had not previously played were entertaining enough, but that is influenced by nostalgia. Overall, I think these games are worth playing, but if you finish the free episode and are not impressed, do not think things will improve.

So… will I finish Commander Keen: Vorticons for the next post? Not likely… But I am running out of ideas for old DOS games to play, so make me some suggestions.

Following on from my last post about mutt, I have been tinkering more with the most suckless of mail clients to get it to deal with—of all things—HTML emails. I will preface this post with a remark or two that amounts to a warning about intemperate language, ranting and a generally cranky disposition: repeated contact with email marketers can do that to you…

A further brief digression is warranted here. HTML email is, without doubt, evidence of the imminent end of civilized life as we know it; much like the Golgafrincham diaspora, it is attributable to a depraved cabal of marketing consultants and provides the same level of social good as syphilis and fistulas. Suffice to say, it is a blight.

Sadly, it is a blight that I have to deal with. Up until now, I had done that by using the excellent w3m to render HTML as text in mutt itself. That is accomplished easily enough with an entry in muttrc and another in a mailcap file.

auto_view text/html                                      # view html automatically
alternative_order text/plain text/enriched text/html     # save html for last

text/html; w3m -I %{charset} -T text/html; copiousoutput;

This works for 80 percent of the HTML email I get. It is enough to confirm that the mail is useless and can be discarded. The other 20 percent, however, is slightly more problematic. The initial view in w3m is enough to tell me that, due to the outstanding incompetence of the idiots that generated the email, there is actually some information that is contained in the email that I need to access but it is not viewable in a text browser.

For this special level of retardation, I need to be able to open the email in a ghraphical browser. As I use vimprobable, that means sending the file to vimprobable running in tabbed. Fortunately, as you would expect with superior software, mutt has a way of handling this gracefully. In mutt's extensive documentation, there is a page on mutt's MIME support that details how to set up a graduated response to this pernicious issue:

In addition, you can use this with Autoview to denote two commands for viewing
an attachment, one to be viewed automatically, the other to be viewed
interactively from the attachment menu.

So, you can have mutt render the HTML abomination in w3m in most cases, and when those instances of particularly cretinous behaviour make it to your inbox, you can choose to view them in your web browser. In my case, by altering the mailcap entry to look like this:

text/html; vimprobtab.sh %s &; test=test -n "$DISPLAY"; needsterminal;
text/html; w3m -I %{charset} -T text/html; copiousoutput;

The first entry tests that X is running, and if it is, it hands the file to vimprobable. The default, however, is determined by the copiousoutput tag. So, in mutt it is just a matter of hitting v to view the attached HTML and then m to send it to mailcap. For convenience, I bind Enter to that function in muttrc:

bind attach <return>    view-mailcap

Pierre Schmitz wrote:

PHP has been updated to its latest major release 5.4. The upstream changelog can be found at php.net/ChangeLog-5.php, a detailed migration guide is at php.net/migration54. In addition to this the following packaging changes have been made:

• The Suhosin patch and extension have been removed. Learn more about the reasoning behind this decision at pierre-schmitz.com/php-5-4-1-in-suhosin-out/ If you had used the suhosin extension before remove the php-suhosin package as it is no longer compatible with PHP 5.4
• Note that the extension API has been changed. If you are using any third party extensions that are not in our repositories, you have to at least recompile them or update to a compatible version.
• The json module is now built into PHP itself. Therefore, remove the extension=json.so directive from your php.ini file.
• The gd extension has been built with webp support using the libvpx library.

Make sure to merge the changes of php.ini into your local copy; or probably easier: adjust the new php.ini to your needs. If you any encounter problems using the APC extension you may try the php-xcache package which has been added to the repository.

Last but not least: especially if you are running old scripts, make sure they do not rely on features like "Safe Mode" or "Magic Quotes". As these features were removed, an update to PHP 5.4 will result in serious security holes.

Here’s a quick reminder to get CGit and Nginx running together on an Arch Linux server. I assume you have already Nginx up and running.

First, we install the require packages.

pacman -S fcgiwrap cgit

The default configuration values for fcgiwrap should be fine, but you can check anyway in /etc/conf.d/fcgiwrap.

Here the Nginx configuration for the CGit part. EDIT: check the update code at the end of the post.

server {
    listen       80;
    server_name  git.nicosphere.net;
    index cgit.cgi;
    gzip off;

    location / {
        root /usr/share/webapps/cgit/;
    }

    if (!-f $request_filename) {
        rewrite ^/([^?/]+/[^?]*)?(?:\?(.*))?$ /cgit.cgi?url=$1&$2 last;
    }
                                                }

    location ~ \.cgi$ {
        gzip off;
        include fastcgi_params;
        fastcgi_pass    127.0.0.1:9001;
        fastcgi_index   cgit.cgi;
        fastcgi_param   SCRIPT_FILENAME /usr/share/webapps/cgit/cgit.cgi;
        fastcgi_param   DOCUMENT_ROOT /usr/share/webapps/cgit/;

    }
}

Obviously, make sure the server_name is setup right. The SCRIPT_FILENAME and DOCUMENT_ROOT are those used with the official Arch Linux package, it might be different if you retrieved your own tarball.

Now, let’s get all this running.

rc.d start fcgiwrap && rc.d restart nginx

Don’t forget to add the fcgiwrap daemon to the corresponding rc.conf line, to get it working when if you restart the server.

Once it’s running, you might setup the /etc/cgitrc file as usual. You certainly want add at least the CSS file as follow:

css=/cgit.css

EDIT: As I wanted a basic authentication, with an exception for the /pub/ directiory, and couldn’t find the correct answer, I asked to StackOverflow. The answer works fine for me, and even better that the previous code. For the record, here the final nginx configuration:

server {
    listen       80;
    server_name  git.nicosphere.net;
    index cgit.cgi;
    gzip off;

    root /usr/share/webapps/cgit;

    # $document_root is now set properly, and you don't need to override it
    include fastcgi_params;
    fastcgi_param   SCRIPT_FILENAME $document_root/cgit.cgi;

    location / {
        try_files $uri @cgit;
    }

    # Require auth for requests sent to cgit that originated in location /    
    location @cgit {
        auth_basic "Restricted";
        auth_basic_user_file /srv/gitosis/.htpasswd;

        gzip off;
        # rewrites in nginx don't match the query string
        rewrite ^/([^/]+/.*)?$ /cgit.cgi?url=$1 break;
        fastcgi_pass    127.0.0.1:9001;
    }

    location ^~ /pub/  {
        gzip off;
        rewrite ^/([^/]+/.*)?$ /cgit.cgi?url=$1 break;
        fastcgi_pass    127.0.0.1:9001;
    }
}

Short but sweet post here. The short version is this: if you are currently running MySQL and don't have /tmp on a tmpfs, you're getting burned and performance will be extremely poor. The graphs below pretty much speak for themselves. We went from far too many slow queries and very high I/O on the only two disks this machine has (the box hosting the Arch Linux AUR) to almost no slow queries at all, normal I/O, and reduced CPU and IO wait.

If it wasn't clear, the reason this makes such a difference is MySQL's usage of temp for sorting and temporary table purposes when things can't fit in memory. Moving to a tmpfs means nothing ever needs to get flushed to disk or journaled, even if it is deleted within milliseconds of it being created.

Today I finally moved PHP 5.4.1 into our [testing] repository. Our previous tests of 5.4.0 were quite successful. In short PHP 5.4 is ready for production now and will be in our [extra] repository in a couple of days.

When I first packaged PHP 5.4.0 one of the most concerning issues was the lack of a compatible Suhosin patch and extension. Unfortunately this situation has not changed within the last two months: there is still no information if or when Suhosin will be released for recent versions of PHP. Therefor I decided to remove the Suhosin patch and its extension from our repository. To not break compatibility on minor updates the current version in [extra], PHP 5.3.11, includes a patched version of Suhosin; the last official one was dedicated for 5.3.9.

Suhosin is a safe guard for scripts and PHP itself. It consists of  patch to PHP and an extension which can be used independently. The patch aims to protect the PHP core against buffer overflows and string format vulnerabilities. Scripts written in PHP are protected by the Suhosin extension. It can be configured to disallow or limit certain user input. There is also a feature to transparently encrypt cookie and session data. More of Suhosin’s features are described on its website.

Certainly some of Suhosin’s features are a good idea as they might protect you in case PHP itself or scripts fail. This is why it was added years ago when I took over maintainership of PHP. Fortunately some of its features got added to PHP itself over time. But today we are confronted with several problems:

• There is no Suhosin for PHP 5.4. This means we would be stuck at 5.3 and cannot ship the most recent packages which is one of the goals of Arch Linux.
• The upstream project has become less and less active in the past years. There are certain indicators which make Suhosin an unhealthy open source project: There is still only one main contributor, the development process is mostly closed, there is no public repository for the patch, no bug tracker, the forums are in maintenance mode for years and the latest news entry is from 2007. Of course things might be different in the background, but this is the situation which is publicly visible.
• The delays between PHP releases and their Suhosin counterparts are increasing. I often have to patch the Suhosin patch itself to use it with latest minor updates of PHP. This leaves us with a PHP version that is neither supported by the PHP project nor the Suhosin authors. While these modifications are mostly trivial I cannot exclude the possibility that I break PHP in a dangerous way. The situation is even worse on major update like we have now. Adjusting Suhosin to PHP 5.4 is not trivial. In addition to this testing development releases or release candidates often means that you have to disable Suhosin.
• There are no plans of the project to ever move any of Suhosin’s features into PHP itself. In fact the Suhosin author is strictly against such a move.

In general we have a “Do not patch” policy in Arch and try to keep our packages as close to upstream as possible. In its current state I no longer have any excuse to break this rule for Suhosin; in fact I now have a lot of arguments which support this policy. So even if a new version of Suhosin was released tomorrow, next week or in a few months, it wont be re-added to our PHP packages. The upstream projects would need some changes to its process to be reliable again.The relationship with the PHP project itself has to improve and it should at least be tried to apply certain features into the PHP core project.

My earlier post about installing Arch Linux on a MacBook Pro 5.5 is one of the most accessed posts on my site, so I figured I should write an update for the newer model.

The basic specs of my MacBook Pro 8.1 (13″) are:

• Intel Core i7-2620M @ 2.7 GHz
• 8GB (2x4GB) 1333MHz DDR3 SDRAM
• 750GB SATA @ 5400 RPM
• Intel HD Graphics 3000
• Broadcom BCM4331 802.11a/b/h/n

Installation: I have gone for a pure x86_64 install this time. The initial install was “fun”… So much so, that I would have probably abandoned Arch altogether if I did not have a vested interest in it. The latest official Arch Linux install CD (2011.08.19) does not even boot so I had to grab a testing iso. The install is fairly routine as far as a single OS install on a MacBook Pro goes. I followed the same strategy as my previous install and changed the partition table format and blessed the partition /boot was on for a faster boot-up. I could have tried GRUB2 with its EFI support, but I just stuck with what I knew worked. But “worked” is a funny term as the current Arch Linux installer will only allow you to install GRUB on the hard-drives MBR and not onto an individual partiation (which is required on MacBook Pros). So I did my first manual GRUB installation and everything booted fine!

Video: Just pacman -S xf86-video-intel and everything works.

Screen Brightness: Worked out of the box.

Keyboard Backlight: I recommend learning to touch type… but I read it works.

Touchpad: Sort of worked out of the box using xf86-input-synaptics-1.5.99.902 (see my previous post about what I consider “bugs” in synaptics finger distance calculations). That includes two/three finger right/middle clicks and two finger scrolling. Also, click with thumb and drag with finger no longer required a patched kernel module.

Wifi: Requires b43-firmware from the AUR.

Suspend to RAM: I set xfce4-power-manager to suspend on lid close it worked fine.

Webcam: Worked out of the box.

Sound: Use alsamixer to unmute the speakers.

Keyboard: Screen brightness keys worked fine. Needed to add shortcuts to XFCE for volume control and disc eject.

Fan: Appears fine out of the box, but I need to test it under more variety of load.

Anything else (bluetooth, thunderbolt) has not been tested because “meh”.

Overall, this install took me far less time to do compared to installing on the 5.5 model. There were no patched modules for the touchpad or screen brightness control and no compiling a proprietary module for the wireless. In fact, after the initial dramas with the installation media, everything basically just worked. I guess some of the reason for that is that the 8.1 model I am using was released some time early in 2011, so many of the issues people may have faced early on appear to have already been fixed.

Hi Arch community,

Arch Linux (archboot creation tool) 2012.04-1, "2k12-R2" has been released.
To avoid confusion, this is not an official arch linux iso release!

Homepage and for more information on archboot:
http://wiki.archlinux.org/index.php/Archboot

Summary:
- Bugfixes and uefi updates

Hybrid image files and torrents are provided, which include
i686 and/or x86_64 core repository. Please check md5sum before using it.

Hybrid image file is a standard CD-burnable image and also a raw disk image.
    - Can be burned to CD(RW) media using most CD-burning utilities.
    - Can be raw-written to a drive using 'dd' or similar utilities.
      This method is intended for use with USB thumb drives.

Please get it from your favorite arch linux mirror:
https://downloads.archlinux.de/iso/archboot/2012.04/
<yourmirror>/iso/archboot/2012.04/

/boot for PXE/Rescue files are provided here:
https://downloads.archlinux.de/iso/arch … 12.04/boot
<yourmirror>/iso/archboot/2012.04/boot

Changelog:

GENERAL:
- kernel 3.3.3/ LTS kernel 3.0.29
- pacman 4.0.3 usage
- RAM recommendations: 768 MB

Kernel changes:
- bump to latest 3.3.x series and bump lts to latest 3.0.x series

Removed features:
- floppy

Environment changes:
- synced with latest mkinitcpio changes
- fixed binary moving to new location in /usr
- added rEFInd boot manager support

hwdetect changes:
- fixed to work with latest userspace
setup changes:
- many grub2 fixes and grub2 cleanup
- added rEFInd support

quickinst changes:
- None

Further documentation can be found on-disk and on the wiki.
Have fun!

greetings
tpowa

One of the great things about starting a new job at an open source company is having the freedom to use the tools that suit your workflow, rather than having to suffer the indignity of whatever the IT department consider to be the lowest comon denominator. Suffice to say, I have had a lot of fun this week setting up my working environment—and the ocassional hiccough as I was forced to learn something new…

One of those “learning opportunities” consisted of trying to get my mail client, mutt to talk to the LDAP directory over SSL so that I could query the shared address book. There are a number of helpful blog posts that describe using lbdb with mutt¹. Unfortunately, after a lot of searching, I was unable to find any documentation on achieving this integration over a secure connection. I kept seeing this error:

Error: Search failed. LDAP server returned an error : 13, description: TLS
confidentiality required at /usr/lib/mutt_ldap_query line 198, <DATA> line 558.

Several hours later, and with some help from @ibeardslee, I managed to set it up, and it was definitely worth the effort.

You will need to install lbdb from the AUR:

cowerd lbdb     # 2

…and a couple of packages from the repos to make it all work:

pacman -S perl-ldap perl-io-socket-ssl netkit-bsd-finger

Then it is a matter of configuring lbdb to both query the LDAP directory and be able to be called from mutt. First, copy the config files into your $HOME:

mkdir .lbdb
cp /etc/lbdb.rc .lbdb/lbdbrc
cp /etc/lbdb_ldap.rc .lbdb/ldap.rc

And then modify the two configuration files to suit your setup: The first, $HOME/.lbdb/lbdbrc, is well commented and self-explanatory; add ldap to the methods and the nickname of your server:

METHODS="m_abook m_ldap"
LDAP_NICKS="catalyst"

The second config file, $HOME/.lbdb/ldap.rc is written in Perl and is a bit of a shocker:

%ldap_server_db = (</p>

<pre><code>'catalyst' =&gt; ['ldaps://ldap.catalyst.net.nz:636',
                'ou=Staff,ou=People,dc=catalyst,dc=net,dc=nz',
                'givenname sn cn mail', 'givenname sn cn mail',
                '${mail}', '${givenname} ${sn}']
</code></pre>

<p>);</p>

<h1>hostname of your ldap server</h1>

<p>$ldap_server = 'ldaps://ldap.catalyst.net.nz:636';
$search_base = 'ou=Staff,ou=People,dc=catalyst,dc=net,dc=nz';
$ldap_search_fields    = 'givenname sn cn mail';
$ldap_expected_answers = 'givenname sn cn mail';
$ldap_result_email     = '${mail}';
$ldap_result_realname  = '${givenname} ${sn}';
$ignorant = 0;
$ldap_bind_dn = '';
$ldap_bind_password = '';
1;

The key is to ensure that you use both the ldaps prefix and explicitely specify the SSL port, 636. Without both of these, you will get the TLS confidentiality error.

You can then test that it is working correctly by running a query:

lbdbq jemima

All going well, if there is indeed a Jemima in the shared address book, you will see her contact details miraculously appear before you. If there is more than one, you will have a list to choose from.

Finally, you just need to set up mutt to query lbdb. In your muttrc, add the following:

set query_command = "lbdbq %s 2>/dev/null"

I found that suppressing the errors made the whole experience a little smoother. You may not require it… Now, hitting Shiftq in mutt brings up a prompt to query the LDAP directory (and my abook address book that I share via dropbox). You can also access the directory by starting to type an email address and then hitting Ctrlt to see a list of possible completions.

Notes

1. Christian Schenk's post got me started.

2. A wrapper script for cower

Creative Commons image by bertop on Flickr

I recently acquired an iPad for work purposes… so the most important thing to know is how to convert video to play on it. Use handbrake, done – short blog post.

But, that would be all too simple. Often I want to watch an entire season of a show that I have collected from various sources over the years and these often have widely varying sound levels. That is quite annoying if you set the season to play and then have to adjust the volume for every episode.

Here is a simple guide to convert your videos into a format suitable for the iPad with equalized volumes. I somewhat deliberately used a variety of tools for illustration purposes, but I think the one I selected tended to be the quickest for each step. The following code snippets assume that your videos have extension “.avi”. They also destroy source files, so make a backup.

Step 1. Extract the audio track using mplayer:
for i in *.avi; do
  mplayer -dumpaudio "$i" -dumpfile "${i//avi}mp3"
done

Step 2. Make sure all audio is mp3 and convert using ffmpeg if not:
for i in *.mp3; do
  if ! file "$i" | grep -q "layer III"; then
    mv "$i" "$i.orig"
    ffmpeg -i "$i.orig" "$i"
    rm "$i.orig"
  fi
done

Step 3. Normalize the audio levels using mp3gain:
mp3gain -r *.mp3

Step 4. Stick the adjusted audio back into the video file using mencoder (part of mplayer):
for i in *.avi; do
   mv "$i" "$i.orig"
  mencoder -audiofile "${i//avi}mp3" "$i.orig" -o "$i" -ovc copy -oac copy
  rm "$i.orig" "${i//avi}mp3"
done

Step 5. Convert to iPad format using the command line version of handbrake:
for i in *.avi; do
  HandBrakeCLI -i "$i" -o "${i//avi/m4v}" --preset="iPad"
  rm "$i"
done

This is probably not the most efficient way of doing this and will become less so once handbrake can normalize volume levels by itself (which appears to be being somewhat worked on by its developers…). But when you have several seasons of a show, each with more than 50 episodes (only a few minutes each), you quickly become glad to be able to make a simple script to do the conversion automatically.

Once you have verified the finger prints of the five master keys you might want to actually configure pacman to verify packages. While this was already possible in the past, it is now way easier to get started thanks to pacman 4.0.3 and the keyring package.

If you already have played around with signature verification you could start from scratch by (re)moving the /etc/pacman.d/gnupg directory. I also assume that you have disabled signature checking by setting SigLevel = Never in /etc/pacman.conf.

All you need to do now is to install the archlinux-keyring package, initialize your local pacman keyring, finally import all the packager keys and set the trust level for the master keys. This is achieved by the following commands:

# pacman -Syu archlinux-keyring
# pacman-key --init
# pacman-key --populate archlinux

The last command will ask you to confirm each of the master keys. Now you can set SigLevel to PackageRequired for all official repositories in /etc/pacman.conf. For example:

[core]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

[extra]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

[community]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

From now on pacman will check the signatures of all packages before installation and the keys are automatically managed by the keyring package.

Note: You can also set the SigLevel globally in the [options] section. The downside is that you wont be able to install your own packages using pacman -U anymore if they are not signed by a trusted key. Future versions of pacman will have a separate SigLevel option for packages that are installed from you local hard drive.

All our packages are finally signed, pacman itself and our development tools are able to create and verify signatures for some time now. There are still some bits and pieces missing untill we can have package verification enabled by default, but we are almost there.

At some point in time you might want to switch from an untrusted setup to one that magically verifies every package. Sadly it wont be that easy and to be entirely correct paranoid users will even need to reinstall from a signed medium (which does not exist yet) as it is hard to tell if your system was already compromised. I assume most users will instead hope for the best and skip this step.

While pacman can tell you if a package was signed by a private key to a given public key, it will be up to you to decide whether that key can be trusted or not. Due to the nature of the web of trust used in PGP you wont need to verify the key of every single developer. Instead you will only need to trust five of us who have created so called master keys.

Graph of the Arch Linux web of trust

These master keys wont be used to sign packages directly but to sign the keys of every packager. If a packager key is signed by at least three of those master keys PGP will automatically trust this key as well. To sum things up, you will only need to verify five keys once and you are done. You neither need to care about verifying each key nor to kee track of developers joining or leaving the team.

You will find the finger prints of all five master keys on our website. However, it is certainly a good idea to check other sources as well. For all what we know that site could already be compromised. The more clues you can gather to verify a key the better. Here are some hints about what you could verify. But don’t stick to that list, be creative.

1. Make sure you are accessing that master key website via https.
2. Verify that the SSL certificate is valid.
3. You may also check that the certificate is issued by StratCom Ltd. to Aaron Griffin.

Each key holder has also signed their master key with their packager key. Therefore it is useful (and probably easier) to verify these packager keys as well. Let’s see what we can find out about these keys

1. Some packager keys are verified by CAcert. If you trust those you now know that their email address and name match a real person. Keep in mind that this does not mean that this person is authorized to create packages for Arch Linux.
2. Some of us have their own homepage where we also publish our keys.:
3. In that case you can check whether the whois entry matches the person’s name.
4. If they provide https access check their certificate and the name it was issued to.
5. Check if that homepage is the one that these developers advertise in their signatures, forum profiles etc..
6. If they sign their mails to the mailing list you can verify these as well. See if you can find mails from different time periods.
7. Be a stalker, use your favorite search engine to look up their names, mail address, homepage addresses but also key fingerprints and ids.

These are some basic checks you can quickly do online with a reasonable amount of effort. Of course none of these checks are absolute and even if you do them all you cannot be certain. At some point you will have to ask yourself how much effort and money an attacker would invest in compromising Arch Linux users. This is a general rule for security related topics: you may not be able to make an attack impossible, but you can try to make it too expensive.

Last but not least, if you are too lazy to verify all this yourself, ask someone you trust instead. It is a web of trust after all. This also means that if you went ahead and verified all master keys as good as possible, tell people about it. You may just write a brief blog article about it and quote the key finger prints.

Of course the best way to let everybody know that you have verified someones key is to sign it using your own. But I would advice you to only do so if you have met that key owner in person. That also means: if you ever meet a developer or trusted user: ask them to sign each others key.

I am leaving you with a list of short cuts to start with:

• Allan’s homepage
• Dan’s homepage
• My PGP keys
• Ionut’s blog

For background reminder, I started last year a ncurses twitter client called Tyrs, as it was almost the only ncurses client, some persons used it. Earlier this year, I wrote a blog post announcing that haven’t enought time and motivation anymore to carry on Tyrs. I was a bit frustrated, as I knew Tyrs had some bugs left, due to the late migration to the Urwid module (awesome python ncurses module by the way), and I knew that some persons was still using it.

The good news is, Tyrs has been forked! A warm welcome to Turses!

You might have some questions. What’s going on? A fork isn’t a big deal if nothing happen behind? Any new stuffs?

A short answer could be that I’m kind of exited about the fork. But here some more details about it.

The fork has been initiated by Alejandro Gómez a few months ago after writing some patchs for Tyrs. Starting from Tyrs, he wasn’t afraid to moves codes around, rearrange stuffs, doing some refactoring, even add some new functionalities. I must say he’s done some good and cleaver work. So it’s obvious that Turses isn’t a fork just to add a patch, but there is a lot going on, and for the best! It might sound like a programmer side view about it, but I’m confident about Turses future, and that’s what’s important for users.

About new features, I really like the new dynamic buffer managment, meaning that you now can add or remove buffers on the fly. For example you can add a buffer of some keyword research or from a user timeline, a bit like you add or remove a channel from an IRC client. That’s sweet! Another feature is the possibility to display two buffer at once. Let say for example, home and mentions timeline. I have to say that I haven’t try yet, but It looks cool!

Here the traditional screenshot, taken from the github page.

So, I’m quite happy about Turses, I really hope Alejandro will enjoy writing it, as I enjoyed writing Tyrs. If you’re using Tyrs, or even if you don’t, you really should give Turses a shot. And don’t be affraid to [reports bugs][5] or even participate, as Turses is naturaly under licence GPLv3!

I hope the best for Turses.

I was happy to note the following in the release announcement of xf86-input-synaptics-1.5.99.901:

“… the most notable features are the addition of multitouch support and support for ClickPads.”

As a MacBook Pro user, that sounded just what I needed. No more patching the bcm5974 module to have some basic drag-and-drop support. I upgraded and everything seemed to work… for a while. I began noticing weird things occurring when I was trying to right and middle click (being a two and three finger click respectively). Specifically, my fingers would sometimes be registered in a click and sometimes not.

The two finger “right” click was easy enough to figure out. If my fingers were too far apart, the two finger click was not registered. It turns out I am what I have decided to call a “finger-spreader”, as my fingers can be quite relaxed across the touchpad when I click. Fair enough, I thought… I just have to train myself to click with my fingers closer together. Then came the three finger click. All three fingers close together did not seem to register as anything. A bit of spreading and a three finger click got registered, but too much spreading and it was down to two fingers. Also, it was not actual distance between fingers that mattered as rotating my hand on the touchpad with my fingers the same distance apart could result in different types of clicks registered.

A bit of looking in the xf86-input-synaptics git shortlog lead me to this commit as a likely candidate for my issues. The commit summary starting with “Guess” and it having a comment labelled “FIXME” were the give-aways… The first thing I noticed was that the calculation of the maximum distance between fingers to register as a multitouch click was done in terms of percentage of touchpad size. That means that the acceptable region where your two fingers need to be for a two finger click is an ellipse, which at least explains why physical distance appeared not to matter.

Attempted fix #1 was to increase the maximum allowed separation between fingers from 30% to 50%. That worked brilliant for two finger clicking, but made three finger clicking even worse, which lead me to another interesting discovery… The number of fingers being used is calculated as the number of pairs of fingers within the maximum separation, plus one. For two fingers, fingers 1 and 2 form one pair, plus one is “two fingers”. However, for three fingers, there are three possible pairs: (1,2), (2,3) and (1,3). This explains the weirdness in three finger clicking; finger pairs (1,2) and (2,3) must be withing the maximum allowed separation while finger pair (1,3) must be outside that. That explains why having fingers too close together did not register as a three finger click (as it was being reported as four fingers – three pairs plus one) and why things became worse when I increased the maximum allowed separation. I filed a bug report upstream and a patch to fix that quickly appeared.

After applying that patch, multifinger clicks all work fine provided your fingers are close enough together. I do not find the required finger closeness natural so I got rid of the finger distance restrictions altogether using this patch. I am not entirely sure what I break removing that, but it appears to be something I do not use as I have not noticed an issues so far. As always, use at your own risk…

With the addition of source code PGP signature checking to makepkg, I have began noticing just how many projects release their source code without any form of verification available. Or even if some form of verification is provided, it is done in a way that absolutely fails (e.g. llvm which was signed by a key that was not even on any keyservers meaning it could not be verified). If code security fails at this point, actually signing packages and databases at a distribution end-point instills a bit of a false sense of security.

To assess how readily validated upstream source code is, I did a survey of what I would consider the “core” part of any Linux distribution. For me, that basically means the packages required to build a fairly minimal booting system. This is essentially the package list from Linux From Scratch with a few additions that I see as needed…

For each source tarball I asked the following questions: 1) Is a PGP signature available and is the key used for signing readily verified? 2) Are checksum(s) for the source available and if they are only found on the same server as the source tarball, are they PGP signed? The packages are rated as: green – good source verification; yellow – verification available but with concerns; red – no verification. Apologies to any colour-blind readers, but the text should make it clear which category each package is in…

Note that some of these packages have additional methods of verification available (e.g. those that are PGP signed may also provide checksums and file sizes), but I stopped looking once I found suitable verification. When I label a key as “readily verifiable”, that means it is either signed by keys I trust, that it is used to sign emails that I can find or it is posted on the developers personal website (which must be different from where the source code is hosted). I personally found my preferred method of verification was packages whose release announcements were signed by the same key as the source.

While you might look at that table and think there is a lot of green (and yellow) there so everything is in reasonable shape, it is important to note that the majority of these are GNU software and all GNU software is signed. Also, 15% of the packages in that list have no source verification at all. From some limited checking, it appears the situation quickly becomes worse as you move further away from this core subset of packages needed for a fairly standard Linux system, but I have not collected actual numbers to back that up yet.

This is good news for persian users and our community to introduce “arch-wiki-docs-fa” pkg in AUR. Our official activity started since more than 1 mounth ago and one of our web site`s part is wiki and persian documentations.
As other users use english offline docs with “arch-wiki-docs” in community repository ، from now persian users can use farsi offline docs via terminal with make this pkg from AUR. We promise our users that this pkg will update every month. Special thanks to Mehdi Ghaffari, our wiki and this pkg maintainer.

Posted from WordPress for Android

Aspen is a unique Python web framework developed by Chad Whitacre. I am still evaluating it, but one thing I can say unequivocally is that it’s different. Very different. Aspen divorces itself from the MVC pattern that every major — and almost every minor — web framework has been relying on for the past decade. While each framework has its perks and annoyances, they all have this underlying MVC-like interface. Aspen throws that all away.

Aspen lodges itself in a small niche between php-like spaghetti code and a complicated package structure full of python models, views, controllers, and templates. In Aspen, template and control code live in the same file, but in separate pages, called Simplates. In a rebellion against regular expression url parsing, Aspen uses the filesystem to define url structure and paths; like in the old days of statically served HTML files.

One thing I love about Aspen is how easy it is to get up and running. Aspen will serve a single HTML file with no python code without any boilerplate. It serves static files for css, javascript, and images just as easily. To add a dynamic page to your Aspen powered site, you simply use the aforementioned simplates, which include python controllers and Tornado inspired templates in the same file. Aspen provides numerous service connectors, so deployment can also be trivial.

Another great feature of Aspen is its documentation. At first glance, the docs seem sparse and rather limited. However, they are well organized, and every page seems to answer every question I have on the topic succinctly and quickly. Chad seems to have a terrific minimalist policy of telling you exactly what you want to know in as few words as possible. As a writer, I know just how difficult this is to accomplish without repeating oneself.

On the negative side, I do have misgivings as to how well Aspen’s filesystem urls it can scale. Chad has assured me that Aspen is up for running large sites. I appreciate the power of regex urls, and I don’t necessarily like having arbitrary file extensions in URLS. I think locators should be succinct and divorced from implementation details. Ideally, I believe there may be a happy medium between Aspen’s enforced folder structure, and the overly-abstract and overly-complicated url scheme Django and its kin provide.

Another downside is that I haven’t figured out a good way to unit test Aspen simplates. You can refactor much of the code into backend libraries and test that, of course, or you can start the service running and mock requests. However, I miss the TestClient feature from Django that allows me to mock a request without running the server. It provides a better analysis of code coverage, and is closer to the actual code.

Now, with the review over, I’d like to introduce a little project I’ve started to provide some vital features that Aspen core is missing.
Trembling is kind of a catch-all project with a few basic helpers and a few useful projects. Namely, trembling provides:

• Basic sessions
• Basic user authentication
• Less than basic single request messages
• A simple redirect response

Aspen itself does not define or require a model layer; you’re free to use the ORM, DBAPI connector, or NoSQL database of your choice. Trembling arbitrarily chooses mongoengine for the models. I made no effort to make pluggable backends like Django does; Aspen does not encourage excessive layers of abstraction.

I also don’t know how secure trembling’s session and auth features are. If you find a security hole, pull requests are welcome. Overall, I think the combination of Trembling and Aspen will be very useful for my future basic sites. I would also like to try it on a more complicated project just to see how well it performs.

Here we are, a new KDE minor release has been packaged!

The upstream ChangeLog is here.

The Arch Linux changes/fixes are:

• [kdebase-workspace] menu cannot navigate the categories using the keyboard FS#25718)
• [kdebase-konsole] qt 4.8 breaks some font rendering (FS#26624)
• [kdebase-workspace] pam_env.so missing in /etc/pam.d/kde (FS#28842)
• [kdebase-kwrite] Split katepart (FS#29186)
• Minor fixes in the PKGBUILDs

Have a nice update!

Hi all, as you already know GNOME 3.4 is in [testing]. And this means: we’ve telepathy-glib 0.18!

Why I’m happy? I do because this means that we can update the whole KDE Telepathy suite, so I did.

Telepathy-qt4, which was mandatory for this update, has already been update to 0.9.1 someday ago; yesterday I renamed it as telepathy-qt, as upstream did.

Today I released the KDE Telepathy 0.3.1 suite in [testing]. Because of the upstream name change, I renamed the package group to kde-telepathy, but I didn’t rename the packages yet.

Also, I introduced the kde-telepathy-meta package; now you’ve two ways to install KTP: using the kde-telepathy group or the kde-telepathy-meta package.

Please report any packaging bugs/features requests to our bug tracker, and the upstream bugs to the KDE bug tracker.

Cheers

Last year’s anime list appeared in July, so I am definitely making progress at getting this done earlier in the year. Who knows, by next year it might appear in January! Then again, it could happen in August…

As always, this is not a review of all of the year’s anime as there is far too much crap out there. Instead it is a collection of my opinions on the anime that I thought were good enough to watch in the first place. There was definitely a lot of decidedly average anime during this year. There was not necessarily anything bad about them, just there is nothing to make them stand out as worthwhile either.

Anime of the Year

Steins;Gate
(TV, 25 episodes)

It took me until at least the eighth episode to have some sort of idea what was going on here. Bananas in microwaves that turn into green goop… But even though I had no idea what was happening, the early episodes still are intriguing enough that you want to watch more and it gets even better once the plot points all begin coming together.

Recommended

Black Lagoon – Roberta’s Blood Trail
(OVA, 5 episodes)

If you like the two previous series of Black Lagoon, then you will probably like these episodes. It is just more of the same thing, but given the previous series were very, very good…

Mawaru Penguin Drum
(TV, 24 episodes)

This series made it into my Recommended list based entirely on its uniqueness. And I am sure there is some deep underlying meaning in there (probably something about abandoned children being able to find happiness) and there appears to be a large number of metaphors too (the boxes, the apple). But there were too many twists and turns for me to fully grasp what was being implied. I still have no idea what the penguins were…

Tiger and Bunny
(TV, 25 episodes)

Remember those days of getting up early on a Saturday morning to watch cartoons? Tiger and Bunny is like an American superhero cartoon, but with more awesome. Almost enough awesome for me to select it as the anime of the year. I even found the shameless sponsor promotion to be quite amusing. But in the end, there is nothing amazing or original about this series, it is just damn entertaining.

Usagi Drop
(TV, 11 episodes)

Not the usual style of show for me to watch, but it was a solid drama so the risk paid off. However, if you like this show and want to find out more by reading the manga, then you should wipe that thought completely from your mind. Pretend the manga never existed. Seriously… No arguing. Just don’t. As one blogger wrote, he should have just let sleeping, scrotum-eating, rabid dogs lie.

Average

A Certain Magical Index II
(TV, 24 episodes)

Worse than the first series and far worse than Railgun. But given both of those were very good, that is not bad in itself. However, my overall impression was that they were running out of ideas so they had to stretch out the little that they had, which resulted in a fairly slow pace. And the superpower that only negates other superpowers gets boring.

Blue Exorcist
(TV, 25 episodes)

There was a few quite good episodes in this series. However, its storyline is overly predictable and has too much filler (including a beach episode…).

Broken Blade
(OVA, 6 episodes)

I have never been a fan of mecha anime, but this was one of the better ones. Well, it was for the first four episodes anyway. I found the finish was bland. Almost as bland as the characters… But at least they were not teenagers because the addition of boring teenage coming of age crap (that normally is in mecha anime), would have killed this for me.

Deadman Wonderland
(TV, 12 episodes)

The theme park style prison where inmates have to compete in fights in order earn points to buy their continued living is not particularly original… But at least the superpowers were interesting. The ending is also far too non-conclusive, but I am not sure if it was because there was nothing to conclude or if everything was kept open to make a sequel.

Dragon Ball Kai
(TV, 97 episodes)

I watched this mainly for nostalgia reasons and because I have never seen the final 50 or so episodes from Dragon Ball Z. While this version was significantly more fast paced than the original (the 97 episodes covered what took 193 in DBZ), I think it could still have moved faster in lots of places. Also, the remake is not complete and so I still have not seen the second half of the Majin Buu Saga. Maybe one day…

Fractale
(TV, 11 episodes)

This anime started off well, but got progressively worse. But I can not actually say how it got worse… it just became bad. And I think I mean genuinely bad and not just bad because it did not live up to the potential shown in the first episode.

Gosick
(TV, 24 episodes)

This tries so hard to be a good show. But I think I am already bored of shows that involve some sort of detective with a side-kick. The first few cases are examples of entirely amateur writing. I think someone just jumped on the internet and searched for “crime cases with minor twist when solving”. It all just seemed too familiar. But the series was rescued as story did go beyond a pure detective drama and turn into something much more interesting.

No. 6
(TV, 11 episodes)

After lots of war, only six pieces of land remain. Guess which number this is? This is another series that I think started off strong but just did not go anywhere spectacular. Just a typical Dystopian Society storyline.

Subpar

The Mystic Archives of Dantalian
(TV, 12 episodes)

You can always tell a show is bad when there is some sort of super power transformation sequence that lasts so long that you starting thinking to yourself that you could have killed both of the lead characters multiple times by now and that would not necessarily be a detriment to the show. It is even worse when that superpower is unlocking a girls chest with a key and then reaching in a pulling out a book. That takes forever, but then the book has to be read aloud for it to work. With only minor exaggeration, in total this sequence takes about three quarters of every episode.

Over the last week I have been moving my blog over to Octopress, a lightweight blogging framework for Jekyll, the static site generator powering Github Pages. I had previously been posting to a tumblr page and, over the nearly four years that I had been doing that I had somehow racked up just over 4000 posts. I was not looking forward to migrating across.

However, the fact that the Jekyll project has a number of scripts for migrating from other platforms assuaged my concerns about the difficulty of this task. That sense of relief was shortlived. Neither of the two tumblr migration scripts were of any assistance: both would die during their initial runs, probably due to some funky characters in the post titles, or perhaps the posts themselves.

I certainly had no intention of trying to wade through the entire back catalogue identifying the rogue posts. Rather that admit defeat, and probably more due to a sense of misguided optimism about the “straightforward” nature of the task, I saw this setback as an opportunity to cull all of the cruft¹ from the blog and decided to manually import the fifty posts that I thought were of some interest.

Being an assiduous record keeper, all of the posts were helpfully bookmarked on Pinboard under one tag, and therefore it was simple enough to create a list of the required URLs. Armed with this list, it was just a matter of cobbling together a script to do the bulk of the work for me.

The first task was to retrieve the posts from the list:

</p>

<h1>grab files</h1>

<p>while read url; do</p>

<pre><code>wget --adjust-extension "${url}"
</code></pre>

<p>done &lt; /home/jason/Scripts/list

Then I needed to remove all of the HTML surrounding the actual posts: an awk one-liner took care of that.

</p>

<h1>strip HTML cruft</h1>

<p>for file in <em>.html; do
  awk '/<h3>/ {flag=1;next} /&lt;\/div>/{flag=0} flag {print}' "$file" > "${file%%.</em>}"
done
mkdir html &amp;&amp; mv *.html html/

The final task of this part of the migration was to convert the HTML into markdown, the format that Octopress uses. Pandoc the "universal document converter" handled that job:

</p>

<h1>convert to markdown format</h1>

<p>for file in *; do</p>

<pre><code>pandoc -f html -t markdown "$file" &gt; "$file".md
</code></pre>

<p>done

The final result was fifty markdown files holding all of my posts, almost ready to be committed to github. I say “almost” because the files still required what turned out to be a reasonable amount of cleaning up. Pandoc did a great job, for example, but would inexpicably break multi word hyperlinks over two lines. Similarly all of the internal links to my other posts pointed to the (meaningless) tumblr URLs².

Setting up Octopress was extremely simple and quick by comparison: the documentation is very helpful. There was one slight hitch, a known issue on Arch x86_64, which was simple enough to deal with.

While the migration was not entirely pain-free, I am pleased that I have done it. Tumblr's service increasingly left a lot to be desired but as it was a free service, I couldn't complain too much. Or, more accurately, when I did complain, no-one actually listened…

Indeed, moving to a paid service like Github (yes, it's free at first, but once you have enough data there you need to pay a small amount every month) makes a lot of sense. The paid services I do use, like Pinboard and Tarsnap are both inexpensive and much more reliable than their free counterparts³; and you get to invest in great software that is a pleasure to use.

Notes

1. Initially, I had set up the site as a simple holding page and dumped a whole lot of feeds into it: twitter, bookmarks, scrobbled music, etc. Those 4000 posts were mostly just that sort of internet detritus…
2. For creating redirections (Github pages do not support .htaccess) I can't recommend enough the Jekyll Alias Generator. Just. Brilliant.
3. And much more scrupulous about how they use your personal data.

After more than a year of development, we are happy to announce Mono 2.11, the first in a series of beta releases that will lead to the next 2.12 stable release.  

Flagging Packages
If you notice that a package is out-of-date (i.e., there is a newer stable release available), then please notify us by using the Flag button in the Package Details screen. This will notify the maintainer(s) responsible for that package so they can update it.
The message box portion of the flag utility is optional, and meant for short messages only. If you need more than 200 characters for your message, then file a bug report, email the maintainer directly, or send an email to the arch-general mailing list with your additional text.
Note: Please do not use this facility if the package is broken! Use the bugtracker instead.

I’m a big user of subtitles, the reason is simple, I always watch movies or series in English, even if I’m French. To help me to understand movies, I use English subtitles. In this way, it’s easier for me to follow the film even if I’m tired. It also help me to learn new words.

Like everything else, you don’t want waste your time looking for subtitles around the web, that why software come naturally for subtitle retrieving tasks. That’s exactly what Submarine does, and well.

I have to say that I didn’t really looked for this kind of program, only tried one, and I’m happy about it. I found Submarine while I was reading one of my favorite section of the Arch Linux official forum called Community Contributons, it worths looking around for some new softwares announcing, some of them has become quite famous since. Anyway, I saw this announcement maybe a month ago of this program called Submarine.

Maybe Submarine isn’t perfect, or doesn’t find every subtitles that I want, but it does a good job for sure.

Here some through about it:

• Command line: That’s fine, who want some GUI for this kind of program anyway?
• Two servers: Submarine try to retrieve from OpenSubtitle and Podnapisi. That’s sounds good to me.
• Download multiple movies at once: I found it useful for series. With series like Big Bang Theory, I like to watch several episode in a row. Therefor download all subtitles at once is faster.
• Download best rated: cool feature again, that it avoid badly ranked subtitles.
• Rename to match the movie: As VLC automatically loads subtitles if it match the movie title, it saves me some time to manually set the subtitle by rename the subtitle if needed.
• Language: More for information that a feature, but it use Vala.

If you’re using Arch Linux, you can find it in AUR and install it via yaourt -S submarine.

Source code can be seen on github, and it’s release under GPLv3. That’s it, no much more to say but thanks to the author of Submarine.

And now, I have the opportunity to be a part of that. After much pondering I decided to go for it. Resigning at the university was hard but smooth, I felt I had to take this chance and they were very supportive.
I'll be working on the infrastructure/backend side of things, I'm actually working on transcoding infrastructure right now. Working from my place in Ghent, a move to NYC at some point in the future might happen, but we'll see...

[*] When they told me the new version would be more appealing than the old, I couldn't believe that's possible. but to my own surprise they succeeded.

Hi Everyone!

I've had a lot of requests that I bring Arch Linux Lanyards back to the Arch Schwag store. After a lot of back and forth on a brand new design, I am proud to say that I finally have lanyards back in stock.

As always, thank you for supporting Arch Linux!

Dusty

Demand for Arch Linux lanyards has been growing steadily. I had intended to have a new order by new years, but I ended up dealing with a different company, and decided to do a completely new design inspired by the updated Arch Linux website. There is a subtle gradient from dark grey to black in the background that looks very dynamic, and the logo itself is crisp and clear. Each lanyard is thin and very light to wear.

The lanyards are $6 for singles, and can be purchased from The Arch Schwag Store.

2012-03-16 09:21:34: (log.c.166) server started 
2012-03-16 09:21:34: (mod_fastcgi.c.977) bind failed for: unix:/var/run/lighttpd/sockets/mywebsite.sock-0 No such file or directory 
2012-03-16 09:21:34: (mod_fastcgi.c.1397) [ERROR]: spawning fcgi failed. 
2012-03-16 09:21:34: (server.c.945) Configuration of plugins failed. Going down.

$ cat /etc/tmpfiles.d/lighttpd.conf 
d /run/lighttpd/sockets 0700 http http

I’ve spent this weekend researching a wide variety of different python libraries for a pet project that may never come to fruition. To my delight, most of the projects I was interested were documented in Sphinx and the documentation had been posted online using Read The Docs or a self-hosted site.

Some of the projects only include API documentation, while others also include helpful guides or tutorials. However, None of the projects I visited had links back to the project website and source code repository in their documentation. Some had such links hidden on a contributing or download page.

A lot of the time, my Duck Duck Go results link me directly to the documentation for a project I haven’t heard of. I read the APIs and think, “hey, I’d like to try this out.” Then I have a bit of trouble actually finding the project’s home page.

So documentation authors: Please add prominent links to your project’s home page, as well as source code repository and issue trackers to your documentation.

Some time in the last couple of days, the last of the packages in the Community repository were signed and, thanks to the tremendous work of the Arch developers and Trusted Users, you can fully implement package signing in your /etc/pacman.conf.

You can check the state of the signed packages with this expac one-liner; it will return a list of any unsigned packages:

expac -S '%r %n %g' | awk '$3=="(null)" {print $1 "/" $2}'

Now that the packages are all signed, I updated my /etc/pacman.conf to take advantage of this. My overall SigLevel setting requires signed packages, and—as of yesterday—I was able to move the last repository entry over to do the same:

SigLevel = Required DatabaseOptional TrustedOnly</p>

<p>[core]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist</p>

<p>[extra]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist</p>

<p>[community]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist</p>

<p>[multilib]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

The next step was to add my key to pacman’s keychain so that I could sign the packages that I build using ABS or from the AUR. Allan has an excellent post on setting this up.

First, import your key into pacman’s keyring:

pacman-key -r 0xB1BD4E40

Then follow the prompts as you edit the key to sign, set a trust level and save your key:

pacman-key --edit-key 0xB1BD4E40
gpg> lsign
...
gpg> trust
...
gpg> save

Then it is just a matter of changing the BUILDENV option in your /etc/makepkg.conf, which is set to !sign by default. Remove the bang and include the details of the key you wish to use:

</p>

<h1>-- Packager: name/email of the person or organization building packages</h1>

<p>PACKAGER="Jason Ryan <a href="&#109;&#97;&#x69;&#x6c;&#116;&#111;&#58;&#106;&#97;&#x73;&#x6f;&#110;&#119;&#x72;&#121;&#97;&#x6e;&#64;&#x67;&#109;&#x61;&#x69;&#x6c;&#46;&#x63;&#x6f;&#109;">&#x6a;&#x61;&#x73;&#x6f;&#x6e;&#119;&#114;&#x79;&#97;&#x6e;&#x40;&#x67;&#x6d;&#x61;&#105;&#108;&#x2e;&#99;&#111;&#x6d;</a>"</p>

<h1>-- Specify a key to use for package signing</h1>

<p>GPGKEY="B1BD4E40"

Now, when you build a package, you will be prompted for your key’s passphrase:

==> Signing package...</p>

<p>You need a passphrase to unlock the secret key for
user: "Jason W Ryan <a href="&#x6d;&#97;&#105;&#x6c;&#116;&#111;&#58;&#106;&#x61;&#x73;&#x6f;&#110;&#119;&#114;&#x79;&#97;&#110;&#64;&#x67;&#x6d;&#x61;&#105;&#108;&#46;&#x63;&#111;&#109;">&#x6a;&#97;&#x73;&#111;&#110;&#x77;&#x72;&#x79;&#x61;&#x6e;&#64;&#103;&#109;&#x61;&#x69;&#108;&#46;&#x63;&#x6f;&#109;</a>"
2048-bit RSA key, ID B1BD4E40, created 2010-08-21</p>

<p>Enter passphrase:

Enter the correct passphrase and your package is built and signed:

-> Created signature file /home/jason/Build/{pkg}.pkg.tar.xz.sig.

Creative Commons image by donovanbeeson on Flickr.

DVD playback in banshee

1. DVD playback, you can now watch your favourite DVDs with banshee. DVD menu support is also included (see the screenshot)
2. UPNP support, browse and play your media provided by a UPNP server. 

The workon script available in virtualenvwrapper is a useful tool for switching virtual environments on the fly. However, it fails for the workflow I prefer, so I wrote my own.

I prefer to keep my virtualenvs in the same folder as whatever source code or project they apply to. Normally, I put them in a folder named venv, although sometimes I use different names, most often when I have multiple virtualenvs applying to the same source tree (e.g. venv27, venvpypy).

My directory structure therefore looks like this:

/home/dusty/code/
+ project1
| + src
| | + pkg1
| | + pkg2
| + doc
| + venv
+ project2
| + src
| + venv27
| + venvpypy
+ project3
| + venv

... and so on

The problem is, I may be anywhere in the folder tree when I want to activate a virtualenv. This means I'm doing stuff like . ../../venv/bin/activate one time and venv/bin/activate another time. Trying to remember which parent directory I want often requires several tab completions to see the directory listing. This is annoying. So I wrote a bash function to activate the optionally named virtualenv in the current directory or any parent directory.

I avoid bash whenever I can, so this may not be the prettiest bash script you've ever seen.

function v {
  # activate a virtualenv
  # usage: v
  #   activate the virtualenv named venv
  # usage: v venvname
  #   activate the virtualenv named venvname
  name=venv
  if [ $1 ] ; then
    name=$1
  fi
  olddir=$(pwd)
  quit=0
  deactivate &>/dev/null
  cwd=$(pwd)
  while [ $quit -eq 0 ]
  do
    cd $cwd
    if [ $cwd == '/' ] ; then
      quit=1
    fi
    if [ -e $name ]  ; then
      source "$name/bin/activate"
      quit=1
    fi

Put this in your ~/.bashrc. Typing v on the command line will look in the current and all parent directories for a directory named venv, and will activate that venv. Typing v [venvname] will walk the same tree looking for a directory named [venvname] to activate.

If you want to make it prettier, feel free to fork it on github.

You might have noticed that we still do not have the latest version of PHP in our repositories. Unfortunately there are still several issues which prevents me from updating to 5.4.0 right away.

Concerning the php package itself the updated version of the Suhosin patch is not available yet.

In addition to this, several modules do not work with the new PHP yet. Among them is of course the suhosin extension but also APC and xdebug. The state of other extensions still needs to be researched; if you are lucky simply recompiling will do it.

While I obviously cannot recommend the use of PHP 5.4.0 on any server or production system yet, I would consider it a good idea to start testing your setup and scripts for compatibility. Also at Arch we like to play with the newest features and get hit by bugs nobody has seen before.

Therefore I set up a repository for all administrators and developers who like their packages broken. These come without suhosin (patch nor extension) and are available for both i686 and x86_64. I also packaged some modules like memcache(d), apc and geoip. Note that I will base these builds upon the [testing] repository. This might not be a problem right now, but once you see a our stock version of PHP in [testing] these package wont work without the usage of [testing] as well.

To use this repo put the following lines on top of all other repositories (including [testing]) in pacman.conf:

[php]
Server = https://repo.pierre-schmitz.com/$repo/os/$arch

For those who are all over the brand new signing stuff: Yes, all these packages are signed using my Arch packager key.

Before you get started have a brief look at the changelog and migration guide which are provided upstream. Then make sure you get all changes in php.ini merged. The easiest was to do so is probably starting with the packaged one and adjust the changes you need. For testing properly it is important to enable error reporting:

error_reporting = E_ALL
display_errors = On
display_startup_errors = On

Note that E_ALL now also includes E_STRICT by default. If you are testing third party code make sure it does not override these settings by either .htaccess or calling error_reporting(). If you find any error this way: You know the drill, report it to the upstream projects.

If you have any suggestions about the packages or configuration, let me know. Especially drop a line if you found any issues regarding extensions or web apps. It’ll help me a lot to know how many breakage such an update will cause.

I spent some time this week helping a colleague learn the ins and outs of Git rebase. He mentioned that he could not find a git introduction that summarized the basic workflow you might use when contributing to an open source project on Github. This article attempts to fill the gap.

Note that there are many possible ways to interact with git and github and none of those ways are canonically correct. The steps here will help a newcomer to git or to collaborative editing on git. I outline the steps in short form first, then summarize alternatives below.

1. Fork the remote project on github.
2. Clone your new fork.

   git clone ssh://git@github.com/[yourname]/[reponame]/
   cd [reponame]

3. Add a shortcut to upstream.

   git remote add upstream https://git@github.com/[developername]/[reponame]/

4. Optionally choose a branch to hack on.

   git checkout development_branch_name

5. Create a local copy of that branch.

   git checkout -b working_branch_name

6. Hack.
7. Commit your changes

   git add [-p] filenames
   git commit

   Repeat the hack,add,commit steps until your commits look the way you want them to.

8. Grab any commits that were added upstream while you were working.

   git checkout development_branch_name
   git pull upstream development_branch_name

9. Rebase your work onto to the end of the development branch.

   git checkout working_branch_name
   git rebase development_branch_name

10. Push your commits to your github fork for the world to see.

    git push origin working_branch_name

11. Make a pull request on github. Pick development_branch_name as the one you want your code pulled into, and working_branch_name as the one you are pulling from in your repo.
12. Oh nos! The upstream devs didn’t like your changes. That’s fine. Repeat the hack/commit step as needed. You may
    be able to just append commits to your previous commits. If you need to alter your history (e.g. because they don’t like how one of your commits looks), check the man pages for git commit --amend and git rebase -i.
13. Once you have the working_branch in a state you like, repeat the remote pull and rebase steps to ensure your commits “look like” they happened at the end of the remote developer’s branch.
14. Push your changes to your github fork, forcing an overwrite of any existing changes.

    git push -f origin

15. Issue another pull request on github; your old pull request will be updated to the new working branch.
16. Your code has been accepted upstream, congratulations on your contribution to an important open source project!

Notes

4. This represents a branch the upstream developer has pushed to their remote repository. The branch chosen really depends on where you want to develop. Some projects do all their development on master. Others push a remote branch named ‘devel’ or branches named after the feature being developed on that branch. Either way, your ultimate goal is to have your commits added to this branch on the remote repository.
5. It is possible to do your development on a single branch and rearrange commits as needed. However, branching is free in git, and storing your changes on a different logical branch will really help you identify where everything fits together.

7. Each commit should contain one logical change.
   A commit message should not contain the word “and”.
   If you have been working on two logical changes at the same time, you can index only relevant hunks using git add -p.
8. You’re checking out the original branch here. You don’t want to pull all the foreign commits into the branch you’ve been developing on; that branch should only contain your commits at this time.
9. This is the same as “moving” all your commits
   to the end of development. The idea is to have your commits as a “tail” that all come after the upstream changes. Upstream will love this because they can do a fast-forward merge of your commits. Hopefully
   there aren’t any conflicts. If there are, resolve them so that your commits can apply cleanly. You’ll want to review the rebase documentation before continuing. Also, use three way merging, it makes life so much easier.

14. There is a philisophical debate as to whether it is acceptable to rewrite public history, as Dieter has discussed in detail. In this case, since you have communicated (via the pull request) an intent to rewrite history, it’s not only acceptable, but desirable.Most upstream developers will consider it good form to edit older commits
    so that they reflect what the history “should have” looked like so that people reading your history can understand it.

In another great Pycon2012 keynote, David Beazely asked the question, “is PyPy easily hackable?” After a great talk, he answered with a decisive, “I still don’t know.” Having sprinted on Python I’d like to answer his question in a bit more detail.

I love David’s presentation style. He has a novel method of using phrases like, “blow your mind” and “this is really scary” repeatedly until they lose their meaning and you no longer feel mindblown or scared. A variety of factors, including Beazely’s thorough keynote address motivated me to join the PyPy team during the Pycon developer sprints.

I’d like to clear up one oversight in Dave’s otherwise impeachable talk. One of the PyPy devs, Holger Kregel explained to me that PyPy does not have over 1 million lines of code. I don’t have exact numbers, but for “historical reasons”, a non-python file containing Base64 encoded data was given a .py extension. When excluding this file from the line count, around half a million lines of actual Python code exist, and about a quarter of these are tests.

I was surprised how trivial it was to get started hacking on PyPy. I don’t really grok the many layers of the translation toolset and PyPy interpreter, but it’s pretty clear that the layers are well separated. I was hacking on the py3k branch of PyPy. I am happy to admit I was working primarily on changing print statements to print() functions and commas in exceptions to the as keyword.

Here are the steps to start hacking on PyPy. Notice that the hour-long translation step is not part of the procedure. PyPy has a solid test framework, and the PyPy crew are focused on a 100% test-driven-development paradigm.

1. Clone pypy (this takes a while):

   hg clone https://bitbucket.org/pypy/pypy/

2. Pick a branch to work on. There are about 80 branches. I don’t know what they all do. Popular ones during the sprints included py3k and numpy-ufuncs2
3. Pick a feature to work on. For py3k support, the list of failing tests in the buildbot is a good place to start. Numpy programmers had a list of fuctions that needed implementing, but I can’t find the link. Ask on IRC, the PyPy crew are very helpful. The bug tracker contains many features and issues that need addressing
4. Add /path/to/pypy/ to your path so you can run the pytest.py command
5. cd into the directory indicated in the buildbot output and run pytest.py path/to/test.py -k testname
6. The test will likely fail. Hack away and fix it.
7. When the test passes, commit, push to a bitbucket repo, and issue a pull request.
8. Repeat!

There are quite a few cons to working on this project. If you run hg in the pypy/modules/ directory, it will try to pick standard library modules from pypy and choke horribly. The pypy developers don’t really believe in documenting their code. Being able to tell the difference between rpython and python (which have identical syntax) is important. In general, if a module starts with “interp_” it contains rpython, but if it starts with “app_” it contains python. The code does not appear to be well-documented.

If you are hacking on Python 3 support, you need to bear in mind that the PyPy interpreter is written in Python 2. You are working on a Python 2 application that executes Python 3 bytecode!

On the positive side, rPython and Python are much easier to read and write than C. The PyPy devs are brilliant, but not intimidating. They are so confident in their test suite that they are comfortable programming in a “cowboy coding” style, hacking randomly until all the tests pass. Any one layer in the toolchain is easy to understand and develop. The IRC channel is full of friendly, knowledgable, helpful people at any time of day.

Overall, I am much less intimidated by this project than I was before I started the dev sprints. I still can’t answer the, “Is Python easily hackable?” question fully. It’s certainly easy to get started, but I don’t know how easy it is to become intimate with the project. Dave Beazely’s keynote made PyPy more approachable, and I approached it. Hopefully this article will encourage you to do the same.

I accidentally blew away my .hgrc and it took me some time to recover my 3-way merge inspired by Dan McGee’s article on the subject. So for my and your future reference here’s what you need in your .hgrc to get hg merge to give you a three merge window in vim:

[ui]
merge = vimdiff

[merge-tools]
vimdiff.executable = vim
vimdiff.args = -f -d -c "wincmd J" "$output" "$local" "$base" "$other"

When running hg merge, the effect you get is best described by Dan McGee himself:

The end result is three panes along the top (local, base, and remote respectively), and the merge work-in-progress is on the bottom. In addition, all four panes are still locked together for scrolling.

In his Pycon 2012 keynote speech on Sunday, Guido van Rossum covered many of the open “Troll” questions related to the Python community. I’ve had occasion to either complain about or defend all of the topics he covered, and he did a wonderful job of addressing them. The issues could largely be divided into two categories: those that come from a fundamental misunderstanding of why Python is wonderful (e.g. whitespace), and those that are not an issue for the core python dev team, but are being actively worked on outside of core Python (e.g event loop).

And then there’s the question of supporting Python in the web browser. I honestly thought I was the only one that cared about this issue, but apparently enough people have complained about it that Guido felt a need to address it. His basic assertion is that the browsers aren’t going to support this because nobody uses it and that nobody uses it because the browsers don’t support it.

This is a political problem. Politics shouldn’t impact technical awesomeness.

The fallacious underlying assumption here is that modern HTML applications must be supported on all web browsers in order to be useful. This is no longer true. Web browser applications are not necessarily deployed to myriad unknown clients. In a way, HTML 5, CSS 3, and DOM manipulation have emerged as a de facto standard MVC and GUI system. For example, many mobile apps are developed with HTML 5 interfaces that are rendered by a packaged web library rather than an unknown browser. Client side local storage has created fully Javascript applications that require no or optional network connectivity. There are even situations where it may not be necessary to sandbox the code because it’s trusted. Many developers create personal or private projects using HTMl 5 because it’s convenient.

Convenient. It would be more convenient if we could code these projects in Python. Web browsers can be viewed as a zero install interface, a virtual machine for these applications. Such a VM has no reason to be language dependent.

It is simply unfair to all the other programming languages and coders of those languages to say, “we can’t displace Javascript, so we won’t try.” Web browsers have evolved into a virtualization layer more like operating systems than single programs. While it is true that the most restrictive operating systems only permit us to code in Objective C, in general, it is not considerate to restrict your developers a single language or environment.

It is time (in fact, long overdue) for Python to be supported in the browser, not necessarily as an equal to Javascript, but as an alternative. The web is a platform, and we must take Guido’s statement as a call to improve this platform, not to give up on it.

Update: I just stumbled across http://www.gnu.org/software/pythonwebkit/ and I can’t wait to play with it!

Update 2: From the number of comments on this article, it appears that my article has hit some web aggregator. To those users suggesting python to javascript compilers, I’ve been a minor contributor to the pyjaco project, a rewrite of the pyjs library. It has the potential to be a great tool, and the head developer, Christian Iversen is very open to outside contributions. Let’s make it better!

I'll join the party, although I missed the actual date everyone keyed onto by a day. Thanks, Allan McRae, Andrea Scarpino, Daniel Isenmann, Dave Reisner, Ionuț Mircea Bîru, and Pierre Schmitz for already doing one of these. Also thanks to every current or former developer, TU, or active community member for all your work over the past ten years.

So here are the dates I can pull together:

• 2006-05-24: First entry in pacman.log on the first computer I installed it on.
• 2006-06-06: Registered on the Arch Linux forums.
• 2006-09-14: First package (if I remember correctly), weka, submitted to the AUR.
• 2006-11-15: First message to the pacman-dev mailing list.
• 2006-12-12: Aaron Griffin asks me if I would be interested in being a pacman developer.
• 2007-01-29: I officially become an Arch developer, focusing on pacman-related stuff.
• 2007-03-25: pacman 3.0.0 is released, the first version to have libalpm included, and the first version I both contributed to and helped drive the final release.
• 2008-06-04: Brought Allan McRae on as a developer for his work on pacman/makepkg as well as his attitude in the forums helping many a misguided user.
• 2008-11-23: My first patch to archweb.
• 2009 and 2010: "Stuff", because I'm too lazy to look up any more events that happened.
• 2011-03-24: Wrote my most commented-on blog post to date, The real story behind Arch Linux package signing.
• 2011-06-13: Sponsored Dave Reisner (aka falconindy) to be a developer as he had contributed significantly to pacman and thought it would be great to get recognition for his work. In hindsight, perhaps this was a mistake. :)
• 2011-10-13: Pacman 4.0.0 with package signing "finally" released.

Dan McGee wrote:

If you follow Arch Planet, you may have already heard the news that we are celebrating a decade of existence, with the release of 0.1 Homer on March 11, 2002. If you haven't already, grab some birthday cake and head over to Arch Planet to read several developers chronologies and wonderful words of praise for Arch Linux. There is also a brief article from The H Open Source as well as discussion on Reddit. With good fortune and a little luck, hopefully we'll be around to celebrate another 10 years!

As a way to see how far we've come, we've put together a yearly snapshot of the home page thanks to our friends at the Wayback Machine. Please don't judge our former (lack of) design too harshly! For your viewing pleasure, the home page as it existed in 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 and 2011.

Here is a gift from me and heftig for our 10th anniversary.

GNOME 3.4beta2 is available in [gnome-unstable] repository. To use this version you need to enable [gnome-unstable] repository, as top repository to pacman.conf, along with [testing] repository.

[gnome-unstable]
Include = /etc/pacman.d/mirrorlist
[testing]
Include = /etc/pacman.d/mirrorlist
[core]
...

Have fun testing these packages. Like always packaging bugs should be reported on our bugtracker, anything else considered bugs should be reported to GNOME bugzilla. Before reporting an upstream bug, be sure you search first on bugzilla in the appropriate project. I keep track on bugs discovered by our community on this todo list.
If you report an issue, please let me know to add it on my todo list.

Today Arch celebrate its 10th birthday. Allan had a very nice idea writing something about his years as archer; Pierre already did the same, and I’m going to write something about my years in this great distro.

I can’t forget the first time I heard about Arch Linux in a conversation with Antonio. I wasn’t fully satisfied by Slackware at that time, and Antonio said me to try Arch. So, I back to home to download the Arch Linux Gimmik iso; I try to install it, but I fail. Some month later the Voodoo iso was released; I try to install Arch again, but this time I success.

My adventure begins.

Probably I’m missing something here because I made this list very quickly, but I guess these are the relevant goals. Five beautiful years in which I learned a lot of new things and I’ve to thank you the whole Arch community.

Happy birthday Arch Linux!!!

Let me start with a quick joke that I saw on reddit:

Q:How do you know that somebody runs Arch?

A: They will tell you…

Now, let me tell you my Arch history.  I tried it for the fist time back in 2005 when somebody from university used it while teaching about operating systems. It didn’t stayed more than 1 hour on my computer because it didn’t had any GUI and I didn’t read anything about installation (go figure).  After using gentoo for about a year and getting angry for compiling 6 hours my system because I broke it, my good friend cippp, who used Arch and was a contributor, pushed me to start using Arch. This time I read the wiki before installing for about a week and since 2006 I’m a happy Arch user.

Here is a brief summary of my involvement with Arch Linux:

• 2006-07-05 – registered on forums
• 2009-04-04 – applied to became a TU because some packages that Allan had are not maintained anymore
• 2009-04-18 -  I’m a TU now (17 yes, 0 no, 0 abstain)
• 2009-07-03 – I launched Arch Linux Romania
• 2009-10-31 – I was invited to became a developer by JGC because I maintained telepathy and empathy in community
• 2009-11-04 – FS#16922, first contribution as developer
• 2010-01-21 -  firefox 3.6 was out and I’m the new Mozilla guy
• 2010-03-09 – GNOME 2.30 was released and it was my first time doing GNOME. At that time i wasn’t prepared at all to be the new GNOME maintainer. Somebody screwed me over!
• 2011-03-15 – resigned as TU
• 2011-04-30 – GNOME 3.0 moves in extra. This was my biggest challenge ever and the most controversial move. We were the first distro to push the major version to users. I don’t want to remember the flames…

This are the moments that I’m most proud along with a lot of trolling in the #archlinux @ freenode channel

I’m glad that I’m a part of this community,  I saw it grow from being a hipster distribution to a mainstream one, I helped growing Arch.  What is really incredible how we managed to still keep the same principles as it were laid down by Judd. Is nice to hear from time to time that people really appreciate our work.

Ten years ago Judd released the first version of Arch Linux. This is quite an age for a Linux distribution and we are still rolling. Allan just posted a brief history of mile stones that were important to him. Inspired by this nice idea…I am going to do exactly the same.

It has been a long time but it seems I started a more serious look at Linux back in April 2003. That time I joined linuxforen.de.

In June 2004 I came in contact with Arch when I was looking for a lightweight and technical easy distribution with rolling releases.

Here is the Arch Linux history from my perspective:

These are some information about my history at Arch I could gather in a reasonable amount of time. It has been fun to dig through all these old mails. Some information are hard to get or lost forever as we changed our mailing lists, forums, wiki and even switched from cvs to svn in the meantime.

I probably could have added dates to the timeline forever. But I’ll just stop here to keep it less boring. If I had missed anything important, let me know. Also I didn’t add other conference we had attended like last years FrOSCon, OpenRheinRuhr or FOSDEM as I simply have no pictures of those.

At this point I am still working on getting PHP 5.4 ready for action and helping to finalize the package signing process. I am also working on repository management and on the package building process in general. There will be separate articles about these ideas.

Arch Linux 0.1 and pacman 1.1

If you like to have a look at the very first version of Arch: I have uploaded arch-0.1-full-i686.iso to our mirrors. It’s md5sum is:

03cece0ee5ba6ffdaedb450baa7589b2

Arch Linux turns 10 today. It’s pretty spunky for a pre-teen! Inspired by some fellow developers, I figured I’d post my (alebit short) involvement with Arch over the past 2 1/2 years:

• 2009-10-22: Wiped Ubuntu, installed Arch and joined the forums
• 2010-03-21: First release of cower
• 2010-04-25: Applied for junior dev position, and was rightfully turned down!
• 2010-07-11: First commit to Arch’s codebase (Thomas later reverts it!)
• 2010-07-24: First commit to pacman
• 2010-11-04: My first breakage — later fixed
• 2010-11-30: Applied to be a TU — thanks Ionuț for sponsoring me
• 2010-12-11: Accepted as TU! (Yes: 29, No: 0, Abstain: 1)
• 2011-03-01: Applied for initscripts maintainership (I passed this role to Tom Gundersen, who’s done an awesome job)
• 2011-03-19: libfetch is removed from pacman, and my curl branch is merged
• 2011-03-20: systemd joins [community]
• 2011-06-13: Dan McGee (aka toofishes) sponsors me to be a developer (thanks, Dan!)
• 2011-10-04: My first release of mkinitcpio (0.7.3) as maintainer
• 2011-10-12: pacman 4.0.0 is released into the wild with curl
• 2011-12-20: kmod enters the repos
• 2012-02-12: systemd moves to [extra]

I consider several of the subsequent mkinitcpio and pacman releases fairly memorable as well, but I’ll leave those off the list. I can honestly say that working with Arch has been an amazing experience. Ignoring the 1s and 0s, the people backing Arch Linux are an amazingly intelligent group of individuals, all with differing strengths working towards the same goal.

Having only picked up Linux as my daily operating system in the summer of 2009, I can honeslty say that without the flexibility and transparency that Arch offers, there’s no way I could have learned as much as I have in the past few years, elsewhere. Thanks to what I’ve learned I’ve become involved, to varying degrees, with other upstream projects. I’ve made numerous contributions to systemd, util-linux, and kmod.

I’m truly honored to be a user and developer of this awesome distro. Here’s to 10 more years!

Today marks the 10 year anniversary since the first release of Arch Linux. I have been involved in Arch for only about half that time, but I thought it might be quite interesting to make a time-line of the major things I remember being involved with in my history with Arch Linux.

I actually first used Arch Linux in a virtual machine for quite some time prior to the start of this time-line. I was running a Linux From Scratch install and was looking to add better package management. Then I realized that it was quite a waste of time building my own packages when Arch was actually quite similar to what I was trying to achieve in a Linux install. The toolchain was the part I liked dealing with the most in LFS and now I get to play with it anyway, so things worked out well!

Here is a brief summary of my involvement with Arch Linux:

2007-06-09 – Joined forums
2007-06-20 – First non-VM Arch install
2007-12-04 – First patch to the pacman project (fixed compressing man pages in makepkg)
2008-01-09 – Applied to be a Trusted User (TU)
2008-01-23 – Voted in as a TU (23 yes, 1 abstain!)
2008-06-03 – Joined the forum moderation team.
2008-06-04 – Became a Arch Linux developer!
2008-08-17 – First big rebuild. I made all ncurses applications use the wide-character library for better locale support.
2008-09-01 – Interviewed in the Arch Newsletter – yes, there used to be one of those…
2009-04-04 – Resigned as a TU
2009-04-22 – First complete toolchain build after becoming the toolchain maintainer (gcc-4.4 update)
2009-06-07 – Broke module-init-tools due to missing a file failing to compile in the PKGBUILD. (makepkg would catch such an error these days and the missing file would be noted using the checkpkg utility)
2009-07-17 – libjpeg-7 rebuilds enter [extra]. I think this was the first major rebuild to severely break KDEmod.
2009-08-02 – pacman-3.3.0 released with package splitting support in makepkg
2010-03-05 – Fix a bug introduced by my first patch to makepkg… In true Allan style, the commit message contains a typo so does not point at the right bug!
2010-07-02 – Resigned as forum moderator. Mainly because I wanted to be able to be grumpy on the forums when…
2010-10-18 – python3 transition – Probably the single most controversial thing I have done in Arch…
2011-02-28 – A minor bash update resulted in non-booting systems…
2011-03-16 – pacman-3.5.0 released for which I contributed reading the sync databases directly from the downloaded tarball, disk space checking and storing package information in a hash table.
2011-10-13 – pacman-4.0.0 release with full PGP signing support for packages and databases.

Of course there have been lots of other things I have contributed to Arch, including 1000s of package updates and about 400 patches to pacman. But these are the ones I found most memorable.

If you are an Arch Linux user and you are not using aurphan, you can take a moment to hang your head in shame before you click on to whatever the next thing is. The package description sums up perfectly what aurphan does:

Finds packages in need of maintainers, bug fixes and patches. Adopt today!

Running aurphan -a, for example, will identify any AUR packages in your database that are not currently maintained; allowing you to log in to the AUR and hit the ‘Adopt’ button, leaving you feeling all worthy and fulfilled.¹

If, like most Archers, you have a compulsive desire to automate everything then clearly typing (or even remembering to type) aurphan -a into a terminal periodically is as ludicrous as it is onerous. The obvious answer is some shell script and a cron job.

First, the cron job:

crontab -e
@hourly aurphan -a >/tmp/aurphans

Then, as part of my dwm-status script, a function to check that there are no new additions to the file and, if there are, flash a highlight:

aur(){</p>

<pre><code>aurphans="$(awk '$0 !~ /^No /' /tmp/aurphans | wc -l)"
if [ $aurphans -gt 0 ]; then
    echo -en "\x03*\x01"
fi
</code></pre>

<p>}

The awk filter excludes a null return, which is “No aurphans found.” Now, if any of the packages that I use from the AUR are disowned, I’ll be the first to know about it.

Notes

1. Actual feeling may differ depending on the package you have adopted and the amount of prescription painkillers that you are currently ingesting…

PGP file formats and data are not the simplest thing to wrap your head around, so pgpdump is a very handy tool to have available. Although similar to gpg --list-packets, pgpdump output is a bit more verbose and descriptive.

pgpdump is a PGP packet visualizer which displays the packet format of OpenPGP (RFC 4880) and PGP version 2 (RFC 1991).

There is an online version available as well if you have never used it before and want to see what it is. Here is the output when parsing a detached signature file.

$ pgpdump testfile.sig
Old: Signature Packet(tag 2)(70 bytes)
    Ver 4 - new
    Sig type - Signature of a binary document(0x00).
    Pub alg - DSA Digital Signature Algorithm(pub 17)
    Hash alg - SHA1(hash 2)
    Hashed Sub: signature creation time(sub 2)(4 bytes)
        Time - Wed Mar  7 15:42:52 CST 2012
    Sub: issuer key ID(sub 16)(8 bytes)
        Key ID - 0x5C2E46A0F53A76ED
    Hash left 2 bytes - d4 1a 
    DSA r(159 bits) - ...
    DSA s(160 bits) - ...
        -> hash(DSA q bits)

With package signing now a reality in Arch Linux, I wanted a way of displaying the signature data stored in the package databases on the website. I was a bit disappointed after searching for tools that could be used to help me do this, and shelling out to gpg or another tool seemed like a bad way of going about it.

It turns out the PGP packet format is completely insane, which is probably why there isn't much in the way of library support nor many binary implementations outside of GnuPG. Luckily the pgpdump code is a lot more terse than diving into GnuPG, so I was able to use it as a good reference point for writing a PGP packet-parsing library in Python.

The two existing projects I found related to PGP in Python appear to be abandoned: OpenPGP, last updated in July 2005, and pgpmsg, which has no listed date, and no source files.

Discovering there was nothing great out there, I present my first package on PyPi, python-pgpdump. The home for the code is on github. Designed so it can hopefully parse every type of PGP packet down the road, the first release did only what I needed it to do- parse detached signature packet data. I've already had one contributor add public key packet parsing support, so that is a good sign that I didn't code things too stupidly. Since then, I have also added some parsers for other packets and public key algorithms.

The end result of this is a usable library to extract the details from the PGP signature data with no dependencies except the Python standard library. This allows us to show the PGP key ID of the person who created the signature on the Arch Linux website, and from there, match the key ID to a developer's known key in the system.

In [1]: import base64, pgpdump

In [2]: data = base64.b64decode("iEYEABECAAYFAk824S0ACgkQXC5GoPU6du3rsgCeOXqjR0K
NIIfdZNhLZtzvU5d7oc0AoJHaJRAgGv4r6kAKgsNjfMBttHwM")

In [3]: pgp_data = pgpdump.BinaryData(data)

In [4]: pgp_data.data
Out[4]: bytearray(b'\x88F\x04\x00\x11\x02\x00\x06\x05\x02O6\xe1-\x00\n\t\x10\\.F
\xa0\xf5:v\xed\xeb\xb2\x00\x9e9z\xa3GB\x8d \x87\xddd\xd8Kf\xdc\xefS\x97{\xa1\xcd
\x00\xa0\x91\xda%\x10 \x1a\xfe+\xea@\n\x82\xc3c|\xc0m\xb4|\x0c')

In [5]: packets = list(pgp_data.packets())

In [6]: packets[0]
Out[6]: <SignaturePacket: DSA Digital Signature Algorithm, SHA1, length 70>

In [7]: packets[0].key_id
Out[7]: '5C2E46A0F53A76ED'

In [8]: packets[0].datetime
Out[8]: datetime.datetime(2012, 2, 11, 21, 44, 13)

I definitely welcome pull requests for any improvements. I would highly recommend looking at the pgpdump C source if you need to figure out what is going on in addition to reading RFC 4880.

Migrating my old posts from blogspot made me think about this kind of posts about KDE packaging changes in Arch Linux.

They were nice, weren’t they? So, I’ll start again to list the Arch Linux changes in ever KDE release, both minor and major.

First of all, the upstream ChangeLog is here. Then, the (very few) Arch Linux changes in this release are:

• [kdeutils-ksecrets] Requires qca-ossl (FS#28746)
• Install the kigo module.dsc file
• New kde-l10n-{he,id,ug} packages
• Drop the kde-l10n-kn package

The upgrade is suggested as this minor release has a lot of upstream bugfixes.

It’s true. I bought again my old domain andreascarpino.it :)
This time I’ll not leave it, I promise!

To begin I’m moving my blog back from blogspot. No, I’ll not use wordpress this time, but octopress! I’m fascinated how it works (static html pages!) and, of course, by ruby too. However, this will require a bit of work (BTW, I already moved half of the posts there).

You should really give octopress a try.

You must be very desperate to read a “Hello World” post on an empty blog using the default WordPress theme. However, I am glad you are still reading. In future you will find articles about Arch Linux development, PHP and anything related here.

There is nothing more to say here, but an article with just one paragraph looks kind of lame, don’t you agree? If you are still with me you should subscribe to the RSS feed which is hidden somewhere in order to get notified about more exciting articles.

Poor MySQL, always getting picked on. However, it is pretty bad that a relational database in the year 2012 cannot handle simultaneous non-conflicting inserts on a table when transactions are involved.

First, some background. The InnoDB documentation does have a section dealing with deadlocks and possible ways to avoid them. There, it offers this wonderful fact:

You can get deadlocks even in the case of transactions that just insert or delete a single row.

A relatively recent update to archweb, the Arch Linux website code, added an inotify-based update script that watches the package repository database files for updates, and then kicks off the script for updating the website database.

Previously this ran on a fixed schedule via cron, so updates to the database were serialized, and things worked as expected. However, it now often occurs that the databases for the same repository for both architectures are updated at nearly the same time (within seconds of each other). The inotify-based script will thus kick off two simultaneous updates; however, these updates should always be non-conflicting and operating on different rows in the various touched tables.

After a few trials, I started noticing updates not running to completion, and tracebacks such as the following (slightly trimmed for brevity).

$ ./manage.py reporead_inotify -v0
Process Process-17:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
...
  File "/srv/http/archweb/devel/management/commands/reporead.py", line 223, in populate_pkg
    populate_files(dbpkg, repopkg, force=force)
  File "/srv/http/archweb/devel/management/commands/reporead.py", line 283, in populate_files
    pkgfile.save(force_insert=True)
  ...
  File "/usr/lib/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (1213, 'Deadlock found when trying to get lock; try restarting transaction')

The first step to debugging deadlocks is to ask the database what the heck happened. Here is the output from SHOW ENGINE INNODB STATUS;.

------------------------
LATEST DETECTED DEADLOCK
------------------------
120130 14:53:49
*** (1) TRANSACTION:
TRANSACTION 32116C73, ACTIVE 1 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 12 lock struct(s), heap size 3112, 8 row lock(s), undo log entries 4
MySQL thread id 393561, OS thread handle 0x7f61cc503700, query id 5884410 localhost 127.0.0.1 archweb-update update
INSERT INTO `package_files` (`pkg_id`, `is_directory`, `directory`, `filename`) VALUES (70297, 1, 'usr/share/', NULL)
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 139734 n bits 224 index `package_files_pkg_id` of table `archweb`.`package_files` trx id 32116C73 lock_mode X insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
 0: len 8; hex 73757072656d756d; asc supremum;;

*** (2) TRANSACTION:
TRANSACTION 32116C74, ACTIVE 1 sec inserting
mysql tables in use 1, locked 1
11 lock struct(s), heap size 1248, 6 row lock(s), undo log entries 2
MySQL thread id 393562, OS thread handle 0x7f61d18ce700, query id 5884411 localhost 127.0.0.1 archweb-update update
INSERT INTO `package_files` (`pkg_id`, `is_directory`, `directory`, `filename`) VALUES (70298, 1, 'usr/', NULL)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 0 page no 139734 n bits 224 index `package_files_pkg_id` of table `archweb`.`package_files` trx id 32116C74 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
 0: len 8; hex 73757072656d756d; asc supremum;;

*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 139734 n bits 224 index `package_files_pkg_id` of table `archweb`.`package_files` trx id 32116C74 lock_mode X insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
 0: len 8; hex 73757072656d756d; asc supremum;;

*** WE ROLL BACK TRANSACTION (2)

Trying to debug this, I came across a great blog post that describes InnoDB deadlocking on concurrent inserts. More specifically, this is due to "gap locking".

What is unfortunate is the post describes this only happening when a UNIQUE index is involved. In the case detailed above, the package_files_pkg_id index is most-definitely not unique, and you can see the values going into that column (70297 and 70298) are different.

At this point, I've been hesitant to "solve" this problem by hacks in the code, as it seems like a total failure at the database level more than anything else. This seems like as good of an excuse as any to get the site switched to using PostgreSQL now that the hosting server isn't responsible for four MySQL databases anymore.

Hello Arch Linux community! If any of you are attending Pycon 2012 in Santa Clara, CA this week, make sure to bump into me. Or contact me to schedule a meetup.

I just had an interesting phone call from someone at the “Windows Service Center”. Note I spelled “Center” the non-Australian way, because I do not think they were based here. Anyway, I get these calls about once a month so I get to have a bit of fun. The conversation went like this…

Caller: “Hello. Is Mr. Mac-Rae there?” (my name was pronounced with a discernible gap…)
Me: “Yes”
Caller: “(pause…) Are you Mr Mac-Rae?”
Me: “Yes”
Caller: “I am calling from the Windows Service Center.”
Me: “My windows are quite clean from all the rain we had lately and they are showing no leaks.”
Caller: “Um… as in Windows on your computer.”
Me: “Oh, right. That would make lots more sense.”
Caller: “Are you the main user of the computer?”
Me: “Which one?”
Caller: “You have more than one?”
Me: “Yes. Six” (which is technically correct – the best kind of correct – but most of them are broken and serve as door stops)
Caller: “We will start with the one you mainly use.”
Me: “OK then. Yes, I am the main user of the computer I mainly use.”
Caller: “There are lots of viruses on the internet these days and our records indicate that your computer is infected.”
Me: “Which one?”
Caller: “Excuse me?”
Me: “Which computer is infected?”
Caller: “Probably all of them, but I will help you check.”
Me: “How would I know such a thing?”
Caller: “Have you ever seen a ‘This page can not be displayed’ message in Internet Explorer?”
Me: “I can honestly say I have never seen that message from Internet Explorer on my computer.”
Caller: “OK. We better check your computer anyway. Can you go to the Control Panel?”
Me: “I don’t think my computer has one of those.” (XFCE calls it a “Settings Manager”)
Caller: “I will guide you to it. Is your computer on?”
Me: “No. I just turned it off because I was told it might be infected with a virus.”
Caller: “Can you please turn it back on?”
Me: “OK, but it will take a while. It seems to be booting really slowly lately for some reason…”
Caller: “OK, I will wait.”

(a few minutes interlude while I respond to an email)

Me: “OK, my computer is on.”
Caller: “Good. Now click on the button in the lower left corner of the screen.”
Me: “I do not have a button there.” (slight lie… I have a “Show Desktop” button there)
Caller: “What operating system do you use?”
Me:“Plan 9″
Caller: “Sorry, what was that?”
Me:“Plan 9″
Caller: “…” (hung up)

So I probably went a bit overboard with choosing Plan 9 as the operating system I was running, but I was getting bored. One day I really should follow instructions and see what they try to get me to do to my computer.