Planet Arch Linux

How I moved from Nginx to Caddy

January 18, 2020 11:57 AM

Nginx has been my webserver of choice for several years now. But I had always some issues with nginx that bothered me for quite a while: Weak defaults (no TLS on default, weak ciphers, no OSCP stapling on default, …) The configuration is very verbose (this doesn’t need to be something bad) New technologies like (QUIC or zstd compression need ages until their are available in downstream) Dealing with Let’s Encrypt / certificates has always been an error-prone process (I never got that working for a longer period of time without issues).
Christian Rebischke

rsync compatibility

January 15, 2020 08:14 PM

Our rsync package was shipped with bundled zlib to provide compatibility with the old-style --compress option up to version 3.1.0. Version 3.1.1 was released on 2014-06-22 and is shipped by all major distributions now.

So we decided to finally drop the bundled library and ship a package with system zlib. This also fixes security issues, actual ones and in future. Go and blame those running old versions if you encounter errors with rsync 3.1.3-3.

Christian Hesse@Official News

Now using Zstandard instead of xz for package compression

January 04, 2020 08:35 PM

As announced on the mailing list, on Friday, Dec 27 2019, our package compression scheme has changed from xz (.pkg.tar.xz) to zstd (.pkg.tar.zst).

zstd and xz trade blows in their compression ratio. Recompressing all packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup.

We already have more than 545 zstd-compressed packages in our repositories, and as packages get updated more will keep rolling in. We have not found any user-facing issues as of yet, so things appear to be working.

As a packager, you will automatically start building .pkg.tar.zst packages if you are using the latest version of devtools (>= 20191227).
As an end-user no manual intervention is required, assuming that you have read and followed the news post from late last year.

If you nevertheless haven't updated libarchive since 2018, all hope is not lost! Binary builds of pacman-static are available from Eli Schwartz' personal repository (or direct link to binary), signed with their Trusted User keys, with which you can perform the update.

Robin Broda@Official News

My pacman.conf file

January 01, 2020 08:12 PM

Many users don’t modify their pacman.conf file. Either because they think there is not so much to configure or because they are afraid to break something. In this short article I want to highlight some nice options, that make my daily use with Arch Linux a lot easier. First of all, here is my pacman.conf without comments: [options] HoldPkg = pacman glibc Architecture = auto IgnorePkg = Color TotalDownload CheckSpace VerbosePkgLists ILoveCandy SigLevel = Required DatabaseOptional LocalFileSigLevel = Optional [testing] Include = /etc/pacman.
Christian Rebischke

Xorg cleanup requires manual intervention

December 20, 2019 01:37 PM

In the process of Xorg cleanup the update requires manual intervention when you hit this message:

:: installing xorgproto (2019.2-2) breaks dependency 'inputproto' required by lib32-libxi
:: installing xorgproto (2019.2-2) breaks dependency 'dmxproto' required by libdmx
:: installing xorgproto (2019.2-2) breaks dependency 'xf86dgaproto' required by libxxf86dga
:: installing xorgproto (2019.2-2) breaks dependency 'xf86miscproto' required by libxxf86misc

when updating, use: pacman -Rdd libdmx libxxf86dga libxxf86misc && pacman -Syu to perform the upgrade.

Andreas Radke@Official News

Traefik BasicAuth

December 05, 2019 09:47 PM

In this short blog article we revisit traefik and add password authentication to our reverse proxy example.Password authentication means we use a (user,password) tuple for the login. We don’t want to safe our password in clear text, therefore we need to encrypt it. At this moment, traefik supports three hash algorithms: MD5, SHA1, BCrypt. Two of them are considered to be broken, hence you should use BCrypt: $ htpasswd -nbB myName myPassword myName:$2y$05$c4WoMPo3SXsafkva.
Christian Rebischke

primus_vk>=1.3-1 update requires manual intervention

November 25, 2019 01:03 PM

The primus_vk package prior to version 1.3-1 was missing some soname links. This has been fixed in 1.3-1 so the upgrade will need to overwrite the untracked soname links. If you get an error like:

primus_vk: /usr/lib/ exists in filesystem
primus_vk: /usr/lib/ exists in filesystem

when updating, use:

pacman -Syu --overwrite=/usr/lib/,/usr/lib/

to perform the upgrade.

Giancarlo Razzolini@Official News

Arch Conf 2019 Report

November 17, 2019 02:00 PM

During the 5th and 6th of October, 21 team members attended the very first internal Arch Conf. We spent 2 days at Native Instruments in Berlin having workshops, discussions and hack sessions together. We even managed to get into, and escape, an escape room! It was a great and productive weekend which we hope will continue in the next years. Hopefully we will be able to expand on this in the future and include more community members and users.
Conference Posts

Reproducible Arch Linux Packages

November 11, 2019 11:00 AM

Arch Linux has been involved with the reproducible builds efforts since 2016. The goal is to achieve deterministic building of software packages to enhance the security of the distribution. After almost 3 years of continued effort, along with the release of pacman 5.2 and contributions from a lot of people, we are finally able to reproduce packages distributed by Arch Linux! This enables users to build packages and compare them with the ones distributed by the Arch Linux team.
Morten Linderud

New kernel packages and mkinitcpio hooks

November 10, 2019 09:41 PM

All our official kernels: linux, linux-lts, linux-zen and linux-hardened, do not install the actual kernel to /boot anymore.

The installation is done by mkinitcpio hooks and scripts, as well as removals. There is no need for any manual intervention.

The intention is to make the kernel packages more self-contained, as well as making the boot process more flexible, while also keeping it backwards compatible.

As of now, only mkinitcpio has hooks for handling kernels installations and removals. We do not ship any for dracut yet, but it will have similar hooks in the near future.

Giancarlo Razzolini@Official News

Traefik as Reverse Proxy

November 06, 2019 04:09 PM

A few days ago I had the joy to configure a reverse proxy. My first thoughts went to Nginx or Apache, but I forced myself to destroy the filter bubble and get in touch with some new software. Therefore I had a lookon traefik. traefik is written in Golang and can act as reverse proxy and loadbalancer. So let’s talk about a specific use case. I have the following services that I want to make available behind a reverse proxy:
Christian Rebischke

Clarification regarding recent email activity on the arch-announce list

October 25, 2019 08:27 PM

Today, one email was sent to the arch-announce mailing list that was able to circumvent the whitelisting checks that are done by the mailman software. This was not due to unauthorized access and no Arch Linux servers were compromised.

We have implemented measures to make sure this does not happen again, by using mailman's poster password feature. We are also making sure, these simple whitelist checks are not used anywhere else.

Edited to add: There was a second email that was also sent today, in order to make sure the poster password feature was working. That email did not circumvent any check and was intentionally sent.

Giancarlo Razzolini@Official News

Pacman 5.2 Release

October 21, 2019 11:48 AM

Nothing like a new pacman release to make me locate the password to this site…

Tradition dictates I thank people who have contributed to the release (as well as genuinely meaning the thanks!). We had 29 people have a patch committed this release, with a few new names. Here is the top ten:

$ git shortlog -n -s v5.1.0..v5.2.0 | head -n10
   108  Eli Schwartz
    38  Allan McRae
    30  morganamilo
    24  Andrew Gregory
    20  Dave Reisner
     9  Jan Steffens
     6  Michael Straube
     4  Jonas Witschel
     4  Luke Shumaker
     3  Que Quotion

We have a clear winner. Although I’m sure that at least half of those are in responses to bugs he created! He claims it is a much smaller proportion… And a new contributor in third.

What has changed in this release? Nothing super exciting as far as I’m concerned, but check out the detailed list here.

We have completely removed support for delta packages. This was a massively underused feature, usually made updates slower for a slight saving on bandwidth, and had a massive security hole. Essentially, a malicious package database in combination with delta packages could run arbitrary commands on your system. This would be less of an issue if a certain Linux distro signed their package databases… Anyway, on balance I judged it better to remove this feature altogether. We may come back to this in the future with a different implementation, but I would not expect that any time soon. Note a similar vulnerability was found with using XferCommand to download packages, but we plugged that hole instead of removing it!

Support for downloading PGP keys using the new Web Key Directory (WKD) was added to pacman. Both pacman-key and makepkg will also look there by default with the latest GnuPG release. This prevents DoS attacks through people adding very large numbers of signatures to PGP keys. The attack scope was limited for Arch Linux anyway, as most people obtain the pacman keyring through the archlinux-keyring package.

The much maligned --force made its way to /dev/null. The --overwrite option has been a replacement for over a year and is a precision surgical instrument compared to the blunt hammer of --force.

There is a small user interface change for searching files databases with -F. Specifying the -s option was redundant, so removed. More information such as package group and installed status is shown in the search results, bringing the output inline with -Ss.

The split of makepkg into smaller and extendable components continued. You can now provide new source download and signature verification routines (e.g. if you are living in the past and want to support cvs:// style URLs). We also added support for lzip, lz4 and zst compressed packages. Arch Linux will switch zst by default in the near future.

Under the hood, we are in the process of changing our build system from autotools to meson. This is relatively complete, but there still was a decent churn of patches to meson files as we approached release. You can build pacman from the release tarball using meson if you want to test. Next release is likely to be meson only. (Edit: you can’t test meson with the 5.2.0 tarball as it is missing a couple of the meson build files.)

Expect the release to land in Arch Linux “soon”. Expect to see another blog post in a year or so when I make the next release…

Allan@Allan McRae

Required update to recent libarchive

October 16, 2019 12:43 PM

The compression algorithm zstd brings faster compression and decompression, while maintaining a compression ratio comparable with xz. This will speed up package installation with pacman, without further drawbacks.

The imminent release of pacman 5.2 brings build tools with support for compressing packages with zstd. To install these packages you need libarchive with support for zstd, which entered the repositories in September 2018. In order for zstd compressed packages to be distributed, we require all users to have updated to at least libarchive 3.3.3-1. You have had a year, so we expect you already did update. Hurry up if you have not.

If you use custom scripts make sure these do not rely on hardcoded file extensions. The zstd package file extension will be .pkg.tar.zst.

Christian Hesse@Official News

Systemd Mail

October 12, 2019 01:46 PM

In this small article I am going to explain how to setup a small systemd service for notifications in case of failing systemd services. You’ll need the following software for it: systemd a mail transfer agent (postfix, qmail, exim, name your poison) sendmail (or any other application that can send mails) I chose sendmail. First create /usr/local/bin/systemd-mail: #!/bin/bash sendmail -i -t <<ERRMAIL To: <your mail address> From: systemd <root@$HOSTNAME> Subject: [$HOSTNAME] $1 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 $(systemctl status --full "$1") ERRMAIL Then create this systemd service:
Christian Rebischke

`base` group replaced by mandatory `base` package - manual intervention required

October 06, 2019 10:09 AM

The base group has been replaced by a metapackage of the same name. We advise users to install this package (pacman -Syu base), as it is effectively mandatory from now on.

Users requesting support are expected to be running a system with the base package.

Be aware that base as it stands does not currently contain:
- A kernel
- An editor
... and other software that you might expect. You will have to install these separately on new installations.

Robin Broda@Official News

Login via Yubikey on Linux (U2F)

September 28, 2019 10:06 PM

I was very happy with my HMAC challenge-response solution for my Yubikey, but when I wanted to configure my i3 status bar to show the current state of the key I ran into issues. The problem was that I couldn’t see the state for the HMAC challenge. Watching the state (shall I press a button now to activate the key) for GPG worked fine, but I had trouble with the HMAC challenge.
Christian Rebischke

Login via Yubikey on Linux (HMAC)

September 27, 2019 04:03 PM

In this small article I want to explain how to use your Yubikey as 2-factor device for logins on Linux. I used the “Yubikey 5” for this article. If you use an older one, some option will maybe not work. Make sure to read before reading further. You need the following Arch Linux packages for this tutorial: yubico-pam yubikey-manager yubikey-personalization yubico-c If you have a fresh Yubikey, the second slot or second configuration should be free, but you can verify this with using the following command:
Christian Rebischke

How to trace Linux signals

September 26, 2019 09:10 PM

Did you ever run into the problem, that a random process on your hosts is running amok and killing other processes? If so, you know how painful it is to find the process. But there is a solution for it: systemtap. Just install systemtap on your system, write a small stap script for it and run it, and it will show you the evil process: #!/usr/bin/stap # sigkill.stp # Copyright (C) 2007 Red Hat, Inc.
Christian Rebischke

astyle>=3.1-2 update requires manual intervention

August 26, 2019 06:39 AM

The astyle package prior to version 3.1-2 was missing a soname link. This has been fixed in 3.1-2, so the upgrade will need to overwrite the untracked soname link created by ldconfig. If you get an error

astyle: /usr/lib/ exists in filesystem

when updating, use

pacman -Suy --overwrite usr/lib/

to perform the upgrade.

Antonio Rojas@Official News

tensorflow>=1.14.0-5 update requires manual intervention

August 20, 2019 10:22 PM

The tensorflow packages prior to version 1.14.0-5 were missing some soname links. This has been fixed in 1.14.0-5, so the upgrade will need to overwrite the untracked soname links created by ldconfig. If you get an errors like so

tensorflow: /usr/lib/ exists in filesystem
tensorflow: /usr/lib/ exists in filesystem
tensorflow: /usr/lib/ exists in filesystem

when updating, use

pacman -Suy --overwrite=usr/lib/,usr/lib/,usr/lib/

to perform the upgrade.

Konstantin Gizdov@Official News

E-ink home display

July 22, 2019 07:37 PM

I've always wanted an e-ink status display in my living room to view the weather forecast, news and public transport information. Previously I've used a SHA2017 Badge with the following app which showed a weather forecast for the following four days. So I've decided to scale up to a nice 7.5" e-ink screen which I can hang on the wall. To control the e-ink screen I've taken an Raspberry Pi Zero W since it's easier to develop with then an ESP32. To hold the e-ink screen I've gotten an Ikea RRIBBA which perfectly fits the e-ink screen and leaves enough space to fit an e-ink SPI controller and a RaspberryPi.

e-ink back panel

When I started playing around with drawing images on the e-ink screen with the official Waveshare Python driver, I noticed a blank and an image update took around 50 seconds with 100% cpu. This is too slow for a status display so I started profiling with a simple test program. The Python profiler concluded that writebytes was called for the most of the time, which is a function of the python SPIDev module. It does a write call to the SPI device for every pixel individually which was the first issue to tackle. A newer version of this driver included the 'writebytes2' function which can write a Python iterable at once, this led to a significant improvement in this commit.

Waveshare also sells e-ink panels with a third color which lead to unrequited looping since my panel is black and white. The example code first clears the panel, then generates a buffer and writes it to the device simply generating the buffer up front saved a small amount of "panel updating" time. The code to generate the buffer was also optimized.

After all these changes the panel updates with a Raspberry Pi Zero W in ~ 10 seconds and a tiny bit faster on a Raspberry Pi 3 in ~ 8 seconds. The driver code can be viewed here. Now all that was left is to write my own status page for my living room. The e-ink panel fetches my local weather, public transport and Dutch news the code which drives the display below can be read here. The final display can be viewed below, the frame hangs on a nail with a barrel jack connector for a 5V power supply.

e-ink wall mount

In the future I would like to include a graph of the predicted rain for the following hour since cycling in the rain isn't always fun :-)

Jelle van der Waa ( Van der Waa

Zsh Performance

July 21, 2019 07:45 PM

I use zsh for a pretty long time now. It began with zsh + grml configuration, went over the famous powerlevel9k (where I helped implementing a few features like svn support) and currently ended with my own zsh configuration: Hikari-ZSH I have to admit I have been quite happy with powerlevel9k. It had a rich feature set and I have been in love with all these shiny UTF-8 icons and powerline graphics.
Christian Rebischke

My Way to Wayland

July 12, 2019 08:03 PM

I guess everybody knows that X11 aka Xorg is a pain in the ass and a security nightmare. Therefore it shouldn’t be such a suprise that I think about switching to Wayland for a long time now. And it looks like it’s finally the day, where I can switch to wayland without effects on my convenience. TL;DR here is the link to my dotfiles with the whole configuration: But first let’s sum up what I need:
Christian Rebischke

libbloom>=1.6-2 update requires manual intervention

July 11, 2019 01:07 PM

The libbloom package prior to version 1.6-2 was missing a soname link. This has been fixed in 1.6-2, so the upgrade will need to overwrite the untracked soname link created by ldconfig. If you get an error

libbloom: /usr/lib/ exists in filesystem

when updating, use

pacman -Suy --overwrite usr/lib/

to perform the upgrade.

Felix Yan@Official News

Reproducing Arch [core] repository packages

June 27, 2019 04:37 PM

As Arch Linux we are working on reproducible builds for a while and have a continuous test framework rebuilding package updated in our repositories. This test does an asp checkout of a package and builds it twice in a schroot, we do not try to reproduce actual repository packages yet. In the end this is however what we want to achieve, giving users the ability to verify a repository package by rebuilding it on their own hardware.

repro was created to achieve this goal, it creates a build chroot with the packages installed during build (from the .BUILDINFO file), sets SOURCE_DATE_EPOCH accordingly, fetches the correct PKGBUILD and then builds the package. This tool however does not run in a CI environment yet, so a bash script was hacked together to build all our [core] (232) packages one by one leading to 0% reproducibility with the following issues:

  • makepkg options differed, these options are recorded in BUILDINFO but not set yet by repro.
  • Packages where not reproducible (108 due to makepkg recording false sizes in .PKGINFO).
  • PKGBUILD fetching logic failed (21 packages).
  • Failed to download source files due to DNS issues (popt, libpipeline, acl, mlocate).
  • Packages did not build due to OOM and other issues (lib32-gcc-libs, gcc-obj, gcc-libs, gcc-go, gcc-fortran, gcc, fakeroot).
  • asp failed to get package due unknown reasons (libusb).
  • Packages not reproducible (s-nail, amd-ucode, syslinux, texinfo, tzdata, patch, .. and more).
  • libpcap GPG verification failed.
  • Builds with different packages installed leading to a different BUILDINFO due to an issue in repro (unknown).

Logs of the process can be found here.

This shows that still a lot has still to be done for reproducible Arch Linux, in the next pacman release the size issue should be resolved. Which will lead to at least some reproducible packages! Repro has to be improved and non reproducible packages sorted out. In a few months I intend to retry reproducing [core] packages and have at least > 0% reproducibility!

Jelle van der Waa ( Van der Waa

mariadb 10.4.x update requires manual intervention

June 27, 2019 01:40 PM

The update to mariadb 10.4.6-1 and later changes configuration layout as recommended by upstream.

The main configuration file moved from /etc/mysql/my.cnf (and its include directory /etc/mysql/my.cnf.d/) to /etc/my.cnf (and /etc/my.cnf.d/). Make sure to move your configuration.

Instantiated services (like mariadb@foo.service) are no longer configured in separate files (like /etc/mysql/myfoo.cnf). Instead move your configuration to configuration blocks with group suffix in main configuration file, one for each service. A block should look something like this:

datadir = /var/lib/mysql-foo
socket = /run/mysqld/mysqld-foo.sock

Like every mariadb feature update this requires the data directory to be updated. With the new configuration in place run:

systemctl restart mariadb.service && mariadb-upgrade -u root -p
Christian Hesse@Official News

Mini DebConf Hamburg 2019

June 20, 2019 11:37 AM

The reproducible builds project was invited to join the mini DebConf Hamburg sprints and conference part. I attended with the intention to get together to work on Arch Linux reproducible test setup improvements, reproducing more packages and comparing results.

The first improvement was adding JSON status output for Arch Linux and coincidently also OpenSUSE and in the future Alpine the commit can be viewed here. The result was deployed and the Arch Linux JSON results are live.

The next day, I investigated why Arch Linux's kernel is not reproducible. The packaging requires a few changes for partial reproducibility:

export KBUILD_BUILD_HOST="arch"

One of the remaining issue is CONFIG_MODULE_SIG_ALL which signs all kernel modules to allow loading of only signed kernel modules. If there is no private key specified a key will be generated which is always non-reproducible. A solution for this problem hasn't been found, as providing a key in the repository might also be non-optimal. Apart from this issue, the vmlinuz-linux image is also non-reproducible which needs to be further investigated.

Further packages where investigated which currently do not reproduce in our test framework.

  • s-nail due to recording of MAKEFLAGS which is under investigation for fixing.

  • keyutils was fixed for embedding the build date in it's binary with this patch

  • nspr has been made reproducible in Arch Linux with the following change.

Plans where made to extend the reproducible builds test framework for Arch Linux and start reproducing real repository packages on the test framework. Pacman was also packaged for Debian inclusion so that it's easier to bootstrap Arch containers/chroots from a Debian install.

A big thanks to all the organizers of mini DebConf Hamburg for organizing the event!

Jelle van der Waa ( Van der Waa

Arch Conf in October

May 26, 2019 02:19 PM

Fellow Archers! We are happy to announce, that we will be hosting a community-centric (developers, trusted users and support staff) event in October (the weekend of the 5th and 6th to be exact) this year. For more information, see the about page. Arch Conf will happen at Native Space (the community space of Native Instruments GmbH). For details on the exact location and how to get there, please check out the travel page.
Conference Posts

External encrypted disk on LibreELEC

May 05, 2019 12:00 AM

Last year I replaced, on the Raspberry Pi, the ArchLinux ARM with just Kodi installed with LibreELEC.

Today I plugged an external disk encrypted with dm-crypt, but to my full surprise this isn’t supported.

Luckily the project is open source and sky42 already provides a LibreELEC version with dm-crypt built-in support.

Once I flashed sky42’s version, I setup automated mount at startup via the script and the corresponding umount via this way:

// copy your keyfile into /storage via SSH
$ cat /storage/.config/
cryptsetup luksOpen /dev/sda1 disk1 --key-file /storage/keyfile
mount /dev/mapper/disk1 /media

$ cat /storage/.config/
umount /media
cryptsetup luksClose disk1

Reboot it and voilà!


If you want to automatically mount the disk whenever you plug it, then create the following udev rule:

// Find out ID_VENDOR_ID and ID_MODEL_ID for your drive by using `udevadm info`
$ cat /storage/.config/udev.rules.d/99-automount.rules
ACTION=="add", SUBSYSTEM=="usb", SUBSYSTEM=="block", ENV{ID_VENDOR_ID}=="0000", ENV{ID_MODEL_ID}=="9999", RUN+="cryptsetup luksOpen $env{DEVNAME} disk1 --key-file /storage/keyfile", RUN+="mount /dev/mapper/disk1 /media"
Andrea Scarpino

Automated phone backup with Syncthing

May 04, 2019 12:00 AM

How do you backup your phones? Do you?

I use to perform a copy of all the photos and videos from my and my wife’s phone to my PC monthly and then I copy them to an external HDD attached to a Raspberry Pi.

However, it’s a tedious job mainly because: - I cannot really use the phones during this process; - MTP works one in 3 times - often I have to fallback to ADB; - I have to unmount the SD cards to speed up the copy; - after I copy the files, I have to rsync everything to the external HDD.

The Syncthing way

Syncthing describes itself as:

Syncthing replaces proprietary sync and cloud services with something open, trustworthy and decentralized.

I installed it to our Android phones and on the Raspberry Pi. On the Raspberry Pi I also enabled remote access.

I started the Syncthing application on the Android phones and I’ve chosen the folders (you can also select the whole Internal memory) to backup. Then, I shared them with the Raspberry Pi only and I set the folder type to “Send Only” because I don’t want the Android phone to retrieve any file from the Raspberry Pi.

On the Raspberry Pi, I accepted the sharing request from the Android phones, but I also changed the folder type to “Receive Only” because I don’t want the Raspberry Pi to send any file to the Android phones.

All done? Not yet.

Syncthing main purpose is to sync, not to backup. This means that, by default, if I delete a photo from my phone, that photo is gone from the Raspberry Pi too and this isn’t what I do need nor what I do want.

However, Syncthing supports File Versioning and best yet it does support a “trash can”-like file versioning which moves your deleted files into a .stversions subfolder, but if this isn’t enough yet you can also write your own file versioning script.

All done? Yes! Whenever I do connect to my own WiFi my photos are backed up!

Andrea Scarpino

Arch signoff

April 02, 2019 07:37 PM

Arch sign off tool

Since some time Arch has been letting users become testers which can sign off packages in [testing] repository's. The idea behind allowing users and not only the Arch team sign off packages as known good is that packages can be moved earlier or bugs and issues found earlier. To sign off a package you need to login into Arch Linux's website and go to the sign off page to sign off a package. Haavard created a tool to be able to sign off packages from the command line which makes it easier to sign off by doing it interatively.

This tool has now been adopted by Arch as the official sign off tool and has been packaged in the extra repository. Issues can be reported here.

If you want to become an Arch Linux tester, feel free to apply here. A special thanks goes out to the current testing team and haavard for creating this awesome tool!

Jelle van der Waa ( Van der Waa

My new hobby

March 06, 2019 09:22 PM

A few years ago, sitting in an emergency room, I realized I'm not getting any younger and if I want to enjoy some highly physical outdoor activities for grownups these are the very best years I have left to go and do them. Instead of aggravating my RSI with further repetitive motions on the weekends (i.e. trying to learn how to suck less at programming) I mostly wrench on an old BMW coupe and drive it to the mountains (documenting that journey, and the discovery of German engineering failures, was best left to social media and enthusiast forums).

Around the same time I switched jobs, and the most interesting stuff I encounter that I could write about I can't really write about, because it would disclose too much about our infrastructure. If you are interested in HAProxy for the enterprise you can follow development on the official blog.

anrxc@Adrian Caval

Bug Day 2019

January 01, 2019 05:20 PM

Hey all. smile

We will be holding a bug day on the weekend of January 5th and 6th, to start off the year with a cleaned up bugtracker.

The community is encouraged to canvass the bugtracker and find old bugs, figure out which ones are still valid, track down fixes and suchlike. smile

Feel free to join #archlinux-bugs at that time in order to reach a bug wrangler and get more input on a bug. Or just post to the bug tracker.

Links: … 29410.html
Open bugs, sorted by last edit date: core/extra and community

eschwartz@Forum Announcements

Arch Linux @ Reproducible Build Summit Paris

December 13, 2018 10:37 PM

Write up of the reproducible summit

Three members of the Arch Linux team attended the Reproducible Build Summit 2018 in Paris this week to work together with the reproducible ecosystem to work on reproducible build issues. The other participants where from a lot of different projects and companies such as Debian, NixOS, Guix, Alpine, openSUSE, OpenWrt, Google, Microsoft and many more. The summit was organized by letting attendees work with a small subset of the attendees on issues which they are interested in and trying to find solutions and discuss ideas. At the end of the day there time for hacking together on solutions. The event was very open and there was a lot of collaboration between projects which have different goals!

The Arch Team has worked on the following topics:

  • Packaging & updating more reproducible build tools in our repos, disorderfs was updated to the latest version and disorderfs was updated after a pytest fix from Chris Lamb for diffoscope. Reprotest, the tool to test if something is reproducible has been added to [community].
  • A note has been made that we should investigate if the Arch ISO is reproducible. At least one possible issue is that squashfs images are not reproducible and Arch should consider switching to squashfskit which creates reproducible squashfs images.
  • Discussed adding a JSON endpoint for fetching the reproducible build status of Arch Linux packages on
  • Sharing reproducible build issues cross distros.
  • Discussed how to rebuild Arch Linux packages and test if they are reproducible.
  • Discussed how to verify before installing a package if a package is reproducible.
  • Debian's Kernel is reproducible, but Arch's isn't. We started investigating why ours isn't reproducible, as one goal is to get [core] reproducible as first repo.
  • Investigate PGO (profile guided optimisation) reproducibility issues for Firefox and Python.

And much more! It has left us with a lot of "homework" to continue making Arch Linux more reproducible!

A huge thanks to the organizers and sponsors of the Reproducible build summit!

Jelle van der Waa ( Van der Waa

Arch Linux ARM on the Allwinner NanoPi A64

December 09, 2018 12:37 PM

Arch Linux ARM on a NanoPi A64

I've obtained two NanoPi A64's a long while ago and recently thought of setting them up as a HA cluster as an exercise. Since setting it up with real hardware is a lot more fun then with VM's or containers. And I wanted to try out aarch64 and see how well that fares on mainline Linux.

The first part of setting it up created the partitions and rootfs on the sd card. For this I've just followed the "Generic AArch64 Installation". The more challenging part was setting up U-boot, clone it and follow the 64 bit board instructions. All that is required now is to install a boot.scr file in /boot on the sdcard, download the boot.cmd file and create a boot.scr with mkimage from uboot-tools with mkimage -C none -A arm64 -T script -d boot.cmd boot.scr.

That should get the NanoPi A64 booting, note that 4.20 is required for the ethernet controller to work, luckily Arch Linux ARM offers an linux-rc package since as of writing this article 4.20 is still not released yet.

Jelle van der Waa ( Van der Waa

archlinux-keyring update required before December 1 2018

October 18, 2018 06:39 PM

archlinux-keyring 20181018-1 re-enables my PGP key for packaging. As any package updates on my behalf requires this version (or greater) to proceed without errors, users should update archlinux-keyring before December 1 2018.

Prior to this date, there will be no new packages signed by my key. The list of affected packages: … ainer=Alad


Alad@Forum Announcements

packer renamed to packer-aur

August 14, 2018 04:31 PM

The famous AUR helper `packer` has been renamed to `packer-aur` in favor of the Hashicorp image builder `packer` (community/packer)

Shibumi@Forum Announcements

Arch User Magazine

August 03, 2018 11:37 AM

A blast from the past, the Arch User Magazine

It's almost 10 years ago that Ghost1227 created the Arch User Magazine and this week I got reminded about it's existence. I found that the original domain where the magazine was hosted was no longer owned by Ghost1227, but by using the way back machine I was able to retrieve two of the three editions of the magazine.

The original forum thread about the first magazine can be found here and the first and second magazine. There should be a third edition, but I couldn't find it via the way back machine.

Enjoy reading this part of Arch history and I hope someone recreates the user magazine!

Jelle van der Waa ( Van der Waa

Arch monthly July

August 01, 2018 03:00 PM

Archweb updates

The Arch Linux website has been updated and it's search functionality was expanded to make it able to find the 'archlinux-keyring' by searching for 'archlinux keyring'. This was contributed by an external!. Another small visual improvement was made by removing some empty spaces in provides.


AURpublish was added to [community] by eschwartz, a tool to manage your AUR packages.

Dropping luxrender packages

Lukas proposed dropping luxrays, luxrender and luxblend25 packages from [community]. The proposal went through without opposition and embree2 was also dropped in the process.

Python 3.7 in [testing]

Python 3.7 finally landed in [testing] after a painful rebuild period with many packages requiring fixes due to the async keyword or C ABI/Compiler changes.

Enforcing 2FA on Github

This does not impact Arch, but the Github repo used for the development of the Arch security Tracker, website and some mirroring now enforces 2FA in light of the recent Gentoo Github repo incident

Removal of openjdk 9, phasing out Java 7

Anthraxx removed openjdk 9 from the repos since it is EOL and nothing depends on it. Java 7 will be phased out as well soon.

New TU

Filipe Laíns has been accepted as a new TU, read his proposal and results here.

New TU applicant

A new application has arrived, voting is currently underway.

Acroread package compromised

The acroread package was compromised by a user who took over the orphan package and uploaded a new version with

aurweb 4.7.0

A new version of the AUR is deployed with new features and bugfixes.

Linux package source moved

Linux package source moved to github along with changes in the PKGBUILD.

Pacman 5.1.1 release

Pacman 5.1.1 was released containing several bugfixes.

Jelle van der Waa ( Van der Waa

libutf8proc>=2.1.1-3 update requires manual intervention

July 14, 2018 04:55 PM

The libutf8proc package prior to version 2.1.1-3 had an incorrect soname link. This has been fixed in 2.1.1-3, so the upgrade will need to overwrite the untracked soname link created by ldconfig. If you get an error

libutf8proc: /usr/lib/ exists in filesystem

when updating, use

pacman -Suy --overwrite usr/lib/

to perform the upgrade.

Antonio Rojas@Official News

Arch Linux at FrOSCon

July 10, 2018 07:37 PM

Yet another shoutout for FrOSCon, which will be held 25th and 26th of August. Arch Linux will have a devroom with talks so far about Linux Pro Audio and our general Infrastructure / Reproducible build.

Thanks to Stickermule there will be Arch Linux sticker to hand out. Stickermule

Jelle van der Waa ( Van der Waa

Arch monthly June

July 02, 2018 07:37 PM

Archive cleanup

The Arch Archive has been cleaned up, the discussion started in this mail thread. The archive server was running out of space and therefore needed some cleaning, all packages which are not required for reproducible builds where removed (and where from 2013/2014/2015). Packages from these years should also be available at the internet archive.


There will be an Arch Linux Devroom on the Sunday of FrOSCon with talks and the possibility to meet members of the team.

Python2 modules cleanup

A proposal has been send out to remove 'orphan' python2 modules. As a start of phasing out python2 packages.

Package guidelines improvements

Foxboron proposed improving the package guidelines.

Core/extra cleanup

Core and extra has been cleaned up a bit, removed packages where pcmciautils, speedtouch and zd1211-firmware.

AUR package compromised

As expected from the AUR, anyone can upload a package or adopt one and change it. This happened to acroread on Sunday and some other packages, always review packages you build from the AUR before building.

Jelle van der Waa ( Van der Waa

Arch monthly May

June 02, 2018 07:37 PM

Pacman release

Finally! A new pacman release, this version adds some critical bits for reproducible builds and the pacman repository has been shed of misc tools which are now in pacman-contrib. More details in the changelog and on reddit


For reproducible builds, every package in the repository build on a users system should create exactly the same package as the repository package. To be able to achieve this the packages which where installed in build chroot are recorded in a BUILDINFO file (man BUILDINFO) which is added in the .pkg.tar.xz package. BUILDINFO files where added a while ago in pacman, but not every package contains them yet! Interestingly enough even a rolling release distro contains packages from 2013, these are now being rebuild! This also ties in to the cleanup of, since the archive server is almost full and the 2013/2014/2015 directories will be removed. If you have a good network connection and want to mirror the archive, reach out!

pkgconf replaces pkg-config

As can be read on the mailing list, pkgconf has now replaced pkg-config.

GCC 8 in [core]

The latest version of GCC 8 lands in [core], this enables more warnings by default so older packages might fail to build if they enable -Werror.

Jelle van der Waa ( Van der Waa

Pacman-5.1 – Don’t Use the Force, Luke!

May 28, 2018 11:18 PM

Wow… look at all the cobwebs around here! No posts in two years. But the need for a pacman release post has dragged me back. I clearly still remembered the password, so that is a bonus!

As is tradition, before I get in to details, I need to thank everyone for their help in making this release. Here are the top 10 committers:

$ git shortlog -n -s v5.0.0..v5.1.0
    82  Allan McRae
    60  Andrew Gregory
    45  Eli Schwartz
    16  Ivy Foster
    10  Dave Reisner
     9  Christian Hesse
     9  Gordian Edenhofer
     8  Alastair Hughes
     7  Rikard Falkeborn
     6  Michael Straube

(I win!) Lots of new names there which is always really appreciated. And as usual a long tail of contributors submitting the occasional patch – there were 48 contributors in total.

Onto what has changed in this release. There is a lack of what I would call a killer feature in this release. Mostly a lot of small changes that improve usability, which is why there was so much time between releases. Here is a detailed list of changes. However, there are a few things worth highlighting.

There is a new option --overwrite, which is a replacement for to often misused --force (hence the release name). This allows fine grained control of what files pacman is safe to ignore conflicts with. Handling the latest upgrade requiring user intervention in Arch Linux would now look like:
pacman -Syu --overwrite usr/lib/ can even use globs when specifying the files to overwrite. Not only is specifying exact files to overwrite a lot safer than the old --force, there are also some common sense restrictions there too (you can’t overwrite a directory with a file, or force package installs with conflicting files).

We have also added a --sysroot option that will replace --root. Basically, this now works the way people will expect – for example, the configuration file used is the one in the specified root, and not the local one. This does require a bit more setup while creating a new install root, but hopefully will be a lot more robust.

We have also added support for reproducible builds. This was mostly ensuring all files had the same timestamp and obeyed the SOURCE_DATE_EPOCH standard. We also added a .BUILDINFO file within each package, recording information about the environment a package was built in. This allows scripts to regenerate the build environment to demonstrate a package is reproducible (particularly important in rolling release distros).

There was also improved support for debugging packages. Split packages now produce a single debug package instead of one for each split package. This makes it easier to get all required debug symbols for a particular package (and hopefully easier for distros to carry these packages…). Also, we include relevant source files in the debug packages, allowing us to step through the code.

Finally, I killed off the “contrib” directory as it was taking excessive amounts of pacman developer time. That means no more checkupdates, paccache, … However, this has been picked up as a separate project, which is available by installing pacman-contrib in Arch Linux.

As always, this is a bug free release. But if you spot something you think is a bug, please file a bug report and we can assign blame – which is more important than fixing! (The pool for developer who created the first pacman bug of this release is still open at the time of posting.)

Allan@Allan McRae

IWD: the new WPA-Supplicant Replacement

May 13, 2018 08:08 PM

I just want to inform you all that I have pushed IWD version 0.3 into community.
IWD is a new wireless daemon and aims to replace wpa_supplicant in the future.
I have created a first wikipage for the package as well here:

IWD comes with a more secure approach. It doesn't use OpenSSL or GnuTLS. Instead it uses different Kernel functions for cryptographic operations.

If you want to know more you can checkout this video here:

Shibumi@Forum Announcements

js52 52.7.3-2 upgrade requires intervention

May 04, 2018 08:27 PM

Due to the SONAME of /usr/lib/ not matching its file name, ldconfig created an untracked file /usr/lib/ This is now fixed and both files are present in the package.

To pass the upgrade, remove /usr/lib/ prior to upgrading.

Jan Alexander Steffens@Official News

glibc 2.27-2 and pam 1.3.0-2 may require manual intervention

April 20, 2018 07:45 AM

The new version of glibc removes support for NIS and NIS+. The default /etc/nsswitch.conf file provided by filesystem package already reflects this change. Please make sure to merge pacnew file if it exists prior to upgrade.

NIS functionality can still be enabled by installing libnss_nis package. There is no replacement for NIS+ in the official repositories.

pam 1.3.0-2 no longer ships pam_unix2 module and pam_unix_*.so compatibility symlinks. Before upgrading, review PAM configuration files in the /etc/pam.d directory and replace removed modules with Users of pam_unix2 should also reset their passwords after such change. Defaults provided by pambase package do not need any modifications.

Bartłomiej Piotrowski@Official News


February 23, 2018 08:34 PM

Solving Battleships with SAT
Kyle Keen

zita-resampler 1.6.0-1 -> 2 update requires manual intervention

February 22, 2018 07:57 AM

The zita-resampler 1.6.0-1 package was missing a library symlink that has been readded in 1.6.0-2. If you installed 1.6.0-1, ldconfig would have created this symlink at install time, and it will conflict with the one included in 1.6.0-2. In that case, remove /usr/lib/ manually before updating.

Antonio Rojas@Official News

Arch monthly January

February 06, 2018 12:37 PM

Arch Linux @ FOSDEM

Arch Linux Trusted Users, Developers and members of the Security team have been at FOSDEM. Next year there will be more stickers hopefully and maybe a talk, but it was great to meet some Arch users in real life, discuss and even hack on the Security Tracker.

TU Application: Ivy Foster

A new TU applied, you can read the sponsorship here.

New DevOps member Phillip Smith

A new member joined the sysadmin/devops team. This is the team which maintains the Arch infrastructure such as the forums, AUR and wiki.

Jelle van der Waa ( Van der Waa

Arch monthly December

January 01, 2018 12:37 PM

Arch Linux @ 34C3

Arch Linux Trusted Users, Developers and members of the Security team have been at 34C3 and even held a small meetup. There was also an assembly where people from the irc channel could meet each other. Seeing how much interest there was this year, it might be worth it to host a self organized session or assembly with more stickers \o/

Fosdem 2018

Arch Linux Trusted Users and Developers will be at Fosdem 2018 in February. We don't have a booth or developer room but you can probably find us by looking for Arch stickers or hoodies :-)

2017 Repository cleanup

The repository's will be cleaned of orphan packages, which will be moved to the AUR, where they can be picked up and taken care of.

AUR 4.6.0 Release

A new version of aurweb has been released on December third. It brings markdown support for comments and more Trusted User specific changes.

Happy 2018!

I wish everyone a happy 2018 and keep on rolling :)

Jelle van der Waa ( Van der Waa

Arch monthly November

December 08, 2017 11:11 AM

New TU Andrew Crerar

Andrew Crerar applied to become a Trusted User and was accepted! Congratulations! His intentions is to move firefox-develop from the AUR to [community]

77% Reproducible packages

Currently 77% of the packages are reproducible, note that we do not vary everything yet in the two builds. For example filesystem, build patch and other options can be varied.

Pro-audio mailing list

For audio enthusiasts there is a new mailing list to discuss audio packaging, development and usage etc..

GCC and GCC-multilib merged

Now that 32 bit support is dropped, the normal GCC package has gained support to build multilib packages, simplifying packaging.

Mime-types replaced with mailcap

Mime-types is now replaced by mailcap in this change.

Arch Linux at 34C3

A few Arch Linux Developers and Trusted users will be at 34C3 in Leipzig, if you are there, meet us there! A certain Arch user was recruited after talks at congress!

Analysis of AUR and Official Arch Repository data

Brian Caffey has made some a analysis of the AUR and the Arch repositories.

Jelle van der Waa ( Van der Waa

Writing text in Unity

December 04, 2017 06:40 AM

Writing text in Unity isn't that easy, at least if you want to generate text with single gameobjects to be displayed in 3D and not only as a flat UI text. Every single letter must be dragged and dropped to its place to form a word.

Personally I was frustrated and didn't find a solution on the internet, so I decided to write this small script on my own which generates text in the editor while you are typing as you can see in the animation. For this example I used the letters from the Unity Asset Store package "Simple Icons - Cartoon Assets" (!/content/59925). Maybe you will find this helpful or has more ideas to improve it, if so then please let me know.😉

A detailed description how to use this small script can be found in my repository on GitHub (
ise ( Isenmann

Start VR development with the right toolkit

November 29, 2017 12:28 AM

More or less one year ago I have started with developing in VR and tried several plugins and tools in Unity to implement all the basic stuff like teleporting, grabbing, triggering and so on.

The first thing you will probably try or see in Unity is the official SteamVR plugin from Valve if you have a Vive like me. Basically this SDK has everything you will need to realize your project, but it's really a very hard start if you try several things with the SteamVR SDK directly. There is no much documentation or example scenes where you can have a look at. For me it was more frustrating than helpful, so searching the Unity Asset Store and also the internet you will maybe find the VRTK (Virtual Reality Toolkit) from Harvey Ball (aka TheStoneFox).

It's the best toolkit you can get if you are try to do something in Unity for VR. You get tons of documentations for nearly all use cases you can imagine. Furthermore he has created lots of examples where you find nearly all that stuff sorted and splitted in different scenes to show you how to use them. Every single script he has written is licensed under the MIT license and can be studied directly in the SDK or in his GitHub repository for the toolkit.

A very active community is discussing stuff at its own Slack channel. Even a Youtube channel exists where he posts tutorials or doing a live Q&A session to answer your questions.

But why do I tell you this? Because Harvey Ball has done an absolute astonishing job on this toolkit. You don't have to fiddle around with each single SDK for each vendor, you don't have to think about grabbing an object, teleporting around, using an object, realize a button, realizing an usable door or anything you can think of. You can such use the VRTK and use nearly all available VR headsets out there directly and start to code your idea or project right away.

And the best thing, he has decided right from the beginning that he give this away all for free! This is even more astonishing if you see all the effort which is behind such a toolkit. These are all reasons to give something back to Harvey. How? You can decide to become a patreon on his patreon page, start contributing directly in the toolkit if you are a coder, contact Harvey and ask how you can help. Right now he is really looking for some donations to go on with the development of VRTK. It would be a shame if VRTK is dying just because of the lack of support. So give and show him a little bit of love and help for this really useful and totally necessary toolkit for VR development in Unity!

Look at all the links I have posted here and decide on your own how important this project is.
ise ( Isenmann

Reproducible Arch Linux?!

November 26, 2017 12:37 PM

The reproducible build initiative has been started a long time ago by Debian and has been grown to include more projects. Arch is now also in the process of getting reproducible build support, thanks to the of hard work of Anthraxx, Sangy, and many more volunteers. In pacman git patches where landed to support reproducible builds which will be included in a hopefully soon next stable release! Meanwhile with help of the rebuild infrastructure rebuilds have been started!

Currently 77% of the 17% tested packages are reproducible as can be found here. This page is fed by the work done by two Jenkins builders, which currently build the whole Arch repository.

The builder builds the package twice in different environments and then uses diffoscope to find differences in packages. Usually the differences are due to timestamps :-). Now that we have some results of rebuilds, we can start fixing our packages. The work I did so far:

  • Fixing 404 sources of our packages, some of the source failures where due to being used and not

    This has been fixed in SVN. Also old pypi links needed to be fixed

  • One package's .install file contained a killall statement, I'm not sure why but it shouldn't be required so it was eradicated

  • Integrity mismatch, so upstream did a ninja re-release, annoying but fixed

  • Imagemagick's convert sets some metadata in the resized png's which makes reproducible builds fail. Since it does not adhere to SOURCE_DATE_EPOCH.

  • Missing checkdepends on pytest-runner, which is automatically downloaded by the build tools but that failed in the reproducible build. Some simply adding the depdency to checkdepends fixed it.

As you can see, only one of the bullet points was really an reproducible build issue the others where packaging issues. So I can conclude that reproducible builds will increase the packaging quality in the Arch repository. Having the packages in our repository always build-able will also help the Arch Linux 32 project.

The Arch reproducible project still needs a lot of work, to make it possible to verify a package build as a user against the repository package.

P.S.: If you are at 34C3 this year and interested, visit the reproducible build assembly.

Jelle van der Waa ( Van der Waa

Using the WRLD Unity SDK with a stencil mask object

November 24, 2017 09:32 AM

Maybe you have heard about the great WRLD project, which provides a great way to display real world map data in your project. Furthermore they provide several different SDKs to access these datas. 

For a small project I needed some map data to visualize them in a 3D scene inside Unity. Using the Unity SDK from WRLD you can easily access those data and they will be displayed in your scene. Sadly they render the map all over your scene and their is no restriction in size. At least I haven't found any, even if you use their script attached to a GameObject with a specific size, the map will be displayed all over the scene. 

After some fails and searching the web I stumbled upon a video showing the usage of WRLD in a AR environment. There they do exactly what I needed. Luckily the video was made by WRLD and they also provided two very good blog posts where they explained how they have done it. With the help of these blog posts I have implemented it without all the AR stuff and came up with the proof of concept you can see in the animation. 

The displayed cube is used as a stencil mask for the map and if you move the cube or the map, only the part of the map which is inside the cube will be rendered. Also new tiles of the map are loaded dynamically depending on the main camera in the scene. I have published the Unity project on GitHub to provide the solution ready to use for your project and also to archive it for myself. You will need a valid API key from WRLD, just register at their website and generate one for your needs. Then insert your API key at the WRLD Map GameObject:

The project includes the WRLD Unity SDK which you also find in the Asset Store of Unity. But be careful if you replace the included one with the official one from the store, because I have done some changes they have mentioned in their blog posts. So make sure to apply the code changes if you replace the integrated WRLD SDK.

Hope you will find it useful. If you find a bug or have useful hints then let me know, because I'm quite new in Unity and thankful for anything related to it. 
ise ( Isenmann

Arch monthly October

November 11, 2017 10:11 AM

This is the second edition of Arch monthly, mostly due to the lack of time to work on Arch weekly. So let's start with the roundup of last month.

New TU David Runge

David Runge applied to become a Trusted User and was accepted! He mentioned to have a huge interest in pro-audio, so hopefully there will be improvements made in that area!

Farewell 32 bit

After nine months of deprecation period, 32 bit is now unsupported on Arch Linux. For people with 32 bit hardware there is the Arch Linux 32 project which intends to keep 32 bit support going.

AUR Changes Affecting Your Privacy

The next aurweb release, which will be released on 2017-12-03, includes a public interface to obtain a list of user names of all registered users. This means that, starting on 2017-12-03, your user name will be visible to the general public. The user name is the account name you specified when registering, and it is the only information included in this list. See this link for more information.

#archlinux-testing irc channel

An irc channel has been created for coordination between Arch Linux testers. See more about becoming an official tester here.

Jelle van der Waa ( Van der Waa

The end of i686 support

November 08, 2017 01:39 PM

Following 9 months of deprecation period, support for the i686 architecture effectively ends today. By the end of November, i686 packages will be removed from our mirrors and later from the packages archive. The [multilib] repository is not affected.

For users unable to upgrade their hardware to x86_64, an alternative is a community maintained fork named Arch Linux 32. See their website for details on migrating existing installations.

Bartłomiej Piotrowski@Official News

Testing your salt states with kitchen-salt

October 04, 2017 04:17 PM

What is Kitchen and why would someone use it.

test-kitchen was originally written as a way to test chef cookbooks. But the provisioners and drivers are pluggable, kitchen-salt enables salt to be the provisioner instead of chef.

The goal of this kitchen-salt is to make it easy to test salt states or formulas independently of a production environment. It allows for doing quick checks of states and to make sure that upstream changes in packages will not affect deployments. By using platforms, users can run checks on their states against the environment they are running in production as well as checking future releases of distributions before doing major upgrades. It is also possible to test states against multiple versions of salt to make sure there are no major regressions.

Example formula

This article will be using my wordpress-formula to demo the major usage points of kitchen-salt.

Installing Kitchen

Most distributions provide a bundler gem in the repositories, but there are some that have a version of ruby that is too old to use kitchen. The easiest way to use kitchen on each system is to use a ruby version manager like rvm or rbenv. rbenv is very similar to pyenv.

Once ruby bundler is installed, it can be used to install localized versions of the ruby packages for each repository, using the bundle install command.

$ bundle install
The latest bundler is 1.16.0.pre.2, but you are currently running 1.15.4.
To update, run `gem install bundler --pre`
Using artifactory 2.8.2
Using bundler 1.15.4
Using mixlib-shellout 2.3.2
Using mixlib-versioning 1.2.2
Using thor 0.19.1
Using net-ssh 4.2.0
Using safe_yaml 1.0.4
Using mixlib-install 2.1.12
Using net-scp 1.2.1
Using net-ssh-gateway 1.3.0
Using test-kitchen 1.17.0
Using kitchen-docker 2.6.1.pre from (at master@9eabd01)
Using kitchen-salt 0.0.29
Bundle complete! 3 Gemfile dependencies, 13 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

This will require having a separate Gemfile to hold the requirements for running test-kitchen.

source ""

gem "test-kitchen"
gem "kitchen-salt"
gem 'kitchen-docker', :git => ''

Because I am also testing opensuse, right now the git version of kitchen-docker is required.

Using kitchen

$ bundle exec kitchen help
  kitchen console                                 # Kitchen Console!
  kitchen converge [INSTANCE|REGEXP|all]          # Change instance state to converge. Use a provisioner to configure one or more instances
  kitchen create [INSTANCE|REGEXP|all]            # Change instance state to create. Start one or more instances
  kitchen destroy [INSTANCE|REGEXP|all]           # Change instance state to destroy. Delete all information for one or more instances
  kitchen diagnose [INSTANCE|REGEXP|all]          # Show computed diagnostic configuration
  kitchen driver                                  # Driver subcommands
  kitchen driver create [NAME]                    # Create a new Kitchen Driver gem project
  kitchen driver discover                         # Discover Test Kitchen drivers published on RubyGems
  kitchen driver help [COMMAND]                   # Describe subcommands or one specific subcommand
  kitchen exec INSTANCE|REGEXP -c REMOTE_COMMAND  # Execute command on one or more instance
  kitchen help [COMMAND]                          # Describe available commands or one specific command
  kitchen init                                    # Adds some configuration to your cookbook so Kitchen can rock
  kitchen list [INSTANCE|REGEXP|all]              # Lists one or more instances
  kitchen login INSTANCE|REGEXP                   # Log in to one instance
  kitchen package INSTANCE|REGEXP                 # package an instance
  kitchen setup [INSTANCE|REGEXP|all]             # Change instance state to setup. Prepare to run automated tests. Install busser and related gems on one or more instances
  kitchen test [INSTANCE|REGEXP|all]              # Test (destroy, create, converge, setup, verify and destroy) one or more instances
  kitchen verify [INSTANCE|REGEXP|all]            # Change instance state to verify. Run automated tests on one or more instances
  kitchen version                                 # Print Kitchen's version information

The kitchen commands I use the most are: - list: show the current state of each configured environment - create: create the test environment with ssh or winrm. - converge: run the provision command, in this case, salt_solo and the specified states - verify: run the verifier. - login: login to created environment - destroy: remove the created environment - test: run create, converge, verify, and then destroy if it all succeeds

For triaging github issues, I regularly use bundle exec kitchen create <setup> and then salt bootstrap to install the salt version we are testing.

Then for running tests, to setup the environment I want to run the tests in I run bundle exec kitchen converge <setup>

Configuring test-kitchen

There are 6 major parts of the test-kitchen configuration file. This is .kitchen.yml and should be in the directory inside of which the kitchen command is going to be run.

  • driver: This specifies the configuration of how the driver requirements. Drivers are how the virtual machine is created. kitchen drivers (I prefer docker)
  • verifier: The command to run for tests to check that the converge ran successfully.
  • platforms: The different platforms/distributions to run on
  • transport: The transport layer to use to talk to the vm. This defaults to ssh, but winrm is also available.
  • suites: sets of different test runs.
  • provisioner: The plugin for provisioning the vm for the verifier to run against. This is where kitchen-salt comes in.

For the driver on the wordpress-fomula, the following is set:

  name: docker
  use_sudo: false
  privileged: true
    - 80

This is using the kitchen-docker driver. If the user running kitchen does not have the correct privileges to run docker, then use_sudo: true should be set. All of the containers that are being used here are using systemd as the exec command, so privileged: true needs to be set. And then port 80 is forwarded to the host so that the verifier can run commands against it to check that wordpress has been setup

For the platforms, the following are setup to run systemd on the container start.

  - name: centos
      run_command: /usr/lib/systemd/systemd
  - name: opensuse
      run_command: /usr/lib/systemd/systemd
        - systemctl enable sshd.service
  - name: ubuntu
      run_command: /lib/systemd/systemd
  - name: debian
      run_command: /lib/systemd/systemd

All of these distributions except for opensuse have sshd.service enabled when the package is installed, so we only have to have one provision command to enable sshd for opensuse. The rest have a command to configure the driver run_command to the correct systemd binary for that distribution.

For suites, there is only one suite.

  - name: wordpress

If multiple sets of pillars or different versions of salt were needed to be tested, they would be configured here.

  - name: nitrogen
  - name: develop
      salt_bootstrap_options: -X -p git -p curl -p sudo git develop

And there would be multiple suites with for each platform created and tested.

And lastly for the verifier.

  name: shell
  remote_exec: false
  command: pytest -v tests/integration/

There are a couple base verifiers. I usually use the shell verifier and use testinfra which has multiple connectors to run pytest type test functions inside of the container.

Kitchen also has a $KITCHEN_SUITE variable that it sets, so if different tests files need to be run for each suite.

  name: shell
  remote_exec: false
  command: pytest -v tests/integration/$KITCHEN_SUITE

For the salt-jenkins, since we are setting up the containers to run the SaltStack testing suite, the verifier is setup to run inside of the container, and run the salt testing suite.

  name: shell
  remote_exec: true
  command: '$(kitchen) /testing/tests/ -v --output-columns=80 --run-destructive<%= ENV["TEST"] ? " -n #{ENV["TEST"]}" : "" %>'

remote_exec will cause the command to be run inside of the container. The kitchen command uses the installed salt to lookup if py3 was used or not, so that the correct python executable is used to run the test suite. Then if the TEST environment variable is set, that test is run, otherwise the full test suite is run.

Configuring kitchen-salt

The documentation for kitchen-salt is located here

  name: salt_solo
  salt_install: bootstrap
  salt_version: latest
  salt_bootstrap_options: -X -p git -p curl -p sudo
  is_file_root: true
  require_chef: false
    - .circleci/
    - Dockerfile
    - .drone.yml
    - .git/
    - .gitignore
    - .kitchen/
    - .kitchen.yml
    - Gemfile
    - Gemfile.lock
    - requirements.txt
    - tests/
    - .travis.yml
    - name: apache
      repo: git
    - name: mysql
      repo: git
    - name: php
      repo: git
        - wordpress
          - wordpress
          - wordpress
            password: quair9aiqueeShae4toh
            host: localhost
              - database: wordpress
                  - all privileges
          admin_user: gtmanfred
          title: "GtManfred's Blog"
  • name: The name of the provisioner is salt_solo
  • salt_install: This defaults to bootstrap which installs using the salt bootstrap. Other options are apt and yum which use the repository. ppa allows for specifying a ppa from which to install salt. And distrib which just uses whatever version of salt is provided by the distribution repositories.
  • salt_bootstrap_options: These are the bootstrap options that are passed to the bootstrap script. -X can be passed here to not start the salt services, because salt_solo runs salt-call and doesn't use the salt-minion process.
  • is_file_root: This is used to say just copy everything from the current directory to the tmp fileserver in the kitchen container. If there were not a custom module and state for this formula, kitchen could be set to have formula: wordpress to copy the wordpress-formula to the kitchen environment.
  • salt_copy_filter: This is a list of files to not copy to the kitchen environment.
  • dependencies: This is the fun part. If the formula depends on other formulas, they can be configured here. The following types are supported:
    • path - use a local path
    • git - clone a git repository
    • apt - install an apt package
    • yum - install a yum package
    • spm - install a spm package
  • state_top: This is the top file that will be used to run at the end of the provisioner
  • pillars: This is a set of custom pillars for configuring the instance. There are a couple other ways to provide pillars that are also useful.

Running test kitchen on pull requests.

Any of the major testing platforms should be usable. If there are complicated setups needed, Jenkins is probably the best, unfortunately I do not know jenkins very well, so I have provided examples for the three I know how to use.

My personal favorite is Drone. You can setup each one of the tests suites to run with a mysql container if you did not have states that need mysql-server installed on the instance. Also, for each job runner for Drone, you just need to setup another drone-agent on a server running docker, and then hook it into the drone-server, then each drone-agent can pick up a job and run it.

@author@@Daniel Wallace

Planet Arch Linux

Planet Arch Linux is a window into the world, work and lives of Arch Linux hackers and developers.

Last updated on January 27, 2020 05:52 AM. All times are normalized to UTC time.