Planet Arch Linux

More Breakage

June 21, 2016 07:02 AM

I broke reddit


Allan@Allan McRae

Choqok 1.6 Beta 1

June 16, 2016 10:00 PM

I’m happy to announce that we will release Choqok 1.6 next month! (mid July)

This will be the first release after the KDE frameworks port and many things have been fixed in those 16 months, including:

  • Twitter: fix user lists loading (BUG:345641)
  • Twitter: allow to select any follower when sending a direct message
  • Twitter: fix searches by username
  • Twitter: fix searches by hashtag
  • Twitter: show original retweet time (BUG:343438)
  • Twitter: fix external URL to access direct messages and tweets
  • Twitter: send direct message without text limits
  • Twitter: support to send and view tweets with quoted text
  • Twitter: allow to delete direct messages
  • Twitter: always show ‘Mark as read’ button
  • GNU Social: fix medium attachment to post
  • GNU Social: allow to send direct messages
  • Pump.IO: do not show resend button for own posts
  • Pump.IO: display avatar image in own posts on the right
  • Pump.IO: do not create a post if there’s no text
  • Fix removal of accounts with spaces in their name
  • Add scalable versions of Choqok icon
  • Check the result of external URL opening to report any failure (BUG:347525)
  • Fixed a bug that overwrite an account with the same alias if you use the same alias
  • Do not allow to send quick posts with no text
  • Always use HTTPS when available
  • ImageView: dropped Twitpic and Tweetphoto support (service are dead)
  • A couple of segmentation fault fixed

Oh, we also added official support for Friendica!

But there’s still a lot to do!

Please join the KDE translation team and help us with translations or try this 1.6 beta and report any bug or, still, join the development team and fix the open bugs.

Together we can make the next release a new starting point for Choqok!

Andrea Scarpino

Maintainers Matter

June 16, 2016 11:31 AM

The case against upstream packaging (postscript)
Kyle Keen

Choqok 1.6 Beta 1

June 15, 2016 10:00 PM

I’m happy to announce that we will release Choqok 1.6 next month! (mid July)

This will be the first release after the KDE frameworks port and many things have been fixed in those 16 months, including:

  • Twitter: fix user lists loading (BUG:345641)
  • Twitter: allow to select any follower when sending a direct message
  • Twitter: fix searches by username
  • Twitter: fix searches by hashtag
  • Twitter: show original retweet time (BUG:343438)
  • Twitter: fix external URL to access direct messages and tweets
  • Twitter: send direct message without text limits
  • Twitter: support to send and view tweets with quoted text
  • Twitter: allow to delete direct messages
  • Twitter: always show ‘Mark as read’ button
  • GNU Social: fix medium attachment to post
  • GNU Social: allow to send direct messages
  • Pump.IO: do not show resend button for own posts
  • Pump.IO: display avatar image in own posts on the right
  • Pump.IO: do not create a post if there’s no text
  • Fix removal of accounts with spaces in their name
  • Add scalable versions of Choqok icon
  • Check the result of external URL opening to report any failure (BUG:347525)
  • Fixed a bug that overwrite an account with the same alias if you use the same alias
  • Do not allow to send quick posts with no text
  • Always use HTTPS when available
  • ImageView: dropped Twitpic and Tweetphoto support (service are dead)
  • A couple of segmentation fault fixed

Oh, we also added official support for Friendica!

But there’s still a lot to do!

Please join the KDE translation team and help us with translations or try this 1.6 beta and report any bug or, still, join the development team and fix the open bugs.

Together we can make the next release a new starting point for Choqok!

Andrea Scarpino

Maintainers Matter

June 15, 2016 11:51 AM

The case against upstream packaging
Kyle Keen

June’s TalkingArch, and Other News

June 08, 2016 07:16 PM

The TalkingArch Release

The TalkingArch team is pleased to bring you the newest TalkingArch release. This version includes version 4.5.4 of the Linux kernel, the ndisc6 package with some IPv6 networking utilities and uefi-tools for booting x86_64 systems with UEFI. Find it in the usual spot.

Torrent Changes

Some users who downloaded Talkingarch using Bittorrent may have noticed that the TalkingArch iso is now being downloaded into a directory, along with its signature files. This is being done for convenience, so that downloading the one torrent file gets everything, rather than forcing users who want to verify the integrity of the iso to go back to the website to download the signatures. This change was introduced with the previous release, but that release wasn’t posted here. In any case, enjoy the new format.

TalkingArch and IPv6

The TalkingArch team is very pleased to announce full IPv6 connectivity for all TalkingArch sites and services. Anyone with IPv6 connectivity will immediately notice the new addresses for the connections they make, including connections to the NetwIRC where the #talkingarch IRC channel resides. TalkingArch has now joined the future of the internet,. Thanks to all who answered questions and made the transition run smoothly. Have a virtual beer, on the house.



April 30, 2016 09:18 AM

Now with more UI
Kyle Keen

TalkingArch has a new home

April 08, 2016 06:18 PM

Thanks to the folks over at Scaleway, TalkingArch, its blog, its torrent tracker and its support e-mail are all now hosted on real ARM hardware in France. This server provides more disk space and processing power than the previous virtualized environment at a lower monthly cost. The transfer speed is also faster, and the connection is unmetered. RAM is the same as on the previous virtual server, although it is dedicated now, so won’t be slowed down by massive Mysql queries that other users on the same host machine may be running. We are looking into the possibility of migrating the git repository to this server as well, as our repository at NotABug has been refusing push requests since February. In spite of the difficulties, a release for the month of April is imminent, so keep checking this space for the April 2016 iso.


Required update to pacman-5.0.1 before 2016-04-23

March 24, 2016 11:50 AM

The release of pacman-5.0 brought support for transactional hooks. These will allow us to (e.g.) run font cache updates a single time during an update rather than after each font package installation. This will both speed up the update process, but also reduce packaging burden for the Developers and Trusted Users.

In order for the use of hooks to be started, we require all users to have updated to at least pacman-5.0.1 before 2016-04-23. Pacman-5.0.1 was released on 2016-02-23, so this will have given everyone two months to update their system.

Allan McRae@Official News

25 Graphite, Grafana and statsd gotchas

March 15, 2016 06:22 AM

This is a crosspost of an article I wrote on the blog
For several years I’ve worked with Graphite, Grafana and statsd on a daily basis and have been participating in the community. All three are fantastic tools and solve very real problems. Hence my continued use and recommendation. However, between colleagues, random folks on irc, and personal experience, I’ve seen a plethora of often subtle issues, gotchas and insights, which today I’d like to share.

I hope this will prove useful to users while we, open source monitoring developers, work on ironing out these kinks. At raintank we’re addressing a bunch of these as well but we have a long road ahead of us.
Before we begin, when trying to debug what’s going on with your metrics (whether they’re going in or out of these tools), don’t be afraid to dive into the network or the whisper files. They can often be invaluable in understanding what’s up.

For network sniffing, I almost always use these commands:
ngrep -d any -W byline port 2003 # carbon traffic ngrep -d any -W byline port 8125 # statsd traffic

Getting the json output from Graphite (just append &format=json) can be very helpful as well. Many dashboards, including Grafana already do this, so you can use the browser network inspector to analyze requests. For the whisper files, Graphite comes with various useful utilities such as whisper-info, whisper-dump, whisper-fetch, etc.

here we go:

1. OMG! Did all our traffic just drop?
2. Null handling in math functions.
3. Null handling during runtime consolidation.
4. No consolidation or aggregation for incoming data.
5. Limited storage aggregation options.
6. Runtime consolidation is detached from storage aggregation.
7. Grafana consolidation is detached from storage aggregation and runtime consolidation.
8. Aggregating percentiles.
9. Deriving and integration in Graphite.
10. Graphite quantization.
11. statsd flush offset depends on when statsd was started.
12. Improperly time-attributed metrics.
13. The relation between timestamps and the intervals they describe.
14. The statsd timing type is not only for timings.
15. The choice of metric keys depends on how you deployed statsd.
16. statsd is “fire and forget”, “non-blocking” & UDP sends “have no overhead”.
17. statsd sampling.
18. As long as my network doesn’t saturate, statsd graphs should be accurate.
19. Incrementing/decrementing gauges.
20. You can’t graph what you haven’t seen
21. Don’t let the data fool you: nulls and deleteIdleStats options.
22. keepLastValue works… almost always.
23. statsd counters are not counters in the traditional sense.
24. What can I send as input?
25. Hostnames and ip addresses in metric keys.

1) OMG! Did all our traffic just drop?
Probably the most common gotcha, and one i run into periodically. Graphite will return data up until the current time, according to your data schema.

Example: if your data is stored at 10s resolution, then at 10:02:35, it will show data up until 10:02:30. Once the clock hits 10:02:40, it will also include that point (10:02:40) in its response.

It typically takes some time for your services to send data for that timestamp, and for it to be processed by Graphite, so Graphite will typically return a null here for that timestamp. Depending on how you visualize (see for example the “null as null/zero” option in grafana), or if you use a function such as sumSeries (see below) it may look like a drop in your graph, and cause panic.

You can work around this with “null as null” in Grafana, transformNull() or keepLastValue() in Graphite, or plotting until a few seconds ago instead of now. See below for some other related issues.

2) Null handling in math functions.
Graphite functions don’t exactly follow the rules of logic (by design). When you request something like sumSeries(diskspace.server_*.bytes_free) Graphite returns the sum of all bytes_free’s for all your servers, at each point in time. If all of the servers have a null for a given timestamp, the result will be null as well for that time. However, if only some - but not all - of the terms are null, they are counted as 0.

Example: The sum for a given point in time, 100 + 150 + null + null = 250.

So when some of the series have the occasional null, it may look like a drop in the summed series. This especially commonly happens for the last point at each point in time, when your servers don’t submit their metrics at the exact same time.

Now let’s say you work for a major video sharing website that runs many upload and transcoding clusters around the world. And let’s also say you build this really cool traffic routing system that sends user video uploads to the most appropriate cluster, by taking into account network speeds, transcoding queue wait times, storage cluster capacity (summed over all servers), infrastructure cost and so forth. Building this on top of Graphite would make a lot of sense, and in all testing everything works fine, but in production the decisions made by the algorithm seem nonsensical, because the storage capacity seems to fluctuate all over the place. Let’s just say that’s how a certain Belgian engineer became intimately familiar with Graphite’s implementation of functions such as sumSeries. Lesson learned. No videos were hurt.

Graphite could be changed to strictly follow the rules of logic, meaning as soon as there’s a null in an equation, emit a null as output. But a low number of nulls in a large sum is usually not a big deal and having a result with the nulls counted as 0 can be more useful than no result at all. Especially when averaging, since those points can typically be safely excluded without impact on the output. Graphite has the xFilesFactor option in storage-aggregation.conf, which lets you configure the minimum fraction of non-null values, which it uses during storage aggregation (rollups) to determine whether the input is sufficiently known for output to be known, otherwise the output value will be null.

It would be nice if these on-demand math requests would take an xFilesFactor argument to do the same thing. But for now, just be aware of it, not using the last point is usually all you need if you’re only looking at the most recent data. Or get the timing of your agents tighter and only request data from Graphite until now=-5s or so.

INTERMEZZO: At this point we should describe the many forms of consolidation; a good understanding of this is required for the next points.

  • storage consolidation aka aggregation, aka rollups: data aggregated to optimize the cost of historical archives and make it faster to return large timeframes of data. Driven by storage-aggregation.conf

  • runtime consolidation: when you want to render a graph but there are more datapoints than pixels for the graph, or more than the maxDataPoints setting. Graphite will reduce the data in the http response. Averages by default, but can be overridden through consolidateBy()

  • Grafana consolidation: Grafana can visualize stats such as min/max/avg which it computes on the data returned by your timeseries system.

  • statsd consolidation: statsd is also commonly used to consolidate usually very high rates of irregular incoming data in properly spaced timeframes, before it goes into Graphite.

  • Unlike all above mechanisms which work per series, the sumSeries(), saverageSeries(), groupByNode() Graphite functions work across series. They merge multiple series into a single or fewer series by aggregating points from different series at each point in time.

  • And then there's carbon-aggregator and carbon-relay-ng which operate on your metrics stream and can aggregate on per-series level, as well as across series, as well as a combination thereof.

3) Null handling during runtime consolidation
Similar to above, when Graphite performs runtime consolidation it simply ignores null values. Imagine a series that measures throughput with points 10, 12, 11, null, null, 10, 11, 12, 10. Let’s say it needs to aggregate every 3 points with sum. this would return 33, 10, 33. This will visually look like a drop in throughput, even though there probably was none.

For some functions like avg, a missing value amongst several valid values is usually not a big deal, but the likeliness of it becoming a big deal increases with the amount of nulls, especially with sums. For runtime consolidation, Graphite needs something similar to the xFilesFactor setting for rollups.

4) No consolidation or aggregation for incoming data.
If you submit multiple values into Graphite for the same interval, the last one overwrites any previous values, no consolidation/aggregation happens in this scenario.

Never send multiple values for the same interval. However, carbon-aggregator or carbon-relay-ng may be of help (see above)

5) Limited storage aggregation options.
In storage-aggregation.conf you configure which function to use for historical rollups. It can be limiting that you can only choose one. Often you may want to retain both the max values as well as the averages for example.(very useful for throughput). This feature exists in RRDtool.

Another issue with this is that often these functions will be misconfigured. Make sure to think about all the possible metrics you’ll be sending (e.g. min and max values through statsd) and set up the right configuration for them.

Proprietary systems tend to be more flexible and I’m sure it will make a come back in the Graphite stack as well. (more on that later)

6) Runtime consolidation is detached from storage aggregation.
The function chosen in storage-aggregation.conf is only used for rollups.

If Graphite performs any runtime consolidation it will always use average unless told otherwise through consolidateBy. This means it’s easy to run into cases where data is rolled up (in whisper) using, say, max or count, but then accidentally with average while creating your visualization, resulting in incorrect information and nonsensical charts. Beware!

It would be nice if the configured roll-up function would also apply here (and also the xFilesFactor, as above), For now, just be careful :)

7) Grafana consolidation is detached from storage aggregation and runtime consolidation.
Like mentioned earlier, Grafana can provide min/max/avg/… values from the received data. For now, it just computes these from the received data, which may already have been consolidated (twice: in storage and in runtime) using different functions, so these results may not be always representative. (more info) We will make this mechanism more powerful and accurate.

8) Aggregating percentiles.
The pet peeve of many, it has been written about a lot: If you have percentiles, such as those collected by statsd, (e.g. 95th percentile response time for each server) there is in theory no proper way to aggregate those numbers.

This issue appears when rolling up the data in the storage layer, as well when doing runtime consolidation, or when trying to combine the data for all servers (multiple series) together. There is real math behind this, and I’m not going into it because in practice it actually doesn’t seem to matter much. If you’re trying to spot issues or alert on outliers you’ll get quite far with averaging the data together or taking the max value seen. This is especially true when the amount of requests, represented by each latency measurement, is in the same order of magnitude. E.g. You’re averaging per-server latency percentiles and your servers get a similar load or you’re averaging multiple points in time for one server, but there was a similar load at each point. You can always set up a separate alerting rule for unbalanced servers or drops in throughput. As we’ll see in some of the other points, taking sensible shortcuts instead of obsessing over math is often the better way to accomplish your job of operating your infrastructure or software.

9) Deriving and integration in Graphite.
The Graphite documentation for derivative() hints at this already (“This function does not normalize for periods of time, as a true derivative would.”), but to be entirely clear:

Maybe for a future stable Graphite release this can be reworked. for now, just something to be aware of.

10) Graphite quantization.
We already saw in the consolidation paragraph that for multiple points per interval, last write wins. But you should also know that any data point submitted gets the timestamp rounded down.

Example: You record points every 10 seconds but submit a point with timestamp at 10:02:59, in Graphite this will be stored at 10:02:50. To be more precise if you submit points at 10:02:52, 10:02:55 and 10:02:59, and have 10s resolution, Graphite will pick the point from 10:02:59 but store it at 10:02:50. So it’s important that you make sure to submit points at consistent intervals, aligned to your Graphite retention intervals (e.g. if every 10s, submit on timestamps divisible by 10)

11) statsd flush offset depends on when statsd was started.
Statsd lets you configure a flushInterval, i.e. how often it should compute the output stats and submit them to your backend. However, the exact timing is pretty arbitrary and depends on when statsd is started.

Example: If you start statsd at 10:02:00 with a flushInterval of 10, then it will emit values with timestamps at 10:02:10, 10:02:20, etc (this is what you want), but if you happened to start it at 10:02:09 then it will submit values with timestamps 10:02:19, 10:02:29, etc. So typically, the values sent by statsd are subject to Graphite’s quantization and are moved into the past. In the latter example, by 9 seconds. This can make troubleshooting harder, especially when comparing statsd metrics to metrics from a different service.

Note: vimeo/statsdaemon (and possibly other servers as well) will always submit values at quantized intervals so that it’s guaranteed to map exactly to Graphite’s timestamps as long as the interval is correct.

12) Improperly time-attributed metrics.
You don’t tell statsd the timestamps of when things happened. Statsd applies its own timestamp when it flushes the data. So this is prone to various (mostly network) delays. This could result in a metric being generated in a certain interval only arriving in statsd after the next interval has started. But it can get worse. Let’s say you measure how long it takes to execute a query on a database server. Typically it takes 100ms but let’s say now many queries are taking 60s, and you have a flushInterval of 10s. Note that the metric is only sent after the full query has completed. So during the full minute where queries were slow, there are no metrics (or only some metrics that look good, they came through cause they were part of a group of queries that managed to execute timely), and only after a minute do you get the stats that reflect queries spawned a minute ago. The higher a timing value, the higher the attribution error, the more into the past the values it represents and the longer the issue will go undetected or invisible.

Keep in mind that other things, such as garbage collection cycles or paused goroutines under cpu saturation may also delay your metrics reporting. Also watch out for the queries aborting all together, causing the metrics never to be sent and these faults to be invisble! Make sure you properly monitor throughput and the functioning (timeouts, errors, etc) of the service from the client perspective, to get a more accurate picture.

Note that many instrumentation libraries have similar issues.

13) The relation between timestamps and the intervals they describe.
When I look at a point at a graph that represents a spike in latency, a drop in throughput, or anything interesting really, I always wonder whether it describes the timeframe before, or after it.

Example: With points every minute, and a spike at 10:17:00, does it mean the spike happened in the timeframe between 10:16 and 10:17, or between 10:17 and 10:18?

I don’t know if a terminology for this exists already, but I’m going to call the former premarking (data described by a timestamp that precedes the observation period) and the latter postmarking (data described by a timestamp at the end). As we already saw statsd postmarks, and many tools seem to do this, but some, including Graphite, premark.

We saw above that any data received by Graphite for a point in between an interval is adjusted to get the timestamp at the beginning of the interval. Furthermore, during aggregation (say, aggregating sets of 10 minutely points into 10min points), each 10 minutes taken together get assigned the timestamp that precedes those 10 intervals. (i.e. the timestamp of the first point)

So essentially, Graphite likes to show metric values before they actually happened, especially after aggregation, whereas other tools rather use a timestamp in the future of the event than in the past. As a monitoring community, we should probably standardize on an approach. I personally favor postmarking because measurements being late is fairly intuitive, predicting the future not so much.

14) The statsd timing type is not only for timings.
The naming is a bit confusing, but anything you want to compute summary statistics (min, max, mean, percentiles, etc) for (for example message or packet sizes) can be submitted as a timing metric. You’ll get your summary stats just fine. The type is just named “timing” because that was the original (and still most common) use case. Note that if you want to time an operation that happens at consistent intervals, you may just as well simply use a statsd gauge for it. (or write directly to Graphite)

15) The choice of metric keys depends on how you deployed statsd.
It’s common to deploy a statsd server per machine or per cluster. This is a convenient way to assure you have a lot of processing capacity. However, if multiple statsd servers receive the same metric, they will do their own independent computations and emit the same output metric, overwriting each other. So if you run a statsd server per host, you should include the host in the metrics you’re sending into statsd, or into the prefixStats (or similar option) for your statsd server, so that statsd itself differentiates the metrics.

Note: there’s some other statsd options to diversify metrics but the global prefix is the most simple and common one used.

16) statsd is “fire and forget”, “non-blocking” & UDP sends “have no overhead”.

This probably stems from this snippet in the original statsd announcement.

“[UDP is] fast — you don’t want to slow your application down in order to track its performance — but also sending a UDP packet is fire-and-forget.”

UDP sends do have an overhead that can slow your application down. I think a more accurate description is that the overhead is insignificant either if you’re lucky, or if you’ve spent a lot of attention to the details. In specific, it depends on the following factors:

  • Your network stack. UDP sends are not asynchronous, which is a common assumption. (i.e. they are not “non-blocking”) Udp write calls from userspace will block until the kernel has moved the data through the networking stack, firewall rules, through the network driver, onto the network. I found this out the hard way once when an application I was testing on my laptop ran much slower than expected due to my consumer grade WiFi adapter, which was holding back my program by hanging on statsd calls. Using this simple go program I found that I could do 115 k/s UDP sends to localhost (where the kernel can bypass some of the network stack), but only 2k/s to a remote host (where it needs to go through all layers, including your NIC).

  • other code running, and its performance. If you have a PHP code base with some statsd calls amongst slow / data-intensive code to generate complex pages, there won’t be much difference. This is a common, and the original setting, for statsd instrumentation calls. But I have conducted cpu profiling of high-performance Golang applications, both at Vimeo and raintank, and when statsd was used, the apps were often spending most of their time in UDP writes caused by statsd invocations. Typically they were using a simple statsd client, where each statsd call corresponds to a UDP write. There are some statsd clients that buffer messages and send batches of data to statsd, reducing the UDP writes, which is a lot more efficient. (see below). You can also use sampling to lower the volume. This does come with some problems however, see the gotcha below.

  • The performance of client code itself can wildly vary as well. Besides how it maps metric updates to UDP writes (see above) the invocations themselves can carry an overhead, which as you can see through these benchmarks of all Go statsd clients I’m aware of, vary anywhere between 15 microseconds to 0.4 microseconds.

    When I saw this, Alex Cesaro’s statsd client promptly became my favorite statsd library for Go. It has zero-allocation logic and various performance tweaks, such as the much needed client side buffering. Another interesting one is quipo’s, which offloads some of the computations into the client to reduce traffic and workload for the server.

  • Finally, it should be noted that sending to non-listening destinations may make your application slower. This is because destination hosts receiving UDP traffic for a closed socket, will return ICMP rejects to the sender, which the sender then has to process. I’ve not seen this with my own eyes, but the people on the #go-nuts channel who were helping me out definitely seemed to know what they’re talking about. Sometimes “disabling” statsd sends is implemented by pointing to a random host or port that’s not listening (been there done that) and is hence not the best idea!

Takeaway: If you use statsd in high-performance situations, use optimized client libraries . Or use client libraries such as go-metrics, but be prepared to pay a processing tax on all of your servers. Be aware that slow NICs and sending to non-listening destinations will slow your app down.

17) statsd sampling.
The sampling feature was part of the original statsd, so it can be seen in pretty much every client and server alternative. The idea sounds simple enough: If a certain statsd invocation causes too much statsd network traffic or overloads the server, sample down the messages. However, there’s several gotchas with this. A common invocation with a typical statsd client looks something like:


In this code, we have 1 statement, and it increments a counter to track http response codes for different backends. It works for different values of backend and http_response, tracking the counts for each. However, let’s say the backend “default” and http_response 200 are causing too much statsd traffic. It’s easy to just sample down and change the line to:

statsd.Increment("requests.$backend.$http_response", 0.1)

There’s 3 problems here though:

  1. How do you select a good sampling ratio? It’s usually based on a gut feeling or some very quick math at best to come up with a reasonable number, without any attention to the impact on statistical significance, so you may be hurting the quality of your data.

  2. While you mainly wanted to target the highest volume metric (something like “requests.default.200”), your sampling setting affects the metrics for all values of backend and http_response. Those variable combinations that are less frequent (for a backend that’s rarely used or a response code not commonly seen) will be too drastically sampled down, resulting in an inaccurate view of those cases. Worst case, you won’t see any data over the course over a longer time period, even though there were points at every interval. You can add some code to use different invocations with different sample rates based on the values of the variables, but that can get kludgy.

  3. Volumes of metric calls tend to constantly evolve over time. It’s simply not feasible to be constantly manually adjusting sample rates. And given the above, it’s not a good solution anyway.

One of the ideas I wanted to implement in was a feedback mechanism where the server would maintain counts of received messages for each key. Based on performance criteria and standard deviation measurements, it would periodically update “recommended sampling rates” for every single key, which then could be fed back to the clients (by compiling into a PHP config file for example, or over a TCP connection). Ultimately I decided it was just too complicated and deployed a statsdaemon server on every single machine so I could sample as gently as possible.

Takeaway: Sampling can be used to lower load, but be aware of the trade offs.

18) As long as my network doesn’t saturate, statsd graphs should be accurate.
We all know that graphs from statsd are not guaranteed to be accurate, because it uses UDP, so is prone to data loss and inaccurate graphs. And we’re OK with that trade off, because the convenience is worth the loss in accuracy. However, it’s commonly believed that the only, or main reason of inaccurate graphs is UDP data loss on the network. In my experience, I’ve seen a lot more message loss due to the UDP buffer overflowing on the statsd server.

A typical statsd server is constantly receiving a flood of metrics. For a network socket, the Linux kernel maintains a buffer in which it stores newly received network packets, while the listening application (statsd) reads them out of the buffer. But if for any reason the application can’t read from the buffer fast enough and it fills up, the kernel has to drop incoming traffic. The most common case are the statsd flushes, which compute all summary statistics and submit them each flushInterval. Typically this operation takes between 50ms and a few seconds, depending mostly on how many timing metrics were received, because those are the most computationally expensive. Usually statsd’s cpu is maxed out during this operation, but you won’t see it on your cpu graphs because your cpu measurements are averages taken at intervals that typically don’t exactly coincide with the statsd flush-induced spikes. During this time, it temporarily cannot read from the UDP buffer, and it is very common for the UDP buffer to rapidly fill up with new packets, at which point the Linux kernel is forced to drop incoming network packets for the UDP socket. In my experience this is very common and often even goes undetected, because only a portion of the metrics are lost, and the graphs are not obviously wrong. Needless to say, there’s also plenty of other scenarios for the buffer to fill up and cause traffic drops other than statsd flushing.

There’s two things to be done here:

  1. Luckily we can tune the size of the incoming UDP buffer with the net.core.rmem_max and net.core.rmem_default sysctl’s. I recommend setting it to the max values

  2. The kernel increments a counter every time it drops a UDP packet. You can see UDP packet drops with netstat -s –udp on the command line. I highly encourage you to monitor this with something like collectd, visualize it over time, and make sure to set an alert on it.

Note also that the larger of a buffer you need, the more delay between receiving a metric and including it in the output data, possibly skewing data into the future.

INTERMEZZO: There's a lot of good statsd servers out there with various feature sets. Most of them provide improved performance (upstream statsd is single threaded JavaScript), but some come with additional interesting features. My favorites include statsite, brubeck and of course vimeo's statsdaemon version

19) Incrementing/decrementing gauges.
Statsd supports a syntax to increment and decrement gauges. However, as we’ve seen, any message can be lost. If you do this and a message gets dropped, your gauge value will be incorrect forever (or until you set it explicitly again). Also by sending increment/decrement values you can confuse a statsd server if it was just started. For this reason, I highly recommend not using this particular feature, and always setting gauge values explicitly. In fact, some statsd servers don’t support this syntax for this reason.

20) You can’t graph what you haven’t seen
Taking the earlier example again:


The metrics will only be created after the values have been sent. Sounds obvious, but this can get surprisingly annoying. Let’s say you want to make a graph of http 500’s, but they haven’t happened yet. The metrics won’t exist and with many dashboarding tools, you won’t be able to create the graph yet, or at least need extra work.

I see this very commonly with all kinds of error metrics. I think with Grafana it will become easier to deal with this over time, but for now I write code to just send each metric once at startup. For counts you can just send a 0, for gauges and timers it’s of course harder/weirder to come up with a fake value but it’s still a reasonable approach.

21) Don’t let the data fool you: nulls and deleteIdleStats options.
statsd has the deleteIdleStats, deleteGauges, and similar options. By default they are disabled, meaning statsd will keep sending data for metrics it’s no longer seeing. So for example, it’ll keep sending the last gauge value even if it didn’t get an update. I have three issues with this:

  • Your graphs are misleading. Your service may be dead but you can’t tell from the gauge graphs cause they will still “look good”. If you have something like counters for requests handled, you can’t tell the difference between your service being dead or it being up but not getting incoming requests. I rather have no data show up if there is none, combined with something else (alerting, different graph) to deduce whether the service is up or not

  • Any alerting rules defined on the metrics have the same fate as humans and can’t reliably do their work

  • Tougher to phase out metrics. If you decommission servers or services, it’s nice that the metrics stop updating so you can clean them up later

My recommendation is, enable deleteIdleStats. I went as far as hard coding that behavior into vimeo/statsdaemon and not making it configurable.

To visualize it, make sure in Grafana to set the “null as null” option, though “null as zero” can make sense for counters. You can also use the transformNull() or drawNullAsZero options in Graphite for those cases where you want to explicitly get 0’s instead of nulls, which I typically do in some of my bosun alerting rules, so I can treat null counts as no traffic received, while having a separate alerting rule to make sure my service is up and running, based on a different metric, while using null as null for visualization.

22) keepLastValue works… almost always.
Another Graphite function to change the semantics of nulls is keepLastValue, which causes null values to be represented by the last known value that precedes them. However, that known value must be included in the requested time range. This is probably a rare case that you may never encounter, but if you have scripts that infrequently update a metric and you use this function, it may result in a graph sometimes showing no data at all if the last known value becomes too old. Especially confusing to newcomers, if the graph does “work” at other times.

23) statsd counters are not counters in the traditional sense.
You may know counters such as switch traffic/packet counters, that just keep increasing over time. You can see their rates per second by deriving the data. Statsd counters however, are typically either stored as the number of hits per flushInterval, or per second, or both (perhaps because of Graphite’s derivative issue?). This in itself is not a huge problem, however this is more vulnerable to loss of data. If your statsd server has trouble flushing some data to Graphite and some data gets lost. If it were using a traditional counter, you could still derive the data and average out the gap across the nulls. In this case however this is not possible and you have no idea what the numbers were. In practice this doesn’t happen often though. Many statsd implementations can buffer a writequeue or use something like carbon-relay-ng as a write queue. Other disadvantages of this approach is that you need to know the flushInterval value to make sense of the count values, and the rounding that happens when computing the rates per second, which is also slightly lossy.

24) What can I send as input?
Neither Graphite, nor statsd does a great job specifying exactly what they allow as input, which can be frustrating. Graphite timestamps are 32bit unix timestamp integers, while values for both Graphite and statsd can be integers or floats, up to float 64bit precision. For statsd, see the statsd_spec project for more details.

As for what characters can be included in the metric keys. Generally, Graphite is somewhat forgiving and may alter your metric keys: it converts slashes to dots (which can be confusing), subsequent dots become single dots, prefix dots get removed, postfix dots will get it confused a bit though and create an extra hierarchy with an empty node at the end if you’re using whisper. Non-alphanumeric characters such as parenthesis, colons, or equals signs often don’t work well or at all.

Graphite as a policy does not go far in validating incoming data, citing performance in large-throughput systems as the major reason. It’s up to the senders to send data in proper form, with non-empty nodes (words between dots) separated by single dots. You can use alphanumeric characters, hyphens and underscores, but straying from that will probably not work well. You may want to use carbon-relay-ng which provides metric validation. See also this issue which aims to formalize what is, and isn’t allowed.

25) Hostnames and ip addresses in metric keys.
The last gotcha relates to the previous one but is so common it deserves its own spot. Hostnames, especially FQDN’s, and IP addresses (due to their dots), when included in metric keys, will be interpreted by Graphite as separate pieces. Typically, people replace all dots in hostnames and IP addresses with hyphens or underscores to combat this.

Closing thoughts:

Graphite, Grafana and statsd are great tools, but there’s some things to watch out for. When setting them up, make sure you configure Graphite and statsd to play well together. Make sure to set your roll-up functions and data retentions properly, and whichever statsd version you decide to use, make sure it flushes at the same interval as your Graphite schema configuration. More tips if you use the nodejs version. Furthermore, I hope this is an extensive list of gotchas and will serve to improve our practices at first, and our tools themselves, down to road. We are definitely working on making things better and have some announcements coming up…

Dieter Plaetinck

Announcing the first TalkingArch iso of 2016

March 03, 2016 11:49 PM

After many broken and failed attempts, the TalkingArch team is please to bring you the latest TalkingArch, which is the first release of 2016. This version includes linux kernel 4.4.1 and all the latest base packages to give you a smooth experience when running and installing TalkingArch on your system. The new release can be found at the usual location.

Just a heads-up, gpg has changed the verification process for signatures. From now on, you should download both linked signature files and verify them individually with a command like
gpg --verify <sig-file> <iso-file>
If one signature is good, both should be, but it’s always good to double check. The download page has been changed to reflect the new verification procedure. As always, have fun with the new release, and do report any problems you find.



February 10, 2016 09:48 AM

A utility for sorting really big files.
Kyle Keen

LibreOffice Ransom Demand

February 01, 2016 06:43 AM

I feel like my document is being held hostage and I just received note made form a bunch of letters cut from newspapers and magazines.

* I have no idea what caused this and can not replicate…

Allan@Allan McRae

Pacman-5.0 Released

January 30, 2016 12:40 AM

As is becoming tradition, I need to make a blog post to accompany a pacman release! This is a big release with a long awaited feature so it needed a major version bump (and, most importantly, we now are back ahead of the Linux kernel in version numbers). I have reclaimed the title as most prolific committer, but that just means Andrew had more patches to point out mistakes in… Here are the top 10 committers:

$ git shortlog -n -s v4.2.0..v5.0.0
   176  Allan McRae
    85  Andrew Gregory
    16  Florian Pritz
     9  Dave Reisner
     9  Johannes Löthberg
     9  Rikard Falkeborn
     7  Pierre Neidhardt
     5  David Macek
     5  Evangelos Foutras
     4  Mohammad Alsaleh

As always, more regular contributors would be helpful. Just have a talk to us first before running ahead implementing a new feature (it is not nice to have to reject a patchset that obviously took a lot of work because it is already being handled in a different way…).

On to the more important stuff… What is new this release?

Hooks: This has been one of the most requested features for a long time, and Andrew gets all the credit for the implementation. So, what exactly are hooks? Hooks are scripts that are run at the beginning and end of a transaction. They can be triggered by either a file or a package name. This will allow us to (e.g.) update the desktop MIME type cache at the end of a transaction (if needed) and only do it one rather than after every package. Andrew has a git repo with some examples. Lets look at the desktop MIME cache one:

Type = File
Operation = Install
Operation = Upgrade
Operation = Remove
Target = usr/share/applications/*.desktop
When = PostTransaction
Exec = /bin/update-desktop-database --quiet

It should be fairly obvious what that does… See the alpm-hooks(5) man page for more information on the hook format.

Files database operations: pacman can now search repository file lists like pkgfile, but slower and probably less flexible. Sort of like Falcon to Captain America… still a super-hero! File bug reports for improvement requests. A separate files database is used so everyday package operations do not require downloading a much larger database. After updating the database (with “pacman -Fy“, you can do things like:

$ pacman -Fo /usr/bin/pacman
usr/bin/pacman is owned by core/pacman 4.2.1-4
$ pacman -Fl pacman
pacman etc/
pacman etc/makepkg.conf
pacman etc/pacman.conf
$ pacman -Fs
core/pacman 4.2.1-4
$ pacman -Fsx kcm.*print.*\.so
extra/print-manager 15.12.1-1
community/kmymoney 4.7.2-3

libmakepkg: makepkg in the pacman-4.2 release was a 3838 line shell script. A bit daunting, hard to test and not reusable… I have started the process of splitting this into a library containing scripts that are a more reasonable size. makepkg is still 2395 lines, so a lot of work remains (help!). One outcome of this splitting is we can drop in extra checks into the PKGBUILD and package checking steps, and even extra passes to (e.g.) optimize svg files. I have started rewriting namcap using this feature (see my github repo). This also provides tools for extracting variables from PKGBUILDs without sourcing the PKGBUILD itself, which does require ensuring that variables that should be arrays are actually arrays and those that are not are not.

There were a bunch of other small changes throughout the code base. Check out the NEWS file for more details.

Allan@Allan McRae

Where Else?

January 12, 2016 03:44 PM

Australia’s “Shadow Minister for Justice” is the representative for the seat of Batman.

Allan@Allan McRae

PHP 7.0 packages released

January 02, 2016 03:13 PM

Packages of the new major version of PHP have been released into the stable repositories. Besides the new PHP 7 features there are the following packaging changes. In general the package configuration is now closer to what was intended by the PHP project. Also refer to the PHP 7 migration guide for upstream improvements.

Removed packages

  • php-pear
  • php-mssql
  • php-ldap The module is now included in the php package
  • php-mongo The new php-mongodb might provide an alternative even though it is not a compatible drop-in replacement
  • php-xcache Consider using the included OPcache and optionally APCu for user data cache
  • graphviz The PHP bindings had to be removed

New packages

Configuration changes

  • open_basedir is no longer set by default
  • openssl, phar and posix extensions are now built in
  • php-fpm does no longer provide a logrotate configuration. Instead syslog/journald is used by default
  • php-fpm's service file does no longer set PrivateTmp=true
  • The configuration and module of php-apache have been renamed to php7_module.conf and
Pierre Schmitz@Official News

How your parents installed Arch

January 02, 2016 12:51 PM

For those who think the new Arch Install Scripts are no longer the Arch Way, here are the original install instructions of Arch Linux 0.1 published in March 2002.

                              ARCH LINUX #VERSION#

1.  Make your swap/root partitions
      # fdisk /dev/discs/disc0/disc

2.  Make your target filesystems
      # mkswap /dev/discs/disc0/partX
      # mkreiserfs /dev/discs/disc0/partY

3.  Activate your swap partition
      # swapon /dev/discs/disc0/partX

4.  Mount your target root filesystem under /mnt
      # mount /dev/discs/disc0/partX /mnt

5.  Initialize the pacman database
      # mkdir -p /mnt/var/lib/pacman
      # touch /mnt/var/lib/pacman/pacman.db

6.  Add the filesystem package from /arch/pkg/filesystem-0.X-Y.tar.gz
      # cd /arch/pkg
      # pacman --add -r /mnt filesystem-0.X-Y.tar.gz

7.  Mount any other data partitions you may have created
      # mount /dev/discs/disc0/partX /mnt/home
      # mount /dev/discs/disc0/partY /mnt/usr

8.  Install some base packages
      # cd /arch
      # ./installworld /mnt

9.  Uncompress the linux source to /mnt/usr/src
      # cd /mnt/usr/src && tar zxvf /arch/linux-2.4.XX.tar.gz

10. Mount a dev and proc under your new system and chroot
      # mount -t devfs none /mnt/dev
      # mount -t proc none /mnt/proc
      # chroot /mnt /bin/bash

11. Build & install your new kernel from /usr/src/linux
      NOTE: make sure you ask for "/dev file system support" and
            "Automatically mount at boot", since we use DevFS.

12. Install a lilo bootloader
      # vi /etc/lilo.conf
      # lilo

13. Edit settings
      # vi /etc/rc.conf
      # vi /etc/resolv.conf
      # vi /etc/fstab
      # vi /etc/modules.conf (if needed)

14. Exit your chroot shell
      # exit

15. Reboot!

16. Verify success and install any additional packages you want.
      # pacman -A /mnt/cd/arch/pkg/whatever-1.1-1.pkg.tar.gz
Pierre@Pierre Schmitz

Random but important

January 02, 2016 12:51 PM

Security in computer programs is often based on some kind of secret. It might be a key, passphrase or the popular password. It is also equally important that this key is not only secret but also random and cannot be easily guessed.

The same concept applies to our package signing method. The security model which is used to ensure validity of our packages is directly based on PGP. In this trust model you mark a certain key as trusted by signing it with your very own private key.

On Arch Linux we ship the archlinux-keyring package to set up and maintain a set of packager keys and their web of trust. On install a new key pair is generated and the five master keys are locally signed. Any package that is signed by a key that itself is signed by at least three of the master keys is considered valid. This also implies that anything signed directly by that private key is automatically valid as well.

The keyring pacman uses is stored in /etc/pacman.d/gnupg. There is no magic involved here and you can even directly use it with the gpg command by appending the --homedir /etc/pacman.d/gnupg parameter.

It is important to underline that the PGP key pair stored there is unique to your machine. It is not part of any package but generated on install and random. An attacker who is able to obtain or guess that key can sign packages that pacman will happily accept as valid. Be aware of this when you create backups, clone your machines or setup virtual machines. It might be a bad idea to share the same private key on different machines.

For our live system image we ship a service that initializes a new random key on every boot. It is stored in RAM using tmpfs and therefor not persistent. We make use of the haveged service to create enough random data to not slow down the boot process.

Sadly the problem does not seem to be a theoretical one and it seems people are not aware of the dangers a shared /etc/pacman.d/gnupg directory implies. This especially affects system images, virtual machines and installer images you can download.

For example the Arch Linux fork Manjaro ships a static private key on their installer image which also gets copied to the target system. The result is that all Manjaro users have the same "private" key and anybody can easily use this to sign any package that the package manager will accept. It might be even worse as users expect package verification is set up correctly and care less about the source they install packages from.

If you just realized you made a terrible mistake, you can reinitialize your keyring with a new random and secret key using these commands:

$ rm -rf /etc/pacman.d/gnupg
$ pacman-key --init
$ pacman-key --populate archlinux

Keep in mind that package verification was essentially unavailable to this point and it’s up to you what consequences you draw from this. They only real solution is to re-install everything from a trusted source.

tl;dr: Make sure pacman’s PGP keys in /etc/pacman.d/gnupg are random and remain secret.

Pierre@Pierre Schmitz

Verify all the packages

January 02, 2016 12:51 PM

Once you have verified the finger prints of the five master keys you might want to actually configure pacman to verify packages. While this was already possible in the past, it is now way easier to get started thanks to pacman 4.0.3 and the keyring package.

If you already have played around with signature verification you could start from scratch by (re)moving the /etc/pacman.d/gnupg directory. I also assume that you have disabled signature checking by setting SigLevel = Never in /etc/pacman.conf.

All you need to do now is to install the archlinux-keyring package, initialize your local pacman keyring, finally import all the packager keys and set the trust level for the master keys. This is achieved by the following commands:

# pacman -Syu archlinux-keyring
# pacman-key --init
# pacman-key --populate archlinux

The last command will ask you to confirm each of the master keys. Now you can set SigLevel to PackageRequired for all official repositories in /etc/pacman.conf. For example:

SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

From now on pacman will check the signatures of all packages before installation and the keys are automatically managed by the keyring package.

Note: You can also set the SigLevel globally in the [options] section. The downside is that you wont be able to install your own packages using pacman -U anymore if they are not signed by a trusted key. Future versions of pacman will have a separate SigLevel option for packages that are installed from you local hard drive.

Pierre@Pierre Schmitz

PHP 5.4.1 in – Suhosin out

January 02, 2016 12:51 PM

Today I finally moved PHP 5.4.1 into our [testing] repository. Our previous tests of 5.4.0 were quite successful. In short PHP 5.4 is ready for production now and will be in our [extra] repository in a couple of days.

When I first packaged PHP 5.4.0 one of the most concerning issues was the lack of a compatible Suhosin patch and extension. Unfortunately this situation has not changed within the last two months: there is still no information if or when Suhosin will be released for recent versions of PHP. Therefor I decided to remove the Suhosin patch and its extension from our repository. To not break compatibility on minor updates the current version in [extra], PHP 5.3.11, includes a patched version of Suhosin; the last official one was dedicated for 5.3.9.

Suhosin is a safe guard for scripts and PHP itself. It consists of  patch to PHP and an extension which can be used independently. The patch aims to protect the PHP core against buffer overflows and string format vulnerabilities. Scripts written in PHP are protected by the Suhosin extension. It can be configured to disallow or limit certain user input. There is also a feature to transparently encrypt cookie and session data. More of Suhosin’s features are described on its website.

Certainly some of Suhosin’s features are a good idea as they might protect you in case PHP itself or scripts fail. This is why it was added years ago when I took over maintainership of PHP. Fortunately some of its features got added to PHP itself over time. But today we are confronted with several problems:

  • There is no Suhosin for PHP 5.4. This means we would be stuck at 5.3 and cannot ship the most recent packages which is one of the goals of Arch Linux.
  • The upstream project has become less and less active in the past years. There are certain indicators which make Suhosin an unhealthy open source project: There is still only one main contributor, the development process is mostly closed, there is no public repository for the patch, no bug tracker, the forums are in maintenance mode for years and the latest news entry is from 2007. Of course things might be different in the background, but this is the situation which is publicly visible.
  • The delays between PHP releases and their Suhosin counterparts are increasing. I often have to patch the Suhosin patch itself to use it with latest minor updates of PHP. This leaves us with a PHP version that is neither supported by the PHP project nor the Suhosin authors. While these modifications are mostly trivial I cannot exclude the possibility that I break PHP in a dangerous way. The situation is even worse on major update like we have now. Adjusting Suhosin to PHP 5.4 is not trivial. In addition to this testing development releases or release candidates often means that you have to disable Suhosin.
  • There are no plans of the project to ever move any of Suhosin’s features into PHP itself. In fact the Suhosin author is strictly against such a move.

In general we have a “Do not patch” policy in Arch and try to keep our packages as close to upstream as possible. In its current state I no longer have any excuse to break this rule for Suhosin; in fact I now have a lot of arguments which support this policy. So even if a new version of Suhosin was released tomorrow, next week or in a few months, it wont be re-added to our PHP packages. The upstream projects would need some changes to its process to be reliable again.The relationship with the PHP project itself has to improve and it should at least be tried to apply certain features into the PHP core project.

Pierre@Pierre Schmitz

Trust the Master Keys

January 02, 2016 12:51 PM

All our packages are finally signed, pacman itself and our development tools are able to create and verify signatures for some time now. There are still some bits and pieces missing untill we can have package verification enabled by default, but we are almost there.

At some point in time you might want to switch from an untrusted setup to one that magically verifies every package. Sadly it wont be that easy and to be entirely correct paranoid users will even need to reinstall from a signed medium (which does not exist yet) as it is hard to tell if your system was already compromised. I assume most users will instead hope for the best and skip this step.

While pacman can tell you if a package was signed by a private key to a given public key, it will be up to you to decide whether that key can be trusted or not. Due to the nature of the web of trust used in PGP you wont need to verify the key of every single developer. Instead you will only need to trust five of us who have created so called master keys.

Graph of the Arch Linux web of trust

These master keys wont be used to sign packages directly but to sign the keys of every packager. If a packager key is signed by at least three of those master keys PGP will automatically trust this key as well. To sum things up, you will only need to verify five keys once and you are done. You neither need to care about verifying each key nor to kee track of developers joining or leaving the team.

You will find the finger prints of all five master keys on our website. However, it is certainly a good idea to check other sources as well. For all what we know that site could already be compromised. The more clues you can gather to verify a key the better. Here are some hints about what you could verify. But don’t stick to that list, be creative.

  1. Make sure you are accessing that master key website via https.
  2. Verify that the SSL certificate is valid.
  3. You may also check that the certificate is issued by StartCom Ltd. to Aaron Griffin.

Each key holder has also signed their master key with their packager key. Therefore it is useful (and probably easier) to verify these packager keys as well. Let’s see what we can find out about these keys

  1. Some packager keys are verified by CAcert. If you trust those you now know that their email address and name match a real person. Keep in mind that this does not mean that this person is authorized to create packages for Arch Linux.
  2. Some of us have their own homepage where we also publish our keys.:
  3. In that case you can check whether the whois entry matches the person’s name.
  4. If they provide https access check their certificate and the name it was issued to.
  5. Check if that homepage is the one that these developers advertise in their signatures, forum profiles etc..
  6. If they sign their mails to the mailing list you can verify these as well. See if you can find mails from different time periods.
  7. Be a stalker, use your favorite search engine to look up their names, mail address, homepage addresses but also key fingerprints and ids.

These are some basic checks you can quickly do online with a reasonable amount of effort. Of course none of these checks are absolute and even if you do them all you cannot be certain. At some point you will have to ask yourself how much effort and money an attacker would invest in compromising Arch Linux users. This is a general rule for security related topics: you may not be able to make an attack impossible, but you can try to make it too expensive.

Last but not least, if you are too lazy to verify all this yourself, ask someone you trust instead. It is a web of trust after all. This also means that if you went ahead and verified all master keys as good as possible, tell people about it. You may just write a brief blog article about it and quote the key finger prints.

Of course the best way to let everybody know that you have verified someones key is to sign it using your own. But I would advice you to only do so if you have met that key owner in person. That also means: if you ever meet a developer or trusted user: ask them to sign each others key.

I am leaving you with a list of short cuts to start with:

Pierre@Pierre Schmitz

Arch before it was cool

January 02, 2016 12:51 PM

Ten years ago Judd released the first version of Arch Linux. This is quite an age for a Linux distribution and we are still rolling. Allan just posted a brief history of mile stones that were important to him. Inspired by this nice idea…I am going to do exactly the same.

It has been a long time but it seems I started a more serious look at Linux back in April 2003. That time I joined

In June 2004 I came in contact with Arch when I was looking for a lightweight and technical easy distribution with rolling releases.

Here is the Arch Linux history from my perspective:

2004-06-26 I asked for a new distribution at and tried Arch for the first time.
2004-07-05 I registered at our great forums.
2004-12-21 My first contribution was setting up a forum for German users. I used to code my own forum system back then and hosted several boards.
2006-10-13 I send my application as trusted user to tur-users. Eric was so kind to sponsor me.
2006-10-29 My application was accepted by most Trusted Users and I joined the team. I started contributing the first set of lib32 package to be able to play games like Doom 3 or Quake 4 on the x86_64 port of Arch.
2007-05-12 I sent my application to the Arch developers in order to help out with x86_64 and KDE.
2007-05-13 I was accepted as a developer along with Eric.
2007-06-30 I wrote an article for the Linux User magazine about Arch Linux. People could actually buy this along with a DVD at their kiosk.
2007-07-10 We started to reorganize our repositories. The [core] repository and its signoff policy was born.
2007-08-26 First meetup of some developers at FrOSCon. Most people didn’t know about Arch yet and only very few found us at a distant room within the conference building.
2007-09-26 I started breaking PHP.
2007-10-01 Judd resigned and handed the leadership over to Aaron.
2007-12-19 I resigned as a Trusted User; in order to concentrate on my role as developer but also to keep the TU group independent from the developers.
2008-07-27 I replaced KDE 3 with KDE 4 and broke everything. I got a lot of mails from people who were really happy about that.
2008-08-24 Again, it was time for FrOSCon. We now had our own booth and a developer room. Thomas and I even gave an interview.
2008-11-06 I launched the pkgstats project. Statistics are still available at a “temporary page“.
2009-02-05 I started maintaining our wiki setup. I already did this for, so I made it independent from the language and the site it was running on.
2009-08-03 I introduced a split up version of KDE.
2009-08-26 We met again at FrOSCon.
2009-09-22 I also split PHP packages.
2010-02-10 “I am done with packaging” was the subject of a mail I sent to the arch-dev mailing list. I wanted to concentrate on our scripts and workflow in general. As such I took over maintainership of dbscripts and devtools. In the process I mainly dropped KDE and Andrea took the lead here.
2010-03-23 Packages could be compressed with xz to reduce their size significantly by about 30%.
2010-06-28 I took over maintainer ship of our forums.
2010-08-11 I started working on the concept of a [staging] repository again. I finally got enough positive feedback to get this implemented a long with a package pool.
2010-08-30 FrOSCon 2010 and we were there.
2010-09-06 I finished support for a package pool which makes moving packages between repositories a matter for adjusting some symlinks and updateing the db files. This improved the usage of the [testing] repository a lot.
2011-05-11 I uploaded the first signed package to the repository.

These are some information about my history at Arch I could gather in a reasonable amount of time. It has been fun to dig through all these old mails. Some information are hard to get or lost forever as we changed our mailing lists, forums, wiki and even switched from cvs to svn in the meantime.

I probably could have added dates to the timeline forever. But I’ll just stop here to keep it less boring. If I had missed anything important, let me know. Also I didn’t add other conference we had attended like last years FrOSCon, OpenRheinRuhr or FOSDEM as I simply have no pictures of those.

At this point I am still working on getting PHP 5.4 ready for action and helping to finalize the package signing process. I am also working on repository management and on the package building process in general. There will be separate articles about these ideas.

Arch Linux 0.1 and pacman 1.1

If you like to have a look at the very first version of Arch: I have uploaded arch-0.1-full-i686.iso to our mirrors. It’s md5sum is:

Pierre@Pierre Schmitz

Using OPcache to speed up your CLI scripts

January 02, 2016 12:51 PM

With PHP 7 it is now possible to use OPcache efficiently. It can be configured to store the compiled byte code on disk and use it on the next run of a script. Complex command line scripts like Composer and PHPUnit tests will benefit from such a cache as it reduces startup time dramatically.

In addition to benefiting cli scripts, this cache is also used as second level cache in general. This should increase performance if the SHM backed OPcache runs out of memory or the PHP process gets restarted.

To enable the second level cache simply ensure the following ini directives are set:

In this case I am going to store the byte code in /tmp which means it wont survive a reboot and will be kept in RAM by default. We now need to ensure that the mentioned directory /tmp/php-opcache is created by systemd on boot. Place the following content in a file called /etc/tmpfiles.d/php-cli-opcache.conf:

d /tmp/php-opcache 1777 root root 1d

This will also prune all files that are older than one day. You might want to adjust these values to your needs or even use a persistent storage in /var/tmp instead of /tmp.
To initially create this folder simply run

# systemd-tmpfiles --create /etc/tmpfiles.d/php-cli-opcache.conf

You may now check your configuration by running some cli scripts:

$ composer -V

The compiled byte code should now appear in /tmp/php-opcache:

$ tree /tmp/php-opcache               
├── 896f8ffaef254102552141eb4ab1b214
│   └── usr
│       └── bin
│           └── composer.bin
└── 896f8ffaef254102552141eb4ab1b214phar:
    └── usr
        └── bin
            └── composer
                ├── bin
                │   └── composer.bin
                ├── src
                │   ├── bootstrap.php.bin
                │   └── Composer
                │       ├── Command
                │       │   ├── AboutCommand.php.bin
                │       │   ├── ArchiveCommand.php.bin
                │       │   ├── ClearCacheCommand.php.bin
Pierre@Pierre Schmitz

PHP 7 on Arch Linux

January 02, 2016 12:51 PM

I have been working on PHP 7 packages for a while. They are available in my repository:

Server =$repo/os/$arch

Insert these lines on top of the other repository definitions in your /etc/pacman.conf
A copy of the PKGBUILDs I used to create the packages are available in my git repository.

Upstream changes

PHP 7 is possibly one of the most interesting releases since version 5 which was published over a decade ago. Besides up to twice the performance PHP now offers scalar type declarations and return types. The official migration guide lists all new features and incompatible changes.

Packaging changes

With PHP 7 I am trying to move closer to upstream defaults and be less opinionated. While this requires a more responsible configuration by the user it also eliminates some surprises. And after all this is the Arch way. As a bonus, scripts like Composer work fine with the default configuration now.

Core package changes

  • php-pear is no longer available
  • php-mssql was removed by upstream
  • php-ldap is now included in the core php package as its dependencies were already met
  • The mysql extension was dropped upstream and is no longer available

Configuration changes

  • openssl, phar and posix modules are now built in. Remove the corresponding directives from you php.ini, e.g.
  • The include_path is no longer set to /usr/share/pear by default
  • open_basedir is no longer defined. This also means that packaged scripts do not need to be put into an awkward /usr/share/webapps directory.
  • PHP-FPM does no longer provide a logrotate configuration. By default errors are logged to syslog (these will be picked up by journald).
  • FPM pool configuration is now stored in /etc/php/php-fpm.d by default.
  • The upstream FPM service file is now installed which does not include PrivateTmp=true.

Third party extensions

We offer a few PHP modules that are maintained by third parties and are not part of the PHP source distribution. So far only modules that are in active development were able to provide updated versions. This means we have to drop most of these PHP extensions, especially those that might be considered dead.

The following list provides an overview of modules I gathered so far. It is possible we could re-upload updated packages once upstream provides us with compatible versions.

Package Status Solution
graphviz incompatible remove PHP bindings
php-apcu compatible update to version 5
php-geoip incompatible remove package
php-memcache incompatible remove package, project seems dead
php-memcached incompatible remove package, move back when upstream version is available
php-mongo incompatible remove package, superseded by the mongodb driver
php-xcache incompatible remove package, project seems dead
uwsgi-plugin-php incompatible remove PHP support, upstream update seems possible
xdebug compatible update to latest release candidate of version 2.4

In addition to this I will introduce a new module called php-apc-bc which will provide the legacy apc_ function for use with the APCu extension. Before version 5 this feature was provided by APCu itself.

Testing in Production

While I have been working on PHP 7 for months now one really only sees how it performs when put into production. So a few days ago I started running all services at (this also includes this very blog) with that latest version of PHP.

So far its running fine. The update went smoother than expected and the needed configuration changes were close to nonexistent.

There are some regular but very few core dumps produced by php-fpm. These did not result in 5xx server responses though and I assume that they are related to shutting down fpm workers. It should be investigated but it does not seem critical.

Before updating I had to make two small source code changes:

  • Engine errors are now of the type \Error. \Error and \Exception implement the \Throwable interface. So I had to adjust a type hint, or in this case remove it so it would also work in PHP 5.6.
  • The return type of the \SessionHandlerInterface are now enforced. If you return anything other than a string in your custom session handler the operation will fail.

Your Feedback

Now it is your turn. The packages provided in my repository are ready and I will ensure an upgrade path once the updated packages are available in the [extra] repository. I do not plan to hold them back for very long but I’d like to get some feedback first. After all it is a major update, we drop a lot of extensions and compatibility with many scripts out there is unknown.

Try these packages, test your own code and send me your feedback and suggestions.

Pierre@Pierre Schmitz

PHP 5.5 on Arch Linux

January 02, 2016 12:51 PM

Packages of PHP 5.5 are available in my repository. Note that these are debug builds and might require the use of the [testing] repository.

Server =$repo/os/$arch

Only try this at home. APC gets replaced by APCu for user data caching. Use the new opcache module to cache the PHP scripts. XCache is also available as a development snapshot.

Let me know of any issues but also successes. More will follow up soon.

Pierre@Pierre Schmitz

Farewell i686!

January 02, 2016 12:51 PM

The first 64 Bit x86 CPU was released ten years ago: the AMD Opteron. Intel followed with compatible processors about one year later. Another two years later, in 2006, a first port of Arch Linux to the x86_64 platform was available. Starting outside the official repositories, the new architecture was merged into the Arch project and became a first class citizen within the following years. Right from the start it was designed as a pure 64 Bit system and kept in sync with the i686 tree. That also meant no multilib support at first and every package was compiled for x86_64.

Of course getting there was not that easy. Our repositories and tools like pacman did not have any concept of different architectures. While most software would just work fine when compiled for x86_64, we occasionally ran into issues that were specific to the 64 Bit architecture.

Years later the relation between 32 and 64 bit architecture seem to have switched roles.  Developers have chosen x86_64 as their primary platform, our i686 packages seem to get tested by less users and we see issues specific to the 32 Bit architecture. We recently relaxed our signoff policy for i686 packages in order to counteract this trend and avoid holding back development due to lack of testing.

To back these observations with some numbers, I analyzed the architecture submitted by users via pkgstats over the past few years. Even if we assume that these data are not absolutely accurate and I could not find all the data gathered between 2009 and 2010, the results are clear: While the x86_64 version of Arch Linux was installed by only 20% of our users in 2008, its usage grew to 80% in 2013. It is fair to say that within the next years i686 usage will fade away.

Architecture Usage by Date

Architecture Usage by Date

For a few month we also collect data about the actual CPU architecture users have. Right now about 93% operate a CPU that would be capable of running a 64 Bit system. There is not much data yet, but it seems to be a slowly rising number.

CPU Architecture Usage by Date

CPU Architecture Usage by Date

We are pretty close to the point where i686 Arch Linux installations have been almost completely replaced by x86_64. This of course means that it might no longer be practical to provide support for an architecture that is barely used. The question is not if 32 bit support might be dropped,  but when.

Don’t Panic! I don’t see us dropping i686 support this year, but I would not dare to predict anything beyond that date. As always these decision are not based on business or politics. We will see when i686 usage will drop below a critical number and developers loose interest in packaging and supporting that architecture.

Pierre@Pierre Schmitz

BananaPi with Arch Linux ARM and a mainline kernel

December 30, 2015 08:37 PM

I posted a guide on getting Arch Linux ARM on the BananaPi last week. Now I was eager to get a mainline kernel working on the BananaPi for some ARM hacking and testing of new patches. In this post I’ll describe the steps required to get a mainline kernel booted next to the “normal” kernel.

Compiling mainline kernel

I am using Hans de Goede’s branch here, since he has some patches for USB OTG support for the BananP in there. But the normal mainline kernel should work with similiar steps

git clone
cd linux-sunxi
git checkout -B sunxi-wip origin/sunxi-wip

mv kernel-config .config
# CROSS_COMPILE may differ per distro. arm-linux-gnu- probably works for the other distro's.
make -j4 ARCH=arm CROSS_COMPILE=arm-none-eabi- oldconfig 
make -j4 ARCH=arm CROSS_COMPILE=arm-none-eabi- dtbs
make -j4 ARCH=arm CROSS_COMPILE=arm-none-eabi- zImage

Now that we have a kernel compiled, mount the sdcard.

mount /dev/sdX1 mnt

cp /path/to/linux-sunxi/arch/arm/boot/zImage mnt/boot/zImagemainline

mkdir mnt/boot/dtbsmainline
cp /path/to/linux-sunxi/arch/arm/boot/dts/*.dtb mnt/boot/dtbsmainline/
umount /dev/sdX1

Now that we have copied the mainline kernel and dts files to the sdcard, it still won’t boot it. Since the boot.scr is specified to look dts/* and zImage. So we will need to copy a new scr file. (note that you cannot edit the scr file on the sdcard) You can either generate a new scr file or create on yourself!

mount /dev/sdX1 mnt
wget -O mnt/boot/boot.scr
umount /dev/sdX1

Or generate a new scr file, which sounds more fun.

pacman -S uboot-tools

mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n "BananPI boot script" -d boot.cmd boot.scr

mount /dev/sdX1 mnt
cp boot.scr mnt/boot/boot.scr
umount /dev/sdX1

I haven’t figured out what the boot.scr exactly does or if there is a way to specify which kernel to boot with one boot.scr file. So expect some more posts while I explore more about U-boot and ARM :-)

BananaPi with Arch Linux ARM and a mainline kernel was originally published by Jelle van der Waa at Jelly's Blog on December 30, 2015.

Jelle van der Waa ( Van der Waa

Logrotate Weechat logs

December 27, 2015 08:45 PM

Today I noticed my Weechat logs have grown a lot! ~/.weechat/logs has grown to 360M. Obviously I need some sort of automation to take care of these logs, enter logrotate. Luckily Earnestly showed me how to use logrotate on it’s own some time ago. So all I what was needed was a logrotate config file (similiar to /etc/logrotate.d/*) and a systemd user service file. First off the ~/.local/share/logrotate.d/irc.

# weechat logrorate file

# Log files.
"~/.weechat/logs/**log" {}

The systemd service file located in ~/.config/systemd/user/weechat-rotate.service.

Description=Logrotate weechat's logs

ExecStart=/usr/bin/logrotate -s %t/logrotate.state  %h/.local/share/logrotate.d/irc -v


Logrotate will run when Systemd reaches, more info about can be found here. You could also use a Systemd timer to schedule the logrotate weekly or monthly. To view log output use journalctl.

journalctl --user -f -u weechat-rotate

Logrotate Weechat logs was originally published by Jelle van der Waa at Jelly's Blog on December 27, 2015.

Jelle van der Waa ( Van der Waa

Arch Linux ARM on the BananaPi

December 22, 2015 06:37 PM

Since some time the Banana Pi is supported in the Linux kernel and U-Boot. Arch Linux ARM does not support the board yet, but it is possible to get it working with an upstream U-Boot.

SD card Image

First we to create an SD card image, since the BananaPi is also ARMv7 we can use the Cubieboard 2 installation instructions Follow the steps until step 6, “Install the U-Boot bootloader”. We will have to compile our own u-boot for the board.


The next step is creating a u-boot image. Make sure you have the following packages installed on Arch Linux.

  • git
  • arm-none-eabi-gcc # ARM cross compiler

Now follow the following steps.

git clone git://
cd u-boot
make -j4 ARCH=arm CROSS_COMPILE=arm-none-eabi- Bananapi_defconfig 
make -j4 ARCH=arm CROSS_COMPILE=arm-none-eabi-

If everything went fine you should have an U-Boot image: u-boot-sunxi-with-spl.bin. Now dd the image to your sdcard, where /dev/sdX is your sdcard.

sudo dd if=u-boot-sunxi-with-spl.bin of=/dev/sdX bs=1024 seek=8

Flush buffers


Now mount the sdcard again.

mount /dev/sdX1 mnt
wget -O mnt/boot/boot.scr
umount /dev/sdX1

Booting & Serial

Instructions on getting serial working are on the sunxi wiki.

Arch Linux ARM on the BananaPi was originally published by Jelle van der Waa at Jelly's Blog on December 22, 2015.

Jelle van der Waa ( Van der Waa

Dropping Plasma 4

December 12, 2015 10:04 AM

Since the KDE 4 desktop has been unmaintained for several months and it's becoming increasingly difficult to support two versions of Plasma, we are removing it from our repositories. Plasma 5.5 has just been released and should be stable enough to replace it.

KDE 4 installations will not be automatically upgraded to Plasma 5. However, we recommend all users to upgrade or switch to a maintained alternative as soon as possible, since any future update may break the KDE 4 desktop without prior notice. See the wiki for instructions on how to upgrade to Plasma 5.

Antonio Rojas@Official News

C++ ABI change

December 10, 2015 06:48 AM

GCC 5.x contains libstdc++ with dual ABI support and we have now switched to the new ABI.

While the old C++ ABI is still available, it is recommended that you build all non-repo packages to have the new ABI. This is particularly important if they link to another library built against the new ABI. You can get a list of packages to rebuild using the following shell script:


while read pkg; do
    mapfile -t files < <(pacman -Qlq $pkg | grep -v /$)
    grep -Fq "${files[@]}" <&- 2>/dev/null && echo $pkg
done < <(pacman -Qmq)

(Original announcement text by Allan McRae [link])

Evangelos Foutras@Official News

C100P Tweaks

November 18, 2015 05:37 AM

Now with charging limits!
Kyle Keen

Xorg 1.18.0 enters [testing]

November 13, 2015 12:42 PM

Xorg 1.18.0 is entering [testing] with the following changes:

  • You can now choose between xf86-input-evdev and xf86-input-libinput.

  • xf86-input-aiptek will not be updated and will be removed when xorg-1.18.0 is moved to [extra]

Update: Nvidia drivers are now compatible with xorg-1.18.0 (ABI 20)

Laurent Carlier@Official News

C100P Tweaks

November 06, 2015 08:06 AM

All the little things I do for the Asus Flip
Kyle Keen

Shooting and Stitching Panoramas in Hugin - Step by step

November 02, 2015 03:35 PM

I was asked to write something about my workflow in creating panoramas. Some of you maybe have seen my huge panoramas on my Google+ or 23 account or if you haven't then have a look at them ;-)

For demonstrating this step by step guide I will use my latest panorama, which was generated from 39 portrait photos. At the end the final processed panorama will look like this:

The shooting

As I mentioned in my G+ post you can see the Kinzig valley near Haslach (Black Forest in Germany). You have this view on the newly builded observation tower on top of Urenkopf, the tower is simply called Urenkopf-Tower. If you need more information about the tower, you will find all information on the website of Haslach.

I had no tripod with me, most of the time I don't use a tripod for my panoramas. If you really hold your camera in the same horizontal position during the shooting you will be able to shoot one shot after the other with a little overlaps of each shot. Sure it will be better and easier if you use a tripod, so decide on your own how you will do the shots. 

For the shots I used a focal length of 28mm on my Tamron 28-75mm f/2.8 lens and an ISO setting of 100. Shutter speed was around 1/250s at f/8. With these setting I had shot 39 photos:
I know that less photos would also work, but if you shoot handheld then you have better more shots than missing important parts of the panorama and then the work were useless.

The processing

For my photo processing and my complete workflow I use open source software under Linux. May favourite distribution is Arch Linux, because I'm a member of the developer team, so it's normal that I use it. My photo management software is Digikam, photo processing and editing is done in GIMP and in RawTherapee. I know that Digikam has a panorama plugin, but I prefer the usage of Hugin.

All these programs are available for Mircosoft Windows, too. I don't know how well they work on Windows, so if you try these on Windows then good luck, but I can't give you any support on this. 

Since some versions Hugin has the "Simple" interface mode which hides most of the expert settings from the GUI. That's okay for the most of panoramas you will generate.

So I have imported all 39 images into hugin. If you start up Hugin and you have the simple interfaces activatd, then you have just to click on the "1. Load images" button, select the panorama images and press "Open".
Hugin starts automatically to analyze the images and tries to find control points for the stitching process.
After some seconds or minutes, depends on the number of shots, Hugin will present you a first preview of your panorama:
Hugin gives you some information about your images in the right corner, how good your shots fits and other maybe useful information.

Now you can change the projection of the panorama, move single images around, straighten the sky and crop the panorama to your needs. Also you can identify the control points or the overlapping images. Hugin gives a huge set of options here. Some options are displayed in the next screenshots:
Showing control points

Identify each single photo

Select the projection

Straighten the horizon

Play around with the setting of the projection, maybe you will find a better projection for your panorama. I used the cylindrical projection.
Most of the time all these settings are not necessary to change, so if everything looks good, then you just have to click the "Create panorama..." button. Hugin is now starting the complete process with stitching and with blending all the images, this takes some time depending on your images. Hugin will inform you about the process all the time with its status window:

After the complete process Hugin should output the final panorama. Normally it looks like the one in the preview. Mine looks like this:

After editing the final panorama in GIMP and RawTherapee the following panorama which you have seen in my posts was created:

Deconstructing Featured Photo

Camera: Canon EOS 7D
Lens: Tamron 28-75mm f/2.4
Focal Length: 28mm
ISO: 100
Aperture: F8
Shutter Speed: 1/250 sec
Tripod: none used for this shot, but my new travel tripod is a Sirui T-005X
Ballhead: C-10X and plate TY-C10

I hope you liked my small step by step tutorial about my workflow in creating panoramas. Leave a comment, a +1, a like or tweet about it.
Daniel Isenmann ( Isenmann

Downtime (rsync, mail)

October 08, 2015 08:10 AM

Update: All fixed now.

I just installed a kernel update on our rsync and mail server and it seems we have broken hardware so it is unable to reboot right now. Mailing lists are running on a different system however you need to use the domain rather than So for arch-general you'd use Mails sent to the normal domain will go through once the server is up again.

The rsync master will stay unavailable for now.

I've asked the hoster to look into the issue, but I can't currently estimate when I'll get a reply/fix.

Sorry for the inconvenience, Florian

Florian Pritz@Official News

From Ghost To Nanoc

October 05, 2015 10:00 PM

I completed my blog migration from Ghost to nanoc.

About 2 years ago I did setup a blog on using Ghost. It’s UI was very minimal and I liked the default theme (Casper) a lot.

However, I kept nanoc for my main website, until I decided to give Hakyll a try. It’s not that nanoc didn’t satisfy me at that time, but that I was affascinated by Haskell - I’m still affascinated by Haskell, but I’ve no much time to play with it, while I play with Ruby more often.

Someday ago I tought it’s time to merge my website and my blog; both could be handled by a static site generator and since I’m fluent in Ruby more then Haskell, I went for nanoc again.

The migration has not been hard because one of the main features of Ghost is that you write your post using Markdown, then I wrote this shell script to migrate my posts from Ghost to a “nanoc compatible format” like:

kind: article
created_at: 2015-10-06
title: My Ghost Post
tags: ['example']
This is a post in **Ghost**!

With that script my posts were split and ready in the content folder to be built by nanoc. Nothing more to do! Well, in truth I had to fix the path to the linked images manually…

The second step was to put some redirect to allow the old links around the web to continue to work, specifically the Ghost pattern was while in nanoc I went for I fixed this in my blog nginx configuration:

location = / {
  rewrite ^ $scheme:// permanent;

location / {
  rewrite ^(.*) $scheme://$request_uri permanent;

location = /rss/ {
  rewrite ^ $scheme:// permanent;

While in the website nginx configuration I put:

location /posts/ {
  root   /srv/http/website;
  if ($request_filename ~* ^.+.html$) {
  if ($request_uri ~* ^.+/$) {
    rewrite ^/(.*)/$ /$1.html permanent;

And that’s!

Hope this helps someone that plans to do the same migration. If you are interested at looking at my nanoc setup, the configuration is here.

Andrea Scarpino

Interview with Matt Reiferson, creator of NSQ

October 02, 2015 08:25 AM

I’m a fan of the NSQ message processing system written in golang. I’ve studied the code, transplanted its diskqueue code into another project, and have used NSQ by itself. The code is well thought out, organized and written.

Inspired by the book coders at work and the systems live podcast, I wanted to try something I’ve never done before: spend an hour talking to Matt Reiferson - the main author of NSQ - about software design and Go programming patterns, and post the video online for whomever might be interested.

We talked about Matt’s background, starting the NSQ project at Bitly as his first (!) Go project, (code) design patterns in NSQ and the nsqd diskqueue in particular and the new WAL (write-ahead-log) approach in terms of design and functionality.

You can watch it on youtube

Unfortunately, the video got cut a bit short. But basically in the cut off part i asked about the new go internals convention that prevents importing packages that are in an internals subdirectory. Matt wants to make it very clear that certain implementation details are not supported (by the NSQ team) and may change, whereas my take was that it’s annoying when i want to reuse code some I find in a project. We ultimately both agreed that while a bit clunky, it gets the job done, and is probably a bit crude because there is also no proper package management yet.

I’ld like to occasionally interview other programmers in a similar way and post on my site later.

Dieter Plaetinck

It's time for a review or my 1 1/2 years with the Renault Zoe

October 01, 2015 07:23 AM

Renault Zoe (photo credits: Renault)
Since 1 1/2 years I drive my all-electrical Renault Zoe. Since my very first drive I know one thing exactly: I'll never go back to a gasoline car.

I could go on and write lots of arguments for an electric car, but I don't. There are lots of pros and cons around the web, I think that anyone must decide on its own if he is willing (or able because of the price) to drive an electric car or not.
I will only post some details about my charging behavior and the total costs I had the last 1 1/2 year, so this will be very objective review. 

So let us start with some facts:
  • daily usage (Monday to Friday) for driving to work and back (12 kilometers one way)
  • some trips during the weekends
  • 14.284 kilometers (~ 8.875 miles) driven
  • from October 1st, 2014 until September 30th, 2015 I charged the Zoe 89 times:
    • 37 charges at home
    • 3 charges at public charging points
    • 49 charges at public charging points with no costs (the local power company near my office provides them)
  • overall charging costs of 85€ (~ $94) between October 2014 and September 2015
  • 1 planned service appointment (90€)
  • 420€ per year for full comprehensive insurance
  • 948€ leasing rate for the battery which is rented in all electric cars of Renault (included are replacement of the battery (if the battery capacity falls under 75% or if the battery fails at all) and free 24/7 assistance service in emergency)
Overall costs (incl. everything from above) for 1 year:  1.543€ (~ $1.720)

The next two diagrams (you can click on each diagram to see it better) will show you how often I have charged and how the charging costs is spread over the year:

In the next diagrams you can see each charging process for every month. I have listed the SoC (state of charge of the battery) before charging and which state the battery had after charging.
I hope that I was able to give you a short review and summary of my usage of my Zoe. If you have any question, just leave a comment, I will try to answer them as good I can.  
Daniel Isenmann ( Isenmann


September 24, 2015 03:03 PM

Hey all

UPDATE: A rebuild has been done against an initial patch. If you already have gdal1, please uninstall and reinstall vtk [6.1.0-10 -> 6.1.0-11] so that gdal1 is removed from the system.

I should have posted this sooner but there is a bit of a mess with VTK and GDAL for which I am responsible. If you have vtk installed in your system along with any one of the following:


Any attempt to update your system will be prevented by the conflict between gdal and a gdal1. [1] A gdal 2.x rebuild [2] was staged for a long while and vtk was preventing it from moving forward (it is not API-compatible with the new gdal, which is a core dependency) -- there were more rebuilds that were waiting in line.

The gdal1 package is only a tentative inconvenience as I did not want to drop vtk. Due to time constraints I could not attempt a patch in the several months that the rebuild was sitting idle and upstream was not ready to migrate. Another developer (anatolik) has stepped up to help and may take over maintenance of the package. Anyone is also welcome to contribute.

Please bear with us in the meantime, thanks!


schivmeister@Forum Announcements

D-Bus now launches user buses

September 20, 2015 08:31 PM

The packages systemd 226-1 plus dbus 1.10.0-3 now launch dbus-daemon once per user; all sessions of a user will share the same D-Bus "session" bus. The pam_systemd module ensures that the right DBUS_SESSION_BUS_ADDRESS is set at login.

This also permits dbus-daemon to defer to systemd for activation instead of spawning server processes itself. However, currently this is not commonly used for session services (as opposed to system services).

kdbus will only support this model, so this is also an opportunity to iron out some bugs and make a future transition to kernel buses easier. Please let us know of any issues.

Jan Steffens@Official News

KDE Telepathy ThinkLight Plugin

September 11, 2015 10:00 PM

Do you own a ThinkPad? Good! Does it have the ThinkLight? Good! Then this post might interest you!

I just wrote a KDE Telepathy plugin that blinks the ThinkLight when you get an incoming message. Sounds almost useless, isn’t? Maybe not.

I found a good use case for it: sometime you could be away from keyboard, but near your ThinkPad (e.g. studying), the screen goes black, sounds are off, but you see the ThinkLight blinking - you got a message!

To enable it you just have to fetch the source code, build and install as usual with CMake.

There’s just an annoyance at the moment: you need write permission over /proc/acpi/ibm/light. I’m looking for a solution for this, but found nothing if not changing that file permissions manually. Any idea?

There’s also a tool, thinkalert (mirror), which allows to turn on/off the ThinkLight without being root by using suid. If you prefer this way, you can fetch the code from the thinkalert branch instead.

Have fun!

Andrea Scarpino

September’s TalkingArch is here

September 10, 2015 02:33 AM

The TalkingArch team is pleased to bring you the latest iso. Based on September’s Arch Linux iso, this TalkingArch features all the latest base and rescue packages, along with Linux kernel 4.1.6. Due to the steadily growing size of both the Arch Linux iso and the TalkingArch iso, this will be the first image that is larger than 700MB, which means that it will need to be burned to a DVD or written to flash media such as a USB drive or a memory card. If you need to have an image that you can burn to a DVD, one is available upon request. Please don’t hesitate to contact support to get your slightly older, but still fully functional CD image. As always, have fun with TalkingArch.


Getting Started with Zshell

September 08, 2015 05:51 PM

Getting Started

The first thing I will say about zsh is that all the extra tools to put more information in your prompt for your every day use, aren't needed with zshell. It can handle all of that fancyness for you.

All of that is done only in zsh code here. I don't have much written in there to do anything, it is mostly just configuring using zstyle.

Parameter Expansion


ZSH Parameter expansion flags

Parameter Expansion documentation

These are probably the links I still use the most. The first one is just to the zsh wiki, there is a ton of good information to get started in there. The second link is what I use the most, because I never remember what all the parameter expamsion stuff does.

The one I use the most is probably (f), which will split output of a parameter on newlines. The below takes the output from ls -l and then splits it on newline, and then grabs the 4th line from the output.

└─╼ echo ${"${(f)"$(ls -l)"}"[4]}
-rw-r--r--  1 daniel daniel   1770341 Jun  2  2014 2014-06-02-182425_1918x1079_scrot.png

You can also do matching. The below uses the (M) to tell the expansion to only show lines that start with 'drwx'.

┌─ daniel at hailey in ~
└─╼ print -l -- ${(M)${(f)"$(ls -l)"}:#drwx*}
drwxr-xr-x 18 daniel daniel      4096 Aug 24 00:06 aurinstall
drwxr-xr-x  2 daniel daniel      4096 Nov  8  2014 bin
drwxr-xr-x  3 daniel daniel      4096 Mar 10 22:17 Colossal Order
drwxr-xr-x  2 daniel daniel      4096 Apr 26 09:44 Desktop
drwxr-xr-x  2 daniel daniel      4096 Apr  1  2014 Documents
drwxr-xr-x  5 daniel daniel      4096 Jul 14 17:34 Downloads
drwxr-xr-x 41 daniel users       4096 Jul 20 18:49 github
drwxr-xr-x  2 daniel daniel      4096 Apr  1  2014 Music
drwxr-xr-x  4 daniel daniel      4096 Sep  8 14:12 Pictures
drwxr-xr-x  2 daniel daniel      4096 Apr  1  2014 Public
drwxr-xr-x  6 daniel daniel      4096 Jul 30 11:04 python-systemd
drwxr-xr-x  2 daniel daniel      4096 Apr  1  2014 Videos

Things to remember about zsh parameter expansion, like bash # means from the beginning of the line, and % means from the end of the line. But, you can chain them together on one line, unlike in bash. The below is a somewhat contrived example. First we use # and % to do replacement on each line of the array to find out what groups are available, then we assign the array to the names variable, and use ${array1:|array2} to get all variables in array1 that aren't in array2.

└─╼ print -l ${${${(M)${(f)"$(ls -l)"}:#drwx*}#* * * }%% *}
┌─ daniel at hailey in ~
└─╼ names=(${${${(M)${(f)"$(ls -l)"}:#drwx*}#* * * }%% *})
┌─ daniel at hailey in ~
└─╼ groups=(daniel)
┌─ daniel at hailey in ~
└─╼ echo ${names:|groups}


Filename Expansion

Just like with parameter expansion, zshell has a bunch of extra flags to expand filenames. One of my favorite examples is to just find filenames that are only directories. Which can also be expanded using the information from before to find all files that aren't directories.

┌─ daniel at hailey in ~/example
└─╼ tree
├── dir1
├── dir2
│   └── file1
├── dir3
├── file1
├── file2
└── file3
3 directories, 4 files
┌─ daniel at hailey in ~/example
└─╼ print -- *(/)
dir1 dir2 dir3
┌─ daniel at hailey in ~/example
└─╼ print -- *(/F) # expand to only Full directories...
┌─ daniel at hailey in ~/example
└─╼ dirs=(*(/))
┌─ daniel at hailey in ~/example
└─╼ everything=(*)
┌─ daniel at hailey in ~/example
└─╼ print ${everything:|dirs}
file1 file2 file3
└─╼ print *(.)      # or just match plain files
file1 file2 file3

Another nice one is being able to recursively glob for files.

┌─ daniel at hailey in ~/example
└─╼ tree
├── dir1
│   ├── dir1
│   │   └── file3
│   ├── dir2
│   │   ├── dir1
│   │   ├── dir2
│   │   └── dir3
│   │       ├── file1
│   │       └── file2
│   └── dir3
├── dir2
├── dir3
├── file1
├── file2
└── file3

9 directories, 6 files
─ daniel at hailey in ~/example
└─╼ for file in **/*(.); do mv $file{,.sh}; done
┌─ daniel at hailey in ~/example
└─╼ tree
├── dir1
│   ├── dir1
│   │   └──
│   ├── dir2
│   │   ├── dir1
│   │   ├── dir2
│   │   └── dir3
│   │       ├──
│   │       └──
│   └── dir3
├── dir2
├── dir3

9 directories, 6 files

and maybe you want to remove suffixes from each file that is found in the glob

┌─ daniel at hailey in ~/example
└─╼ print -- **/file*.sh(:r)
dir1/dir1/file3 dir1/dir2/dir3/file1 dir1/dir2/dir3/file2 file1 file2 file3

or maybe you only want to show the files, with the removed path, like using basename.

┌─ daniel at hailey in ~/example
└─╼ rename .sh .zip *.sh
┌─ daniel at hailey in ~/example
└─╼ ls
dir1  dir2  dir3
┌─ daniel at hailey in ~/example
└─╼ tree
├── dir1
│   ├── dir1
│   │   └──
│   ├── dir2
│   │   ├── dir1
│   │   ├── dir2
│   │   └── dir3
│   │       ├──
│   │       └──
│   └── dir3
├── dir2
├── dir3
┌─ daniel at hailey in ~/example
└─╼ print -- **/*(.:t)
┌─ daniel at hailey in ~/example
└─╼ print -- **/*(.:h)      # or head
. . . dir1/dir1 dir1/dir2/dir3 dir1/dir2/dir3

The :h modifier is great for moving into directories with files you just opened or referenced on the commandline. (!$ works just like in bash, it grabs the last argument of the previous line)

┌─ daniel at hailey in ~/example
└─╼ ls dir1/dir2/dir3/ -l
-rw-r--r-- 1 daniel daniel 0 Sep  8 14:56 dir1/dir2/dir3/
┌─ daniel at hailey in ~/example
└─╼ cd !:1:h        # index 1 of the previous line array, grab the head
cd dir1/dir2/dir3
┌─ daniel at hailey in ~/example/dir1/dir2/dir3
└─╼ pwd

You can do the above with !:1 in bash, but you would need to put it in a subshell and run it through dirname first... cd $(dirname !:1)

And there are a ton more things you can do with this that I haven't even covered.


Alaises in zsh are for the most part similar to bash. You have your regular command replacement, but you also have extra stuff like global aliases.

┌─ daniel at hailey in ~
└─╼ alias -g AWK="|awk"
┌─ daniel at hailey in ~
└─╼ ip a AWK '/^\w/'
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000

This comes in handy from time to time when you want to do something like

alias -g NULL="2>&1 >/dev/null"

and redirect everything to dev null just by appending the word NULL to the end of the line.

Commands and fpath!

Anyone that has ever done bash scripting before has to have run into doing this.

if ! which git 2>&1 >/dev/null; then
    yum install -y git

In zsh, you get an array with all of the commands in it.

┌─ daniel at hailey in ~
└─╼ echo ${commands[pacman]}
f [[ -n ${commands[pacman]} ]]; then echo yes; else echo no; fi

You also have a new variable $path that is an array for your $PATH variable, so that you can manage it using +=.

┌─ daniel at hailey in ~
└─╼ echo $path[@]
/home/daniel/bin /usr/local/sbin /usr/local/bin /usr/bin

And a new variable called $fpath which is the location of all the zsh function stuff. The first thing that is really in the function path is your completion files, and it defines the inheritance.

┌─ daniel at hailey in ~
└─╼ print -l -- ${^fpath}/**/_pacman(N)

You can use this to maintain your own completion files outside of the system locations, but have them automatically loaded. To enable completion, you have to load the compinit module in zsh, so you don't have to source stuff like you do in bash. That is another one of the big differences.

autoload -U is used to load files as functions. I have this function here that I use to manage dtach sessions, since the only thing I used to use of tmux or screen was the dtach feature and I use a tiling window manager, and tach will put the socket files in my $XDG_RUNTIME_DIR. Then it is just loaded as a function as if it was defined like tach() {...;} in a zsh script.

┌─ daniel at hailey in ~
└─╼ which tach
tach () {
        # undefined
            builtin autoload -X

To load and enable completion, you do basically the same thing, load the compinit function, and run it.

autoload -U compinit && compinit

And zsh will look for compdef options at the top of files in your fpath to specify tab completions.

The other big things that are specified in fpath are the different prompts. First you have to autoload them, then you can choose from one of the default prompts, or preview them.

hailey% autoload compinit promptinit && compinit && promptinit
hailey% prompt
Usage: prompt <options>
    -c              Show currently selected theme and parameters
    -l              List currently available prompt themes
    -p [<themes>]   Preview given themes (defaults to all)
    -h [<theme>]    Display help (for given theme)
    -s <theme>      Set and save theme
    <theme>         Switch to new theme immediately (changes not saved)

Use prompt -h <theme> for help on specific themes.
hailey% prompt redhat
[daniel@hailey ~]$ prompt zefram
[2/5.0.8]daniel@hailey:~> prompt adam1
daniel@hailey ~ % prompt -l
Currently available prompt themes:
adam1 adam2 bart bigfade clint elite2 elite fade fire off oliver pws redhat suse walters zefram

This makes it easy for other people to share the same setup too, if that is what you want to do, and you can just set up different prompts and switch between them.

I have my 2 prompts stored in ~/.config/zsh/themes/prompt_<name>_setup and can switch back and forth between them. (I try to put stuff in ~/.config when possible, if you specify ZDOTDIR to ~/.config/zsh in your zprofile, or on linux in your .pam_environment, which could be moved as well, then you should be all set to go.)

Prompt docs here


This is where zsh gets fun. Instead of having a bunch of variables to configure in your environment, zsh uses zstyles to configure things. The best example I have is in my prompt.

zstyle ':vcs_info:*' enable bzr git hg svn                      # enable different source control plugins
zstyle ':vcs_info:*' check-for-changes true                     # check for changes when in a vcs directory
zstyle ':vcs_info:*' stagedstr '%F{g}●%f'                       # if we have an staged, uncommitted file, put a green dot
zstyle ':vcs_info:*' unstagedstr '%F{y}!%f'                     # if there are unstaged files, that are tracked, put a yellow !
zstyle ':vcs_info:*' formats 'on %F{m}%b%c%u%m%F{n}'            # display the branch and bold it in magento
zstyle ':vcs_info:*' actionformats "%b%c%u|%F{c}%a%f"           # display the branch/commit during an action (bisect, etc)
zstyle ':vcs_info:(sv[nk]|bzr):*' branchformat '%b|%F{c}%r%f'   # different branch information for svn and bzr
zstyle ':vcs_info:git*+set-message:*' hooks git-status          # I do not remember

Also for this to work, you have to include the vcs_info function as part of your zsh precmd.

# Example 1
autoload -Uz vcs_info
precmd(){ vcs_info }

# Example 2
autoload -Uz add-zsh-hook vcs_info
add-zsh-hook precmd prompt_gtmanfred_precmd

Then you just need to put your vcs_info variables into your prompt ${vcs_info_msg_0_}

More info about vcs_info can be found here

The other big one that I configure is zsh completion, with some documentation here with my configuration here


I would really strongly encourage anyone wanting to get started with zshell to dive into the docs. I really started diving in when I wanted to write some zshell completion for different stuff I was using. You will get more out of it if you spend time actually learning the different ins and outs of how other people are configuring plugins instead of just using stuff already out there.

If you do want to get a jumpstart, I really don't like oh-my-zsh, back when I would hangout in the #zsh irc channel on freenode, the majority of the problems that came in there were caused by something weird with omz. The better one, is the fork that was made off of it, and then basically made independent, prezto is really solid, if you need help beginning with plugins.

Happy Hacking :)!


Extra links

setting up bindkeys and zle

Daniel Wallace@Daniel Wallace

Transplanting Go packages for fun and profit

September 02, 2015 04:25 PM

crazy Gopher scientist

A while back I read coders at work, which is a book of interviews with some great computer scientists who earned their stripes, the questions just as thoughtful as the answers. For one thing, it re-ignited my interest in functional programming, for another I got interested in literate programming but most of all, it struck me how common of a recommendation it was to read other people’s code as a means to become a better programmer. (It also has a good section of Brad Fitzpatrick describing his dislike for programming languages, and dreaming about his ideal language. This must have been shortly before Go came about and he became a maintainer.)

I hadn’t been doing a good job reading/studying other code out of fear that inferior patterns/style would rub off on me. But I soon realized that was an irrational, perhaps slightly absurd excuse. So I made the decision to change. Contrary to my presumption I found that by reading code that looks bad you can challenge and re-evaluate your mindset and get out with a more nuanced understanding and awareness of the pros and cons of various approaches.

I also realized if code is proving too hard to get into or is of too low quality, you can switch to another code base with negligible effort and end up spending almost all of your time reading code that is worthwhile and has plenty of learnings to offer. There is a lot of high quality Go code, easy to find through sites like Github or Golang weekly, just follow your interests and pick a project to start reading.

It gets really interesting though once you find bodies of code that are not only a nice learning resource, but can be transplanted into your code with minimal work to solve a problem you’re having, but in a different context then the author of the code originally designed it for. Components often grow and mature in the context of an application without being promoted as reusable libraries, but you can often use them as if they were. I would like to share 2 such success cases below.

Nsq’s diskqueue code

I’ve always had an interest in code that manages the same binary data both in memory and on a block device. Think filesystems, databases, etc. There’s some interesting concerns like robustness in light of failures combined with optimizing for performance (infrequent syncs to disk, maintaining the hot subset of data in memory, etc), combined with optimizing for various access patterns, this can be a daunting topic to get into.

Luckily there’s a use case that I see all the time in my domain (telemetry systems) and that covers just enough of the problems to be interesting and fun, but not enough to be overwhelming. And that is: for each step in a monitoring data pipeline, you want to be able to buffer data if the endpoint goes down, in memory and to disk if the amount of data gets too much. Especially to disk if you’re also concerned with your software crashing or the machine power cycling.

This is such a common problem that applies to all metrics agents, relays, etc that I was longing for a library that just takes care of spooling data to disk for you without really affecting much of the rest of your software. All it needs to do is sequentially write pieces of data to disk and have a sequential reader catching up and read newer data as it finishes processing the older.

NSQ is a messaging platform from bitly, and it has diskqueue code that does exactly that. And it does so oh so elegantly. I had previously found a beautiful pattern in bitly’s go code that I blogged about and again I found a nice and elegant design that builds further on this pattern, with concurrent access to data protected via a single instance of a for loop running a select block which assures only one piece of code can make changes to data at the same time (see bottom of the file), not unlike ioloops in other languages. And method calls such as Put() provide a clean external interface, though their implementation simply hooks into the internal select loop that runs the code that does the bulk of the work. Genius.

func (d *diskQueue) Put(data []byte) error {
  // some details
  d.writeChan <- data
  return <-d.writeResponseChan

In addition the package came with extensive tests and benchmarks out of the box.

After finding and familiarizing myself with this diskqueue code about a year ago I had an easy time introducing disk spooling to Carbon-relay-ng, by transplanting the code into it. The only change I had to make was capitalizing the Diskqueue type to export it outside of the package. It has proven a great fit, enabling a critical feature through little work of transplanting mature, battle-tested code into a context that original authors probably never thought of.

Note also how the data unit here is the []byte, the queue does not deal with the higher level nsq.Message (!). The authors had the foresight of keeping this generic, enabling code reuse and rightfully shot down a PR of mine that had a side effect of making the queue aware of the Message type. In NSQ you’ll find thoughtful and deliberate api design and pretty sound code all around. Also, they went pretty far in detailing some lessons learned and providing concrete advice, a very interesting read, especially around managing goroutines & synchronizing their exits, and performance optimizations. At Raintank, we had a need for a messaging solution for metrics so we will so be rolling out NSQ as part of the raintank stack. This is an interesting case where my past experience with the NSQ code and ideas helped to adopt the full solution.

Bosun expression package

I’m a fan of the bosun alerting system which came out of Stack Exchange. It’s a full-featured alerting system that solves a few problems like no other tool I’ve seen does (see my linked post), and timeseries data storage aside, comes with basically everything built in to the one program. I’ve used it with success. However, for litmus I needed an alerting handler that integrated well into the Grafana backend. I needed the ability to do arbitrarily complex computations. Graphite’s api only takes you so far. We also needed (desired) reduction functions, boolean logic, etc. This is where bosun’s expression language is really strong. I found the expression package quite interesting, they basically built their own DSL for metrics processing. so it deals with expression parsing, constructing AST’s, executing them, dealing with types (potentially mixed types in the same expression), etc.

But bosun also has incident management, contacts, escalations, etc. Stuff that we either already had in place, or didn’t want to worry about just yet. So we could run bosun standalone and talk to it as a service via its API which I found too loosely coupled and risky, hook all its code into our binary at once - which seemed overkill - or the strategy I chose: gradually familiarize ourself and adopt pieces of Bosun on a case by case basis, making sure there’s a tight fit and without ever building up so much technical debt that it would become a pain to move away from the transplanted code if it becomes clear it’s not/no longer well suited. For the foreseeable future we only need one piece, the expression package. Potentially ultimately we’ll adopt the entire thing, but without the upfront commitment and investment.

So practically, our code now simply has one line where we create a bosun expression object from a string, and another where we ask bosun to execute the expression for us, which takes care of parsing the expression, querying for the data, evaluating and processing the results and distilling everything down into a final result. We get all the language features (reduction functions, boolean logic, nested expressions, …) for free.

This transplantation was again probably not something the bosun authors expected, but for us it was tremendously liberating. We got a lot of power for free. The only thing I had to do was spend some time reading code, and learning in the process. And I knew the code was well tested so we had zero issues using it.

Much akin to the NSQ example above, there was another reason the transplantation went so smoothly: the expression package is not tangled into other stuff. It just needs a string expression and a graphite instance. To be precise, any struct instance that satisfies the graphiteContext interface that is handily defined in the bosun code. While the bosun design aims to make its various clients (graphite, opentsdb, …) applicable for other projects, it also happens to let us do opposite: reuse some of its core code - the expression package - and pass in a custom graphite Context, such as our implementation which has extensive instrumentation. This lets us use the bosun expression package as a “black box” and still inject our own custom logic into the part that queries data from graphite. Of course, once we want to change the logic of anything else in the black box, we will need come up with something else, perhaps fork the package, but it doesn’t seem like we’ll need that any time soon.


If you want to become a better programmer I highly recommend you go read some code. There’s plenty of good code out there. Pick something that deals with a topic that is of interest to you and looks mature. You typically won’t know if code is good before you start reading but you’ll find out really fast, and you might be pleasantly surprised, as was I, several times. You will learn a bunch, possibly pretty fast. However, don’t go for the most advanced, complex code straight away. Pick projects and topics that are out of your comfort zone and do things that are new to you, but nothing too crazy. Once you truly grok those, proceed to other, possibly more advanced stuff.

Often you’ll read reusable libraries that are built to be reused, or you might find ways to transplant smaller portions of code into your own projects. Either way is a great way to tinker and learn, and solve real problems. Just make sure the code actually fits in so you don’t end up with the software version of Frankenstein’s monster. It is also helpful to have the authors available to chat if you need help or have issues understanding something, though they might be surprised if you’re using their code in a way they didn’t envision and might not be very inclined to provide support to what they consider internal implementation details. So that could be a hit or miss. Luckily the people behind both nsq and bosun were supportive of my endeavors but I also made sure to try to figure out things by myself before bothering them. Another reason why it’s good to pick mature, documented projects.

Gopher frankenstein

Part of the original meaning of hacking, extended into open source, is a mindset and practice of seeing how others solve a problem, discussion and building on top of it. We’ve gotten used to - and fairly good at - doing this on a project and library level but forgot about it on the level of code, code patterns and ideas. I want to see these practices come back to life.

We also apply this at Raintank: not only are we trying to build the best open source monitoring platform by reusing (and often contributing to) existing open source tools and working with different communities, we realize it’s vital to work on a more granular level, get to know the people and practice cross-pollination of ideas and code.

Next stuff I want to read and possibly implement or transplant parts of: dgryski/go-trigram, armon/go-radix, especially as used in the dgryski/carbonmem server to search through Graphite metrics. Other fun stuff by dgryski: an implementation of the ARC caching algorithm and bloom filters. (you might want to get used to reading Wikipedia pages also). And mreiferson/wal, a write ahead log by one of the nsqd authors, which looks like it’ll become the successor of the beloved diskqueue code.

Go forth and transplant!

Also posted on the Raintank blog

Dieter Plaetinck

openssh-7.0p1 deprecates ssh-dss keys

August 14, 2015 05:10 AM

In light of recently discovered vulnerabilities, the new openssh-7.0p1 release deprecates keys of ssh-dss type, also known as DSA keys. See the upstream announcement for details.

Before updating and restarting sshd on a remote host, make sure you do not rely on such keys for connecting to it. To enumerate DSA keys granting access to a given account, use:

    grep ssh-dss ~/.ssh/authorized_keys

If you have any, ensure you have alternative means of logging in, such as key pairs of a different type, or password authentication.

Finally, host keys of ssh-dss type being deprecated too, you might have to confirm a new fingerprint (for a host key of a different type) when connecting to a freshly updated server.

Gaetan Bisson@Official News

Firefox vulnerability

August 07, 2015 07:09 PM

Hi everyone,

A serious vulnerability has been found in firefox yesterday (in the PDFjs module).
You will find more details at:
- … sa2015-78/
- … -the-wild/

The second link mentions that an exploit already existed before mozilla was informed about this problem, and this exploit was found to read the ssh files among others.
Archlinux repo already contains a fixed version of firefox (39.0.3-1).

Note for moderators: I am not sure wether the Archlinux forum is the right place for this topic, many security blogs and website have already published about this, but I thought quite a lot of us might be interested by this issue. Please feel free to close the topic if you wish.

pixelou@Forum Announcements

Hardening Android’s Bionic libc

July 28, 2015 12:53 PM

Good post by Daniel Micay on hardening Android’s Bionic libc:

Allan@Allan McRae

Climbing in Gimmeldingen (2)

July 20, 2015 06:49 AM

It wasn't the perfect weather for climbing yesterday. It was totally a contrary weather with sun, cloudy and in the morning hours some rain. Nevertheless we tried it in the evening and hoped that the rock will be dry enough for climbing. Luckily the rock was dry 1-2 meters above the ground.

We started with "Allgäu" (UIAA: 6-, French: 5b, USA: 5.8) to warm up. Last time we climbed it as last route, yesterday it was our first one. Really a nice route, but because of the rain it was a littble bit sloppy in the start section. On the left photo it's the route which starts on the left plate and goes up the corner in the wall. 

Next route was "Sonnenaufgang (Sunrise)" (UIAA: 6+, French: 6a, USA: 5.10a), which is a little crack in the wall. You can see it in the left photo on the right side where the rope starts and if you follow up the rope you see the route. Very nice route, but it was a little bit too hard in the beginning for a 6+ (at least this was my impression). Therefor it will be easier in the last section, maybe that's the compensation for the hard start.

The third one for this evening was "Tugarez" (UIAA: 6-, French: 5b, USA: 5.8). The start and the last move is harder, the rest is more or less easy. It's a good route, but nothing special. You can see it in the right photo. Just follow the rope there and you see the route.

The last one has no name in our climbing guide, but it was rated with UIAA 5 (French: 4c, USA: 5.6), personally I would say it's a 5+ (French: 5a, USA: 5.7). Perfect route to end a climbing session in the evening, a short route with a relative easy difficulty.

It's not me on the photos, it's my climbing partner Christoph, just to mention it.
Daniel Isenmann ( Isenmann

Climbing in Gausbach

July 16, 2015 06:00 AM

Yesterday I was climbing in Gausbach (Murg valley, Northern Black Forest). Gausbach has a nice climbing area with 56 routes and a height of 40 metres. The area is directly at an old and nearly unused road, no long walking distance from the parking place or train station.

I helped out my girlfriend who was climbing with her school class. I had to prepare the routes for top roping, so there was a wide range of difficulty.

So, we started with the easiest route there because the children weren't climbing on a rock before, which was "Komm mit Ski (Come with ski)" (UIAA: 3+, French: 3, USA: 5.3). A really really easy one, but perfect for those who weren't climbing on a rock before. The next ones were a little bit harder, "Signo di oro" (UIAA: 5-/5, French: 4c, USA: 5.6), "Toprope ahoi" (UIAA: 5-, French: 4c, USA: 5.6) and "Via bambini" (UIAA: 5, French: 4c, USA: 5.6). For some children we also prepared two harder routes: "Stairway to heaven" (UIAA: 5+, French: 5a, USA: 5.7) which is a very nice and long route (about 28 meters),  "Komm mit Ski (links) (Come with ski (left alternative))" (UIAA: 6/6+, French: 5c/6a, USA: 5.9/5.10a) which is good route of slab climbing.

After the children are gone and the official part was over, I tried "Veni vidi vici" (UIAA: 7-, French: 6a+, USA: 5.10b). Sadly I had to abort it after 3/4 of the route. I think it was the key part of the route, or maybe not (I don't know it), but it was just a little bit too hard for me this day. I definitely have to go back there and try it again, I'm sure that this route isn't that complicated at this point.

A small hint at the end, climb in the morning hours, after midday you are climbing without any shadow and it will be definitely hot there.
Daniel Isenmann ( Isenmann

Bug squashing evening for Arch Linux

July 13, 2015 09:41 PM

This evening I started my personal bug squashing day for Arch Linux. After a very long delay and no contribution the last several weeks for Arch Linux, I decided to start again and what will be better than fixing long outstanding bugs.

So, I decided to fix the following bugs:

  • FS#23933 - [dhcp] Separate v4 and v6 configuration files
  • FS#43405 - [dhcp] unable to start dhcpd6
  • FS#33927 - [monodevelop] Add xsp as an optional dependency
  • FS#39779 - [monodevelop] Razor solution wizard fails  
  • FS#28496 - [mod_mono] .conf located elsewhere  
All bugs are now fixed, at least I hope so. The dhcp package has now seperate configuration files for IPv4 and IPv6, also the systemd service files are referring to the correct configuration file.

MonoDevelop has now xsp as optional dependency and the Razor solution wizards works now and all necessary files are on its place.

mod_mono's configuration file is now located in the correct directory and must be changed in the httpd.conf file if you refer to the old directory.

Additionally I have updated the Mono package and MonoDevelop package to the latest stable release.

I hope that I will find another evening like today to do my next personal bug squashing day.

The next bug squashing day will come... ;-)

Daniel Isenmann ( Isenmann

Anime Guide 2014

July 12, 2015 12:54 PM

After many repeated requests (which was kind of surprising), here is my overview of the 2014 anime I watched. I am told that I was too negative in previous lists and I want to avoid that this time… probably not going to happen!

This year had a LOT of sequels. I prefer to watch something new unless the original was outstanding, so I ignored some of these (how can Fate/Stay have so many spin-offs?). And for some reason, many ~22 episode series were split in half and had a break in the middle. I am counting those as one series. Speaking of both sequels and split anime…

Anime of the Year

Mushi-Shi: Next Passage
(TV Series, 11+10 episodes + special)

This is how you do a sequel! And it came out of nowhere about 10 years after the original. The stunning visuals remain and are improved on as we expect in the HD era. The atmospheric soundtrack sits very well with the stories, with the intro “Shiver” by Lucy Rose fitting in perfectly. As with the original series, the story is largely episodic but there is a subtle underlying flow brings it together. Not only do I consider this the anime of 2014, it would be a strong contender for anime of the year in any year. Look out for the movie in 2015.


Kill la Kill
(TV Series, 24 episodes)

Half naked girls fighting while being manipulated by clothing… This is made by the same people as “Gurren Lagann”, and it is just as insane, just with less robots and more fan-service. And there is a lot of fan-service, to the point where it would have been awkward if I watched this when taking public transport to work! That also caused some controversy around the internet. I watched this for the craziness and enjoyed it for that – the fan-service seemed to fit in to that insanity. This is not an anime that needs watched with extensive thought. (The uproar about the nudity makes it awkward to justify enjoying this series. But I am sure that if I only wanted to see naked people, I could just find some somewhere in the dark recesses of the internet…)

Ping Pong the Animation
(TV Series, 11 episodes)

I only watched this after it being repeatedly recommended to me, because I do not enjoy sport anime. It turns out, Ping Pong is not really a sport anime… despite being about ping pong. This is a show about the growth of the main characters, their friendships and rivalries develop while revolving around the game. Saying that, I saw some ping pong on TV the other day and I was now an expert on it, so there was some sport there. I found the distinctive art style aided in skimming over the actual games without losing the main focus of the show.

Samurai Flamenco
(TV Series, 22 episodes)

This is a series that people I have talked to either love or hate. I think it is because the series starts out reasonably serious and realistic before going off on wild tangents. I am still not sure if the script was planned, or if the writers genuinely went crazy (although I may lean towards planned after completing the series). But you can not go past this if you want a fun superhero story that has the ability to go from low-tech and gritty to the other extreme. I watch anime for fun, and not to search for some higher meaning, so this was a close contender for anime of the year!

Silver Spoon 2
(TV Series, 11 episodes)

Much like my comments on the first series, there is nothing individually outstanding about the sequel. It is just consistently good. The second season does well in moving on from the issues dealt with during the first (far less “this is where my food comes from” jokes) and focusing on new challenges. You know, exactly what a good sequel should do…


Akame ga Kill!
(TV Series, 24 episodes)

This was one hyped anime for some reason. But there was nothing outstanding in it (some would call it “average” even…). The animation and music during the action sequences was well done, and that makes this fun to watch. However, it goes from dramatic battle to light-hearted romp with the flick of a switch, which ruins both.

Ghost in the Shell: Arise
(OVA, 4 episodes)

Sort of a prequel to Stand Alone Complex, but the backstories have changed… so not a prequel. But there is enough familiarity with the characters that it feels like a prequel (is any of this making sense?). The result is something neither better or worse than the original series.

Psycho Pass 2
(TV Series, 11 episodes)

Not how to do a sequel! I really enjoyed the darkness and mystery of the first series, but that had been mostly resolved and that left not a lot to add. The sequel proceeded to add that not a lot (or even just blatantly rip-off ideas from the first series), resulting in something that was both mildly entertaining and unnecessary at the same time.

Space Brothers
(TV Series, 99 episodes)

I usually check how many episodes a series has before I start watching it – I forgot for this one! Saying that, I watched each episode of this immediately as it broadcast. I would say this is highly recommended apart from the episodes being a bit slow at times and a bit predictable. Also, there was a trend to a substantial recap at the start of episodes. If they fixed that by reducing the series by ~20 episodes, this would be a great(er) anime.

Space Dandy
(TV Series, 13+13 episodes)

After picking a cow for Silver Spoon, I knew what picture I needed here too! I had great hope for this anime – it is by the makers of some of my favourites. However… I never got caught up in this series. Everything about it is simple, right down to the “comedy” which has none of the unexpected surprise needed to be funny. For example the “Boobies” restaurant, which is Hooters in space – oh, it is also shaped like boobs. That is the joke… There is also nothing riding on the outcome of an episode (spoiler follows). The entire cast die at the end of the first episode and are back as if nothing happened in episode two. So who cares what happens?

Tiger and Bunny the Movie: The Rising
(Movie, 100 minutes)

The second part of a two movie series that may or may not be a recap of the TV series (I can not be bothered checking…). If they were not a recap, it added very little beyond what the TV series did in terms of actual development of the storyline. Still enjoyable.

Tokyo Ghoul
(TV Series, 12 episodes)

I usually like short series (good risk/reward investment ratio), but I think being too short was the failure here. There was character development, building a world where ghouls have their own society, and then someone asked “how many episodes do we have left?”. Queue the final three episodes being packed with action and not a lot else. I realize there is a second series, but why would you watch it after what they did to the first?


A Certain Magical Index The Movie: The Miracle of Endymion
(Movie, 90 minutes)

This was the final straw in me watching “A Certain …” series. I really liked the first Scientific Railgun series (I think everyday superpowers is still a great concept), but this took much of what I did not enjoy about that series and added it to Magical Index (which was never as good), and come up with something that I really did not enjoy. Maybe this will serve as a reminder to me and I will not watch the next one (maybe).

Hozuki no Reitetsu
(TV Series, 13 episodes)

A slice-of-life anime set in Japanese Hell. It gets some points for being fairly original… However, I found the jokes fell flat more often than not, and that can leave a big gap when there is nothing else going on. With two stories per show (that is 10 minutes a plot line), nothing really caught my interest and I took a long time to finish watching this.

Log Horizon
(TV Series, 25 episodes)

Another “stuck in a game” anime, with the twist that there seems to be the lack of motivation to leave. Too much filler (especially near the end), forgettable action scenes, and a storyline that resembled a poor version of Spice and Wolf. The most entertained I was in this series was when other people started mocking the main character for always touching his glasses…

(TV Series, 12 episodes)

I think this series was not particular bad in itself, but I think it could have been good and failed. I am still not sure what it was missing, but I find the whole thing completely forgettable (which may be why this “review” is so vague).

Terror in Resonance (Zankyou no Terror)
(TV Series, 11 episodes)

The initial build-up is OK – good even – but everything becomes overly contrived in the second half. Overall, this is probably a reasonable watch if you are not hoping for greatness, but I rate those series that screw up their potential harshly. And I just noticed the director was Shinichiro Watanabe, who has been involved in many of my favourite series. He did not have a great year according to this blog post!

Allan@Allan McRae

Rolling Release Pizza

July 12, 2015 04:19 AM

/u/XSSpants on reddit:

There’s a lot to be said for rolling release pizza. (without going hardcore “pizza fountain” like Arch)

Allan@Allan McRae

Climbing in Gimmeldingen

July 10, 2015 07:49 AM

Starting from today, I will write down some notes and information about my sport activities. This is more or less for my personal recording (no, I will not use a smartphone app for this) than for information. Nevertheless, there will also some public information which anyone can use from these posts. Maybe you will find new climbing places or hiking routes or whatever. Maybe you just enjoy to read my posts.

Yesterday I was climbing at the quarry of Gimmeldingen (near Neustadt an der Weinstraße, Rhineland-Palatinate) after work. The quarry is a sandstone climbing area with about 70 routes and a height of 20 metres.

In total we "just" climb three routes there, but all three routes are great for itself. We started with "Mitten im Winter (In the middle of winter)" which has a difficulty of 6 in the UIAA scale (in French it's a 5c and in USA (YDS) 5.9). The next route was "Ponyexpress" (UIAA: 6+, French: 6a, USA: 5.10a) which is a really nice crack and corner in the wall, you can see route in the left photo. I can really recommend this route, I have climbed it the first time yesterday, but it has become one of my favourites there already. The last route was "Allgäu" (UIAA: 6-, French: 5b, USA: 5.8). It's a great route to end a climbing session. It has one hard spot and the rest is more or less easy.

This climbing session was kind of after work climbing to end the day with some sport activity and to relax.

Daniel Isenmann ( Isenmann

July 2015 TalkingArch

July 08, 2015 06:33 PM

The TalkingArch team is happy to announce the availability of the TalkingArch release for July 2015. This release includes Linux Kernel 4.0.7 and all the latest base and rescue packages. Additionally, there is a new release info section on the Download pabe,, which shows the download size, included Linux kernel and the current release date, similar to the Arch Linux download page, which will be updated with each release.

The current TalkingArch iso appears to be the last version smaller than 700MB, which is the capacity of a writable CD. The general concensus so far is to continue building each subsequent release normally with no changes. This will mean that anyone burning to a CD will either need to burn the image to a DVD instead, or to write it to a USB stick, SD card or other similar rewritable media with a capacity of 1GB or more. If there is any strong objection to this release policy, either mention it in the comments, hop into the #talkingarch IRC channel on or send an e-mail to TalkingArch support.


Focusing on open source monitoring. Joining raintank.

July 03, 2015 04:22 PM

Goodbye Vimeo

It's never been as hard saying goodbye to the people and the work environment as it is now.
Vimeo was created by dedicated film creators and enthusiasts, just over 10 years ago, and today it still shows. From the quirky, playful office culture, the staff created short films, to the tremendous curation effort and staff picks including monthly staff screenings where we get to see the best of the best videos on the Internet each month, to the dedication towards building the best platform and community on the web to enjoy videos and the uncompromising commitment to supporting movie creators and working in their best interest.
Engineering wise, there has been plenty of opportunity to make an impact and learn.
Nonetheless, I have to leave and I'll explain why. First I want to mention a few more things.

vimeo goodbye drink

In Belgium I used to hitchhike to and from work so that each day brought me opportunities to have conversations with a diverse, fantastic assortment of people. I still fondly remember some of those memories. (and it was also usually faster than taking the bus!)
Here in NYC this isn't really feasible, so I tried the next best thing. A mission to have lunch with every single person in the company, starting with those I don't typically interact with. I managed to have lunch with 95 people, get to know them a bit, find some gems of personalities and anecdotes, and have conversations on a tremendous variety of subjects, some light-hearted, some deep and profound. It was fun and I hope to be able to keep doing such social experiments in my new environment.

Vimeo is also part of my life in an unusually personal way. When I came to New York (my first ever visit to the US) in 2011 to interview, I also met a pretty fantastic woman in a random bar in Williamsburg. We ended up traveling together in Europe, I decided to move the US and we moved in together. I've had the pleasure of being submerged in both American and Greek culture for the last few years, but the best part is that today we are engaged and I feel like the luckiest guy in the world. While I've tried to keep work and personal life somewhat separate, Vimeo has made an undeniable ever lasting impact on my life that I'm very grateful for.

At Vimeo I found an area where a bunch of my interests converge: operational best practices, high performance systems, number crunching, statistics and open source software. Specifically, timeseries metrics processing in the context of monitoring. While I have enjoyed my opportunity to make contributions in this space to help our teams and other companies who end up using my tools, I want to move out of the cost center of the company, I want to be in the department that creates the value. If I want to focus on open source monitoring, I should align my incentives with those of my employer. Both for my and their sake. I want to make more profound contributions to the space. The time has come for me to join a company for which the main focus is making open source monitoring better.

Hello raintank!

Over the past two years or so I've talked to many people in the industry about monitoring, many of them trying to bring me into their team. I never found a perfect fit but as we transitioned from 2014 into 2015, the stars seemingly aligned for me. Here's why I'm very excited to join the raintank crew:
  • I'm a strong believer in open source. I believe fundamental infrastructure tooling should be modifiable and in your control. It's partially a philosophical argument, but also what I believe will separate long lasting business from short term successes.
    SaaS is great, but not if it's built to lock you in. In this day and age, I think you should "lock in" your customers by providing a great experience they can't say no to, not by having them build technical debt as they integrate into your stack.
    That said, integrating with open source also incurs technical debt, and some closed source service providers are so good that the debt is worth it. But I don't believe this lasts long term, especially given the required pricing models. I think you can entice customers more by lowering the debt they build up (i.e. use standardized protocols, tooling and making it easy for them to leave) as they adopt your service.
    raintank's commitment to open source is not a gimmick but stems from a fundamental conviction that 100% open source is the right thing to do. That by providing freedom, you get loyalty. I think they came up with a good business model that combines the benefits of SaaS with those of open source. At least it's a bet I want to take. It's not an easy feat but I think we got a good formula. (basically openSAAS, the stack is open source, integrates in the wider ecosystem, you can run it yourself, you own your data, you can use our SAAS, or create a mixed setup.)

  • Something I value a lot in coworkers, besides "technical" abilities is emotional intelligence and professional maturity. Part of me thinks it's a vastly underestimated quality in general, although perhaps I just happen to find it more important than average. Either way, the new team seems solid on both fronts. I'm mostly impressed by the founder/CEO Raj Dutt who has shown a very personal and graceful approach. Strong high-stakes relationships with people in general, and coworkers in particular are a very worthy pursuit and two months in I can still vouch for the level of alignment and maturity that I haven't experienced before.

    Yours truly and Raj Dutt

  • I (and my fiancee) want to see the world. The common conception that significant leisure travel can't be combined with hard, or even normal amounts of work seems so backwards to me. Today more than ever, we have globalization, we are developing a sharing economy (airbnb, lyft, ...). There's no reason we can't do great work while enjoying our personal time to the fullest. Working remotely, and/or with remote colleagues requires more discipline but I simply only want to work with disciplined people. Working on a fixed schedule, in a fixed office location is needlessly arcane, constrained and inefficient. It puts a damper on life. Almost all software companies I know see open office plans as a great thing, but it's usually an attempt at compensating for employee's poor communication skills, forcing them to talk to the detriment of people who need focus and get distracted. Isolation of workers does not preclude healthy communication and collaboration. I get a lot more done in isolation, with sync-ups and face-to-face when appropriate, especially if it's on my schedule. Working remote is a great way to facilitate this and I'm happy that raintank not only agrees with the idea, but actually encourages travel, and encourages me to follow whatever time allocation works best for me. I work hard but we travel where ever we want, we'll have plenty of time to spend with our families in Belgium and Cyprus. And yet I don't think I ever worked this closely with anyone.

  • office mount shasta

  • I've always wanted to be in a company from the start and experience the "true" start-up feel. I have a lot of opinions on organization, culture, and product and so I'm glad to have the opportunity to make that kind of impact as well.

OK, so what am I really up to?

Grafana is pretty much the leading open source metrics dashboard right now. So it only makes sense that raintank is a heavy Grafana user and contributor. My work, logically, revolves around codifying some of the experience and ideas I have, and making them accessible through the polished interface that is Grafana, which now also has a full time UX designer working on it. Since according to the Grafana user survey alerting is the most sorely missed non-feature of Grafana, we are working hard on rectifying this and it is my full-time focus. If you've followed my blog you know I have some thoughts on where the sweet spot lies in clever alerting. In short, take the claims of anomaly detection via machine learning with a big grain of salt, and instead, focus on enabling operators to express complex logic simply, quickly, and in an agile way. My latest favorite project, bosun exemplifies this approach (highly recommend giving this a close look).

The way I'm thinking of it now, the priorities (and sequence of focus) for alerting within Grafana will probably be something like this:

  • cover the low hanging fruit: simple threshold checks with email notifications actually go a long way
  • gradually provide more power and sophistication (reduction functions, boolean logic, etc), while keeping things in a nice UI
  • provide full-on integration with advanced alerting systems such as Bosun. Iterative workflow, Signal/noise analysis, etc
There's a lot of thought work, UX and implementation details around this topic, I've created a github ticket to kick off a discussion and am curious to hear your thoughts. Finally, if any of this sounds interesting to you, you can sign up to the grafana newsletter or the raintank newsletter which will get you info on the open source platform as well as the SaaS product. Both are fairly low volume.

office sausolito It may look like I'm not doing much from my temporary Mill Valley office, but trust me, cool stuff is coming!
Dieter Plaetinck

Moved blog to hugo, fastly and comma

July 02, 2015 11:35 PM

  • I noticed what a disservice I was doing my readers when I started monitoring my site using litmus. A dynamic website in python on a cheap linode… What do you expect? So I now serve through fastly and use a static site generator.

  • pyblosxom was decent while it lasted. It can generate sites statically, but the project never got a lot of traction and is slowly fading out. There were a bit too many moving parts, so …

  • I now use the hugo static site generator, which is powerful, quite complete and gaining momentum. Fast and simple to use.

  • Should also keep an eye on the caddy webserver since it has some nice things such as git integration which should work well with hugo.

  • Trying to get disqus going was frustrating. Self hosted options like talkatv and isso were too complex, and kaiju is just not there yet and also pretty complex. I wrote comma which is a simple comment server in Go. Everything I need in 100 lines of Go and 50 lines of javascript! Let me know if you see anything funky.

  • migrated all content.

Dieter Plaetinck

TalkingArch June 2015

June 18, 2015 01:55 AM

It’s been a rocky road, but it’s actually been out for some time. It’s the TalkingArch iso for June 2015. The delay in the posting of the release has been due to a server OS upgrade that took some rather unexpected turns, resulting in breakage of the GNU Social node that hosts the social media timeline for TalkingArch. Although the node is not yet fixed, it is certainly past time to go ahead and post the announcement to the blog, which is still working. This month’s iso includes all the latest software to install a fully working Arch Linux system, including Linux kernel 4.0.4. Get your copy from the usual place. Have fun!


The Case of GCC-5.1 and the Two C++ ABIs

June 08, 2015 12:58 AM

Recently, Arch Linux updated to gcc-5.1. This brought a lot of new features, but the one that I am focusing on today is the new C++ ABI that appears when the compiler is built using default option.

Supporting the C++11 standard required incompatible changes to libstdc++ ABI. Instead of bumping the library soname (I still do not understand why that was not done…), the GCC developers decided to have a dual ABI. This is achieved using C++11 inlined namespaces which will give a different mangled name allowing types that differ across the two ABIs to coexist. When that is insufficient, a new abi_tag attribute is used which adds [abi:cxx11] to the mangled name.

With the initial upload of gcc-5.1 to the Arch repos, we configured GCC to use the old ABI by default. Now the initial slew of bugs have been dealt with (or mostly dealt with…), we want to switch to the new ABI. Because this was going to be a much larger rebuild than usual, one of the Arch developers (Evangelos Foutras) developed a server that automatically orders the rebuilds, and provides the next rebuild when a client requests it (this may be the future of rebuild automation in Arch).

This discovered an issue when building software using the new C++ ABI with clang, which builds against the new ABI (as instructed in the GCC header file), but does not know about the abi_tag attribute. This results in problems such as (for example) any function in a library with a std::string return type will be mangled with a [abi:cxx11] ABI tag. Clang does not handle these ABI tags, so will not add the tag to the mangled name and then linking will fail.

This issue was pointed out on a GCC mailing list in April, and a bug was filed independently for LLVM/clang. Until it is fixed, Arch Linux will not be able to switch to the new ABI. Note, this also has consequences for all older versions of GCC wanting to compile C++ software against system libraries…

Allan@Allan McRae

Cleaning an Arch Linux installation

June 05, 2015 10:00 PM

We are in springtime and usually in springtime you make some cleaning. Today I decided to clean my old desktop that is running the same Arch Linux installation since 2007.

Warning: this whole task requires a bit of knowledge about what a package does and why it is installed on your system. You could break something, but IMHO in the worst case you will re-install it later ;-)

So, I started by listing any package I didn’t install:

  • $ pacman -Sqg base base-devel | sort -u > /tmp/essentials
  • $ pacman -Qqe | sort > /tmp/explicit
  • $ comm -13 /tmp/essentials /tmp/explicit > /tmp/unknown

Read /tmp/unknown and mark as “installed as dependence” any package you didn’t install or simply you don’t know. This last step requires a knowledge of what a package do; pacman -Qi and pacman -Ql are your friend here! To mark a package as dependence run: # pacman -D --asdeps <pkg>.

Now, the harmful part! Run # pacman -Rscn $(pacman -Qqtd). It will remove any package installed as dependence plus the packages needed by it because installed as dependence that are no more required without it. Do not rush and read the packages you are going to remove!

In my case, even after all those years I only got 14 packages…I’m too careful. I always mark as dependence the package I don’t use anymore and I periodically check the output of pacman -Qqtd.

Now, you could also remove the orphaned files on your system by using lostfiles (download from AUR). In fact, # lostfiles > /tmp/lostfiles will produce a list of files that are owned by no package, but it also excludes something from that list which makes things easier for your brain.

Update: an user suggested to also check the output of pacman -Qqttd (note the extra ‘t’); this one also lists optional dependencies. Thanks!

Happy cleaning!

Andrea Scarpino

Planet Arch Linux

Planet Arch Linux is a window into the world, work and lives of Arch Linux hackers and developers.

Last updated on June 24, 2016 11:00 PM. All times are normalized to UTC time.