This post is mostly a note to myself, as I can never remember the exact steps. But I think it would be useful for other people.

As much as I dislike systemd, it has to be said that their implementation of a container manager is neat. Simple, lightweight, and well integrated across the systemd tools.

As usual, the Arch wiki has excellent documentation on the matter. I will write another post on how to set up the host, but not today (as I can't remember the details!). Suffice to say that a pre-requisite is to have the systemd-container package installed and configured.

Then, to spin a new instance that will have its own init system and will be started at boot, you only need the following commands:

# Sample values.
DISTRO=stretch
MACH=my_container
# Install base system.
$ debootstrap $DISTRO /var/lib/machines/$MACH
# You need dbus for proper integration from the host.
$ chroot /var/lib/machines/$MACH apt install dbus
# Now it is the time to do any other customisation directly on the filesystem (see below).
# Place your customisations in the override file.
$ mkdir -p /etc/systemd/system/systemd-nspawn@$MACH.service.d/
$ echo $CUSTOM > /etc/systemd/system/systemd-nspawn@$MACH.service.d/override.conf
# Reload systemd, and enable and start the container.
$ systemctl daemon-reload 
$ systemctl enable systemd-nspawn@$MACH
$ systemctl start systemd-nspawn@$MACH
# Profit!
$ machinectl shell $MACH /bin/bash

The override file is important if you -like me- want to use the containers in the same network namespace as the host.

The default configuration for systemd-nspawn uses the following arguments:

$ grep ExecStart /lib/systemd/system/systemd-nspawn@.service
ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-veth -U --settings=override --machine=%i

Note the --network-veth parameter. This creates a virtual ethernet connection between host and guest, which you must then route and/or masquerade properly. It is a sane setting for isolation, but in my case, I usually use these containers as glorified chroots, so I want to share the network. So, in the override file, I clear the ExecStart setting, and then put a modified command line:

$ cat /etc/systemd/system/systemd-nspawn@$MACH.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest -U --settings=override --machine=%i

If you are using this configuration, you should also copy network-related files into the container, before the first start (see below why):

$ cp /etc/hosts /etc/resolv.conf /var/lib/machines/$MACH/etc/

It is important to know that during the first execution with the -U flag (which is the default), systemd-nspawn will chown all the files in the container to use private user IDs. So, you should never modify these files directly, or log in with the chroot command, as you will make them inaccessible for the processes inside the container. The libnss-mymachines package does a nice job of mapping these private users in the host environment.

MiniDebCamp Hamburg - Friday 18/5, Saturday 19/5

Friday and Saturday have been very productive days, I love events where there is time to hack!

I had more chats about contributors.d.o with Ganneff and Formorer, and if all goes according to plan, soon salsa will start streaming commit information to contributors and populate information about different teams: not only about normal packaging repos, but also about websites, tools, native packages, etc.

Note that the latter require special configuration, and the same goes if you want to have separate stats for your team (like for the Go team or the Perl team). So if you want to offer proper attribution to members of your team, please get in touch!

I spent loads of time working on Prometheus packages, and finally today (after almost a year) I uploaded a new version of prometheus-alertmanager to experimental. I decided to just drop all the web interface, as packaging all the Elm framework would take me months of work. If anybody feels like writing a basic HTML/JS interface, I would be happy to include it in the package!

While doing that, I found bugs in the CI pipeline for Go packages in Salsa. Solving these will hopefully make the automatic testing more reliable, as API breakage is sadly a big problem in the Go ecosystem.

I am loving the venue here. Apart from hosting some companies and associations, there is an art gallery which currently has a photo exhibition called Echo park; there were parties happening last night, and tonight apparently there will be more. This place is amazing!

MiniDebCamp Hamburg - Thursday 17/5

I missed my flight on Wednesday, and for a moment I thought I would have to cancel my attendance, but luckily I was able to buy a ticket for Thursday for a good price.

I arrived at the venue just in time for a "stand-up" meeting, where people introduced themselves and shared what are they working on / planning to work on. That gave me a great feeling, having an idea of what other people are doing, and gave me motivation to work on my projects.

The venue seems to be some kind of cooperative, with office space for different associations, there is also a small guest house (where I am sleeping), and a "cantina". The building seems very pretty, but is going through some renovations, so the scaffolding does not let you see it much. It also has a big outdoors area, which is always welcomed.

I had a good chat about mapping support in IkiWiki, so my rewrite of the OSM plugin might get some users even before it is completely merged!

I also worked for a while on Prometheus packages, I am hoping to finally get a new version of prometheus-alertmanager packaged soon.

I realised I still had some repos in my home directory in alioth, so I moved these away to salsa. On the same vein, I started discussions about migrating my data-collection scripts for contributors.d.o to salsa; this is quite important if we want to keep contributors being relevant and useful.

After a lovely car journey through the Alps yesterday, I had a good sleep and I am now in the airport waiting to fly back to Dublin.

I think most attendees will agree that the SnowCamp was a success; I was certainly sad to leave.. It always feels too short!

After my first report, I spent a few hours on fixing a long-standing bug in the KGB bot, which caused it to take several minutes to sync channels and start emitting notifications. I also used for the first time the salsa merge requests feature! The next release of the bot will include this patch and take just a few seconds to be up and running.

I also worked on another RC bug opened on a package I had fixed only the day before, due to another test failure: #891356: golang-google-cloud FTBFS: FAIL google.golang.org/cloud/spanner; which I have finished and uploaded a few minutes ago.

Finally, I had some more talks which can't be reported upon, and then the Camp was over

As Nicolas already reported, a bunch of Debian folk gathered in the North of Italy for a long weekend of work and socialisation.

Valhalla had the idea of taking the SunCamp concept and doing it in another location, and along with people from LIFO they made it happen. Thanks to all who worked on this!

I arrived late on Wednesday, after a very relaxed car journey from Lyon. Sadly, on Thursday I had to attend some unexpected personal issues, and it was not very productive for Debian work. Luckily, between Friday and today, I managed to get back in track.

I uploaded new versions of Prometheus-related package to stretch-backports, so they are in line with current versions in testing:

• prometheus-alertmanager, which also provides a fix for #891202: False owner/group for /var/lib/prometheus.
• python-prometheus-client, carrying some useful updates for users.

I fixed two RC bugs in important Go packages, both caused by the recent upload of Golang 1.10:

• #890927: golang-golang-x-tools: FTBFS and Debci failure with golang-1.10-go
• #890938: golang-google-cloud FTBFS: FAIL: TestAgainstJSONEncodingNoTags

I also had useful chats about continuous testing of Go package, and improvements to git-buildpackage to better support our workflow. I plan to try and write some code for it.

Finally, I had some long discussions about joining an important team in Debian, but I can't still report on that

Since about 15 years ago, I have been thinking of creating a geo-referenced wiki of pubs, with loads of structured data to help searching. I don't know if that would be useful for anybody else, but I know I would use it!

Sadly, the many times I started coding something towards that goal, I ended blocked by something, and I keep postponing my dream project.

Independently of that, for the past two years I have been driving a regular social meeting in Dublin for CouchSurfers, called the Dublin Mingle. The idea is pretty simple: to go every week to a different pub, and make friends.

I wanted to make a map marking all the places visited. Completely useless, but pretty! So, I went back to looking into IkiWiki internals, as the current osm plugin would not fulfill all my needs, and has a few annoying bugs.

After a few days of work, I made it: a refurbished osm plugin that uses the modern and pretty Leaflet library. If the javascript is not lost in the way (because you are reading from an aggregator, for example), below you should see the result. Otherwise, you can see it in action on its own page: Mingle)

The code is still not ready for merging into Ikiwiki, as I need to write tests and documentation. But you can find the changes in my GitHub repo.

It is still a long way to go before I can create my pubs wiki, but it is the first building block! Now I need a way to easily import and sync data from OSM, and then to create a structured search function.

Only six more days to go before SunCamp! If you are still considering it, hurry up, you might still find cheap tickets for the low season.

It will be a small event (about 20-25 people), with a more intimate atmosphere than DebConf. There will be people fixing RC bugs, preparing stuff for after the release, or just discussing with other Debian folks.

There will be at least one presentation from a local project, and surely some members of nearby communities will join us for the day like they did last year.

See you all in Lloret!

Just over 2 years ago, I started packaging Prometheus for Debian. It was a daunting task, mainly because the Golang ecosystem is so young, almost none of its dependencies were packaged, and upstream practices are very different from Debian's.

Today I want to announce that this is complete.

The main part of the Prometheus stack is available in Debian testing, which very soon will become the Stretch release:

• prometheus 1.5.2+ds-2 - Monitoring system and time series database
• prometheus-alertmanager 0.5.1+ds-7 - Handle and deliver alerts created by Prometheus
• prometheus-node-exporter 0.13.0+ds-1 - Prometheus exporter for machine metrics
• prometheus-pushgateway 0.3.1+ds-1 - Prometheus exporter for ephemereal jobs

These are available for a bunch of architectures (sadly, not all of them), and are the most important building blocks to deploy Prometheus in your network.

I have also finished preparing backports for all the required dependencies, and jessie-backports has a complete Prometheus stack too!

Adding to these, the archive already has the client libraries for Go, Perl, Python, and Django; and a bunch of other exporters. Except for the Postgres exporter, most of these are going to be in the Stretch release, and I plan to prepare backports for Jessie too:

• mtail - Extract monitoring data from logs for collection in a timeseries database
• prometheus-blackbox-exporter - Blackbox prober for Prometheus
• prometheus-mailexporter - exports prometheus-style metrics about mail server functionality
• prometheus-mongodb-exporter - Prometheus exporter for MongoDB
• prometheus-mysqld-exporter - Prometheus exporter for MySQL server.
• prometheus-pgbouncer-exporter - Export metrics from pgbouncer to Prometheus
• prometheus-postgres-exporter - Prometheus exporter for PostgreSQL server metrics
• prometheus-varnish-exporter - Prometheus exporter for Varnish

Note that not all of these have been packaged by me, luckily other Debianites are also working on Prometheus packaging!

I am confident that Prometheus is going to become one of the main monitoring tools in the near future, and I am very happy that Debian is the first distribution to offer official packages for it. If you are interested, there is still lots of work ahead. Patches, bug reports, and co-maintainers are welcome!

Update 3/3/2017: Today the Perl client library was uploaded to unstable, and it is waiting for ftp-master approval.

For over a year now, I have been using CyanogenMod in my Nexus phone. At first I just installed some bundle that brought all the proprietary Google applications and libraries, but later I decided that I wanted more control over it, so I did a clean install with no proprietary stuff.

This was not so great at the beginning, because the base system lacks the geolocation helpers that allow you to get a position in seconds (using GSM antennas and Wifi APs). And so, every time I opened OsmAnd (a great maps application, free software and free maps), I would have to wait minutes for the GPS radio to locate enough satellites.

Shortly after, I found about the UnifiedNLP project that provided a drop-in replacement for the location services, using pluggable location providers. This worked great, and you could choose to use the Apple or Mozilla on-line providers, or off-line databases that you could download or build yourself.

This worked well for most of my needs, and I was happy about it. I also had F-droid for installing free software applications, DAVdroid for contacts and calendar synchronisation, K9 for reading email, etc. I still needed some proprietary apps, but most of the stuff in my phone was Free Software.

But sadly, more and more application developers are buying into the vendor lock-in that is Google Play Services, which is a set of APIs that offer some useful functionality (including the location services), but that require non-free software that is not part of the AOSP (Android Open-Source Project). Mostly, this is because they make push notifications a requirement, or because they want you to be able to buy stuff from the application.

This is not limited to proprietary software. Most notably, the Signal project refuses to work without these libraries, or even to distribute the pre-compiled binaries on any platform that is not Google Play! (This is one of many reason why I don't recommend Signal to anybody).

And of course, many very useful services that people use every day require you to install proprietary applications, which care much less about your choices of non-standard platforms.

For the most part, I had been able to just get the package files for these applications1 from somewhere, and have the functionality I wanted.

Some apps would just work perfectly, others would complain about the lack of Play Services, but offer a degraded experience. You would not get notifications unless you re-opened the application, stuff like that. But they worked. Lately, some of the closed-source apps I sometimes use stopped working altogether.

So, tired of all this, I decided to give the MicroG project a try.

MicroG

MicroG is a direct descendant of UnifiedNLP and the NOGAPPS project. I had known about it for a while, but the installation procedures always put me off.

LWN published an article about them recently, and so I decided to spend a good chunk of time making it work. This blog post is to help making this a bit less painful.

Some prerequisites:

• No Gapps installed.
• Rooted phone (at least for the mechanism I describe here).
• Working ADB with root access to the phone.
• UnifiedNLP needs to be uninstalled (MicroG carries its own version of it).

Signature spoofing

The main problem with the installation, is that MicroG needs to pretend to be the original Google bundle. It has to show the same name, but most importantly, it has to spoof its cryptographic signatures so other applications do not realise it is not the real thing.

OmniROM and MarshROM (other alternative firmwares for Android) provide support for signature spoofing out of the box. If you are running these, go ahead and install MicroG, it will be very easy! Sadly, the CyanogenMod refused allowing signature spoofing, citing security concerns2, so most users will have to go the hard way.

The options for enabling this feature are basically two: either patch some core libraries H4xx0r style, or use the "Xposed framework". Since I still don't really understand what this Xposed thing is, and it is one of these projects that distributes files on XDA forums, I decided to go the dirty way.

Patching the ROM

Note: this method requires a rooted phone, java, and adb.

The MicroG wiki links to three different projects for patching your ROM, but turns out that two of them would not work at all with CyanogenMod 11 and 13 (the two versions I tried), because the system is "odexed" (whatever that means, the Android ecosystem is really annoying).

I actually upgraded CM to version 13 just to try this, so a couple of hours went there, with no results.

The project that did work is Haystack by Lanchon, which seems to be the cleaner and better developed of the three. Also the one with the most complex documentation.

In a nutshell, you need to download a bunch of files from the phone, apply a series of patches to it, and then reupload it.

To obtain the files and place them in the maguro (the codename for my phone) directory:

$ ./pull-fileset maguro

Now you need to apply some patches with the patch-fileset script. The patches are located in the patches directory:

$ ls patches
sigspoof-core
sigspoof-hook-1.5-2.3
sigspoof-hook-4.0
sigspoof-hook-4.1-6.0
sigspoof-hook-7.0
sigspoof-ui-global-4.1-6.0
sigspoof-ui-global-7.0

The patch-fileset script takes these parameters:

patch-fileset <patch-dir> <api-level> <input-dir> [ <output-dir> ]

Note that this requires knowing your Android version, and the API level. If you don't specify an output directory, it will append the patch name to the input directory name. Note that you should also check the output of the script for any warnings or errors!

First, you need to apply the patch that hooks into your specific version of Android (6.0, API level 23 in my case):

$ ./patch-fileset patches/sigspoof-hook-4.1-6.0 23 maguro
>>> target directory: maguro__sigspoof-hook-4.1-6.0
[...]
*** patch-fileset: success

Now, you need to add the core spoofing module:

$ ./patch-fileset patches/sigspoof-core 23 maguro__sigspoof-hook-4.1-6.0
>>> target directory: maguro__sigspoof-hook-4.1-6.0__sigspoof-core
[...]
*** patch-fileset: success

And finally, add the UI elements that let you enable or disable the signature spoofing:

$ ./patch-fileset patches/sigspoof-ui-global-4.1-6.0 23 maguro__sigspoof-hook-4.1-6.0__sigspoof-core
>>> target directory: maguro__sigspoof-hook-4.1-6.0__sigspoof-core__sigspoof-ui-global-4.1-6.0
[...]
*** patch-fileset: success

Now, you have a bundle ready to upload to your phone, and you do that with the push-fileset script:

$ ./push-fileset maguro__sigspoof-hook-4.1-6.0__sigspoof-core__sigspoof-ui-global-4.1-6.0

After this, reboot your phone, go to settings / developer settings, and at the end you should find a checkbx for "Allow signature spoofing" which you should now enable.

Installing MicroG

Now that the difficult part is done, the rest of the installation is pretty easy. You can add the MicroG repository to F-Droid and install the rest of the modules from there. Check the installation guide for all the details.

Once all the parts are in place, and after a last reboot, you should find a MicroG settings icon that will check that everything is working correctly, and give you the choice of which components to enable.

So far, other applications believe this phone has nothing weird, I get quick GPS fixes, push notifications seem to work... Not bad at all for such a young project!

Hope this is useful. I would love to hear your feedback in the comments!

Many people reading this have already suffered me talking to them about Prometheus. In personal conversation, or in the talks I gave at DebConf15 in Heidelberg, the Debian SunCamp in Lloret de Mar, BRMlab in Prague, and even at a talk on a different topic at the RABS in Cluj-Napoca.

Since their public announcement, I have been trying to support the project in the ways I could: by packaging it for Debian, and by spreading the word.

Last week the first ever Prometheus conference took place, so this time I did the opposite thing and I spoke about Debian to Prometheus people: I gave a 5 minutes lightning talk on Debian support for Prometheus.

What blew me away was the response I've got: from this tiny non-talk I prepared in 30 minutes, many people stopped me later to thank me for packaging Prometheus, and for Debian in general. They told me they were using my packages, gave me feedback, and even some RFPs!

At the post-conference beers, I had quite a few interesting discussions about Debian, release processes, library versioning, and culture clashes with upstream. I was expecting some Debian-bashing (we are old-fashioned, slow, etc.), instead I had intelligent and enriching conversations.

To me, this enforces once again my support and commitment to community conferences, where nobody is a VIP and everybody has something to share. It also taught me the value of intersecting different groups, even when there seems to be little in common.