Equo rewrite, Sabayon 10 and Google

The following month are expected to be really exciting (and scary, eheh), for many reasons. Explanation below.

My life is going to rapidly change in roughly one month, and when these things happen in your life, you feel scared and excited at the same time. I always tried to cope with these events by just being myself, an error-prone human being (My tech. English teacher doesn’t like me to use “human being”, but where’s the poetry then!) that always tries to enjoy life and computer science with a big smile on his face.

So, let’s start in reverse order. I have the opportunity to do the university internship at Google starting from October, more precisely at Google Ireland, which is located in Dublin. I think many of the Googlers had the same feelings I currently have before me, scared and excited at the same time, with questions like “do I deserve this?”, “am I good enough?”. As I wrote above, the only answer I have found so far is that, well, it will be challenging but, do I like boredom after all? Leveraging on professionality and humbleness is probably what makes you a good team-mate all the time. Individuals cannot scale up infinitely, that is why scaling out (as in team work) is a much better approach.

It’s been two years since I started working at Weswit, the company behind the award-winning Lightstreamer Push technology, and next month is going to be my last one there. Even though, you never know what will happen next year, once back from the internship at Google. Sure thing is, I will need a job again, and I will eventually graduate (yay!).
So yeah, during the whole University period, I kept working and besides it’s been tough, it really helped me out bidirectionally. In the end, I kept accumulating real-world expertise during this time.
Anything in my life has been risk-free, and I took the risk of leaving a great job position to pursue something I would have regretted for the rest of my life, I’m sure. On the other hand, I’m sure that at the end of the day, it will be a win-win situation. Weswit is a great company, with great people (that I want to thank for the trust they gave me) and I’m almost sure that the next one might not be my last month there (in absolute terms I mean). You never know what is going to happen in your life, and I believe there’s always a balance between bad and good things. Patience, passion and dedication is the best approach to life, by the way.

Before leaving for Dublin, we (as in the Sabayon team) are planning to release Sabayon 10. improved ZFS support,  improved Entropy & Rigo experience (all the features users asked me about have been implemented!), out of the box KMS improvements, BFQ iosched as default scheduler (I am a big fan of Paolo Valente’s work) a load of new updates (from the Linux kernel to X.Org, from GNOME to KDE through MATE) and if we have time, more Gentoo-hardened features.

Let me mention here one really nice Entropy feature I implemented last month: Entropy adopted SQLite3 as its repository model engine since day one (and it’s been a big win!), even though, the actual implementation has been always abstracted away so that upper layers never had to deal with it directly (and up to here, there is nothing exciting). Given that a file-based database, like SQLite is, is almost impossible to scale out [1], and given that I’ve been digging into MySQL for some time now, I decided it was time to write an entropy.db connector/adapter for MySQL, specifically designed for the InnoDB storage engine. And 1000 LOC just did it [2]!

As you may have seen if you’re using Sabayon and updating it daily, Entropy version has been bumped from 1.0_rcXXX to just XXX. As of today though, the latest Entropy version is 134. It might sound odd or even funny, but I was sick of seeing that 1.0_rc prefix that was just starting to look ridiculous. Entropy is just about continuous development and improvement, when I fully realized this, it was clear that there won’t be any “final”, “one-point-oh” and “one-size-fits-all done && done” version, ever. Version numbers have been always overrated, so f**k formally defined version numbers, welcome monotonically increasing sequences (users won’t care anyway, they just want the latest and greatest).

I know, I mention “Equo rewrite” in the blog post title. And here we go. The Equo codebase was one of the first and long living part of Entropy I wrote, some of the code is there since 2007, even though it went through several refinement processes, the core structure is still the same (crap). Let me roll back the clock a little bit first, when the Eit codebase [3] replaced old equo-community, reagent and activator tools, it was clear that I was going to do exactly the same thing with the Equo one, thus I wrote the whole code in an extremely modular way, to the point that extra features (or “commands” in this case) could be plugged in by 3rd parties without touching the Eit kernel at all. After almost one year, Eit has proven to be really powerful and solid to the extent that now, its architecture is landing into the much more visible next-gen Equo app.
I tell you, the process of migrating the Equo codebase over will be long. It is actually one of many background tasks I usually work on during rainy weekends. But still, expect me to experiment with new (crazy, arguable, you name it) ideas while I make progress on this task. The new Equo is codenamed “Solo”, but it’s just a way to avoid file names clashing while I port the code over. You can find the first commits on the entropy.git repo, under the “solo” branch [4].

Make sure to not miss the whole picture: we’re a team and Sabayon lives on incremental improvements (continous development, agile!). This has the big advantage that we can implement and deploy features without temporal constraints. And in the end, it’s just our (beloved) hobby!

[1] imagine a web service cluster, etc — I know, SQL in general is known for not scaling out well without sharding or other techniques, but this is outside the scope of this paragraph, and I think NoSQL is sometimes overrated as well.
[2] http://git.sabayon.org/entropy.git/tree/lib/entropy/db/mysql.py
[3] Eit is the server-side (and community-repo side) command line tool, “Eit” stands for “Entropy Infrastructure Toolkit” and it exposes repo management in a git-like fashion.
[4] http://git.sabayon.org/entropy.git/log/?h=solo

Sabayon on Amazon EC2

During the last week, while I was enjoying my vacations, I’ve also had a lot of fun preparing a new EC2-friendly kernel (and sources) based off our kernel git repo (which is based on Linus’ kernel tree + some patches like the BFQ scheduler, fbcondecor and aufs3).

The outcome of my puzzle game (trying to figure out why an instance doesn’t boot on EC2 is like solving puzzles at times) is that sys-kernel/ec2-sources and sys-kernel/linux-ec2 (precompiled binaries) are now available on the sabayon-distro overlay and the Sabayon Entropy repository “sabayonlinux.org”.

As you may expect, I rapidly started to get bored again. For this very simple reason, and since I always wanted to have a fallback website/webservices infra ready on EC2 (in case of a disaster) I started cooking an EBS baked AMI, copycating the current Virtual Machines snapshots from our backup server.

As you may expect, I rapidly started to get bored once again. So, I prepared a molecule .spec file that automatically creates a ready-to-go ext4-based Sabayon Server filesystem image tarball ready to be dumped into a spare EBS volume. Once you have an EBS volume you just need to snapshot it and create the AMI from there (fyi).

As you may expect, I was getting bored of course. So I started preparing a “BuildBot” AMI that could be launched programmatically (say, from a cronjob) and once started (@reboot cronjob target is <3), attaches an existing EBS volume containing a Sabayon chroot, runs equo update && equo upgrade and other stuff, then detaches the volume, makes a snapshot, creates a versioned AMI.
Yes, boring stuff deserve a lot of bash scripting, can’t be otherwise. In this way, I can continuosly build updated Sabayon AMIs for EC2 without much human intervention (of course the BuildBot AMI mails back to me the upgrade result (both stderr and stdout)).
If anybody is interested in my “BuildBot” scripts, just drop a line here.

I don’t know yet where to go from here, but you may be interested in reading this wiki entry: “Sabayon on EC2“. Moreover, you may be also interested in knowing that the aforementioned filesystem image tarballs are already available on Sabayon mirrors, inside the iso/daily directory.

You can have a look at the currently available Sabayon AMIs here:

sysadm for a weekend

Probably many of you do not realize how much time it takes to maintain a healthy distro infrastructure. With “distro infrastructure” I mean all the servers and virtual machines that are serving Sabayon users and random website visitors.

We currently have 5 servers spread around several Italian Universities and TOP-IX, all of them are running Sabayon and all of them need to be kept up-to-date, rebuilt from time to time to accomodate new services or new management architectures (like automated content provisioning for throw-away virtual machines, etc). On top of two of them, there are a bunch of OpenVZ virtual machines, for sandboxing several critical components.

During the last weekend, and the weekend before it, I’ve been busy rebuilding the whole WWW virtual machines (OpenVZ) in order to transform these simple static virtual machines into  “clusterable” ones (for instance, implementing boot-time content provisioning, decoupling static data from webapps,  etc). Moreover, I eventually found some time to also decouple the Database Server (MySQL, at this time) from the WWW virtual machine. At some point, we’ll be able to migrate some of the infrastructure to the cloud with almost no effort and, most importantly, we will be able to scale out quite easily during post-release periods. Unfortunately, at this time, cloud computing is still quite expensive and we would need a lot more donations to be able to pay for it, so we need to squeeze all the RAM and CPU available from the current infra.

On a unrelated note, I am really proud of my little monster, Entropy, which is really doing a great job on our servers. Time to get back to Rigo coding now…

Sabayon runs Android Market / Google Play Apps natively

I’m sure this is going to blow your mind completely.

During the last two months I’ve been busy re-thinking the way Package Management works in Linux distros in terms of User interaction and I ended up designing the new Sabayon Entropy GUI using a “less is more” approach. In layman terms, the idea is to carry out all the activities through two, and only two, main “interaction points”: a search bar and a notification bar (for sending feedback to User).

But why not going further and supporting Android Market Google Play Applications directly in Rigo?
This is exactly what I did, and it’s been even easier than implementing the whole Entropy Package Manager thanks to the fact that Android apps are self-contained and no dependency management is required.

So, yeah, Sabayon now supports Android Market Applications natively, both installation and runtime execution (through the Dalvik Java VM).

Rigo Application Browser, getting closer

So far other two weeks are past since my last blog post about Rigo. I have been able to implement several features since then, mainly thanks to RedBull I’d say. Jokes apart, I’m used to work on Rigo in the evening and early (6am) morning and, despite hackers are all used to nightly coding marathons (including me, actually), the best is coming out of me when birds start singing.

git-log would tell you what changes have been made in this timeframe, without me wasting time here instead of coding, but I think communicating it explicitly would help me getting some feedback from all of you in order to enrich my next UI design iteration. I’m not an interaction designer myself, don’t really want to be, but I learned a lot of things working with them in the past. Although we sometimes see interaction designers as kids with a pencil abusing of erasers (i’m not serious!) they are vital in the success of your App (I learned it the hard way as well).

Dbus Service

RigoDaemon is the new Dbus-powered service that handles the privileged operations on behalf of (concurrent) Rigo clients, such as Repositories Update, Application Management (install/remove/update) and System Upgrade. These are in fact the three main tasks RigoDaemon is going to carry out. The hard part has been implementing a way to do Resource Passing from Rigo clients (holding shared Entropy Resources locks) and RigoDaemon (wanting to acquire exclusive access to the same Entropy Resources), i called the whole process: rendezvous (as it is, in fact). The rendezvous has to be fault-tolerant in the sense that, being for both releasing and acquiring back shared locks by Rigo clients, these can go away anytime and RigoDaemon is required to make sure to not hold resources if no Rigo instances are connected.
RigoDaemon is going to replace Magneto Dbus Service, once Rigo replaced Sulfur.


Multiple Rigo Clients can be used concurrently, as images show below. The IPC architecture allows RigoDaemon to “arbitrate” resources access, kindly asking all the clients connected to release them for example (due to the beginning of a specific Activity, see below). At the same time, it is possible to launch an activity (like repository update or system upgrade), then close Rigo, reopen it and have it resumed to the current activity state. This makes Rigo fully fault tolerant against bugs and critical system updates where GTK3 or other Rigo dependencies could (but shouldn’t) be subject to potential instabilities. Running GTK3 unprivileged is also another indirect goal achieved.


Both RigoDaemon and Rigo implement their daemon and UI state through a Finite State Machine, moving from an activity to another atomically. This makes possible to aggressively use multithreading without worrying too much about UI state and the likes. I don’t want to start describing the boring details here, but since I introduced this way of seeing the whole User <-> Rigo interaction story, everything started to make sense: spawning a repo update event now means “hey, i want to carry out this activity, gimme the UI control” and “hey dude [dude is another Rigo Client process] the other guy here wants to do Activity-X, could you please lock down and start listening to his events?”.
The concept is the same you can find developing for Android, and is about how the human being can deal with tools, you cannot use a hammer and a chainsaw at the same time in real life!


Any privileged activity inside RigoDaemon is controlled by PolicyKit actions. There are three main actions (the same listed below): Repository Update, System Upgrade and Application Management. This way administrators could even setup policies allowing users to update, upgrade or install apps with no privileges at all (read: *could*). PolicyKit gives us this fantastic flexibility though.

Repositories Update

The Activity I’ve been working on is Repositories Update. Even if this is usually carried out by Magneto (automatically updating repositories on our behalf), there are cases in which this is still important for Users: they might not have Magneto loaded, repositories could have been just added to the list and Rigo would need to download them, etc.
This Activity is also a good testbed for RigoDaemon <-> Rigo Client IPC architecture. Fact is, it took me 5 iterations before getting everything working the way I wanted (I’m a picky guy you know…).

Repositories Update Activity, like all the other activities that are going to be implemented soon, require RigoDaemon running. Now, there’s a trick to start it in devel mode, and start Rigo (you need pygobject-cairo, all the GLib introspection stuff, including introspection support in policykit):

$ git clone git://git.sabayon.org/projects/entropy.git
$ cd entropy/rigo/RigoDaemon && sudo sh  devel-start-daemon.sh &
$ cd ../ && python2 rigo_app.py –debug

What’s missing

The UI is not 100% complete first of all.
System Upgrade, Application Management activies are not implemented yet.
The Rigo Preferences View is not there (this will contain handy buttons to launch secondary activies as well as forcing repositories to update).
If all goes well, I expect to have everything working in a month from now, even because I need to go back studying then.

This blog post took me 1 whole evening (split across several), if you think it’s been worth it, don’t think twice and donate to Sabayon now, http://www.sabayon.org/donate .

Eit: the stupid package tracker, reinvented

If you are maintaining either a Community Repository or a full blown one (inside a chroot, etc), please go ahead reading this post, important changes are going to happen very soon.

As the average FLOSS developer does, one morning a few months ago I woke up and decided that I needed to break stuff. Three months later, it turned out I fully achieved this goal, for your great joy.

Eit is going to ease your life as Entropy repository maintainer (and packages cook!) in a very dramatic way. Largely inspired to the “Git-way of doing things” and to its awesomeness, it represents a complete rewrite of the “Community Repositories/Entropy Server” functionalities exposes by “equo community”, “reagent” and “activator”.

It will be available starting from Entropy 1.0_rc60 and will completely replace the old tools and their syntax carrying out a lot of minor improvements as well.
Please note that Community Repositories maintainers, shall need to set “community-mode = enable” inside /etc/entropy/server.conf (see server.conf.example for more info) in order to make Entropy Server behave correctly.

How the syntax changed actually? Just a few examples below.

eit status [<repository>], shows repository status, unstaged packages, packages ready to push, etc.
eit commit [<repository>], commits unstaged packages to given repository (or the current one).
eit checkout <repository>, switches to the given Entropy repository.
eit add [–to <repository>] <package> [<package> …], add the given unstaged packages to repository (current repository if none given).
eit push [<repository>] push staged changes to the remote repository.

This is just a subset of all the commands available, but they’re enough to start using Eit. “Top-level commands” are implemented as plugins to allow maximum modularity and 3rd-party extensions. Just subclass eit.commands.command.EitCommand and implement the respective methods.

Entropy API Tutorial #4: inverse dependencies

Before moving on to other Entropy codebase areas, I need to also mention inverse dependencies. The same are used for calculating a removal schedule basing on currently installed packages. Let’s say you want to remove app-foo/bar, are there packages needing it? If so, are there packages needing packages needing it? And so on… If you know something about Algorithms, you can easily understand that the whole thing is just a topological sort out of a DAG (actually package dependencies unfortunately come with cycles, so, even if it wouldn’t make much sense to have dep A pulling dep B itself pulling dep A, this is what we have in real life, for reasons out of the scope of this blog post…).

But let’s keep things simple, especially if you’re new to these algorithmic concepts (scratch them!).
First of all we need to match a package name in the installed packages repository, then query Entropy Client asking for a removal schedule, and perhaps execute the actual removal.

import os
from entropy.client.interfaces import Client
entropy_client = Client()

installed_repository = entropy_client.installed_repository()
# let’s match app-foo/bar out of the installed packages repository
package_id, pkg_rc = installed_repository.atomMatch(“app-foo/bar”)
if pkg_rc != 0:
    print(“cannot find app-foo/bar”)
    raise SystemExit(1)

# now ask Entropy Client to build up a sorted removal list
# in case of system packages, if system_packages keyword arg
# is True (see API documentation), DependenciesNotRemovable
# exception is raised. But we’re not going to discuss these
# details here for brevity.
packages_ids = entropy_client.get_removal_queue([package_id])

# now let’s remove them from the Live filesystem!
exit_st = os.EX_OK
for package_id in packages_ids:
    pkg = entropy_client.Package()
    pkg.prepare([package_id], “remove”)
    rc = pkg.run()
    if rc != os.EX_OK:
        exit_st = rc
raise SystemExit(exit_st)

That is pretty much it!

kernel.org abbreviated is k.o.

Not that I dislike it. I mean, I always thought that the release process was just a crazy mix of different flavours of madness. Especially with the stable repo, containing “facepalm”-like fixes (hey, this is software engineering, not rocket science — but still!).

After a full month, git.kernel.org (both http and git ports), http://www.kernel.org, are still down, and I’m starting to get really bored. Linus temporarily (permanently?) moved his git repo to github.com in order to keep 3.1 development going and this seems to have worked for a good cut of consumers. But how about the other 100+ repos that were sitting on git.kernel.org? They’re not even accessible in read-only.

Being the Linux Kernel a vital part of the whole Linux-based FLOSS world, how can we accept a 1-month downtime? Why isn’t there anybody out there starting to raise the voice? That is completely ridiculous. While I understand the QA and Security teams involved in bisecting the kernel.org logs etc, I cannot really accept that git.kernel.org is still down, for the following reasons:

  • git repositories themselves haven’t been harmed (as they said — not my own words), so git.k.o could be restored at least to a read-only state without being much scared
  • ssh keys could be revoked for everybody and their access limited to git pushing

Can’t wait to ditch my frustration seeing kernel.org back to life.

Entropy API Tutorial #3: installing packages

Basing on my previous Entropy API Tutorial #2 regarding how to calculate package dependencies through the Entropy Python API, let’s move forward and see how to actually install packages from a repository.

Let’s say we have a list called “package_matches” containing all the package matches we want to install (dependencies have been expanded already). At this point we’re ready to ask Entropy to fetch from mirrors and merge them into the live filesystem. The following example is very simple. For the same reason, it doesn’t cover all the functionalities, such as, for example, multi-fetching or fetching of package source code.

import os
from entropy.client.interfaces import Client

entropy_client = Client()
# this is out package_matches list (fake content)
package_matches = [(12345, ‘sabayon-weekly’),
    (12343, ‘sabayon-weekly’)]

# first fetch, then merge
exit_st = os.EX_OK
for pkg_match in package_matches:
    pkg = entropy_client.Package()
    pkg.prepare(pkg_match, “fetch”, {})
    rc = pkg.run()
    if rc != os.EX_OK:
        exit_st = rc
if exit_st != os.EX_OK:
    raise SystemExit(exit_st)

# now merge into our live filesystem
for pkg_match in package_matches:
    pkg = entropy_client.Package()
    pkg.prepare(pkg_match, “install”, {})
    rc = pkg.run()
    if rc != os.EX_OK:
        exit_st = rc

raise SystemExit(exit_st)

Some curiosities. For http{s,}://, ftp{s,}://, file://, Entropy uses Python urllib2 with custom User-Agent. For rsync:// Entropy uses the external rsync executable.
As some of you pointed out, exit statuses are a heritage of early Entropy codebase stages and are going to be replaced by exceptions in future. Entropy Client refactoring is going to happen in a few months. But for now, enjoy your hacking ;-).

In the next tutorial, before starting to show you the power of Entropy Services, I’ll continue my excursus through Entropy Client, focusing on metadata retrieval from repositories.

Quick and dirty: why sudo is bad for security

I always hated sudo. It’s like trying to fix a tyre with a chewing gum, close eyes and hope it works.
Just a quick and dirty, straightforward explanation why it is bad for your systems’ security.
Let’s say you have a password-less ssh keypair attached to a remote user allowed to run anything as root through prefixing commands with “sudo”. *cough* this is the default setup for any Amazon EC2 instance running Amazon Linux AMIs (and also Ubuntu AMIs I guess, and perhaps even Fedora ones?)
What happens if your keypair slips out, gets leaked somehow or somebody steals it from your hard drive? The result is simple, the attacker automatically gains root access anywhere you have the above setup.
That’s quite dangerous, especially if you’re paranoid. Funny enough, many distros are forcing users to use sudo against their will, in a password-less setup.

Of course, this also happens in the unlikely (ahaha) case where your user account gets compromised. Having two levels of passwords is always better than one.

sudo FTL!

hello, twitter

  • Eventually, the Big Kernel Lock is leaving the Linux kernel. #linux /me likes it 6 years ago

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 584 other followers