I've been working hard lately to finish a report summing up my last two years of development on Licorn®. The report is in french because a great part of the development was sponsored by  ADEME (see also  Energy Efficiency in Europe), which is a French organization.

Focuses of the document :

• brief presentation,
• Licorn® structural advantages / weaknesses,
• what we have accomplished in two years,
• the current architecture
• still open developments

If you can read french and want to know more about the internals and externals of Licorn®, this is a must read. The document is far from exhaustive, but gives a good insight of how Licorn® works. If you prefer to learn what Licorn® do, head up to  the documentation website.

Download  the report (PDF, french, 55 pages, 22Mb).

I've finally managed to finish the WMI2 schemas. I created them for the needs of a scientific report, sponsored by  ADEME ( Agence de l'Environnement et de la Maîtrise de l'Énergie; Agency for Environment and Energy proficiency). 2 schemata representing licornd internal architecture, before and after the work partially sponsored by ADEME and  META IT (the company I work for).

Architecture of Licorn®, 1 year ago:

Architecture of Licorn®, as in the WMI2 development branch (undergoing review for merge in the stable tree). This schema is an enhanced version of the one published in the previous blogpost:

The schemata still need to be annontated to be fully understandable. They are not strictly formal in any way. They are just meant to illustrate how licorn works internaly to be understandable by a newcomer. At some time, they will be — and are already, partially — explained on our  documentation website.

The following schema tries to explain how the future WMI2 is working. This architecture is already fully functionnal in our development repository and will enter stable branch soon (official due date: March 5th, 2012). We are currently in the process of polishing it before releasing the patch.

The new WMI will be a huge step forward interactive and web-2.0-like interface ; everything can be handled asynchronously, and the WMI can update any part of its interface without refreshing the whole page. It is even a fully-featured Django + Jinja2 + jQuery application, which runs on top of our new webserver. The webserver is fully WSGI compliant, and built on top of the great gevent co-routine-based library.

As a side note, I've now integrated the WMI2 repository into Trac to help follow the changes and global Licorn® activity. Without this, one could think that Licorn® development is halted, because it all happens in this separate branch, until its merged into the stable one.

This makes team-development easier, by pushing patches very quick between our repos. In your .bashrc, insert this code, and adapt $REPO to fit your needs :

function push () {

	REPO="dev.licorn.org:/home/groups/licorn.wmi2"
	make darcs_record_prehook
	darcs wh -l

	echo -n "OK to record? ([additionnal record message] + [Enter] or Control-C to quit): "
	read -e MESSAGE

	if [ -n "${MESSAGE}" ]; then
		darcs rec -a -m "Work in progress `date '+%Y-%m-%d %H:%M:%S'` (${MESSAGE})."

	else
		darcs rec -a -m "Work in progress `date '+%Y-%m-%d %H:%M:%S'`.";

	fi

	echo -n "OK to push? ([Enter] or Control-C to abort): "
	read DUMMY

	darcs push -a ${REPO}

}

I just found  D3, while researching for various things related to Licorn® evolutions. I'm pretty impressed (the word is weak).

I just experimented a little with it, and the result is very cool. See before:

And After:

What you can't see it that the graph is completely dynamic, made of SVG, and refreshed every 5 seconds with smooth animations, scale morphing, label smart-placing and other cool things. There are still some rough edges (in the way scale lines come in), but this is just an experiment. Which is very very positive, BTW.

Hope you like it.

I've just updated the Licorn® documentation:

• the  modules states are now clearly defined; the code is not yet up-to-date, but it's the next thing to do on core, backends and extensions.
• backends documentation is a little more clean than before. The URL to access it is better spelled and doesn't display a 403 HTTP error anymore:  http://docs.licorn.org/core/backends.
• French translation has progressed a little, on the related subjects.

Yes. Finally, this long-awaited feature is here: you can now attach to an already-forked daemon, and type commands inside it, remotely. Just enter get in (from "get inside") on your prefered CLI prompt.

I have been thinking about this remote debugger for a while. Crunching rlcompleter, readline and a derived InteractiveConsole in my head for days, It seemed really feasible and nearly easy, but I still missed a key to write it. I wasn't far from the result, which I found nearly complete in the  rfoo python package.

Reading rfoo code inspired to me the #645, because it seems to have everything we need, while beeing much more shorter code-wise, and perhaps much faster than Pyro: the hard part of rfoo is implemented in C (via  Cython). When performance problem arise in the cluster, I think we *will* have to switch to rfoo.

The easy part of the switch is that rfoo is "call compatible" with Pyro (both are completely transparent to python programs). Only the PyroProxyWithAttr cannot be implemented in rfoo, which seems wise enough from the security standpoint (everything in Licorn® will be reimplemented via methods to fullfit the security decorators, anyway).

All I have implemented ahead of Pyro seems to be re-usable "out-of-the-box" with rfoo, which makes me think the security decorators I'm thinking about will be, too.

On the road, while reading and derivating rconsole code for Licorn®, I found the fix for #582. I will surely offer myself a beer, or perhaps my collegues will offer it to me, If they read this. But the persons who really owe a beer are  the rfoo developpers.

A big "thank you" to them. They saved me a great time, and showed me how to fix one the most obscure bugs.

NOTE: as a consequence, the local interactive console has vanished from the daemon. We can still interact with it, but this is becoming more and more useless, as much more things are available remotely (get status, get events, get inside), and usable more than once, in parallel, with different options.

This file contains filenames that darcs should not care about. So here we go (only the relevant part), in _darcs/prefs/boring on every of your repos:

^tests/data/scenarii
^tests/data/\..*_status
^tests/data/\.owner
^tests/data/wmitest

docs/_build
^locale/fr
^interfaces/wmi/.*_donnut.png
^interfaces/wmi/js/json

\.pye$
^tailoritself\.
\.mo$

I don't know if it's interesting, but still  a worthly read.

Recently I added 2 new functionnalities to the Licorn® daemon and the CLI interface. The first is a simple extension of the existing get status. There is now a -m CLI switch, which implies that the CLI process stays connected to the daemon, and updates the status on the terminal. The refresh delay and various parameters (fixed refresh count / maximum refresh time / clear screen between outputs) are configurable via other cli switches; see get status --help for details. Just remember this command:

get status -m

Default parameters are cool anyway (refresh time: 1 second, clear screen: yes, refresh count: infinite, max. refresh time: infinite). Just hit Control-C to stop the status updates when you're done.

The second functionnality could be seen as an internal sniffer, in the daemon. It's called monitor mode. With a new CLI command (get events, not to name it), you can access any internal event, decision or message generated in the daemon. This monitor mode will eventually supersede the LTRACE facility, but it's not very clear at this point because the monitor could impact the daemon performance much more than LTRACE (it's always compiled in, whereas LTRACE is usable only when the Python interpreter is in debug mode).

You use the monitor mode like you use LTRACE (levels are the same for both). E.g. to display std (standard) events:

get events

Or to dump only the inotifier related events:

get events -f inotifier

You can combine them to your liking, just the same you would do with LTRACE:

get events -f 'core^system|daemon^threads'

The bonus is the full sniffing mode: just add 'logging' to the facilities you want to follow, and every output message sent by the daemon to any other CLI command (run by you or by others) will be pushed to your local monitor session:

get events -f 'core|logging'

The bonus: you can even monitor a remote daemon from your local machine. Just set the special environment variable before launching the command:

export LICORN_SERVER='IP_or_hostname'
get events

And you're done! Happy monitoring, and remember: these things can easily flood your terminal!

NOTE: currently, only inotifier and logging are fully converted to monitor mode. Other parts will soon follow. The rest of the code still uses the LTRACE facility.

In a few words, on the server side (Ubuntu Maverick):

/etc/rsyncd.conf:

[Sauvegardes]
	# not beiing root seems safer to me.
	uid = rsync
	# the gid is the tricky part and must be forced, else tranfers fail. 
	# rsp-* implies the daemon on the server side has always full permissions.
	gid = rsp-Sauvegardes
	path = /home/groups/Sauvegardes
	comment = Sauvegardes
	read only = no

/etc/default/rsync:

RSYNC_ENABLE=true
RSYNC_OPTS='--address=192.168.111.1'
RSYNC_NICE=''
RSYNC_IONICE='-c3'

in the root shell:

# not that needed, but seems clean to me
add group --system rsync

# this one is really needed
add user --system rsync --force

# not that needed too, but seems clean to me
add user rsync rsp-Sauvegardes

On the client side:

rsync -a <other options> dir1 dir2 ... leto::Sauvegardes/

Speed note: my server Leto beiing a "poor" and old monocore AMD Sempron, using rsyncd permits to speed up the transfers up to the client disk reading speed (roughly 25Mb/s in best moments) using only 30% CPU, instead of being CPU clamped to 100% with ~8Mb/s maximum speed when wrapped inside SSH (which is totally useless on a trusted local network, IMHO).

You may have noticed or not, but recent changes in the core lead to enhanced console output for get CLI tool. We use utf-8 as much as possible, to include icons and special characters and colors which give a bunch of informations without cluttering your terminal with things to read.

Thanks to the recent core rewrite, we have now high-level methods and properties for building these output informations, and get has an internal cache / invalidation mechanism to avoid wasting CPU cycles.

I'm currently reading a nice  TS-framework-related article, and some others (search for " nose versus py.test with google"), and I'm seriously thinking about (sort of) getting rid of, or enhancing a lot our home-made testsuite.

Sure, our testsuite has advantages of very-small-scenarii writing, very-high-level (command combinations, varying context, etc) testing, auto-included teardown commands and much-much-beautiful output (very human friendly IMHO). But we now need low-level unit testing in Licorn®, and the TS can't handle this without a consequent rewrite.

There is currently no way to test something as simple as a method raising an exception when given bad input, and I miss this. BTW, now that our code base is increasing fast, I'm looking for a way to split core.py into smaller files, and spread the test methods in the relevant part of Licorn®. Neither to say that auto-collecting test methods would be more clever than needing to chain them in core.py.

Perhaps we will acheive some sort of combinations between the two worlds. Beiing capable to test things in the background while hand-checking a current failure is something I'm not yet ready to loose.

The new ConfigurationFile class (currently beiing written and tested, thus the research and this blog entry), based on  the pygments parser is something that needs to be very-very carefully tested before use, and the current TS can't do this easily.

More to comes in the very-near future, both works are progressing alongside.

PS: additionnaly, have a look at  django specific testing. Generaly speaking,  This might me a worth read, too.

With #500 implemented, Licorn® now supports mounting any kind of volumes on the local machine, provided you add extensions.volumes.mount_all_fs = True in your /etc/licorn/licorn.conf file.

This makes licornd provide the same functionnalities as udisks, allowing to avoid it completely when you run licornd on your laptop or desktop machine.

End-User-formatted (vfat/nfts/iso9660/udf) volumes will be mounted with the UID of the console user. Other types (ext*/btrfs/reiserfs/xfs/jfs) will be mounted the standard way, with acl and user_xattr options.

If for any reason you plug your device in before logging in on your machine, just launch the following commands:

del vol -a
add vol -a

And licornd will remount the volumes accordingly for you.

Note: I didn't test the behavior when logged in remotely. I don't think an ssh remote login will be treated like a normal local login. This will probably be implemented in a future enhancement if needed.

Every part of Licorn® can now emit events, and any other part of it can setup a callback to receive the event arguments, to do what is needed. Events callbacks can be run synchronously (with method L_event_run()) or not (with method L_event_dispatch()). In this case, there is no guarantee of callback order call, jobs can be parallel because they are handled by the service facility. Example:

Emitter side:

from licorn.daemon import priorities, InternalEvent

# L_event_dispatch is a builtin, defined by the daemon at start
# no need to import anything, to use it.

L_event_dispatch(priorities.NORMAL,
                 InternalEvent('main_configuration_file_changed'))

Receiver side:

[...]

def main_configuration_file_changed_callback():
    """ this method will be auto-collected if the surrounding object 
        is a controller, or a CoreUnitObject whose controller defines
        self.__look_deeper_for_callbacks (typically ModulesManager 
        does it). """
    #
    # do whatever needed with LMC.configuration, whose
    # contents already reflect the change.

The EventManager is ready to operate even before the INotifier, and just after the LMC is setup. It will collect all callbacks of the controllers, backend and extensions on start.

It is stopped last, at the end of the daemon life.

Note: the InternalEvent instance can be given a callback argument (any callable() can apply for the job), to resync the Emitter at the end of event callback execution.

I'm proud to announce the new inotifier rewrite (and its bunch of small enhancements), internally and lovely named "hopefully-this-one-will-work-as-expected" (private joke to me). It's shorter than previous version in terms of codelines, albeit more complex when dealing with special cases (large directories, multiple concurrent accesses to same files, re-born just-deleted files or dirs, etc). The new version is many times faster than all previous ones (including the external C-implemented gamin one). When you untar an archive, you can expect more or less the time of the untar process, after it finished, for complete ACLs application. Previously, it could take minutes to do the same (specifically when untarring the linux kernel in a shared dir). licornd is also very smart when talking about resources-consumption: it takes the CPU for ACLs intensive tasks (but only ONE CPU), and doesn't take it long. For what it has to do, I find it well balanced from the functionnality/resource point of view.

The new inotifier and related core.classes additions allow users homes to be watched now, and offer dedicated functionnalities to handle configuration files, and report *real* changes to them (not 'all access', generating a lot of false positives).

dnsmasq backend, privileges directly benefit from this new functionnalities. shadow configuration files watch is more robust and verifies everything when they reload (one could create inconsistencies, editing the files manually; this is taken in account).

There are still some rough edges and evil sub-sonic bugs (perhaps they are all the same, I can't hunt it down for now), but only on very-very heavily loaded systems, where users and groups pop in an out very fast. I will fix them in the next coding cycle.

Hopefully, you won't need the chk group command anymore. If you do, please provide a full trace:

export LTRACE=std
licornd -rvD

<whatever command in your other terminal>

In the new-but-small-but-cool features category, you'll find the command fuzzy matching:

get u
get us
get usr
get users

(and so one, with identical counterparts for add/mod/del/chk)

Will bring you the list of users. In the same kind:

get g     -> groups
get pro   -> profiles
get pri   -> privileges
get kw    -> keywords

And so on. Everything is computed when you type it, there are no so-called "fixed values".

In the not-so-small-but-very-cool category, you will find that every part of Licorn® is now fully multi-lingual, on-the-fly: the daemon starts in the system lang, but every thread inside of it can switch to another language, and the client languages are pulled in from the web headers or the calling CLI environment. This makes everything dynamic, at will.

Documentation has been updated for permissions parts.

French tranlation is progressing notably: WMI part is finished, CLI is 90% done, and the rest is more or less 70% done (it doesn't matter anyway, as no user really sees it in real life).

I voluntarily don't mention the core object rewrite. It's very technical and doesn't bring new end-users functionalities, but guarantees that everything is cleaner and easier to extend inside licornd, regarding the users/groups/profiles/privileges/machines point of view.

I probably forgot many things here, but if I had written a book, you won't have read it anyway. Code and *use* the code is better. Many bugs have been fixed, and the code is generally more pythonic and lighter tht before: there are more generators, less hard-coded things, and abstractions (when necessary) got in the right places. At least, this how I wanted to implement them.

Enjoy,

The past 2 weeks have featured a great core rewrite that I wanted to achieve for a long time. Core objects (Users, Groups & Profiles) are now clean objects, implemented with all the pythonic-fancyness that modern code can have (most notably properties, weak references and internal generators where applicable). Controllers manipulates them in a clean way too, doing things the CoreUnitObject can't do because they are not aware of the controller context (which make the whole thing totally logical, finally; things as they were meant to be, at last; [put your favorite self-satisfaction sentence here]).

Controllers and unit objects moderately use the daemon service facility, to make things more instant to clients and avoid long-activity stoppers: for example, setting permissive ON/OFF on groups launches a background check on shared data and returns instantly (among others).

As a consequence, the service facilities are initialized very early in the daemon (even before the LMC initializes) and are usable everywhere: Each individual object has a licord R/O property (named after this avoid collision with the Thread's daemon attribute), offering directly the {service,aclcheck,network}_* methods. The imports overhead and dirtiness of the previous implementation is totally avoided.

The patch lies in the  development repository (not referenced in trac but accessible to SSHers). I will not push it to the stable branch until the new inotifier has landed, but it is very stable (testsuite has run many times on it).

The Full-i18n milestone will soon be closed (or nearly), because #2, #70, #541, #542 and #544 are implemented or closed in this patch. Thus, besides the full-object rewrite, we now have on-the-fly in-thread language transparent switching, and this really rocks for the WMI.

Starting from now, the only thing you've got to do to enable LTRACE is to set the environment variable. If the daemon finds it at start, it will refork itself with the appropriate options to enable it.

The LTRACE status is maintained while restarting from inside the daemon (this was not as easy as it seems to implement).

Bonus: just export LTRACE=std to get a nice but not so verbose LTRACE output, you will notice that the only places where the daemon hands on stop/restart are justified (__unihibit_udisks() and PyroFinder queue not empty). In this case, just wait. The udisks thing can take up to 10 secs, and the PyroFinder one, up to 20 secs (i'm searching how to lower this one, the Pyro timeout is already set to 5 secs).

Theses extensions are ready for wider testing. They have their documentation setup ( volumes and  rdiffbackup).

A major thread rewrite has been done during this dev cycle.

CLI and WMI related parts are coming in.

Tonight I sprint-coded the  OpenSSH extension, removed all openssh-related code in LMC.configuration, and added system-service-related code to  SystemController class.

Guess what:

• It works like a charm :-)
• I readded the missing functionnalities from previous releases (the AllowGroups configuration directive), and added much more (enable/disable ssh extension and service, check other configuration directives, verify the sshd is running...)

By the way, members of remotessh are allowed to connect, and members of admins too (regardless of their remotessh member status: this is cool because you (the reader) are probably already a member of admins, and you won't ever notice the sshd group-authorization enforcement ;-)

The  OpensshExtension module can serve as an extension model: it seems quite clean (and short) to me.

This work would not have been so easy without the ConfigFile class Robin wrote. I enhanced it a little, but it's so practical!!