Developers Planet

February 21, 2018

Alex Bennée

Workbooks for Benchmarking

While working on a major re-factor of QEMU’s softfloat code I’ve been doing a lot of benchmarking. It can be quite tedious work as you need to be careful you’ve run the correct steps on the correct binaries and keeping notes is important. It is a task that cries out for scripting but that in itself can be a compromise as you end up stitching a pipeline of commands together in something like perl. You may script it all in a language designed for this sort of thing like R but then find your final upload step is a pain to implement.

One solution to this is to use a literate programming workbook like this. Literate programming is a style where you interleave your code with natural prose describing the steps you go through. This is different from simply having well commented code in a source tree. For one thing you do not have to leap around a large code base as everything you need is on the file you are reading, from top to bottom. There are many solutions out there including various python based examples. Of course being a happy Emacs user I use one of its stand-out features org-mode which comes with multi-language org-babel support. This allows me to document my benchmarking while scripting up the steps in a variety of “languages” depending on the my needs at the time. Let’s take a look at the first section:

1 Binaries To Test

Here we have several tables of binaries to test. We refer to the
current benchmarking set from the next stage, Run Benchmark.

For a final test we might compare the system QEMU with a reference
build as well as our current build.

Binary title
/usr/bin/qemu-aarch64 system-2.5.log
~/lsrc/qemu/qemu-builddirs/ master.log
~/lsrc/qemu/qemu.git/aarch64-linux-user/qemu-aarch64 softfloat-v4.log

Well that is certainly fairly explanatory. These are named org-mode tables which can be referred to in other code snippets and passed in as variables. So the next job is to run the benchmark itself:

2 Run Benchmark

This runs the benchmark against each binary we have selected above.

    import subprocess
    import os


    for qemu,logname in files:
    cmd="taskset -c 0 %s ./vector-benchmark -n %s | tee %s" % (qemu, tests, logname), shell=True)

        return runs

So why use python as the test runner? Well truth is whenever I end up munging arrays in shell script I forget the syntax and end up jumping through all sorts of hoops. Easier just to have some simple python. I use python again later to read the data back into an org-table so I can pass it to the next step, graphing:

set title "Vector Benchmark Results (lower is better)"
set style data histograms
set style fill solid 1.0 border lt -1

set xtics rotate by 90 right
set yrange [:]
set xlabel noenhanced
set ylabel "nsecs/Kop" noenhanced
set xtics noenhanced
set ytics noenhanced
set boxwidth 1
set xtics format ""
set xtics scale 0
set grid ytics
set term pngcairo size 1200,500

plot for [i=2:5] data using i:xtic(1) title columnhead

This is a GNU Plot script which takes the data and plots an image from it. org-mode takes care of the details of marshalling the table data into GNU Plot so all this script is really concerned with is setting styles and titles. The language is capable of some fairly advanced stuff but I could always pre-process the data with something else if I needed to.

Finally I need to upload my graph to an image hosting service to share with my colleges. This can be done with a elaborate curl command but I have another trick at my disposal thanks to the excellent restclient-mode. This mode is actually designed for interactive debugging of REST APIs but it is also easily to use from an org-mode source block. So the whole thing looks like a HTTP session:

:client_id = feedbeef

# Upload images to imgur
Authorization: Client-ID :client_id
Content-type: image/png

< benchmark.png

Finally because the above dumps all the headers when run (which is very handy for debugging) I actually only want the URL in most cases. I can do this simply enough in elisp:

#+name: post-to-imgur
#+begin_src emacs-lisp :var json-string=upload-to-imgur()
  (when (string-match
         (rx "link" (one-or-more (any "\":" whitespace))
             (group (one-or-more (not (any "\"")))))
    (match-string 1 json-string))

The :var line calls the restclient-mode function automatically and passes it the result which it can then extract the final URL from.

And there you have it, my entire benchmarking workflow document in a single file which I can read through tweaking each step as I go. This isn’t the first time I’ve done this sort of thing. As I use org-mode extensively as a logbook to keep track of my upstream work I’ve slowly grown a series of scripts for common tasks. For example every patch series and pull request I post is done via org. I keep the whole thing in a git repository so each time I finish a sequence I can commit the results into the repository as a permanent record of what steps I ran.

If you want even more inspiration I suggest you look at John Kitchen’s scimax work. As a publishing scientist he makes extensive use of org-mode when writing his papers. He is able to include the main prose with the code to plot the graphs and tables in a single source document from which his camera ready documents are generated. Should he ever need to reproduce any work his exact steps are all there in the source document. Yet another example of why org-mode is awesome 😉

by Alex at February 02, 2018 20:34

February 19, 2018

Marcin Juszkiewicz

Hotplug in VM. Easy to say…

You run VM instance. Nevermind is it part of OpenStack setup or just local one started using Boxes, virt-manager, virsh or other that kind of fronted to libvirt daemon. And then you want to add some virtual hardware to it. And another card and one more controller…

Easy to imagine scenario, right? What can go wrong, you say? “No more available PCI slots.” message can happen. On second/third card/controller… But how? Why?

Like I wrote in one of my previous posts most of VM instances are 90s pc hardware virtual boxes. With simple PCI bus which accepts several cards to be added/removed at any moment.

But not on AArch64 architecture. Nor on x86-64 with Q35 machine type. What is a difference? Both are PCI Express machines. And by default they have far too small amount of pcie slots (called pcie-root-port in qemu/libvirt language). More about PCI Express support can be found in PCI topology and hotplug page of libvirt documentation.

So I wrote a patch to Nova to make sure that enough slots will be available. And then started testing. Tried few different approaches, discussed with upstream libvirt developers about ways of solving the problem and finally we selected the one and only proper way of doing it. Then discussed failures with UEFI developers. And went for help to Qemu authors. And explained what I want to achieve and why to everyone in each of those four projects. At some point I had seen pcie-root-port things everywhere…

Turned out that the method of fixing it is kind of simple: we have to create whole pcie structure with root port and slots. This tells libvirt to not try any automatic adding of slots (which may be tricky if not configured properly as you may end with too small amount of slots for basic addons).

Then I went with idea of using insane values. VM with one hundred PCIe slots? Sure. So I made one, booted it and then something weird happen: landed in UEFI shell instead of getting system booted. Why? How? Where is my storage? Network? Etc?

Turns out that Qemu has limits. And libvirt has limits… All ports/slots went into one bus and memory for MMCONFIG and/or I/O space was gone. There are two interesting threads about it on qemu-devel mailing list.

So I added magic number into my patch: 28 — this amount of pcie-root-port entries in my aarch64 VM instance was giving me bootable system. Have to check it on x86-64/q35 setup still but it should be more or less the same. I expect this patch to land in ‘Rocky’ (the next OpenStack release) and probably will have to find a way to get it into ‘Queens’ as well because this is what we are planning to use for next edition of Linaro Developer Cloud.

Conclusion? Hotplug may be complicated. But issues with it can be solved.

by Marcin Juszkiewicz at February 02, 2018 18:06

February 15, 2018

Marcin Juszkiewicz

One-hit wonders and their other hits

There are so many musical bands and signers that not every one can get popular. Sometimes they are popular in their country/region but not necessary worldwide. Or they get one good song and nothing else gets such popularity. So called ‘one hit wonders’.

One of my friends recently shared “one hit wonders” playlist. But as it is with all those lists created during parties it contained several false entries which rather shown that someone did not know other hits for some bands. Anyway it was interesting enough to play in background.

Music was playing, letters were scrolling in terminal so I took a bit of time and created something more fancy: playlist with less known hits of ‘one-hit wonders’.

Sure, there are many missing entries and that some of listed artists/bands were more popular here and there. I am open for suggestions ;D

by Marcin Juszkiewicz at February 02, 2018 08:09

February 13, 2018

Riku Voipio

Making sense of /proc/cpuinfo on ARM

Ever stared at output of /proc/cpuinfo and wondered what the CPU is?

processor : 7
BogoMIPS : 2.40
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 3
Or maybe like:

$ cat /proc/cpuinfo
processor : 0
model name : ARMv7 Processor rev 2 (v7l)
BogoMIPS : 50.00
Features : half thumb fastmult vfp edsp thumbee vfpv3 tls idiva idivt vfpd32 lpae
CPU implementer : 0x56
CPU architecture: 7
CPU variant : 0x2
CPU part : 0x584
CPU revision : 2
The bits "CPU implementer" and "CPU part" could be mapped to human understandable strings. But the Kernel developers are heavily against the idea. Therefor, to the next idea: Parse in userspace. Turns out, there is a common tool almost everyone has installed does similar stuff. lscpu(1) from util-linux. So I proposed a patch to do ID mapping on arm/arm64 to util-linux, and it was accepted! So using lscpu from util-linux 2.32 (hopefully to be released soon) the above two systems look like:

Architecture: aarch64
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 2
NUMA node(s): 1
Vendor ID: ARM
Model: 3
Model name: Cortex-A53
Stepping: r0p3
CPU max MHz: 1200.0000
CPU min MHz: 208.0000
BogoMIPS: 2.40
L1d cache: unknown size
L1i cache: unknown size
L2 cache: unknown size
NUMA node0 CPU(s): 0-7
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid

$ lscpu
Architecture: armv7l
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
Vendor ID: Marvell
Model: 2
Model name: PJ4B-MP
Stepping: 0x2
CPU max MHz: 1333.0000
CPU min MHz: 666.5000
BogoMIPS: 50.00
Flags: half thumb fastmult vfp edsp thumbee vfpv3 tls idiva idivt vfpd32 lpae
As we can see, lscpu is quite versatile and can show more information than just what is available in cpuinfo.

by Riku Voipio ( at February 02, 2018 14:33

February 11, 2018

Siddhesh Poyarekar

Optimizing toolchains for modern microprocessors

About 2.5 years ago I left Red Hat to join Linaro in a move that surprised even me for the first few months. I still work on the GNU toolchain with a glibc focus, but my focus changed considerably. I am no longer looking at the toolchain in its entirety (although I do that on my own time whenever I can, either as glibc release manager or reviewer); my focus is making glibc routines faster for one specific server microprocessor; no prizes for guessing which processor that is. I have read architecture manuals in the past to understand specific behaviours but this is the first time that I have had to pore through the entire manual and optimization guides and try and eek out the last cycle of performance from a chip.

This post is an attempt to document my learnings and make a high level guide of the various things me and my team looked at to improve performance of the toolchain. Note that my team is continuing to work on this chip (and I continue to learn new techniques, I may write about it later) so this ‘guide’ is more of a personal journey. I may add more follow ups or modify this post to reflect any changes in my understanding of this vast topic.

All of my examples use ARM64 assembly since that’s what I’ve been working on and translating the examples to something x86 would have discouraged me enough to not write this at all.

What am I optimizing for?

CPUs today are complicated beasts. Vulnerabilities like Spectre allude to how complicated CPU behaviour can get but in reality it can get a lot more complicated and there’s never really a universal solution to get the best out of them. Due to this, it is important to figure out what the end goal for the optimization is. For string functions for example, there are a number of different factors in play and there is no single set of behaviours that trumps over all others. For compilers in general, the number of such combinations is even higher. The solution often is to try and ensure that there is a balance and there are no exponentially worse behaviours.

The first line of defence for this is to ensure that the algorithm used for the routine does not exhibit exponential behaviour. I wrote about algorithmic changes I did to the multiple precision fallback implementation in glibc years ago elsewhere so I’m not going to repeat that. I will however state that the first line of attack to improve any function must be algorithmic. Thankfully barring strcmp, string routines in glibc had a fairly sound algorithmic base. strcmp fall back to a byte comparison when inputs are not mutually aligned, which is now fixed.

Large strings vs small

This is one question that gets asked very often in the context of string functions and different developers have different opinions on it, some differences even leading to flamewars in the past. One popular approach to ‘solving’ this is to quote usage of string functions in a popular benchmark and use that as a measuring stick. For a benchmark like CPU2006 or CPU2017, it means that you optimize for smaller strings because the number of calls to smaller strings is very high in those benchmarks. There are a few issues to that approach:

  • These benchmarks use glibc routines for a very small fraction of time, so you’re not going to win a lot of performance in the benchmark by improving small string performance
  • Small string operations have other factors affecting it a lot more, i.e. things like cache locality, branch predictor behaviour, prefether behaviour, etc. So while it might be fun to tweak behaviour exactly the way a CPU likes it, it may not end up resulting in the kind of gains you’re looking for
  • A 10K string (in theory) takes at least 10 times more cycles than a 1K string, often more. So effectively, there is 10x more incentive to look at improving performance of larger strings than smaller ones.
  • There are CPU features specifically catered for larger sequential string operations and utilizing those microarchitecture quirks will guarantee much better gains
  • There are a significant number of use cases outside of these benchmarks that use glibc far more than the SPEC benchmarks. There’s no established set of benchmarks that represent them though.

I won’t conclude with a final answer for this because there is none. This is also why I had to revisit this question for every single routine I targeted, sometimes even before I decide to target it.

Cached or not?

This is another question that comes up for string routines and the answer is actually a spectrum - a string could be cached, not cached or partially cached. What’s the safe assumption then?

There is a bit more consensus on the answer to this question. It is generally considered safe to consider that shorter string accesses are cached and then focus on code scheduling and layout for its target code. If the string is not cached, the cost of getting it into cache far outweighs the savings through scheduling and hence it is pointless looking at that case. For larger strings, assuming that they’re cached does not make sense due to their size. As a result, the focus for such situations should be on ensuring that cache utilization is optimal. That is, make sure that the code aids all of the CPU units that populate caches, either through a hardware prefetcher or through judiciously placed software prefetch instructions or by avoiding caching altogether, thus avoiding evicting other hot data. Code scheduling, alignment, etc. is still important because more often than not you’ll have a hot loop that does the loads, compares, stores, etc. and once your stream is primed, you need to ensure that the loop is not suboptimal and runs without stalls.

My branch is more important than yours

Branch predictor units in CPUs are quite complicated and the compiler does not try to model them. Instead, it tries to do the simpler and more effective thing; make sure that the more probably branch target is accessible through sequential fetching. This is another aspect of the large strings vs small for string functions and more often than not, smaller sizes are assumed to be more probable for hand-written assembly because it seems to be that way in practice and also the cost of a mispredict hits the smaller size more than it does the larger one.

Don’t waste any part of a pig CPU

CPUs today are complicated beasts. Yes I know I started the previous section with this exact same line; they’re complicated enough to bear repeating that. However, there is a bit of relief in the fact that the first principles of their design hasn’t changed much. The components of the CPU are all things we heard about in our CS class and the problem then reduces to understanding specific quirks of the processor core. At a very high level, there are three types of quirks you look for:

  1. Something the core does exceedingly well
  2. Something the core does very badly
  3. Something the core does very well or badly under specific conditions

Typically this is made easy by CPU vendors when they provide documentation that specifies a lot of this information. Then there are cases where you discover these behaviours through profiling. Oh yes, before I forget:

Learn how to use perf or similar tool and read its output it will save your life

For example, the falkor core does something interesting with respect with loads and addressing modes. Typically, a load instruction would take a specific number of cycles to fetch from L1, more if memory is not cached, but that’s not relevant here. If you issue a load instruction with a pre/post-incrementing addressing mode, the microarchitecture issues two micro-instructions; one load and another that updates the base address. So:

   ldr  x1, [x2, 16]!

effectively is:

  ldr   x1, [x2, 16]
  add   x2, x2, 16

and that increases the net cost of the load. While it saves us an instruction, this addressing mode isn’t always preferred in unrolled loops since you could avoid the base address increment at the end of every instruction and do that at the end. With falkor however, this operation is very fast and in most cases, this addressing mode is preferred for loads. The reason for this is the way its hardware prefetcher works.

Hardware Prefetcher

A hardware prefetcher is a CPU unit that speculatively loads the memory location after the location requested, in an attempt to speed things up. This forms a memory stream and larger the string, the more its gains from prefetching. This however also means that in case of multiple prefetcher units in a core, one must ensure that the same prefetcher unit is hit so that the unit gets trained properly, i.e. knows what’s the next block to fetch. The way a prefetcher typically knows is if sees a consistent stride in memory access, i.e. it sees loads of X, X+16, X+32, etc. in a sequence.

On falkor the addressing mode plays an important role in determining which hardware prefetcher unit is hit by the load and effectively, a pre/post-incrementing load ensures that the loads hit the same prefetcher. That combined with a feature called register renaming ensures that it is much quicker to just fetch into the same virtual register and pre/post-increment the base address than to second-guess the CPU and try to outsmart it. The memcpy and memmove routines use this quirk extensively; comments in the falkor routines even have detailed comments explaining the basis of this behaviour.

Doing something so badly that it is easier to win

A colleague once said that the best targets for toolchain optimizations are CPUs that do things badly. There always is this one behaviour or set of behaviours that CPU designers decided to sacrifice to benefit other behaviours. On falkor for example, calling the MRS instruction for some registers is painfully slow whereas it is close to single cycle latency for most other processors. Simply avoiding such slow paths in itself could result in tremendous performance wins; this was evident with the memset function for falkor, which became twice as fast for medium sized strings.

Another example for this is in the compiler and not glibc, where the fact that using a ‘str’ instruction on 128-bit registers with register addressing mode is very slow on falkor. Simply avoiding that instruction altogether results in pretty good gains.

CPU Pipeline

Both gcc and llvm allow you to specify a model of the CPU pipeline, i.e.

  1. The number of each type of unit the CPU has. That is, the number of load/store units, number of integer math units, number of FP units, etc.
  2. The latency for each type of instruction
  3. The number of micro-operations each instruction splits into
  4. The number of instructions the CPU can fetch/dispatch in a single cycle

and so on. This information is then used to sequence instructions in a function that it optimizes for. This may also help the compiler choose between instructions based on how long those take. For example, it may be cheaper to just declare a literal in the code and load from it than to construct a constant using mov/movk. Similarly, it could be cheaper to use csel to select a value to load to a register than to branch to a different piece of code that loads the register or vice versa.

Optimal instruction sequencing can often result in significant gains. For example, intespersing load and store instructions with unrelated arithmetic instructions could result in both those instructions executing in parallel, thus saving time. On the contrary, sequencing multiple load instructions back to back could result in other units being underutilized and all instructions being serialized on to the load unit. The pipeline model allows the compiler to make an optimal decision in this regard.

Vector unit - to use or not to use, that is the question

The vector unit is this temptress that promises to double your execution rate, but it doesn’t come without cost. The most important cost is that of moving data between general purpose and vector registers and quite often this may end up eating into your gains. The cost of the vector instructions themselves may be high, or a CPU might have multiple integer units and just one SIMD unit, because of which code may get a better schedule when executed on the integer units as opposed to via the vector unit.

I had seen an opposite example of this in powerpc years ago when I noticed that much of the integer operations were also implemented in FP in multiple precision math. This was because the original authors were from IBM and they had noticed a significant performance gain with that on powerpc (possible power7 or earlier given the timelines) because the CPU had 4 FP units!

Final Thoughts

This is really just the tip of the iceberg when it comes to performance optimization in toolchains and utilizing CPU quirks. There are more behaviours that could be exploited (such as aliasing behaviour in branch prediction or core topology) but the cost benefit of doing that is questionable.

Despite how much fun it is to hand-write assembly for such routines, the best approach is always to write simple enough code (yes, clever tricks might actually defeat compiler optimization passes!) that the compiler can optimize for you. If there are missed optimizations, improve compiler support for it. For glibc and aarch64, there is also the case of impending multiarch explosion. Due to the presence of multiple vendors, having a perfectly tuned routine for each vendor may pose code maintenance problems and also secondary issues with performance, like code layout in a binary and instruction cache utilization. There are some random ideas floating about for that already, like making separate text sections for vendor-specific code, but that’s something we would like to avoid doing if we can.

by Siddhesh at February 02, 2018 19:37

Naresh Bhat

A dream come true: Himalayan Odyssey - 2016 (Day-0 to 5)


THE HIMALAYAS as most everyone knows are the highest mountains in the world, with 30 peaks over 24,000 feet. The adventure of a lifetime doesn't get much bigger or higher than riding and chasing mountains of the Himalayas.

The Royal Enfield (RE) motorcycles are manufactured and sold in INDIA since 1907. These motorcycles are best suited for INDIAN road conditions. These motorcycles are used by INDIAN ARMY from the period of second world war.

There is a saying "FOUR WHEELS MOVE THE BODY-BUT TWO WHEELS MOVE THE SOUL". I am a motorcycle enthusiast from my childhood days. I always had dreams to own a RE motorcycle after getting into a job. Right now I own two variants of RE motorcycles, “Royal Enfield Thunderbird Twinspark” (TBTS) and “Squadron Blue Classic Dispatch” which is a Limited Edition.

Thunder Bird Twin Spark 350cc 2011 model

Squadron Blue Dispatch 500cc  2015 model

The TBTS is 350CC, good for cruising on long stretched highways.  The dispatch is 500CC EFI engine which gives quick response to throttle.  Hence I decided to take classic 500CC motorcycle for Himalayan Odyssey (HO).

In INDIA, Royal Enfield conducts different motorcycling tours e.g. HO, Tour of Tibet, Tour of Nepal, Tour of Rajasthan..etc.  But out of all these tours it is  considered that HO is the toughest one.  The reason is very simple, riding on Himalayan mountains are not that easy by considering road conditions, unpredictable weather, high altitudes..etc.  The Himalayan mountain roads are completely shutdown for 6 months. The INDIAN army clears the snow, opens and maintains it for another 6 months. Every year, army announces the open and close dates.

From past 12 years RE is conducting HO. I took part in HO-2016, the 13th HO - “18 DAYS LIKE NO OTHER IN RIDING” . It was conducted between 6th to 23rd July 2016. Our group had 70 men and 14 women from all over the world.  The men and women odyssey route were different, but they meet at LADAKH. Again take separate route and meet last day celebration party in Chandigarh. Men's group route map is as below.

HO Preparation:

It takes lot efforts to convince our family and making suitable arrangements at office. I was planning HO ride from last 5 years by accumulating leaves. I was trying to physically be fit as much as possible by doing exercises on regular basis.  After doing registration it is required to go through physical fitness test and submit those documents.  The physical fitness test includes 5KM run and 50 push-ups in 45 min.  There is also a physical fitness certificate from local doctor and you need to submit to RE. Documents to be submitted include medical test reports for blood, urine and Treadmill test (TMT), Medical history by self, Medical check-up fitness certificate by doctor and Indemnity bond.

The HO team includes a doctor, backup van, mechanics, media people, 3-4 lead riders from RE etc. All the information will be communicated to you post registration.

The HO ride starts from Delhi and ends at Chandigarh.  I am located in  Bangalore and hence I also had to plan to reach Delhi on July 7th with my Motorcycle.  I knew I would need 3 days to reach Delhi from Bangalore via road.  Since I had  very limited amount of time, I planed to ship my motorcycle via containers and fly to Delhi.  The transport of my motorcycle I coasted INR Rs.5780.00 one way. Actually the cost of transportation of my motorcycle was more than my air tickets 😅.  The flight tickets round trip cost INR Rs. 7000.00.  Once you register for the HO trip they will include you in closed facebook, whatsapp group.  It will be very easy to discuss all your questions in those groups.

Ready to ship
I used VRL logistics (Vijayanand Road Lines) to ship my bike from Banglore to Delhi.  Many of you may ask why can't I just rent a motorcycle at Delhi ?  This is just because if I ride my own motorcycle in mountains,  I will understand my motorcycle in a better way and the personal attachment with motorcycle will be more.  That's the reason RE suggests to take our own motorcycle for any rides.
locked in container
Luggage types and split-up:

When we start our ride, our overall luggage will be split into two. 

1. The luggage that we carry on the motorcycle, We call it as “satellite luggage”
    A duffel bag is a good choice. You can fasten it to you motorcycle using bungee cords or luggage straps. Remember to waterproof this bag well as this is exposed to the elements of outside nature whatever terrain you ride.  Packing of this bag is very crucial,  Distribute the weight evenly. If there is some space left in the bag use compression straps to ensure stuff does not move around inside the bag.  Tie the bag after checking its placement thoroughly and do so only on the centre stand. We will end up doing this even at camp sites and finding a flat piece of land could be tricky, use stones to ensure that your motorcycle is as upright as possible when you're fastening your luggage.  It is very tempting to use saddle bags for satellite luggage, but this will leave you with more empty space. Avoid starting the trip with saddlebags on your bike and then shifting them to the luggage vehicle.

What my satellite luggage will definitely have
1. A change of clothes- a pair of denims/cargos, a T-shirt and a casual jacket
2. A hat
3. A pair of running shoes
4. Winter gloves - depending on where we're on the Odyssey
5. Toiletries - I'll have my lip balm/guard and sunscreen
6. GoPro, some mounts, batteries and a power bank
7. a Beanie or a woolen buff
8. a Torch
9. Spare cables and a tube

2. The luggage that is carried in the luggage vehicle.
     This will be minus the riding gear that you bring, as that will be worn by you for the duration of the entire ride.  This luggage will have to be restricted to one piece per rider with a max limit of 15 kilos.  Why 15 kilos? After you have removed all the gear and your satellite luggage, we have found that this is comfortable cut-off. This is also a comfortable weight for you to carry to your rooms, unload/load to the luggage vehicle every day. This luggage will need to be loaded and unloaded every day and in case of rain, the bags can get soiled and wet. It is best that we use some level of waterproofing so as to safeguard what's in the bags. a waterproof cover or waterproofing from the inside could do the job.


Everybody needs to reach two days before the HO trip.  They will book the accommodation you. The very first day I just did a check-in and collected my motorcycles at Delhi.  

The next day schedule was as below

Flag Off day and complete Itinerary:

The 13th edition of the Royal Enfield Himalayan Odyssey will flag off from New Delhi on 9th July 2016. This is a call to all those who love to ride on tough and testing terrain and have the passion to ride with RE. In the year 2016 will see 75 riders riding on one of the most spectacular motorcycle journeys in the world. 

Here is our detailed itinerary

Day-1: Delhi To Chandigarh

The first day started as below

5 AM luggage loading - HO
6:30 AM - breakfast
7:15 AM HO start to India gate
Let this begin!

Group photo @INDIA Gate
The first day ride always starts from India Gate, Delhi.  We took a group photo and did some Buddhist
 rituals and prayed for a safe ride. The briefing includes regroup point, road conditions and some common
mistakes committed by riders.

We were 12 people from Karnataka State and grouped together to take some group photos.

Karnataka State riders

The flag off is done by RE sales director.  The video just after flagoff there are some news channel coverage:  Auto Today  and NDTV

Chandigarh, the capital of the northern Indian states of Punjab and Haryana, was designed by the Swiss-French modernist architect, Le Corbusier.  Chandigarh is a city and a union territory in India that serves as the capital of both neighboring states of Haryana and Punjab. The city is not part of either of the two states and is governed directly by the Union Government, which administers all such territories in the country.

Afternoon we reached Chandigarh and checked-in into hotel. The Chandigarh is a very well planned and beautiful city. The city is having lot of tree's and parks. So we did  a quick tour of couple of places in the city.

Day-2: Chandigarh To Manali

In HO every day is a learning day.  You will become much more closer to your motorcycle each day.  In another words you will understand the motorcycle handling in a better way.   The day starts with luggage loading, breakfast, briefing and ride out.  The time which are followed same on each day.

The briefing will be about 10-15 min.  This is very important for a rider.  Because the briefing contains about the kind of road you are going to ride on that day and important riding tips.
We reached Manali Highland hotel by 5PM.  We visited local market to purchase  required items fir the ride.  This will be the last city on our onward journey to Leh.  After Manali, the real ride will start. There will be less tarmac and more rough roads.  After Manali you will see all shops in tents till you reach Leh.  I also met couple of cyclist who were cycling up to Leh.

Cyclists @Manali hotel
Manali is a high-altitude Himalayan resort town in India’s northern Himachal Pradesh state. It has a reputation as a backpacking center and honeymoon destination. Set on the Beas River, it’s a gateway for skiing in the Solang Valley and trekking in Parvati Valley. It's also a jumping-off point for paragliding, rafting and mountaineering in the Pir Panjal mountains, home to 4,000m-high Rohtang Pass.

Day-3: Manali To Keylong (Jispa)

The road from Manali to Rothang pass is a single road.  Although it had tarmac, it was not in a good condition.  We took a break at Rothang pass base camp.
Base camp
We started slowly climbing the pass.  I could feel that thin air and altitude change.  My motorcycle was also giving slow response to the throttle.  The machine also need oxygen for the combustion.   The weather on Rothang pass will change every 15 min.  The last leg climb was very foggy and hardly I could see the road.
Rothang Pass roads
After couple of kms it was very sunny and bright.  We were warned not to stay more than 10 min at high altitude region.
Top of Rothang Pass
We just took couple of photos and started descending the Rothang pass. It is good that after crossing the Rothang pass, the road is completely empty and traffic free.  You can only see some Indian Army trucks or some goods carriages on the road. But suddenly, the road becomes too rough, dusty.After travelling few kms on these rough roads my motorcycle started behaving in a weird way.  The headlight, horn and indicators stopped working.  Hence I stopped by to check the problem. Fortunately, there I spotted one of the RE HO trip co-coordinators.  He just did a basic check and identified that a fuse is burned.  In couple of minutes he replaced the fuse which is readily available in the side panel box.  I continued my ride till the lunch break.

Lunch time..:)

Dusty roads on the way to Tandi
At some places the roads were under construction.  Since they had put wet mud with stones, it was very difficult to handle the motorcycle which is of around 200 KG weight.
Road construction
Finally reached Tandi fuel pump. Filled up the tank full, since there will not be any filling station till next 365KM.
Tandi gas station
The rough and dusty road continues.  At some places the dust settled on the road might be nearly 10-15 cms too.

We continued to ride and reached Jispa camp.  The river was flowing just behind our tents.  It is really a heaven on the earth.  Very beautiful village.
Jispa camp

Our Tent
We had snacks and had hangout.  Evening onwards it was too cold because of wind and  the cold river was just behind our tents.  I was feeling like I could have taken room instead of tent. That was purely our mistake since we reached early we garbed a tent to stay.

Day-4: Keylong (Jispa) To Sarchu

Jispa is a village in Lahaul, in the Indian state of Himachal Pradesh. Jispa is located 20 km north of Keylong and 7 km south of Darcha, along the Manali-Leh Highway and the Bhaga river. There are approximately 20 villages between Jispa and Keylong.

The briefing we were given instructions on how to do water crossing.  In all the water crossing there will be small pebbles and water very chilled. One should make sure that the motorcycle tires should not get stuck between these small stone beds.
Ready to leave Jispa valley
The distance between Jispa to Sarchu is very less.  It makes difficult to ride on no road terrain. We finished the morning briefing and started riding.
Briefing @Jispa
We crossed couple of water streams before reaching Sarchu. The technique to cross water stream is very simple.  First you should hold your motorcycle tanks tightly with knee's.  Next you have to free your upper body, give focus and look ahead on water flowing road and give the throttle.
Riding beside Bhaga river

Water crossing
Valley view
Lunch break
We had a break for the lunch.  I had some noodles.  You will not get any other kind of foods in these tents other than omlet, noodles, plain rice.

We reached Sarchu very early around 3-4PM.  But after reaching Sarchu with-in 15-20mins the headache started. Almost all had mountain sickness.  Acute Mountain Sickness (AMS) is the mildest form and it's very common. The symptoms can feel like a hangover – dizziness, headache, muscle aches, nausea. Camp doctors checked heartbeats for all affected people.

We were unable to eat anything,  could not sleep or take rest.  Even if we walk for 100mtr we were unable to breath.  That day was a very horrible day which I will never forget in my life.

Sarchu camp
We were having again tented accommodation.  There will be only solar charged lights. There will not be any army hospitals nearby.  After sun goes there will be sudden drop in temperature. It felt like the situation was life threatening.

Sarchu is a major halt point with tented accommodation in the Himalayas on the Leh-Manali Highway, on the boundary between Himachal Pradesh and Ladakh (Jammu and Kashmir) in India. It is situated between Baralacha La to the south and Lachulung La to the north, at an altitude of 4,290 m (14,070 ft).

Day-5: Sarchu To Leh

I was very eager to start from Sarchu. The stay at high altitude and very cold weather I could not get a good sleep.  The RE guys bought petrol (gas) in backup van.  All of us queued up to top up petrol. The stay at Sarchu tent was the most uncomfortable stay.  But it is true once you get acclimatize to Sarchu altitude,  you are more prepared to travel further.
I  shifted my satellite luggage to backup van. As the experience speaks, it is very uncomfortable to ride with saddle bags on the motorcycle. After Sarchu the roads are open, no traffic for several kms. I was going alone and stoped to take pictures.  When I reached "Gata Loops" bottom, couple of my friends joined.

GATA Loops begin
Gata Loops is a name that is unknown to everyone except for a few who have traveled on the Manali Leh highway; or planning to do so. It is a series of twenty one hairpin bends that takes you to the top of the 3rd high altitude pass on this highway, Nakeela; at a height of 15,547 ft.
More (Mo-ray) plains
I have covered hundreds of mountain miles but never seen a plateau. When I came upon the More (pronounced ‘mo-ray’) Plains, they were much bigger than what I’d visualized of plateaus from school geography books.
They are endless. Well, 50 km of flatlands at an elevation of 15,000 feet deserves that epithet! And they are flat, for miles after miles, till they run into the surrounding mountains.  Camp here for the evening and you’ll see the most stunning of sunsets. The area is surprisingly active here. You will always have workers building or repairing roads.

We continue the ride towards Leh after taking few pics at More plains.  We were pass through Pang, Meroo, Debring and at Rumtse we had a lunch break.  The Indus river flows parallel to road and other side steep cliff of the mountains.  I remember each mountain had of different colors after Debring. By evening we reached Leh and check-into hotel "Namgyal Palace". 


by Naresh ( at February 02, 2018 06:46

February 07, 2018

Marcin Juszkiewicz

Developers planet is online

People write blogs. People read blogs. But sometimes it is hard to find blogs of all those interesting people. That’s where so called “planets” are solution.

Years ago there was “Planet Linaro” website filled with blog posts from Linaro developers. Then it vanished. Later it got replaced by poor substitute.

But I do not want to have to track each Linaro developer to find their blog and add it into Feedly. So instead I decided to create new planet website. And that’s how Developers Planet got born.

So far it lists a bunch of blogs of Linaro developers. I used venus to run it. Few years old code but runs. Will adapt HTML/CSS template to be a bit more modern.

And why .cf domain? It is free — that’s why.

by Marcin Juszkiewicz at February 02, 2018 08:09

February 06, 2018

Marcin Juszkiewicz

Graphical console in OpenStack/aarch64

OpenStack users are used to have graphical console available. They take it for granted even. But we lacked it…

When we started working on OpenStack on 64-bit ARM there were many things missing. Most of them got sorted out already. One thing was still in a queue: graphical console. So two weeks ago I started looking at the issue.

Whenever someone tried to use it Nova reported one simple message: “No free USB ports.” You can ask what it has to console? I thought similar and started digging…

As usual reason was simple: yet another aarch64<>x86 difference in libvirt. Turned out that arm64 is one of those few architectures which do not have USB host controller in default setup. When Nova is told to provide graphical console it adds Spice (or VNC) graphics, video card and USB tablet. But in our case VM instance does not have any USB ports so VM start fails with “No free USB ports” message.

Solution was simple: let’s add USB host controller into VM instance. But where? Should libvirt do that or should Nova? I discussed it with libvirt developers and got mixed answers. Opened a bug for it and went to Nova hacking.

Turned out that Nova code for building guest configuration is not that complicated. I created a patch to add USB host controller and waited for code reviews. There were many suggestions, opinions and questions. So I rewrote code. And then again. And again. Finally 15th version got “looks good” opinion from all reviewers and got merged.

And how result looks? Boring as it should:

OpenStack graphical console on aarch64

by Marcin Juszkiewicz at February 02, 2018 19:16

Alex Bennée


I’ve just returned from a weekend in Brussels for my first ever FOSDEM – the Free and Open Source Developers, European Meeting. It’s been on my list of conferences to go to for some time and thanks to getting my talk accepted, my employer financed the cost of travel and hotels. Thanks to the support of the Université libre de Bruxelles (ULB) the event itself is free and run entirely by volunteers. As you can expect from the name they also have a strong commitment to free and open source software.

The first thing that struck me about the conference is how wide ranging it was. There were talks on everything from the internals of debugging tools to developing public policy. When I first loaded up their excellent companion app (naturally via the F-Droid repository) I was somewhat overwhelmed by the choice. As it is a free conference there is no limit on the numbers who can attend which means you are not always guarenteed to be able to get into every talk. In fact during the event I walked past many long queues for the more popular talks. In the end I ended up just bookmarking all the talks I was interested in and deciding which one to go to depending on how I felt at the time. Fortunately FOSDEM have a strong archiving policy and video most of their talks so I’ll be spending the next few weeks catching up on the ones I missed.

There now follows a non-exhaustive list of the most interesting ones I was able to see live:

Dashamir’s talk on EasyGPG dealt with the opinionated decisions it makes to try and make the use of GnuPG more intuitive to those not versed in the full gory details of public key cryptography. Although I use GPG mainly for signing GIT pull requests I really should make better use it over all. The split-key solution to backups was particularly interesting. I suspect I’ll need a little convincing before I put part of my key in the cloud but I’ll certainly check out his scripts.

Liam’s A Circuit Less Travelled was an entertaining tour of some of the technologies and ideas from early computer history that got abandoned on the wayside. These ideas were often to be re-invented in a less superior form as engineers realised the error of their ways as technology advanced. The later half of the talk turns into a bit of LISP love-fest but as an Emacs user with an ever growing config file that is fine by me 😉

Following on in the history vein was Steven Goodwin’s talk on Digital Archaeology which was a salutatory reminder of the amount of recent history that is getting lost as computing’s breakneck pace has discarded old physical formats in lieu of newer equally short lived formats. It reminded me I should really do something about the 3 boxes of floppy disks I have under my desk. I also need to schedule a visit to the Computer History Museum with my children seeing as it is more or less on my doorstep.

There was a tongue in check preview that described the EDSAC talk as recreating “an ancient computer without any of the things that made it interesting”. This was was a little unkind. Although the project re-implemented the computation parts in a tiny little FPGA the core idea was to introduce potential students to the physicality of the early computers. After an introduction to the hoary architecture of the original EDSAC and the Wheeler Jump Mary introduced the hardware they re-imagined for the project. The first was an optical reader developed to read in paper tapes although this time ones printed on thermal receipt paper. This included an in-depth review of the problems of smoothing out analogue inputs to get reliable signals from their optical sensors which mirrors the problems the rebuild is facing with nature of the valves used in EDSAC. It is a shame they couldn’t come up with some way to involve a valve but I guess high-tension supplies and school kids don’t mix well. However they did come up with a way of re-creating the original acoustic mercury delay lines but this time with a tube of air and some 3D printed parabolic ends.

The big geek event was the much anticipated announcement of RISC-V hardware during the RISC-V enablement talk. It seemed to be an open secret the announcement was coming but it still garnered hearty applause when it finally came. I should point out I’m indirectly employed by companies with an interest in a competing architecture but it is still good to see other stuff out there. The board is fairly open but there are still some peripheral IPs which were closed which shows just how tricky getting to fully-free hardware is going to be. As I understand the RISC-V’s licensing model the ISA is open (unlike for example an ARM Architecture License) but individual companies can still have closed implementations which they license to be manufactured which is how I assume SiFive funds development. The actual CPU implementation is still very much a black box you have to take on trust.

Finally for those that are interested my talk is already online for those that are interested in what I’m currently working on. The slides have been slightly cropped in the video but if you follow the link to the HTML version you can read along on your machine.

I have to say FOSDEM’s setup is pretty impressive. Although there was a volunteer in each room to deal with fire safety and replace microphones all the recording is fully automated. There are rather fancy hand crafted wooden boxes in each room which take the feed from your laptop and mux it with the camera. I got the email from the automated system asking me to review a preview of my talk about half and hour after I gave it. It took a little longer for the final product to get encoded and online but it’s certainly the nicest system I’ve come across so far.

All in all I can heartily recommend FOSDEM for anyone in an interest is FLOSS. It’s a packed schedule and there is going to be something for everyone there. Big thanks to all the volunteers and organisers and I hope I can make it next year 😉

by Alex at February 02, 2018 09:36

February 01, 2018

Marcin Juszkiewicz

Everyone loves 90s PC hardware?

Do you know what is the most popular PC machine nowadays? It is “simple” PCI based x86(-64) machine with i440fx chipset and some expansion cards. Everyone is using them daily. Often you do not even realise that things you do online are handled by machines from 90s.

Sounds like a heresy? Who would use 90s hardware in modern world? PCI cards were supposed to be replaced by PCI Express etc, right? No one uses USB 1.x host controllers, hard to find PS/2 mouse or keyboard in stores etc. And you all are wrong…

Most of virtual machines in x86 world is a weird mix of 90s hardware with a bunch of PCI cards to keep them usable in modern world. Parallel ATA storage went to trash replaced by SCSI/SATA/virtual ones. Graphic card is usually simple framebuffer without any 3D acceleration (so like 90s) and typical PS/2 input devices connected. And you have USB 1.1 and 2.0 controllers with one tablet connected. Sounds like retro machines my friends prepare for retro events.

You can upgrade to USB 3.0 controller, graphic card with some 3D acceleration. Or add more memory and cpu cores that i440fx based PC owner ever dreamt about. But it is still 90s hardware.

Want to have something more modern? You can migrate to PCI Express. But nearly no one does that in x86 world. And in AArch64 world we start from here.

And that’s the reason why working with developers of projects related to virtualization (qemu, libvirt, openstack) can be frustrating.

Hotplug issues? Which hotplug issues? My VM instance allows to plug 10 cards while it is running so where is a problem? The problem is that your machine is 90s hardware with simple PCI bus and 31 slots present on virtual mainboard while VM instance with PCI Express (aarch64, x86 with q35 model) has only TWO free slots present on motherboard. And once they are used no new slots arrive. Unless you shutdown, add free slots and power up again.

Or my recent stuff: adding USB host controller. x86 has it because someone ‘made a mistake’ in past and enabled it. But other architectures got sorted out and did not get it. Now all of them need to have it added when it is needed. Have a patch to Nova for it and I am rewriting it again and again to get it into acceptable form.

Partitioning is fun too. There are so many people who fear switching to GPT…

by Marcin Juszkiewicz at February 02, 2018 09:40

January 23, 2018

Leif Lindholm

Fun and games with gnu-efi

gnu-efi is a set of scripts, libraries, header files and code examples to make it possible to write applications and drivers for the UEFI environment directly from your POSIX world. It supports i386, Ia64, X64, ARM and AArch64 targets ... but it would be dishonest to say it is beginner friendly in its current state. So let's do something about that.

Rough Edges

gnu-efi comes packaged for most Linux distributions, so you can simply run

$ sudo apt-get install gnu-efi


$ sudo dnf install gnu-efi gnu-efi-devel

to install it. However, there is a bunch of Makefile boilerplate that is not covered by said packaging, meaning that getting from "hey, let's check this thing out" to "hello, world" involves a fair bit of tedious makefile hacking.

... serrated?

Also, the whole packaging story here is a bit ... special. It means installing headers and libraries into /usr/lib and /usr/include solely for the inclusion into images to be executed by the UEFI firmware during Boot Services, before the operating system is running. And don't get me started on multi-arch support.


Like most other programming languages, Make supports including other source files into the current context. The gnu-efi codebase makes use of this, but not in a way that's useful to a packaging system.

Now, at least GNU Make looks in /usr/include and /usr/local/include as well as the current working directory and any directories specified on the command line with -L. This means we can stuff most of the boilerplate in makefile fragments and include where we need them.

Hello World

So, let's start with the (almost) most trivial application imaginable:

#include <efi/efi.h>
#include <efi/efilib.h>

    EFI_HANDLE image_handle,
    EFI_SYSTEM_TABLE *systab
    InitializeLib(image_handle, systab);

    Print(L"Hello, world!\n");

    return EFI_SUCCESS;

Save that as hello.c.

Reducing the boiler-plate

Now grab Make.defaults and Make.rules from the gnu-efi source directory and stick them in a subdirectory called efi/.

Then download this I prepared earlier, and include it in your Makefile:


ifeq ($(HAVE_EFI_OBJCOPY), y)
FORMAT := --target efi-app-$(ARCH)      # Boot time application
#FORMAT := --target efi-bsdrv-$(ARCH)   # Boot services driver
#FORMAT := --target efi-rtdrv-$(ARCH)   # Runtime driver
#SUBSYSTEM=$(EFI_SUBSYSTEM_BSDRIVER)    # Boot services driver

all: hello.efi

    rm -f *.o *.so *.efi *~

The hello.efi dependency for the all target invokes implicit rules (defined in Make.rules) to generate hello.efi from, which is generated by an implicit rule from hello.o, which is generated by an implicit rule from hello.c.

NOTE: there are two bits of boiler-plate that still need addressing.

First of all, in, GNUEFI_LIBDIR needs to be manually adjusted to fit the layout implemented by your distribution. Template entries for Debian and Fedora are provided.

Secondly, the bit of boiler-plate we cannot easily get rid of - we need to inform the toolchain about whether the desired output is an application, a boot-time driver or a runtime driver. Templates for this is included in the Makefile snippet above - but note that different options must currently be set for toolchains where objcopy supports efi- targets directly and ones where it does not.

Building and running

Once the build environment has been completed, build the project as you would with any regular codebase.

$ make
gcc -I/usr/include/efi -I/usr/include/efi/x86_64 -I/usr/include/protocol -mno-red-zone -fpic  -g -O2 -Wall -Wextra -Werror -fshort-wchar -fno-strict-aliasing -fno-merge-constants -ffreestanding -fno-stack-protector -fno-stack-check -DCONFIG_x86_64 -DGNU_EFI_USE_MS_ABI -maccumulate-outgoing-args --std=c11 -c hello.c -o hello.o
ld -nostdlib --warn-common --no-undefined --fatal-warnings --build-id=sha1 -shared -Bsymbolic /usr/lib/crt0-efi-x86_64.o -L /usr/lib64 -L /usr/lib /usr/lib/gcc/x86_64-linux-gnu/6/libgcc.a -T /usr/lib/ hello.o -o -lefi -lgnuefi
objcopy -j .text -j .sdata -j .data -j .dynamic -j .dynsym -j .rel \
        -j .rela -j .rel.* -j .rela.* -j .rel* -j .rela* \
        -j .reloc --target efi-app-x86_64 hello.efi
rm hello.o

Then get the resulting application (hello.efi) over to a filesystem accessible from UEFI and run it.

UEFI Interactive Shell v2.2
UEFI v2.60 (EDK II, 0x00010000)
Mapping table
FS0: Alias(s):HD1a1:;BLK3:
BLK2: Alias(s):
BLK4: Alias(s):
BLK0: Alias(s):
BLK1: Alias(s):
Press ESC in 5 seconds to skip startup.nsh or any other key to continue.
Shell> fs0:
FS0:\> hello
Hello, world!

Wohoo, it worked! (I hope.)


gnu-efi provides a way to easily develop drivers and applications for UEFI inside your POSIX environment, but it comes with some unnecessarily rough edges. Hopefully this post makes it easier for you to get started with developing real applications and drivers using gnu-efi quickly.

Clearly, we should be working towards getting this sort of thing included in upstream and installed with distribution packages.

by Leif Lindholm at January 01, 2018 16:07

Ard Biesheuvel

Per-task stack canaries for arm64

Due to the way the stack of a thread (or task in kernelspeak) is shared between control flow data (frame pointer, return address, caller saved registers) and temporary buffers, overflowing such buffers can completely subvert the control flow of a program, and the stack is therefore a primary target for attacks. Such attacks are referred to as Return Oriented Programming (ROP), and typically consist of a specially crafted array of forged stack frames, where each return from a function is directed at another piece of code (called a gadget) that is already present in the program. By piecing together gadgets like this, powerful attacks can be mounted, especially in a big program such as the kernel where the supply of gadgets is endless.

One way to mitigate such attacks is the use of stack canaries, which are known values that are placed inside each stack frame when entering a function, and checked again when leaving the function. This forces the attacker to craft his buffer overflow attack in a way that puts the correct stack canary value inside each stack frame. That by itself is rather trivial, but it does require the attacker to discover the value first.

GCC support

GCC implements support for stack canaries, which can be enabled using the various ‑fstack-protector[‑xxx] command line switches. When enabled, each function prologue will store the value of the global variable __stack_chk_guard inside the stack frame, and each epilogue will read the value back and compare it, and branch to the function __stack_chk_fail if the comparison fails.

This works fine for user programs, with the caveat that all threads will use the same value for the stack canary. However, each program will pick a random value at program start, and so this is not a severe limitation. Similarly, for uniprocessor (UP) kernels, where only a single task will be active at the same time, we can simply update the value of the __stack_chk_guard variable when switching from one task to the next, and so each task can have its own unique value.

However, on SMP kernels, this model breaks down. Each CPU will be running a different task, and so any combination of tasks could be active at the same time. Since each will refer to __stack_chk_guard directly, its value cannot be changed until all tasks have exited, which only occurs at a reboot. Given that servers don’t usually reboot that often, leaking the global stack canary value can seriously compromise security of a running system, as the attacker only has to discover it once.

x86: per-CPU variables

To work around this issue, Linux/x86 implements support for stack canaries using the existing Thread-local Storage (TLS) support in GCC, which replaces the reference to __stack_chk_guard with a reference to a fixed offset in the TLS block. This means each CPU has its own copy, which is set to the stack canary value of that CPU’s current task when it switches to it. When the task migrates, it just takes its stack canary value along, and so all tasks can use a unique value. Problem solved.

On arm64, we are not that lucky, unfortunately. GCC only supports the global stack canary value, although discussions are underway to decide how this is best implemented for multitask/thread environments, i.e., in a way that works for userland as well as for the kernel.

Per-CPU variables and preemption

Loading the per-CPU version of __stack_chk_guard could look something like this on arm64:

adrp    x0, __stack__chk_guard
add     x0, x0, :lo12:__stack_chk_guard
mrs     x1, tpidr_el1
ldr     x0, [x0, x1]

There are two problems with this code:

  • the arm64 Linux kernel implements support for Virtualization Host Extensions (VHE), and uses code patching to replace all references to tpidr_el1 with tpidr_el2 on VHE capable systems,
  • the access is not atomic: if this code is preempted after reading the value of tpidr_el1 but before loading the stack canary value, and is subsequently migrated to another CPU, it will load the wrong value.

In kernel code, we can deal with this easily: every emitted reference to tpidr_el1 is tagged so we can patch it at boot, and on preemptible kernels we put the code in a non-preemtible block to make it atomic. However, this is impossible to do in GCC generated code without putting elaborate knowledge of the kernel’s per-CPU variable implementation into the compiler, and doing so would severely limit our future ability to make any changes to it.

One way to mitigate this would be to reserve a general purpose register for the per-CPU offset, and ensure that it is used as the offset register in the ldr instruction. This addresses both problems: we use the same register regardless of VHE, and the single ldr instruction is atomic by definition.

However, as it turns out, we can do much better than this. We don’t need per-CPU variables if we can load the task’s stack canary value directly, and each CPU already keeps a pointer to the task_struct of the current task in system register sp_el0. So if we replace the above with

movz    x0, :abs_g0:__stack__chk_guard_offset
mrs     x1, sp_el0
ldr     x0, [x0, x1]

we dodge both issues, since all of the values involved are per-task values which do not change when migrating to another CPU. Note that the same sequence could be used in userland for TLS if you swap out sp_el0 for tpidr_el0 (and use the appropriate relocation type), so adding support for this to GCC (with a command line configurable value for the system register) would be a flexible solution to this problem.

Proof of concept implementation

I implemented support for the above, using a GCC plugin to replace the default sequence

adrp    x0, __stack__chk_guard
add     x0, x0, :lo12:__stack_chk_guard
ldr     x0, [x0]


mrs     x0, sp_el0
add     x0, x0, :lo12:__stack_chk_guard_offset
ldr     x0, [x0]

This limits __stack_chk_guard_offset to 4 KB, but this is not an issue in practice unless struct randomization is enabled. Another caveat is that it only works with GCC’s small code model (the one that uses adrp instructions) since the plugin works by looking for those instructions and replacing them.

Code can be found here.

by ardbiesheuvel at January 01, 2018 11:12

January 19, 2018

Marcin Juszkiewicz FOSDEM? PTG? Linaro Connect?

What connects those names? All of them are conference or team meeting names. Spread over Europe + Asia. And I will be on all of them this year.

First will be in Brno, Czechia. Terrible city to travel to but amount of people I can meet there is enormous. Some guys from my Red Hat team will be there, my boss’ boss (and boss of my boss’ boss) and several people from CentOS, Fedora, RHEL (and some other names) communities. Meetings, sessions… Nice to be there. Will be in A-Sport hotel as it is closest to Red Hat office where I have some meetings to attend.

Then FOSDEM. The only “week-long conference squeezed into two days” I know. Good luck with meeting me probably. As usual my list of sessions covers all buildings and have several conflicts. Will meet many people, miss even more, do some beers. This year I stay in Floris Arlequin Grand-Place hotel.

Next one? OpenStack PTG in Dublin, Ireland. Finally will meet all those developers reviewing my patches, helping me with understanding source code of several OpenStack projects. And probably answering several questions about state of AArch64 support. Conference hotel.

Linaro Connect. Hong Kong again. Not registered yet, not looked at flights. Have to do that sooner than later and checking which airline sucks less on intercontinental connections sucks too. With few members of SDI team I will talk about our journey through/with OpenStack on our beloved architecture. Conference hotel.

What after those? Probably OpenSource Day in Poland, maybe some other ones. Will see.

by Marcin Juszkiewicz at January 01, 2018 18:29

January 17, 2018

Alex Bennée

Edit with Emacs v1.15 released

After a bit of hiatus there was enough of a flurry of patches to make it worth pushing out a new release. I’m in a little bit of a quandary with what to do with this package now. It’s obviously a useful extension for a good number of people but I notice the slowly growing number of issues which I’m not making much progress on. It’s hard to find time to debug and fix things when it’s main state is Works For Me. There is also competition from the Atomic Chrome extension (and it’s related emacs extension). It’s an excellent package and has the advantage of a Chrome extension that is more actively developed and using a bi-directional web-socket to communicate with the edit server. It’s been a feature I’ve wanted to add to Edit with Emacs for a while but my re-factoring efforts are slowed down by the fact that Javascript is not a language I’m fluent in and finding a long enough period of spare time is hard with a family. I guess this is a roundabout way of saying that realistically this package is in maintenance mode and you shouldn’t expect to see any new development for the time being. I’ll of course try my best to address reproducible bugs and process pull requests in a timely manner. That said please enjoy v1.15:


* Now builds for Firefox using WebExtension hooks
* Use chrome.notifications instead of webkitNotifications
* Use

with style instead of inline for edit button
* fake “input” event to stop active page components overwriting text area


* avoid calling make-frame-on-display for TTY setups (#103/#132/#133)
* restore edit-server-default-major-mode if auto-mode lookup fails
* delete window when done editing with no new frame

Get the latest from the Chrome Webstore.

by Alex at January 01, 2018 16:47

January 12, 2018

Marcin Juszkiewicz


Let me introduce new awesome project: YADIBP. It is cool, foss, awesome, the one and only and invented here instead of there. And it does exactly what it has to do and in a way it has to be done. Truly cool and awesome.

Using that tool you can build disk images with several supported Linux distributions. Or maybe even with any BSD distribution. And Haiku or ReactOS. Patches for AmigaOS exist too!

Any architecture. Starting from 128 bit wide RUSC-VI to antique architectures like ia32 or m88k as long as you have either hardware or qemu port (patches for ARM fast models in progress).

Just fetch from git and use. Written in BASIC so it should work everywhere. And if you lack BASIC interpreter then you can run it as Python or Erlang. Our developers are so cool and awesome!

But let’s get back to reality — there are gazillions of projects of tool which does one simple thing: builds a disk image. And gazillion will be still written because some people have that “Not Invented Here” syndrome.

And I am getting tired of it.

by Marcin Juszkiewicz at January 01, 2018 13:34

January 06, 2018

Bin Chen


To understand what Blockchain is, we need to go a little bit low level and understand what is Transactionand Block.
Transaction is also called record. It maps to a real-life event. Such as Rob pay Lucy $100; or Bin pays $100 for Roger Waters’ tour in Sydney 2018 (Yes, that’s true.)
By contrast, Block and Blockchain are abstract entities that are used to making sure all the Transaction happened will be recorded permanently and once it is recorded it is trustworthy and unmutable, but without a centralized authority says so. Blockchain is, well, a chain of Blocks.
The decentralized trust is the beauty and value of blockchain. And it’s power and usefulness is manifested by the success of the application out of which it is invented - BitCoin.
With the power be decentralized, you don’t need to hand over your power and privacy to others in exchange for a service. If there is a distributed social network platform, probably you want to give it a try if you are concerned about your privacy with Facebook? Or people, out of various reasons, don’t want to go through a bank, Paypal, WeChat for their financial transactions, that’s one of the primary reasons that drive the rise of BitCoin.

Distributed Consensus

For such as a system to work, the center of the problems is: How peers in a distributed system can agree on something?
“They can vote!” I hear you screaming. Yes…but if I can control 51% of the machines, I can control the whole system, for my benefit. It is a lot easier than controlling 51% people, all you need is money. If the economic incentive to do so out-weights the cost, people will do it! Mind you that currently BitCoin worth $100Bn.
BitCoin didn’t solve the general problem of distributed consensus, but it provides a solution that works extremely well in practice, using things called proof-of-work, and confirmation. It is not only about technology, but also some genius social innovation to ensure the whole system works.

Other Problems

In the context of Bitcoin, where it’s all about money. Other than making sure there will be a consensus regarding what are the valid transactions, following things are also very important design goals. If you can’t get them working, the whole system won’t work.
  1. A can’t spend B’s coins.
  2. A can’t spend more than she had.
  3. A can’t double-spend her coins.


Goal #1 is guaranteed by the cryptography.
A will need to sign the transaction using her private key. As long as A doesn’t have B’s private key, he won’t be able to spend B’s money.
A big shout here: take good care of your private key! Otherwise, you will lose all your money!
Goal #2 is guaranteed by having immutable transaction history, thus it is easy to verify if A is over-spend her coins: just look up the unique global blockchain find out how much she remains.
The immutability of blockchain is achieved by having the newer blocks including the digest of old blocks, so to modify a block you will also require modifying all the blocks that come after it. The computation power requires to make such modification is so huge (we’ll discuss this in detail in proof-of-work, and mining) that outweighs the benefits you might get so in practices make nobody will do it. Therefore, we consider the blockchain used by BitCoin is safe and can be trusted.
For Goal #3, we need a little bit explanation of what double-spendis since that’s one of the core problems for any digital currency to be useful. When using paper money, there won’t be any issue of double-spending. You hand out your money and you get the good. You can use that same money again. In case of Bitcoin, due to its distributed nature, it will take a while (say 1 hour) for the transaction to be made to the final blockchain, or to be confirmed. During that period, if A start another transaction, using the same coin he has just spent but is now in a to-be-confirmed status, he is double-spending the coin. Due to the distributed nature of the system, there is no easy way to arbitrate which transaction comes first, since there is not a global timing used by each transaction. This is called Race Attack, one of the ways to double-spend. Bitcoin solves the problem by something called confirmation. Newly created transaction has zero confirmation; It gets one confirmation when it is included in a Block and be chained; another confirmation when a new block appended; the more confirmations you get, the more confidence you get that transaction will make into the blockchain and won’t be reversed (thus double-spend). Currently, 6 confirmations are what most people will require.

Journey of a Transaction in BitCoin system

In previous sections, we go through some problems a distributed ledge system have to solve and touch gently how BitCoin solve the problems from a high level. Here we go a little bit low level, walking you through a journey the life cycle of a transaction in BitCoin system.
  1. BitCoin runs in a distributed peer-to-peer network. Everyone equals. We will call each peer aNode.
  2. A Node will create a Transaction and propagate it to the network.
  3. That transaction will be picked up by a few other nodes, be validated by them, and it is considered to be a valid one, it will be put into a Block. Note that there might be more than one Block being created that contains that transaction.
  4. At a certain time, a random node get picked up and asked to propose a candidate as next Block to be added to the global blockchain.
  5. The proposed block will be validated by other nodes, and it can be accepted or rejected by other Node. When being accepted, the Node accepts it will add it to its current longest blockchain; When being rejected, it just gets ignored.
  6. If enough nodes agree the proposed block is a valid one, it will be added to the global unique the blockchain; and all the nodes need to sync up their local blockchain with the global one.
  7. Now, the transaction created in 2 will be stored in the global blockchain forever and be trusted by all that it is a valid transaction. No dispute.
This is basically the protocol used by BitCoin to propagate & validate the transaction, arriving at a consensus if the transaction is valid or not, in a fully distributed peer-to-peer system.


We’ll look at in the details in next article some of the concepts we touched lightly such as proof-of-workand what is mining really. It will be more technically focused.
Stay tuned.

by Bin Chen ( at January 01, 2018 05:32

January 04, 2018

Marcin Juszkiewicz

Today I was fighting with Nova. No idea who won…

I am working on getting OpenStack running on AArch64 architecture, right? So recently I went from “just” building images to also using them to deploy working “cloud” setup. And that resulted in new sets of patches, updates to patches, discussions…

OpenStack is supposed to make virtualization easier. Create accounts, give access to users and they will make virtual machines and use them without worrying what kind of hardware is below etc. But first you have to get it working. So this week instead of only taking care of Kolla and Kolla-ansible projects I also patched Nova. The component responsible for running virtual machines.

One patch was simple edit of existing one to make it comply with all comments. Took some time anyway as I had to write some proper bug description to make sure that reviewers will know why it is so important for us. And once merged we will have UEFI used as default boot method on AArch64. Without any play with hw_firmware_type=uefi property on images (which is easy to forget). But this was the easy one…

Imagine that you have a rack of random AArch64 hardware and want to run a “cloud”. You may end in a situation where you have a mix of servers for compute nodes (the ones where VM instances run). In Nova/libvirt it is handled by cpu_mode option:

It is also possible to request the host CPU model in two ways:

  • “host-model” – this causes libvirt to identify the named CPU model which most closely matches the host from the above list, and then request additional CPU flags to complete the match. This should give close to maximum functionality/performance, which maintaining good reliability/compatibility if the guest is migrated to another host with slightly different host CPUs. Beware, due to the way libvirt detects host CPU, CPU configuration created using host-model may not work as expected. The guest CPU may confuse guest OS (i.e. even cause a kernel panic) by using a combination of CPU features and other parameters (such as CPUID level) that don’t work.

  • “host-passthrough” – this causes libvirt to tell KVM to passthrough the host CPU with no modifications. The difference to host-model, instead of just matching feature flags, every last detail of the host CPU is matched. This gives absolutely best performance, and can be important to some apps which check low level CPU details, but it comes at a cost wrt migration. The guest can only be migrated to an exactly matching host CPU.

Nova assumes host-model when KVM/QEMU is used as hypervisor. And crashes terribly on AArch64 with:

libvirtError: unsupported configuration: CPU mode ‘host-model’ for aarch64 kvm domain on aarch64 host is not supported by hypervisor

Not nice, right? So I made a simple patch to get host-passthrough to be default on AArch64. But when something is so simple then it’s description is probably not so simple…

Reported bug on nova with some logs attached. Then digged for some information which would explain issue better. Found Ubuntu’s bug on libvirt from Ocata times. They used same workaround.

So I thought: let’s report a bug for libvirt and request support for host-model option. There I got link to an another bug in libvirt with set of information why it does not make sense.

The reason is simple. No one knows what you run on when you run Linux on AArch64 server. In theory there are fields in /proc/cpuinfo but still you do not know do cpu cores in compute01 are same as compute02 servers. At least from nova/libvirt/qemu perspective. This also blocks us from setting cpu_mode to custom and selecting cpu_model which could be some way of getting same cpu for each instance despite of types of compute node processor cores.

The good side is that VM instances will work. The problem may appear when you migrate VM to cpu with other core — it may work. Or may not. Good luck!

by Marcin Juszkiewicz at January 01, 2018 16:52

December 30, 2017

Gema Gomez

Add new ball for knitting

I knit less than I crochet, and this means that I forget all the basic things from time to time. Up until now, I had never had to join a new ball of yarn to a project, because my projects were small and used just one skein.

After some research, I have found this video quite clear on how to add a new ball of yarn safely:


  1. In the middle of a row, insert the needle as if getting ready to knit a stitch normally.
  2. Instead of using the old yarn end, create a loop with the new one, and finish the stitch with it.
  3. Loop the old end of yarn over the top of the two new ones, this prevents a hole from forming.
  4. Holding both strands of the new ball of yarn do three or four more regular stitches to secure everything.
  5. Drop the short end from the new ball and just pick up the long strand and continue as normal.

Note: be careful on the way back not to work increases on the stitches that have been knitted with two strands, work them together. If the loose ends loosen up whilst you are working, give them little tugs, then weave them in.

by Gema Gomez at December 12, 2017 00:00

December 29, 2017

Gema Gomez

Autumn Knitting and Stiching Show 2017

This year, once again I took a day off during October and headed to Alexandra Palace in London to enjoy a day off looking at knitting/sewing supplies and ideas. This year’s Autumn Knitting and Stitching Show has been as interesting as always. I started the day doing some fabrics shopping (everything was so colorful):


Then, inevitably, admired all the art that was on display at the show. This time I was quite surprised by two scenes made of yarn, a railway station and a church. Here is proof that it can be knitted and it can look gorgeous:

railway station church

Awesome day out, as always with the Knitting and Stitching Show, cannot wait to see what things are there next year!

by Gema Gomez at December 12, 2017 00:00

December 02, 2017

Bin Chen

AWS Services Vs OpenStack

AWS has numerous services and it’s easy to get lost for beginners regarding what is for what. Meanwhile, as an open source advocator, I’m always interested to know what are the open source alternatives. To be fair, without open source code, none of existing cloud computing and big data platform would even exist.
Hence, I come up with the following table categorizing the key AWS services, each with a one-line interpretation; In addition, it also shows its corresponding OpenStack component, if there is one. Hopeful it’s helpful for you when either wandering through the AWS services or OpenStack one.

As you probably have noticed, AWS has much more services than OpenStackcan offer. That’s true.
OpenStack is more an Infrastructure As A Service(IaaS) solution, while AWS offers the solutions for all the other Xass - you name it, they have it: PaaSCaaSFaaS. And actually it is not just a “MeToo” solution, AWS actually leading the trend in some cases, such as Lambda, which is an offering for FaaS(Function As A service), or serverless, if you like. We might have more comparison regarding the open source solutions and AWS on those areas in the future, but this table primary compares the OpenStack and AWS.
WhatFor WhatOpenStack
ELBLoad Balancer
AutoScalingAuto Scaling
ECSContainer, Docker based
LambdaFunction/runtime cloud
S3Object storageSwift
EBSBlock storageCinder
EFSNetwork filesystem service used by EC2
GlacierData archive/backup
VPCVirtual Private Cloud
Route 53DNS Service & RoutingNeutron
RDSRelation Database ServerTrove
AuroraAmazon’s managed RDS
DynamoDBNoSQL data store
ElastiCachein-memory cache use redis,memcached
RedshiftData warehouse
AthenaAnalysis by sql
KinesisStream Analysis
EMRHadoop/Spark on AWSSahara
AWS IoTIoT Devices, Mqtt broker
GreengrassIoT Gateway ,Lambda on Gateway
LexSpeech to text & NLP/NLU, think Alexa
PollyText to speech
RekognitionImage Analysis
MLClassification and prediction
MobileSDKaccess/use AWS services on mobile
Device Farmapp test on devices
Appl. Svs
API GatewayREST API to access AWS services
SQSMessage QueueZaqar
SNSNotification Service
IAMidentity&access controlKeystone
CloudFmtionService OrchestrationHEAT
CloudWatchAWS resource monitor
CloudTrailAWS API call log
AdvisorAWS best practise Advisor

by Bin Chen ( at December 12, 2017 10:52

October 01, 2017

Bin Chen

AWS IoT Pipelines In Action: IoT Core, Lambda, Kenisis, Analytics

Today we will show you two end to end pipeline using AWS IoT Core and other AWS services.
  1. devices publish their status to the cloud, and cloud will process the events and write abnormal status to a noSQL database.
  2. device publish their status to the cloud, and we'll do real-time stream analytics on the events using Kenisis Analytics.

Pipeline 1 : Process data using Lambda

    +-----------+      +----------+     +------------+    +------------+
| | | Message | | | | |
| IoT Device| | Broker | | Rules | | Kenisis |
| +----> | +---> | Engine +--> | |
| | | | | | | |
+-----------+ +----------+ +------------+ +-----+------+
+------------+ +------------+
| DynamoDB | | |
| | ---+ Lambda |
| | | |
+------------+ +------------+
IoT Devices publish its status to the Message Broker, which is one components of AWS IoT core, using MQTT. Rules Engine (again one components of AWS IoT) is set up to channel the message to a Kenisis stream, which is setup as the trigger event to the Lambda function. The Lambda function is where the process happens, or business logic, if you like.


The lambda function will take records (Kenisis stream's terminology) from the Kenisis stream, i.e it's trigger event, then filter, process, store, and pass to other stage.
Check the code section below for details.

Pipeline 2 : Real-time analytics using Kenisis Analytics

    +-----------+      +----------+     +------------+    +------------+
| | | Message | | | | |
| IoT Device| | Broker | | Rules | | Kenisis |
| +----> | +---> | Engine +--> | |
| | | | | | | |
+-----------+ +----------+ +------------+ +-----+------+
+------------+ +------------+
| Output | | Kenisis |
| |----+ Analytics |
| | | |
+------------+ +------------+
The first four stages of the pipeline is the same as pipeline 1. But here we channel the Kenisis stream to the Kenisis Analytics to do real time analysis. If you know about Hadoop/Spark ecosystem, Kenisis Analytics is equivalent to Spark Stream.

Kenisis Analytics

You can use a syntax similar SQL to analysis the stream data over a window period.
  • An example: filter data
  • result
You'll see new events being added to the result as time goes by. Note that the time stamp is added automatically by Kenisis Analytics.
2017-10-01 03:24:37.978        50      2
2017-10-01 03:24:57.904 50 2
2017-10-01 03:25:05.914 50 2
2017-10-01 03:25:25.978 50 2
2017-10-01 03:25:44.001 50 2
2017-10-01 03:26:02.005 50 2
2017-10-01 03:26:11.898 50 2
2017-10-01 03:26:19.947 50 2
2017-10-01 03:26:29.922 50 2
2017-10-01 03:26:39.973 50 2


I'm not going to show you each and every steps of creating an IoT device, setting the Rules Engine, creating Kenisis and connecting those components to create a pipeline.
What I will show you are:
  1. A device simulator that can be used to drive the whole pipeline. You can easily spin up multiply devices and send message to simulate real user cases.
  2. The complete lambda handler that will parse the Kenisis record, filter the data, and write to a DynamoDB.
  3. A simple Kenisis Analytics SQL that used to filter out abnormal events and generate live report.

IoT device simulator

Launch one or more simulated devices and pushing data to a topic. It is used to drive the whole pipeline.
Basic usage: ./ deviceId. It will start a device as ${deviceId} and publish events declared in the file simulated_events.json in a loop fashion. To quit use ctr+c.
Or, use ./ to launch several devices in the background that will continuously publishing the events.
from AWSIoTPythonSDK.MQTTLib import AWSIoTMQTTClient
import argparse
import json
import logging
import time
import signal

AllowedActions = ['publish', 'subscribe', 'both']

def subscribe_callback(client, userdata, message):
print("[<< Receive]: ", "topic", message.topic, message.payload)

def args_parser():
parser = argparse.ArgumentParser()
parser.add_argument("-e", "--endpoint", action="store", required=True,
dest="host", help="Your AWS IoT custom endpoint")
parser.add_argument("-r", "--rootCA", action="store", required=True,
dest="rootCAPath", help="Root CA file path")
parser.add_argument("-c", "--cert", action="store", dest="certificatePath",
help="Certificate file path")
parser.add_argument("-k", "--key", action="store", dest="privateKeyPath",
help="Private key file path")
parser.add_argument("-w", "--websocket", action="store_true",
dest="useWebsocket", default=False,
help="Use MQTT over WebSocket")
parser.add_argument("-id", "--clientId", action="store", dest="clientId",
help="Targeted client id")
parser.add_argument("-t", "--topic", action="store", dest="topic",
default="sensors", help="topic prefix")
parser.add_argument("-d", "--deviceId", action="store", dest="deviceId",
help="device serial number, used as last part of "
parser.add_argument("-m", "--mode", action="store", dest="mode",
help="Operation modes: %s" % str(AllowedActions))

args = parser.parse_args()

return parser, args

def config_logger():
logger = logging.getLogger("AWSIoTPythonSDK.core")
streamHandler = logging.StreamHandler()
formatter = logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(message)s')

def main():
parser, args = args_parser()

mqtt_client = create_mqtt_client(args)

topic = args.topic + "/" + args.deviceId
print("topic:", topic)

if args.mode == 'subscribe' or args.mode == 'both':
mqtt_client.subscribe(topic, 1, subscribe_callback)

if args.mode == 'subscribe':
print("endless wait.. ctrl+c to finish");

# wait a while to make sure subscribe take effect

if args.mode == 'publish' or args.mode == 'both':
publish_events(topic, args.deviceId, mqtt_client)

def publish_events(topic, deviceId, mqtt_client):
if len(topic) == 0:
print("topic can't be empty")

with open('simulated_events.json') as f:
events = json.load(f)

while True:
for event in events:
# override the deviceId using pass-in value
event['deviceId'] = deviceId
message = json.dumps(event)
mqtt_client.publish(topic, message, 1)
print('[>> publish]', 'topic', topic, message)

def create_mqtt_client(args):
host =
rootCAPath = args.rootCAPath
certificatePath = args.certificatePath
privateKeyPath = args.privateKeyPath
useWebsocket = args.useWebsocket
clientId = args.clientId

mqttclient = None
if useWebsocket:
mqttclient = AWSIoTMQTTClient(clientId, useWebsocket=True)
mqttclient.configureEndpoint(host, 443)
mqttclient = AWSIoTMQTTClient(clientId)
mqttclient.configureEndpoint(host, 8883)
mqttclient.configureCredentials(rootCAPath, privateKeyPath,

# AWSIoTMQTTClient connection configuration
mqttclient.configureAutoReconnectBackoffTime(1, 32, 20)

return mqttclient

if __name__ == "__main__":

python \
-e ${endpoint} \
-r root-CA.crt \
-c device.cert.pem \
-k device.private.key \
--topic 'sensors' \
--deviceId $1

Lambda Handler

In the handler, we filter all the records whose temperature > 50, and write it DynamoDB table - warning_events.
import base64
import boto3
import json

def lambda_handler(event, context):
for record in event['Records']:
event = decode_kenisis_data(record["kinesis"]["data"])
if filter_event(event):
print("warning: temperature too high, write it to dynamoDB")
event = process_event(event)

return "ok"

def filter_event(event):
return event['temperature'] > 30

def process_event(event):
return event

def decode_kenisis_data(data):
"""create event from base64 encoded string"""
return json.loads(base64.b64decode(data.encode()).decode())
except Exception:
print("malformed event data")
return None

def encode_kenisis_data(event):
"""encode event to base64"""
return base64.b64encode(json.dumps(event).encode()).decode()
except Exception:
return None

# dynamoDB table handling
dynamodb = boto3.resource('dynamodb')
dbclient = boto3.client('dynamodb')

def is_table_exist(name):
# Not strong enough, a table might not full ready or in destroy status
# just loose the constraint for now
return True if name in dbclient.list_tables()['TableNames'] else False

def get_or_create_table(name):
if not is_table_exist(name):
print("table", name, " doesn't exist, go and create one")
return create_table(name)

return dynamodb.Table(name)

def create_table(table_name):
table = dynamodb.create_table(
'AttributeName': 'deviceId',
'KeyType': 'HASH'

'AttributeName': 'deviceId',
'AttributeType': 'S'
'ReadCapacityUnits': 5,
'WriteCapacityUnits': 5

# Wait until the table exists.
return table

def add_event(event):

def delete_table(name):


  • debug message broker
Use, open the script file and add --mode 'both'. It will receive the message when publishing it.
You can also use the Test Page[1] in the IoT console. You can subscribe/publish to a topic. You can publish using ./ and subscribe using the webpage. It is more "real" than using --mode both.
One thing worth noting the namespace for topic is implicit: /account/region/topic. It means your topic won't collide with my topic. And even for my account, topic `sensors/001' in us-east-1 region are different the one in ap-southeast-2.
  • debug rules engine
Rules engine connect the message broker and other AWS services, for example, Kenisis, lambda.
Seems there is nothing too much to debug, it is kind of black-box AFAIK. It is suggested to use the Cloud Watch to debug it, but I wasn't lucky enough to get it working. A good idea maybe before debugging the connector, making sure the relevant source/sink are working independently. Say, in the case connect message broker to the Kenisis stream. Make sure we unit tested the topic/message broker and the Kenisis stream first.
Hint: If you run out of ideas, try re-create the rules. And it is what I did and the result is very good :).
  • debug Kenisis
Use to check if there is any records in the stream.

SHARD_ITERATOR=$(aws kinesis get-shard-iterator \
--shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON \
--region ${region} \
--stream-name ${stream_name} \
--query 'ShardIterator')
aws kinesis get-records --region ${region} --shard-iterator ${SHARD_ITERATOR}
If not, something wrong is with the stream publish.
Use aws kinesis put-record to publish a record manually and try to issue get_stream_record.shagain to see if anything changes. If return records now, it means something wrong with the pipeline - no one is produce the record.


We show cases two pipelines/architectures, which can be used to implement two common user case. Actually, you don't have to use one pipeline over the other. In practice, the two pipeline are usually combined. It can be easily be implemented by creating multiple Rules Engines with two output Kenisis Streams, one for Lambda processing and one for analytics.
The beauty of this architectures are:
  1. Very scalable. You can have thousands of devices connected at the same time without worry about the throughput. 
  2. High available. Components such as DynamoDB and the Lambda are built on top of AWS high available services.
  3. Server-less. No single machine, even virtual one, you have to setup and maintain. Agility improves.

by Bin Chen ( at October 10, 2017 07:53

September 16, 2017

Bin Chen

Android Things on RPi3 - 2

This is a follow up to our last report on Android things status on RPi3. Compared with last report, the biggest difference is Andriod 5.1 is running with Android O. We'll go through the services list, system processes, properties, features, dispaly, cameara, wifi, bluetooth, sensors, and a little bit Android O Treble.
Flash, boot up and connect. Wait a little while before you using adb connect.
  • rpi3:/ $ service list
    Found 122 services:
    0 devicemanagementservice: []
    1 gpsdriverservice: []
    2 contexthub: [android.hardware.location.IContextHubService]
    3 netd_listener: []
    4 connmetrics: []
    5 bluetooth_manager: [android.bluetooth.IBluetoothManager]
    6 imms: []
    7 iotsystemservice: []
    8 media_projection: []
    9 launcherapps: []
    10 shortcut: []
    11 media_router: []
    12 media_session: []
    13 restrictions: [android.content.IRestrictionsManager]
    14 graphicsstats: [android.view.IGraphicsStats]
    15 dreams: [android.service.dreams.IDreamManager]
    16 commontime_management: []
    17 network_time_update_service: []
    18 samplingprofiler: []
    19 diskstats: []
    20 trust: []
    21 soundtrigger: []
    22 jobscheduler: []
    23 hardware_properties: [android.os.IHardwarePropertiesManager]
    24 serial: [android.hardware.ISerialManager]
    25 usb: [android.hardware.usb.IUsbManager]
    26 DockObserver: []
    27 audio: []
    28 search: []
    29 country_detector: [android.location.ICountryDetector]
    30 location: [android.location.ILocationManager]
    31 devicestoragemonitor: []
    32 notification: []
    33 updatelock: [android.os.IUpdateLock]
    34 servicediscovery: []
    35 connectivity: []
    36 ethernet: []
    37 rttmanager: []
    38 wifiscanner: []
    39 wifi: []
    40 netpolicy: []
    41 netstats: []
    42 network_score: []
    43 textservices: []
    44 network_management: [android.os.INetworkManagementService]
    45 clipboard: [android.content.IClipboard]
    46 statusbar: []
    47 device_policy: []
    48 deviceidle: [android.os.IDeviceIdleController]
    49 lock_settings: []
    50 uimode: []
    51 storagestats: []
    52 mount: []
    53 accessibility: [android.view.accessibility.IAccessibilityManager]
    54 input_method: []
    55 pinner: []
    56 vrmanager: [android.service.vr.IVrManager]
    57 input: [android.hardware.input.IInputManager]
    58 window: [android.view.IWindowManager]
    59 alarm: []
    60 consumer_ir: [android.hardware.IConsumerIrService]
    61 vibrator: [android.os.IVibratorService]
    62 settings: []
    63 content: [android.content.IContentService]
    64 account: [android.accounts.IAccountManager]
    65 [android.hardware.ICameraServiceProxy]
    66 telephony.registry: []
    67 scheduling_policy: [android.os.ISchedulingPolicyService]
    68 sec_key_att_app_id_provider: []
    69 webviewupdate: [android.webkit.IWebViewUpdateService]
    70 overlay: []
    71 usagestats: []
    72 battery: []
    73 sensorservice: [android.gui.SensorServer]
    74 dropbox: []
    75 processinfo: [android.os.IProcessInfoService]
    76 permission: [android.os.IPermissionController]
    77 cpuinfo: []
    78 dbinfo: []
    79 gfxinfo: []
    80 meminfo: []
    81 procstats: []
    82 activity: []
    83 user: [android.os.IUserManager]
    84 otadexopt: []
    85 package: []
    86 display: [android.hardware.display.IDisplayManager]
    87 recovery: [android.os.IRecoverySystem]
    88 power: [android.os.IPowerManager]
    89 appops: []
    90 batterystats: []
    91 device_identifiers: [android.os.IDeviceIdentifiersPolicyService]
    92 []
    93 media.sound_trigger_hw: [android.hardware.ISoundTriggerHwService]
    94 [android.hardware.IRadioService]
    95 media.aaudio: [IAAudioService]
    96 media.audio_policy: []
    97 audiodriverservice: []
    98 media.extractor: []
    99 media.resource_manager: []
    100 media.player: []
    101 media.audio_flinger: []
    102 gpu: [android.ui.IGpuService]
    103 SurfaceFlinger: [android.ui.ISurfaceComposer]
    104 [android.hardware.ICameraService]
    105 drm.drmManager: [drm.IDrmManagerService]
    106 media.metrics: []
    107 media.codec: []
    108 android.brillo.UpdateEngineService: [android.brillo.IUpdateEngine]
    109 android.brillo.metrics.IMetricsCollectorService: [android.brillo.metrics.IMetricsCollectorService]
    110 media.cas: []
    111 media.drm: []
    112 android.brillo.metrics.IMetricsd: [android.brillo.metrics.IMetricsd]
    113 inputdriverservice: []
    114 netd: []
    115 sensordriverservice: []
    116 []
    117 wificond: []
    118 android.service.gatekeeper.IGateKeeperService: [android.service.gatekeeper.IGateKeeperService]
    119 storaged: [Storaged]
    120 installd: [android.os.IInstalld]
    121 batteryproperties: [android.os.IBatteryPropertiesRegistrar]
rpi3:/ $ service list | grep things
0   devicemanagementservice: []
1 gpsdriverservice: []
92 []
97 audiodriverservice: []
113 inputdriverservice: []
115 sensordriverservice: []
  • rpi3:/ $ ps -ef
    UID            PID  PPID C      TIME CMD
    root 1 0 4 init
    root 2 0 0 [kthreadd]
    root 3 2 0 [ksoftirqd/0]
    root 4 2 0 [kworker/0:0]
    root 5 2 0 [kworker/0:0H]
    root 6 2 0 [kworker/u8:0]
    root 7 2 0 [rcu_preempt]
    root 8 2 0 [rcu_sched]
    root 9 2 0 [rcu_bh]
    root 10 2 0 [migration/0]
    root 11 2 0 [migration/1]
    root 12 2 0 [ksoftirqd/1]
    root 13 2 0 [kworker/1:0]
    root 14 2 0 [kworker/1:0H]
    root 15 2 0 [migration/2]
    root 16 2 0 [ksoftirqd/2]
    root 17 2 0 [kworker/2:0]
    root 18 2 0 [kworker/2:0H]
    root 19 2 0 [migration/3]
    root 20 2 0 [ksoftirqd/3]
    root 21 2 0 [kworker/3:0]
    root 22 2 0 [kworker/3:0H]
    root 23 2 0 [kdevtmpfs]
    root 24 2 0 [netns]
    root 25 2 0 [perf]
    root 26 2 0 [khungtaskd]
    root 27 2 0 [writeback]
    root 28 2 0 [ksmd]
    root 29 2 0 [crypto]
    root 30 2 0 [bioset]
    root 31 2 0 [kblockd]
    root 32 2 0 [kworker/0:1]
    root 33 2 0 [cfg80211]
    root 34 2 0 [rpciod]
    root 35 2 1 [kswapd0]
    root 36 2 0 [vmstat]
    root 37 2 0 [fsnotify_mark]
    root 38 2 0 [nfsiod]
    root 64 2 0 [kthrotld]
    root 65 2 0 [kworker/1:1]
    root 66 2 0 [kworker/2:1]
    root 67 2 0 [bioset]
    root 68 2 0 [bioset]
    root 69 2 0 [bioset]
    root 70 2 0 [bioset]
    root 71 2 0 [bioset]
    root 72 2 0 [bioset]
    root 73 2 0 [bioset]
    root 74 2 0 [bioset]
    root 75 2 0 [bioset]
    root 76 2 0 [bioset]
    root 77 2 0 [bioset]
    root 78 2 0 [bioset]
    root 79 2 0 [bioset]
    root 80 2 0 [bioset]
    root 81 2 0 [bioset]
    root 82 2 0 [bioset]
    root 83 2 0 [bioset]
    root 84 2 0 [bioset]
    root 85 2 0 [bioset]
    root 86 2 0 [bioset]
    root 87 2 0 [bioset]
    root 88 2 0 [bioset]
    root 89 2 0 [bioset]
    root 90 2 0 [bioset]
    root 91 2 0 [VCHIQ-0]
    root 92 2 0 [VCHIQr-0]
    root 93 2 0 [VCHIQs-0]
    root 94 2 0 [iscsi_eh]
    root 96 2 0 [dwc_otg]
    root 97 2 0 [DWC Notificatio]
    root 98 2 0 [VCHIQka-0]
    root 99 2 0 [dm_bufio_cache]
    root 100 2 0 [kworker/u8:1]
    root 101 2 0 [irq/92-mmc1]
    root 102 2 0 [bioset]
    root 103 2 1 [mmcqd/0]
    root 104 2 0 [kworker/0:2]
    root 105 2 0 [binder]
    root 106 2 0 [kworker/u8:2]
    root 107 2 0 [kworker/u8:3]
    root 108 2 0 [kworker/u8:4]
    root 109 2 0 [brcmf_wq/mmc1:0]
    root 110 2 0 [brcmf_wdog/mmc1]
    root 111 2 0 [ipv6_addrconf]
    root 112 2 0 [SMIO]
    root 113 2 0 [deferwq]
    root 114 2 0 [kworker/2:2]
    root 115 2 0 [jbd2/mmcblk0p6-]
    root 116 2 0 [ext4-rsv-conver]
    root 117 2 0 [kworker/3:1]
    root 118 1 0 ueventd
    root 123 2 0 [jbd2/mmcblk0p15]
    root 124 2 0 [ext4-rsv-conver]
    root 125 2 0 [ext4-rsv-conver]
    root 126 2 0 [jbd2/mmcblk0p13]
    root 127 2 0 [ext4-rsv-conver]
    logd 128 1 0 logd
    system 129 1 0 servicemanager
    system 130 1 0 hwservicemanager
    system 131 1 0 vndservicemanager /dev/vndbinder
    root 136 2 0 [kauditd]
    root 140 1 0 android.hardware.boot@1.0-service
    system 141 1 0 android.hardware.keymaster@3.0-service
    root 142 1 0 vold --blkid_context=u:r:blkid:s0 --blkid_untrusted_context=u:r:blkid_untrusted:s0 --fsck_context=u:r:fsck:s0 --fsck_untrusted_context=u:r:fsck_untrusted:s0
    root 151 1 0 netd
    root 152 1 2 zygote
    root 154 151 0 iptables-restore --noflush -w -v
    root 155 151 0 ip6tables-restore --noflush -w -v
    system 156 1 0 android.hidl.allocator@1.0-service
    audioserver 157 1 0
    bluetooth 158 1 0 android.hardware.bluetooth@1.0-service
    system 159 1 0 android.hardware.configstore@1.0-service
    system 160 1 0
    system 161 1 0
    system 162 1 0 android.hardware.power@1.0-service
    system 163 1 0 android.hardware.sensors@1.0-service
    system 164 1 0 android.hardware.usb@1.0-service
    wifi 165 1 0 android.hardware.wifi@1.0-service
    root 166 1 0 healthd
    root 167 1 0 lmkd
    system 168 1 2 surfaceflinger
    shell 169 1 0 sh
    shell 170 1 0 adbd --root_seclabel=u:r:su:s0
    audioserver 171 1 1 audioserver
    cameraserver 172 1 0 cameraserver
    drm 173 1 0 drmserver
    root 174 1 0 installd
    keystore 175 1 0 keystore /data/misc/keystore
    media 177 1 0 mediadrmserver
    mediaex 178 1 0 media.extractor aextractor
    media 179 1 0 media.metrics diametrics
    media 180 1 0 mediaserver
    root 181 1 1 peripheralman
    root 182 1 0 storaged
    wifi 183 1 0 wificond
    mediacodec 184 1 0 media.codec hw/
    root 186 1 0 sh /system/bin/periodic_scheduler 3600 14400 crash_sender /system/bin/crash_sender
    system 187 1 0 gatekeeperd /data/misc/gatekeeper
    system 188 1 0 userinputdriverservice
    metrics_coll 189 1 0 metrics_collector --foreground --logtosyslog
    metricsd 190 1 0 metricsd --foreground --logtosyslog
    tombstoned 192 1 0 tombstoned
    root 193 1 0 update_engine --logtostderr --foreground
    root 216 186 0 sleep 310
    mdnsr 222 1 0 mdnsd
    root 253 2 0 [kworker/1:2]
    system 310 152 12 system_server
    root 349 2 0 [kworker/0:1H]
    root 351 2 0 [kworker/1:1H]
    u0_a28 424 152 0
    u0_a8 431 152 0
    media_rw 436 142 0 sdcard -u 1023 -g 1023 -m -w /data/media emulated
    webview_zygote 467 1 1 webview_zygote32
    system 497 152 0
    root 551 2 0 [kworker/3:2]
    system 592 152 0
    u0_a7 623 152 2
    system 637 152 1
    u0_a11 669 152 1
    u0_a10 695 152 0
    u0_a9 718 152 0
    u0_a16 736 152 0
    system 748 152 0
    u0_a11 756 152 2
    u0_a3 783 152 0
    u0_a11 789 152 9
    u0_a11 846 152 8
    system 851 152 0
    root 908 2 0 [kworker/2:1H]
    u0_a11 954 152 1
    u0_a11 973 152 7
    u0_a4 998 152 1 android.process.acore
    root 1027 2 0 [kworker/3:1H]
    u0_a5 1070 152 0
  • rpi3:/ $ getprop
[camera.disable_zsl_mode]: [1]
[crash_reporter.coredump.enabled]: [1]
[dalvik.vm.appimageformat]: [lz4]
[dalvik.vm.dex2oat-Xms]: [64m]
[dalvik.vm.dex2oat-Xmx]: [512m]
[dalvik.vm.dexopt.secondary]: [true]
[dalvik.vm.heapsize]: [256m]
[dalvik.vm.image-dex2oat-Xms]: [64m]
[dalvik.vm.image-dex2oat-Xmx]: [64m]
[dalvik.vm.isa.arm.features]: [default]
[dalvik.vm.isa.arm.variant]: [generic]
[dalvik.vm.lockprof.threshold]: [500]
[dalvik.vm.stack-trace-file]: [/data/anr/traces.txt]
[dalvik.vm.usejit]: [true]
[dalvik.vm.usejitprofiles]: [true]
[debug.atrace.tags.enableflags]: [0]
[debug.force_rtl]: [0]
[debug.input.timeout_mode]: [none]
[dev.bootcomplete]: [1]
[hwservicemanager.ready]: [true]
[init.svc.adbd]: [running]
[]: [running]
[init.svc.audioserver]: [running]
[init.svc.bluetooth-1-0]: [running]
[init.svc.boot-hal-1-0]: [running]
[init.svc.bootanim]: [stopped]
[init.svc.cameraserver]: [running]
[init.svc.configstore-hal-1-0]: [running]
[init.svc.console]: [running]
[init.svc.crash_reporter]: [stopped]
[init.svc.crash_sender]: [running]
[init.svc.drm]: [running]
[init.svc.gatekeeperd]: [running]
[init.svc.gralloc-2-0]: [running]
[init.svc.healthd]: [running]
[init.svc.hidl_memory]: [running]
[init.svc.hostapd]: [stopped]
[init.svc.hwcomposer-2-1]: [running]
[init.svc.hwservicemanager]: [running]
[init.svc.inputdriverserv]: [running]
[init.svc.installd]: [running]
[init.svc.keymaster-3-0]: [running]
[init.svc.keystore]: [running]
[init.svc.lmkd]: [running]
[init.svc.logd]: [running]
[init.svc.logd-reinit]: [stopped]
[init.svc.mdnsd]: [running]
[]: [running]
[init.svc.mediacodec]: [running]
[init.svc.mediadrm]: [running]
[init.svc.mediaextractor]: [running]
[init.svc.mediametrics]: [running]
[init.svc.metricscollector]: [running]
[init.svc.metricsd]: [running]
[init.svc.netd]: [running]
[init.svc.peripheralman]: [running]
[init.svc.power-hal-1-0]: [running]
[init.svc.sensors-hal-1-0]: [running]
[init.svc.servicemanager]: [running]
[init.svc.storaged]: [running]
[init.svc.surfaceflinger]: [running]
[init.svc.tombstoned]: [running]
[init.svc.ueventd]: [running]
[init.svc.update_engine]: [running]
[init.svc.usb-hal-1-0]: [running]
[init.svc.vndservicemanager]: [running]
[init.svc.vold]: [running]
[init.svc.webview_zygote32]: [running]
[init.svc.wifi_hal_legacy]: [running]
[init.svc.wificond]: [running]
[init.svc.wpa_supplicant]: [stopped]
[init.svc.zygote]: [running]
[log.tag.Hyphenator]: [SUPPRESS]
[log.tag.WifiHAL]: [D]
[logd.logpersistd.enable]: [true]
[]: [Android]
[net.dns1]: []
[net.dns2]: []
[net.qtaguid_enabled]: [1]
[net.tcp.default_init_rwnd]: [60]
[]: [false]
[persist.sys.dalvik.vm.lib.2]: []
[persist.sys.profiler_ms]: [0]
[persist.sys.timezone]: [GMT]
[persist.sys.ui.hw]: [disable]
[persist.sys.usb.config]: [adb]
[persist.sys.webview.vmsize]: [104857600]
[pm.dexopt.ab-ota]: [speed-profile]
[]: [speed-profile]
[pm.dexopt.boot]: [verify]
[pm.dexopt.first-boot]: [quicken]
[pm.dexopt.install]: [quicken]
[ro.allow.mock.location]: [0]
[ro.baseband]: [unknown]
[ro.board.platform]: []
[ro.boot.hardware]: [rpi3]
[ro.boot.selinux]: [permissive]
[ro.boot.serialno]: [00000000b359057f]
[ro.boot.slot_suffix]: [_a]
[]: [Fri Aug 18 02:52:42 UTC 2017]
[]: [1503024762]
[]: [Things/iot_rpi3/rpi3:8.0.0/OIR1.170720.017/4284968:userdebug/test-keys]
[ro.bootloader]: [unknown]
[ro.bootmode]: [unknown]
[ro.boottime.adbd]: [7601300153]
[]: [7563352809]
[ro.boottime.audioserver]: [7604559893]
[ro.boottime.bluetooth-1-0]: [7565701611]
[ro.boottime.boot-hal-1-0]: [6893695518]
[ro.boottime.bootanim]: [10393633329]
[ro.boottime.cameraserver]: [7607383278]
[ro.boottime.configstore-hal-1-0]: [7568247705]
[ro.boottime.console]: [7598286143]
[ro.boottime.crash_reporter]: [7654761924]
[ro.boottime.crash_sender]: [7657323851]
[ro.boottime.drm]: [7611007445]
[ro.boottime.gatekeeperd]: [7660283382]
[ro.boottime.gralloc-2-0]: [7570879059]
[ro.boottime.healthd]: [7587036195]
[ro.boottime.hidl_memory]: [7560812861]
[ro.boottime.hwcomposer-2-1]: [7573384632]
[ro.boottime.hwservicemanager]: [6644135778]
[ro.boottime.init]: [3799]
[ro.boottime.init.cold_boot_wait]: [374]
[ro.boottime.init.mount_all.default]: [1999]
[ro.boottime.init.selinux]: [150]
[ro.boottime.inputdriverserv]: [7662961090]
[ro.boottime.installd]: [7613993122]
[ro.boottime.keymaster-3-0]: [6896106716]
[ro.boottime.keystore]: [7617538590]
[ro.boottime.lmkd]: [7589607393]
[ro.boottime.logd]: [6618810414]
[ro.boottime.logd-reinit]: [7497335361]
[ro.boottime.mdnsd]: [8393546767]
[]: [7638129893]
[ro.boottime.mediacodec]: [7649625518]
[ro.boottime.mediadrm]: [7628786507]
[ro.boottime.mediaextractor]: [7632200413]
[ro.boottime.mediametrics]: [7634985361]
[ro.boottime.metricscollector]: [7665596403]
[ro.boottime.metricsd]: [7668492132]
[ro.boottime.netd]: [7286089424]
[ro.boottime.peripheralman]: [7640988070]
[ro.boottime.power-hal-1-0]: [7575882080]
[ro.boottime.sensors-hal-1-0]: [7578534059]
[ro.boottime.servicemanager]: [6641712810]
[ro.boottime.storaged]: [7643825153]
[ro.boottime.surfaceflinger]: [7592422497]
[ro.boottime.tombstoned]: [7726276455]
[ro.boottime.ueventd]: [4198132029]
[ro.boottime.update_engine]: [7729457809]
[ro.boottime.usb-hal-1-0]: [7581457757]
[ro.boottime.vndservicemanager]: [6648329008]
[ro.boottime.vold]: [6899477601]
[ro.boottime.webview_zygote32]: [28021782541]
[ro.boottime.wifi_hal_legacy]: [7584342080]
[ro.boottime.wificond]: [7646544840]
[ro.boottime.zygote]: [7288532289]
[]: [true]
[]: [embedded]
[]: [Fri Aug 18 02:52:42 UTC 2017]
[]: [1503024762]
[]: [iot_rpi3-userdebug 8.0.0 OIR1.170720.017 4284968 test-keys]
[]: [iot_rpi3-userdebug 8.0.0 OIR1.170720.017 4284968 test-keys]
[]: [Things/iot_rpi3/rpi3:8.0.0/OIR1.170720.017/4284968:userdebug/test-keys]
[]: [iot_rpi3-userdebug]
[]: []
[]: [OIR1.170720.017]
[]: [rpi3]
[]: [true]
[]: [test-keys]
[]: [userdebug]
[]: [android-build]
[]: [REL]
[]: []
[]: [REL]
[]: [4284968]
[]: [0]
[]: [8.0.0]
[]: [26]
[]: [2017-08-05]
[ro.carrier]: [unknown]
[ro.config.alarm_alert]: [Alarm_Classic.ogg]
[ro.config.notification_sound]: [OnTheHunt.ogg]
[ro.crypto.state]: [unsupported]
[ro.dalvik.vm.native.bridge]: [0]
[ro.debuggable]: [1]
[ro.hardware]: [rpi3]
[]: [iot]
[]: [v4l2]
[ro.hardware.gps]: [iot]
[ro.hardware.gralloc]: [gbm]
[ro.hardware.hwcomposer]: [drm]
[ro.hardware.sensors]: [iot]
[ro.metricsd.product_id]: [android-things:r2nfla]
[ro.metricsd.product_version]: [0]
[ro.persistent_properties.ready]: [true]
[ro.product.board]: [rpi3]
[ro.product.brand]: [Things]
[ro.product.cpu.abi]: [armeabi-v7a]
[ro.product.cpu.abi2]: [armeabi]
[ro.product.cpu.abilist]: [armeabi-v7a,armeabi]
[ro.product.cpu.abilist32]: [armeabi-v7a,armeabi]
[ro.product.cpu.abilist64]: []
[ro.product.device]: [rpi3]
[ro.product.manufacturer]: [Google]
[ro.product.model]: [iot_rpi3]
[]: [iot_rpi3]
[ro.property_service.version]: [2]
[]: [yes]
[ro.revision]: [0]
[ro.rfkilldisabled]: [1]
[ro.runtime.firstboot]: [1230768023099]
[]: [1]
[ro.serialno]: [00000000b359057f]
[ro.sf.lcd_density]: [240]
[ro.treble.enabled]: [false]
[ro.wifi.channels]: []
[ro.zygote]: [zygote32]
[security.perf_harden]: [1]
[service.bootanim.exit]: [1]
[service.sf.present_timestamp]: [1]
[sys.boot_completed]: [1]
[sys.logbootcomplete]: [1]
[sys.rescue_boot_count]: [1]
[sys.sysctl.extra_free_kbytes]: [24300]
[sys.sysctl.tcp_def_init_rwnd]: [60]
[sys.usb.config]: [adb]
[sys.usb.configfs]: [0]
[sys.usb.state]: [adb]
[sys.wifitracing.started]: [1]
[vold.has_adoptable]: [0]
[wifi.interface]: [wlan0]
[wifi.supplicant_scan_interval]: [15]
  • HDMI Display
Initially, monitor says "no cable connected" but after switching the source back and force for my monitor and at the time I'm about to give up, android things home screen show up.
1920x1080 is used and that the resolution negotiation between monitor and the board is working properly.
rpi3:/ $ dumpsys SurfaceFlinger

Wide-Color information:
hasWideColorDisplay: 0
Display 0 color modes:
Current color mode: HAL_COLOR_MODE_NATIVE (0)

Sync configuration: [using: EGL_KHR_fence_sync EGL_KHR_wait_sync]
DispSync configuration: app phase 1000000 ns, sf phase 1000000 ns, present offset 0 ns (refresh 16666667 ns)

Buffering stats:
[Layer name]

Visible layers (count = 1)
+ Layer 0xb2dcd000 (
Region transparentRegion (this=0xb2dcd278, count=1)
[ 0, 0, 0, 0]
Region visibleRegion (this=0xb2dcd008, count=1)
[ 0, 0, 1920, 1080]
Region surfaceDamageRegion (this=0xb2dcd044, count=1)
[ 0, 0, -1, -1]
layerStack= 0, z= 21000, pos=(0,0), size=(1920,1080), crop=( 0, 0,1920,1080), finalCrop=( 0, 0, -1, -1), isOpaque=1, invalidate=0, alpha=1.000, flags=0x00000002, tr=[1.00, 0.00][0.00, 1.00]
format= 2, activeBuffer=[1920x1080:1920, 2], queued-frames=0, mRefreshPending=0
mTexName=5 mCurrentTexture=0
mCurrentCrop=[0,0,0,0] mCurrentTransform=0
-BufferQueue mMaxAcquiredBufferCount=1, mMaxDequeuedBufferCount=2, mDequeueBufferCannotBlock=0 mAsyncMode=0, default-size=[1920x1080], default-format=2, transform-hint=00, FIFO(0)={}
>[00:0xb4b0ea00] state=ACQUIRED, 0xb4b0ebe0 [1920x1080:1920, 2]
[01:0xb4b0ec80] state=FREE , 0xb4b0ee60 [1920x1080:1920, 2]
[02:0x0] state=FREE
Displays (1 entries)
+ DisplayDevice: Built-in Screen
type=0, hwcId=0, layerStack=0, (1920x1080), ANativeWindow=0xb517c008, orient= 0 (type=00000000), flips=70, isSecure=1, powerMode=2, activeConfig=0, numLayers=1
v:[0,0,1920,1080], f:[0,0,1920,1080], s:[0,0,1920,1080],transform:[[1.000,0.000,-0.000][0.000,1.000,-0.000][0.000,0.000,1.000]]
-BufferQueue mMaxAcquiredBufferCount=1, mMaxDequeuedBufferCount=1, mDequeueBufferCannotBlock=0 mAsyncMode=0, default-size=[1920x1080], default-format=1, transform-hint=00, FIFO(0)={}
[00:0xb514cdc0] state=DEQUEUED, 0xb514e620 [1920x1080:1920, 1]
[01:0x0] state=FREE
SurfaceFlinger global state:
EGL implementation : 1.4 (DRI2)
EGL_ANDROID_framebuffer_target EGL_ANDROID_image_native_buffer EGL_ANDROID_recordable EGL_EXT_image_dma_buf_import EGL_KHR_cl_event2 EGL_KHR_config_attribs EGL_KHR_create_context EGL_KHR_fence_sync EGL_KHR_get_all_proc_addresses EGL_KHR_gl_colorspace EGL_KHR_gl_renderbuffer_image EGL_KHR_gl_texture_2D_image EGL_KHR_gl_texture_3D_image EGL_KHR_gl_texture_cubemap_image EGL_KHR_image_base EGL_KHR_no_config_context EGL_KHR_reusable_sync EGL_KHR_surfaceless_context EGL_KHR_wait_sync EGL_MESA_configless_context EGL_MESA_drm_image EGL_MESA_image_dma_buf_export
GLES: Broadcom, Gallium 0.4 on VC4 V3D 2.1, OpenGL ES 2.0 Mesa 17.0.4
GL_EXT_debug_marker GL_EXT_blend_minmax GL_EXT_multi_draw_arrays GL_EXT_texture_format_BGRA8888 GL_OES_compressed_ETC1_RGB8_texture GL_OES_depth24 GL_OES_element_index_uint GL_OES_fbo_render_mipmap GL_OES_mapbuffer GL_OES_rgb8_rgba8 GL_OES_stencil8 GL_OES_texture_3D GL_OES_texture_npot GL_OES_vertex_half_float GL_OES_EGL_image GL_OES_depth_texture GL_OES_packed_depth_stencil GL_EXT_texture_type_2_10_10_10_REV GL_OES_get_program_binary GL_APPLE_texture_max_level GL_EXT_discard_framebuffer GL_EXT_read_format_bgra GL_NV_fbo_color_attachments GL_OES_EGL_image_external GL_OES_EGL_sync GL_OES_vertex_array_object GL_EXT_unpack_subimage GL_NV_draw_buffers GL_NV_read_buffer GL_NV_read_depth GL_NV_read_depth_stencil GL_NV_read_stencil GL_EXT_draw_buffers GL_EXT_map_buffer_range GL_KHR_debug GL_OES_surfaceless_context GL_EXT_separate_shader_objects GL_EXT_draw_elements_base_vertex GL_EXT_texture_border_clamp GL_KHR_context_flush_control GL_OES_draw_elements_base_vertex GL_OES_texture_border_clamp
Wide-color: Off
Region undefinedRegion (this=0xb51477ac, count=1)
[ 0, 0, 0, 0]
orientation=0, isDisplayOn=1
last eglSwapBuffers() time: 494.270000 us
last transaction time : 64.688000 us
transaction-flags : 00000000
refresh-rate : 59.999999 fps
x-dpi : 92.014999
y-dpi : 91.440002
gpu_to_cpu_unsupported : 0
eglSwapBuffers time: 0.000000 us
transaction time: 0.000000 us
VSYNC state: disabled
soft-vsync: disabled
events-delivered: 75
0xb2e66c50: count=-1
0xb2e66da0: count=-1
0xb2e66ec0: count=-1
0xb4b19060: count=-1
0xb4b190c0: count=-1
0xb4b190f0: count=-1
0xb4b19120: count=-1
0xb4b19150: count=-1
0xb4b191b0: count=-1
0xb511f4b0: count=-1
0xb511f4e0: count=-1
0xb511f510: count=-1

Display 0 HWC layers:
Layer name
Z | Comp Type | Disp Frame (LTRB) | Source Crop (LTRB)
21000 | Device | 0 0 1920 1080 | 0.0 0.0 1920.0 1080.0
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

h/w composer state:
h/w composer enabled
Allocated buffers:
0xb2e75e20: 8100.00 KiB | 1920 (1920) x 1080 | 1 | 1 | 0x1a00 |
0xb3016f00: 8100.00 KiB | 1920 (1920) x 1080 | 1 | 1 | 0x1a00 |
0xb4b0ebe0: 8100.00 KiB | 1920 (1920) x 1080 | 1 | 2 | 0x933 |
0xb4b0ee60: 8100.00 KiB | 1920 (1920) x 1080 | 1 | 2 | 0x933 |
0xb514e620: 8100.00 KiB | 1920 (1920) x 1080 | 1 | 1 | 0x1a00 | FramebufferSurface
0xb514f3e0: 8100.00 KiB | 1920 (1920) x 1080 | 1 | 1 | 0x1a00 |
Total allocated (estimate): 48600.00 KB
  • partitions
still same. no change.
rpi3:/ $ ls /dev/block/platform/soc/3f202000.sdhost/by-name/ -l
total 0
lrwxrwxrwx 1 root root 20 1970-01-01 00:00 boot_a -> /dev/block/mmcblk0p4
lrwxrwxrwx 1 root root 20 1970-01-01 00:00 boot_b -> /dev/block/mmcblk0p5
lrwxrwxrwx 1 root root 21 1970-01-01 00:00 gapps_a -> /dev/block/mmcblk0p13
lrwxrwxrwx 1 root root 21 1970-01-01 00:00 gapps_b -> /dev/block/mmcblk0p14
lrwxrwxrwx 1 root root 21 1970-01-01 00:00 misc -> /dev/block/mmcblk0p10
lrwxrwxrwx 1 root root 21 1970-01-01 00:00 oem_a -> /dev/block/mmcblk0p11
lrwxrwxrwx 1 root root 21 1970-01-01 00:00 oem_b -> /dev/block/mmcblk0p12
lrwxrwxrwx 1 root root 20 1970-01-01 00:00 rpiboot -> /dev/block/mmcblk0p1
lrwxrwxrwx 1 root root 20 1970-01-01 00:00 system_a -> /dev/block/mmcblk0p6
lrwxrwxrwx 1 root root 20 1970-01-01 00:00 system_b -> /dev/block/mmcblk0p7
lrwxrwxrwx 1 root root 20 1970-01-01 00:00 uboot_a -> /dev/block/mmcblk0p2
lrwxrwxrwx 1 root root 20 1970-01-01 00:00 uboot_b -> /dev/block/mmcblk0p3
lrwxrwxrwx 1 root root 21 1970-01-01 00:00 userdata -> /dev/block/mmcblk0p15
lrwxrwxrwx 1 root root 20 1970-01-01 00:00 vbmeta_a -> /dev/block/mmcblk0p8
lrwxrwxrwx 1 root root 20 1970-01-01 00:00 vbmeta_b -> /dev/block/mmcblk0p9
  • init.rpi3.rc
on fs
mount_all /fstab.${ro.hardware}

on early-init
mount debugfs debugfs /sys/kernel/debug mode=0755

service dhcpcd_wlan0 /system/bin/dhcpcd -dABKL
group dhcp

service wpa_supplicant /vendor/bin/hw/wpa_supplicant \
-iwlan0 -Dnl80211 -c/data/misc/wifi/wpa_supplicant.conf \
-I/system/etc/wifi/wpa_supplicant_overlay.conf \
-O/data/misc/wifi/sockets \
-e/data/misc/wifi/entropy.bin -g@android:wpa_wlan0
class main
socket wpa_wlan0 dgram 660 wifi wifi

on boot
chmod 0666 /dev/sw_sync

# Add a cpuset for the camera daemon
# We want all cores for camera
mkdir /dev/cpuset/camera-daemon
write /dev/cpuset/camera-daemon/cpus 0-3
write /dev/cpuset/camera-daemon/mems 0
chown system system /dev/cpuset/camera-daemon
chown system system /dev/cpuset/camera-daemon/tasks
chmod 0644 /dev/cpuset/camera-daemon/tasks
  • rpi3:/ $ cat ueventd.rpi3.rc
    /dev/video*               0660   system     camera
    /dev/ttyAMA0 0660 bluetooth bluetooth
    /dev/rfkill 0660 bluetooth bluetooth
    /sys/class/rfkill/rfkill0/state 0660 bluetooth bluetooth
  • bluetooth
Bluetooth is disabled by default, but we can enable it from cli.
rpi3:/ # dumpsys bluetooth_manager       
Bluetooth Status
enabled: false
state: OFF
address: null
name: RPI3
Bluetooth never enabled!
Bluetooth crashed 0 times

No BLE Apps registered.

Bluetooth Service not connected
rpi3:/ # service call bluetooth_manager 6  
Result: Parcel(00000000 00000001 '........')

rpi3:/ # dumpsys bluetooth_manager
Bluetooth Status
enabled: true
state: ON
address: 22:22:ED:D0:75:9F
name: RPI3
time since enabled: 00:00:02.555

Enable log:
08-30 04:46:18 Enabled by null
Bluetooth crashed 0 times

No BLE Apps registered.

Bonded devices:

Profile: BtGatt.GattService
mMaxScanFilters: 0

GATT Scanner Map
Entries: 0

GATT Client Map
Entries: 0

GATT Server Map
Entries: 0

GATT Handle Map
Entries: 0
Requests: 0

Profile: HeadsetService
mCurrentDevice: null
mTargetDevice: null
mIncomingDevice: null
mActiveScoDevice: null
mMultiDisconnectDevice: null
mVirtualCallStarted: false
mVoiceRecognitionStarted: false
mWaitingForVoiceRecognition: false
StateMachine: HeadsetStateMachine:
total records=2
rec[0]: time=08-30 04:46:20.124 processed=Disconnected org=Disconnected dest=<null> what=10(0xa)
rec[1]: time=08-30 04:46:20.124 processed=<null> org=Disconnected dest=<null> what=11(0xb)

mAudioState: 10

Profile: A2dpSinkService
mCurrentDevice: null
mTargetDevice: null
mIncomingDevice: null
StateMachine: A2dpSinkStateMachine:
total records=0

Profile: HidService
mTargetDevice: null

Profile: HealthService

Profile: PanService
mMaxPanDevices: 5
mPanIfName: bt-pan
mTetherOn: false

Profile: BluetoothMapService
mRemoteDevice: null
sRemoteDeviceName: null
mState: 0
mIsWaitingAuthorization: false
mRemoveTimeoutMsg: false
mPermission: 0
mAccountChanged: false
mBluetoothMnsObexClient: null
null : MasId: 0 Uri:null SMS/MMS:true

Profile: AvrcpControllerService
StateMachine: AvrcpControllerSM:
total records=0

Profile: BluetoothPbapService

Connection Events:

Bond Events:
Total Number of events: 0

A2DP State:
Counts (enqueue/dequeue/readbuf) : 0 / 0 / 0
Last update time ago in ms (enqueue/dequeue/readbuf) : 0 / 0 / 0
Frames per packet (total/max/ave) : 0 / 0 / 0
Counts (flushed/dropped/dropouts) : 0 / 0 / 0
Counts (max dropped) : 0
Last update time ago in ms (flushed/dropped) : 0 / 0
Counts (underflow) : 0
Bytes (underflow) : 0
Last update time ago in ms (underflow) : 0
Enqueue deviation counts (overdue/premature) : 0 / 0
Enqueue overdue scheduling time in ms (total/max/ave) : 0 / 0 / 0
Enqueue premature scheduling time in ms (total/max/ave) : 0 / 0 / 0
Dequeue deviation counts (overdue/premature) : 0 / 0
Dequeue overdue scheduling time in ms (total/max/ave) : 0 / 0 / 0
Dequeue premature scheduling time in ms (total/max/ave) : 0 / 0 / 0

A2DP Codecs State:
Current Codec: None

A2DP LDAC State:
Priority: 5001
Encoder interval (ms): 20
Config: Invalid
Selectable: Invalid
Local capability: Rate=44100|48000|88200|96000 Bits=16|24|32 Mode=STEREO
Packet counts (expected/dropped) : 0 / 0
PCM read counts (expected/actual) : 0 / 0
PCM read bytes (expected/actual) : 0 / 0
LDAC quality mode : HIGH
LDAC transmission bitrate (Kbps) : -1
LDAC saved transmit queue length : 0

A2DP AAC State:
Priority: 2001
Encoder interval (ms): 20
Config: Invalid
Selectable: Invalid
Local capability: Rate=44100 Bits=16 Mode=STEREO
Packet counts (expected/dropped) : 0 / 0
PCM read counts (expected/actual) : 0 / 0
PCM read bytes (expected/actual) : 0 / 0

A2DP SBC State:
Priority: 1001
Encoder interval (ms): 20
Config: Invalid
Selectable: Invalid
Local capability: Rate=44100 Bits=16 Mode=STEREO
Packet counts (expected/dropped) : 0 / 0
PCM read counts (expected/actual) : 0 / 0
PCM read bytes (expected/actual) : 0 / 0
Frames counts (expected/dropped) : 0 / 0

Bluetooth Config:
Config Source: New file
Devices loaded: 0
File created/tagged: 2017-08-30 04:46:18
File source: Empty

Bluetooth HF Client BTA Statistics

Bluetooth Wakelock Statistics:
Is acquired : true
Acquired/released count : 22 / 21
Acquired/released error count : 0 / 0
Last acquire/release error code: 0 / 0
Last acquired time (ms) : 1019
Acquired time min/max/avg (ms) : 1 / 1019 / 52
Total acquired time (ms) : 1160
Total run time (ms) : 1196

Bluetooth Memory Allocation Statistics:
Total allocated/free/used counts : 1172 / 661 / 511
Total allocated/free/used octets : 370075 / 79808 / 290267

Bluetooth Alarms Statistics:
Total Alarms: 2

Alarm : btif.config (SINGLE)
Action counts (sched/resched/exec/cancel) : 5 / 0 / 0 / 0
Deviation counts (overdue/premature) : 0 / 0
Time in ms (since creation/interval/remaining) : 1008 / 3000 / 1992
Callback execution time in ms (total/max/avg) : 0 / 0 / 0
Overdue scheduling time in ms (total/max/avg) : 0 / 0 / 0
Premature scheduling time in ms (total/max/avg): 0 / 0 / 0

Alarm : btm_ble_addr.refresh_raddr_timer (SINGLE)
Action counts (sched/resched/exec/cancel) : 2 / 0 / 0 / 0
Deviation counts (overdue/premature) : 0 / 0
Time in ms (since creation/interval/remaining) : 1016 / 900000 / 898984
Callback execution time in ms (total/max/avg) : 0 / 0 / 0
Overdue scheduling time in ms (total/max/avg) : 0 / 0 / 0
Premature scheduling time in ms (total/max/avg): 0 / 0 / 0

--- BEGIN:BTSNOOP_LOG_SUMMARY (5757 bytes in) ---
  • supported features
rpi3:/system/etc/permissions # ls -l
total 100
-rw-r--r-- 1 root root 820 android.hardware.bluetooth.xml
-rw-r--r-- 1 root root 830 android.hardware.bluetooth_le.xml
-rw-r--r-- 1 root root 933
-rw-r--r-- 1 root root 834 android.hardware.ethernet.xml
-rw-r--r-- 1 root root 942 android.hardware.location.gps.xml
-rw-r--r-- 1 root root 868
-rw-r--r-- 1 root root 829 android.hardware.wifi.xml
-rw-r--r-- 1 root root 748
-rw-r--r-- 1 root root 828
-rw-r--r-- 1 root root 828
-rw-r--r-- 1 root root 820
-rw-r--r-- 1 root root 977 iot_features.xml
-rw-r--r-- 1 root root 8786 platform.xml
-rw-r--r-- 1 root root 19690 privapp-permissions-google.xml
-rw-r--r-- 1 root root 20273 privapp-permissions-platform.xml
cat ot_features.xml:

<feature name="android.hardware.type.embedded" />

<library name=""
file="/system/framework/" />

  • camera
rpi3:/system/vendor # dumpsys                                                                                                                                
== Service global info: ==

Number of camera devices: 0
Number of normal camera devices: 0
Active Camera Clients:
Allowed user IDs: 0

== Camera service events log (most recent at top): ==
01-01 00:00:03 : USER_SWITCH previous allowed user IDs: , current allowed user IDs: 0

== Camera Provider HAL legacy/0 (v2.4, passthrough) static info: 0 devices: ==

== Vendor tags: ==

Dumping vendor tag descriptors for vendor with id 3854507339
Dumping configured vendor tag descriptors: None set

== Camera error traces (0): ==
No camera traces collected.
  • sensor
There are two sensor services, one is standard android one, the other is android things specific. Not sure the relationship between them.
rpi3:/system/vendor # service list | grep sensor  
73 sensorservice: [android.gui.SensorServer]
115 sensordriverservice: []

rpi3:/system/vendor # dumpsys sensorservice
Sensor Device:
Total 1 h/w sensors, 1 running:
0x00000001) active-count = 1; sampling_period(ms) = {1.0}, selected = 1.00 ms; batching_period(ms) = {0.0}, selected = 0.00 ms
Sensor List:
0x00000001) User-Driver Dynamic Meta-Sensor | Google | ver: 1 | type: android.sensor.dynamic_sensor_meta(32) | perm: n/a
special-trigger | maxDelay=0us | minDelay=0us | no batching | wakeUp |
Fusion States:
9-axis fusion disabled (0 clients), gyro-rate= 200.00Hz, q=< 0, 0, 0, 0 > (0), b=< 0, 0, 0 >
game fusion(no mag) disabled (0 clients), gyro-rate= 200.00Hz, q=< 0, 0, 0, 0 > (0), b=< 0, 0, 0 >
geomag fusion (no gyro) disabled (0 clients), gyro-rate= 200.00Hz, q=< 0, 0, 0, 0 > (0), b=< 0, 0, 0 >
Recent Sensor events:
Active sensors:
User-Driver Dynamic Meta-Sensor (handle=0x00000001, connections=1)
Socket Buffer size = 39 events
WakeLock Status: not held
1 active connections
Connection Number: 0
Operating Mode: NORMAL | WakeLockRefCount 0 | uid 1000 | cache size 0 | max cache size 0
User-Driver Dynamic Meta-Sensor 0x00000001 | status: active | pending flush events 0
0 direct connections
Previous Registrations:
00:00:03 + 0x00000001 pid= 310 uid= 1000 samplingPeriod=0us batchingPeriod=0us

rpi3:/system/vendor # dumpsys sensordriverservice
# nothing

Android O HAL - Treble Architecture

Thing are of particular interests is libandroidthings*.so. That indicates that Android Things is more a platform extension, much less change in scope than the Brillo was set out to do.
total 1068
|- mediadrm
|- soundfx
|- hw
rpi3:/system/vendor/bin/hw # ls -l
root shell
bluetooth bluetooth android.hardware.bluetooth@1.0-service
root shell android.hardware.boot@1.0-service
root shell android.hardware.configstore@1.0-service
root shell
root shell
root shell android.hardware.keymaster@3.0-service
root shell
root shell android.hardware.power@1.0-service
root shell android.hardware.sensors@1.0-service
root shell android.hardware.usb@1.0-service
wifi wifi android.hardware.wifi@1.0-service
root shell wpa_supplicant

by Bin Chen ( at September 09, 2017 02:37

September 12, 2017

Siddhesh Poyarekar

Across the Charles Bridge - GNU Tools Cauldron 2017

Since I joined Linaro back in 2015 around this time, my travel has gone up 3x with 2 Linaro Connects a year added to the one GNU Tools Cauldron. This year I went to FOSSAsia too, so it’s been a busy traveling year. The special thing about Cauldron though is that it is one of those conferences where I ‘work’ as well as have a lot of fun. The fun bit is because I get to meet all of the people that I work with almost every day in person and a lot of them have become great friends over the years.

I still remember the first Cauldron I went to in 2013 at Mountain View where I felt dwarfed by all of the giants I was sitting with. It was exaggerated because it was the first time I met the likes of Jeff Law, Richard Henderson, etc. in personal meetings since I had joined the Red Hat toolchain team just months before; it was intimidating and exciting all at once. That was also the first time I met Roland McGrath (I still hadn’t met Carlos, he had just had a baby and couldn’t come), someone I was terrified of back then because his patch reviews would be quite sharp and incisive. I had imagined him to be a grim old man hammering out those words from a stern laptop, so it was a surprise to see him use the same kinds of words but with a sarcastic smile, completely changing the context and tone. That was the first time I truly realized how emails often lack context. Years later, I still try to visualize people when I read their emails.

Skip to 4 years later and I was at my 5th Cauldron last week and despite my assumptions on how it would go, it was a completely new experience. A lot of it had to do with my time at Linaro and very little to do with technical growth. I felt like an equal to Linaro folks all over the world and I seemed to carry that forward here, where I felt like an equal with all of the people present, I felt like I belonged. I did not feel insecure about my capabilities (I still am intimately aware of my limitations), nor did I feel the need to constantly prove that I belonged. I was out there seeking toolchain developers (we are hiring btw, email me if you’re a fit), comfortable with the idea of leading a team. The fact that I managed to not screw up the two glibc releases I managed may also have helped :)

Oh, and one wonderful surprise was that an old friend decided to drop in an Cauldron and spend a couple of days.

This year’s Cauldron had the most technical talks submitted in recent years. We had 5 talks in the glibc area, possibly also the highest for us; just as well because we went over time in almost all of them. I won’t say that it’s a surprise since that has happened in every single year that I attended. The first glibc talk was about tunables where I briefly recapped what we have done in tunables so far and talked about the future a bit more at length. Pedro Alves suggested putting pretty printers for tunables for introspection and maybe also for runtime tuning in the coming future. There was a significant amount of interest in the idea of auto-tuning, i.e. collecting profiling data about tunable use and coming up with optimal default values and possibly even eliminating such tunables in future if we find that we have a pretty good default. We also talked about tuning at runtime and the various kinds of support that would be required to make it happen. Finally there were discussions on tuning profiles and ideas around creating performance-enhanced routines for workloads instead of CPUs. The video recording of the talk will hopefully be out soon and I’ll link the video here when it is available.

Florian then talked about glibc 3.0, a notional concept (i.e. won’t be a soname bump) where we rewrite sections of code that have been rotting due to having to support some legacy platforms. The most prominent among them is libio, the module in glibc that implements stdio. When libio was written, it was designed to be compatible with libstdc++ so that FILE streams could be compatible with C++ stdio streams. The only version of gcc that really supports that is 2.95 since libstdc++ has since moved on. However because of the way we do things in glibc, we cannot get rid of them even if there is just one user that needs that ABI. We toyed with the concept of a separate compatibility library that becomes a graveyard for such legacy interfaces so that they don’t hold up progress in the library. It remains to be seen how this pans out, but I would definitely be happy to see this progress; libio was one of my backlog projects for years. I had to miss Raji’s talk on powerpc glibc improvements since I had to be in another meeting, so I’ll have to catch it when the video comes out.

The two BoFs for glibc dealt with a number of administrative and development issues, details of which Carlos will post on the mailing list soon. The highlights for me were the malloc instrumented benchmarks that Carlos wants to add to benchtests and build and review tools. Once I clear up my work backlog a bit, I’ll attempt to set up something like phabricator or gerrit and see how that works out or the community instead of patchwork. I am convinced that all of the issues that we want to solve like crediting reviewers, ensuring good git commit logs, running automated builds and tests, etc. can only be effectively solved with a proper review tool in place to review patches.

There was also a discussion on redoing the makefiles in glibc so that it doesn’t spend so much time doing dependecy resolution, but I am going to pretend that it didn’t happen because it is an ugly ugly task :/

I’m back home now, recovering from the cold that worsened while I was in Prague before I head out again in a couple of weeks to SFO for Linaro Connect. I’ve booked tickets for whale watching tours there, so hopefully I’ll be posting some pictures again after a long break.

by Siddhesh at September 09, 2017 06:16

September 09, 2017

Bin Chen


Cryptography is the fundamental technology underlying lots of hot topics nowadays, such as security of IoT system, or blockchain and its primary application - cryptocurrency (Bitcoin is one of them). Hence, a basic understanding of what problems are the classic cryptography are trying to solve is vital to get a genuine of appreciation of their today's trending applications.
I tried to fix that on myself, by taking a Cryptography course on Couresa. It is an excellent course, taught by Professor Dan Boneh, from Standford University. And it is free! It has everything you need on Cryptography, probably a little bit too much for most of us. I strongly encourage you to take a look at the course if you really want to understand what security means technically.
Below are the notes I took during the course. I'm not sure how useful it can be for others since by simpling looking at the notes the chance to understand the Cryptography is slim - you'll have to really have to think it through very hard.


Security Goal

  • Confidentiality : data is kept secret
  • Integrity : data isn't modified
  • Message Authentication : the sender of the message is authentic and the message itself isn't modified
  • nonrepudiation : sender of the message can't deny the creation of the message
For example: Alice send message to Bob
  • Confidentiality means only Alice and Bob understand the message
  • Integrity mean Bob can be sure the message haven't been modified
  • Authentication Bob is sure the message is send from Alice and message isn't modified
  • Nonrepudiation Alice can't deny it's she create the message
  • Confidentiality use Symmetric Cipher or less common Symmetric cipher
  • Integrity use MAC
  • Authentication use digital signature and MAC
  • Nonrepudiation use digital signature

Symmetric Cipher

  • Provide Confidentiality
  • Def enc(plaintext, key) -> cyphered text; dec(cypered_text, key) -> plain text
  • Stream Cipher
    • RC-4
  • Block Cypher
    • DES,3-DES
    • AES


  • Provide Integrity and Authentication
  • Base on either Symmetric Cipher or Hash

Asymmetric Cipher, (G,E,D)

  • Provide Confidentiality (but mainly for key exchange, not for long message) and Nonrepudiation
  • Def G: generate key pair, (sk, pk) E: enc(plaintext, public key) -> cyphered text; D: dec(cyphered text, private key) -> plain text
  • RSA <- factorization="" integer="" li="">
  • ElGamal <- discrete="" li="" logarithms="">
  • Ellie Curve <- curve="" ellie="" li="">

Digital Signature

  • Provide Nonrepudiation
  • Base on Asymmetric Cipher Or DSA
  • Theory: sender sign the message with it's private key, so it can't deny the creation of the message because only she has the private key.
  • Notes: private/security key is used to sign (or encode), public key is used to verify. Compared with encryption, public key is used encrypt, private key is used to decrypt.
sign with private key, decrypt with public key.

Key Establishment

  • Certificate
  • PKI

Stream Ciphers

  • Cipher (K,M,C)
    • E(k,m) = k XOR m
    • D(k,c) = k XOR c
  • One Time Pad (OPT) : a stream cypher for which
    1. key stream k1,k2,k3.. is random generated
    2. key stream bit ki is used only once
    3. key stream is known to legitimate parties only
    OPT is perfect secure, but requires the key stream generator to be true random and len(k) >= len(m), so it is hard to use in practice.
  • Pseudorandom Generators (PRG) Making OPT practical, by replace true random with pseudorandom. E(k,m) = m XOR G(k) D(k,c) = c XOR G(k) where G : {0,1} s size -> {0,1} n size
  • The Stream Cipher still alive - RC4 link

Block Cyphers

  • Confusion and Diffusion
    • Confusion: obscure key and value
    • Diffusion: one plaintext spread over all the cipher text -> 1 bit change in plain text result in big change in cipher text
  • Operate on a fixed block size, e.g 64bits, 128bits(AES)
  • Modes
    • Block Modes : making same block of plaintext output different cyphertext
      • ECB (Electronic Code Book)
      • CBC (Cipher Blocking Chain)
      • CFB (Cipher Feedback)
      • OFB (Output Feedback)
      • CTR (Counter)
      • OCB (Offset Codebook)
      • GCM
    • Padding Modes
      • PKCS7
  • DES, considered not secure nowadays.
    • PRF, PRP
    • Feistel network
    • 3-round Feistel network
  • AES, DES replacement
    • To counter the plain text attack
      • CBC model, using random Initialization Vector,
      • Random ctr mode
  • Example: DES, JCA(Java Cryptography Architecture) API
// sender: 
// 1. generate the key
secretKey = KeyGenerator.getInstance("DES").generateKey();
// 2. encrypt with mode
Cipher encryptor = Cipher.getInstance("DES/CBC/PKCS5Padding");
encryptor.init(Cipher.ENCRYPT_MODE, secretKey);
// 3. share the key
keyString = Base64.encodeToString(secretKey.getEncoded());
// send the Base64 encoded shared key, the receiver will be able
// to reconstruct the shared key and use it to decrypt

// receiver:
// 1. get key, encrypted data and other encrypt specific data
// 2. decrypt

Message Integrity

  • code or data wasn't be modified (secure boot or disk encryption)


  • Definition : MAC I = (S,V) over (Key,Message,Tag)
    • S(k, m) -> t, [m || t] is sent
    • V(k, [m, t]) -> OK?
  • Secure MAC Attacker can't produce valid tag for new message.
  • Important concept and properties
    1. Cryptographic checksum A MAC generates a cryptographically secure authentication tag for a given message.
    2. Symmetric MACs are based on secret symmetric keys. The signing and verifying parties must share a secret key.
    3. Arbitrary message size MACs accept messages of arbitrary length.
    4. Fixed output length MACs generate fixed-size authentication tags.
    5. Message integrity MACs provide message integrity: Any manipulations of a message during transit will be detected by the receiver.
    6. Message authentication The receiving party is assured of the origin of the message.
    7. No nonrepudiation Since MACs are based on symmetric principles, they do not provide nonrepudiation.
  • Secure PRF (Pseudorandom Function) -> Secure MAC
    • AES is a secure PRF so it is secure MAC but works only on 16 bytes
    • Given PRF for short message construct PRF for long message, or convert Small MAC to Big MAC, there are two way doing this
      • CBC-MAC - build it from block cypher
      • HMAC - build it from Hash function


Hash Collision Resistance

  • Let H: M -> T to be a hash function a collision is H(m0) = H(m1) and m0 != m1
  • Requirements/features:
    • preimage resistance, or One-wayness Can't get the message from the hash
    • second image resistance, or Collision Resistance: For a hash function H, the probability that attacker can find a m1 and m2 so that H(m1) = H(m2) is negligible.
  • MAC from collision Resistance Let I = (S,V) be MAC for short message over (K, M, T), e.g AES Let H : Mbig -> M, called Digest/hash algorithm, e.g md5/sha1/sha256/etc.. Define Ibig = (Sbig, Vbig) over (K, Mbig, T) as Sbig(k, m) = S(k, H(m)), Vbig(k, m, t) = V(k, H(m), t)
    If I is secure MAC and H is collision resistant, then Ibig is secure.
    Example: let H = SHA256, S = AES ; then Sbig(k,m) = AES(k, SHA256(m))
    Note: this is not HMAC!!
  • Properties
    • block size : Usually the hash function is blocked based, the hash is base on block, and the output of one block will feed into another stage as the input. see Merkle–Damgård construction. Remember, 512.
    • max message size : Remember, 2^64-1
    • digest size : the output of hash is fixed for certain algorithm (e.g 256 for SHA256).

Merlke-Damgard Paradigm

  • Given C.R function for short message, build C.R function for long message. given compression function h : T * X -> T get H: XBig->T Chain variables:
  H0 = h(IV, m0) 
Hi = h(Hi-1, mi)
  • Build compression function from block cypher. Davies-Meyer compression function h(H,m) = E(m, H) XOR H Use the message as the key, hash value (from previous output) as the value to be encrypted. The output is the same size as the input hash value.


  • A Collision Resistance Hash building using
    • Merlke-Damgard Paradigm
    • Davies-Meyer compression function
    • Block cipher: SHACAL-2
    block size: 512 digest size: 256
  • Very useful!
    • Used by HMACSHA256 to build a Hash based MAC
    • Used to RSA
    • Used by EC

HMAC - a way of building secure MAC from Hash

  • The obvious but not secure way of building the MAC using hash H(k||x) or H(x||k)
  • HMAC - a secure way of building MAC from Hash H (k XOR opad || H( k XOR ipad) || m)
    k is expanded from the symmetric key, with 0 on the left, until the block size is reached, i.e 0000...key_s ipad is a repetition of 0011 0110, until the block size is reached opad is a repetition of 0101 1100, until the block size is reached
    e.g from HmacSHA256 the block size is 512.
    • HMAC is not acronym referring a generic way building hash from MAC but the specific way as described above. The difference is in what Hash function is used, e.g when the hash is SHA256 it is called HmacSHA256. similar, we have HmacSHA224,...
    • key size is defined the Hash function
    • tag size is defined by the digest size of the Hash function.
    • more parameters maybe required depend on the Hash function, e.g SHA256 requires an random IV, this IV will be needed in the verification side.
  • HmacSHA256
  • Despite Merlke-Damgard hash collision resistance and can work on long message but it is not safe to be used as a MAC directly. We need a PRF (e.g) and that is called Hash-MAC or HAMC.

Authenticated Encryption

  • MAC and then Encrypt (SSL)
  • Encrypt and then MAC (IPsec)
    • see android keystore client implementation
  • Encrypt and MAC (SSH) - Not safe
  • Example: TLS


  • Use mac and then encrypt
  • unidirectional key : both client and server has two keys. use symmetric key!
    • browser_to_sever_key_mac
    • browser_to_sever_key_enc
    • server_to_browser_key_mac
    • server_to_browser_key_enc
  • browser -> server
    • record == [ header || data || tag || pad ]
    • tag <- browser_to_sever_key_mac="" ctr_b_s="" data="" header="" li="" sign="">
    • pad [header||data||tag] to AES block
    • encrypt: CBC encrypt with browser_to_sever_key_enc and new random IV
    • prepend header
  • server -> browser
    • decrypt with browser_to_sever_key_enc
    • check pad
    • check tag on [ctr_b_s || header || data]

Key derivation

  • generate many keys from the the source key
  • if Source Key is uniformed distributed, use KDF
  • if not, extract and then expand
    • extract: making it indistinguishable from random distribution, using salt
    • expand : use KDF
  • HKDF : HMAC based KDF
    • extract: k <- hmac="" li="" salt="" source_key="">
  • PBKDF : Password Based KDF
    • Don't use HKDF to generate key from password, since password has low entropy
    • standard: PKCS#5(PBKDF1)
      • iterate c times using H(salt|pwd), slow down each time

Problem with Symmetric Encryption

  • Shared key exchange
  • Number of keys : A key is need between any 2 of N
  • No protection against cheating by Alice or Bob since that have same key

Usage/Purpose of Public-Key Algorithms:

It is important to realize that Public-Key Algorithms is not just for Encryption. Actually, since Public-Key Encryption is very slow, it is rarely used directly to encrypt large data, instead, it is often used to encrypt/exchange the symmetric key that will be used to encrypt the actual message.
  • Key Establishment e.g Diffie–Hellman key exchange (DHKE) or RSA key transport protocols.
  • Encryption Encrypt messages using as RSA or Elgamal.
  • Nonrepudiation Providing nonrepudiation and message integrity can be realized with digital signature algorithms, e.g., RSA, DSA or ECDSA.
  • Identification We can identify entities using challenge-and-response pro- tocols together with digital signatures.

3 Type of Public-Key Algorithm Families

  • Integer-Factorization Schemes RSA
  • Discrete Logarithm Schemes Diffie–Hellman key exchange, Elgamal encryption or the Digital Signature Algorithm (DSA).
  • Elliptic Curve (EC) Schemes Elliptic Curve Diffie–Hellman key exchange (ECDH) and the Elliptic Curve Digital Signature Algorithm (ECDSA). ECDSA used by bitcoin

Basic Key Exchange


  • online Trust 3rd Party
  • Deffie-Hellman Protocol
  • Public-Key Encryption The key exchange problem triggered the start of public cryptography

online Trust 3rd Party

  • Toy protocol: When alice when to share a key with Bob, she ask TTP for a shared key. TTP will send alice [ enc(shared_key, alice_privateKey), enc(shared_key, bob_privateKey)]. The late message will be send to Bob by Alice. So, Alice and Bob set up a shared key
  • insecure to replay attack

Diffie-Hellman Key Exchange

Public-key encryption

Number Theory

  • gcd : gcd(x,y) = ax + by;
  • Modular invertible x is invertible in Zp, if there is a y in Zp making z*y = 1 in Zp Lemma: x is invertible if gcd(x,P) = 1
    (Zp)* represents all invertible in Zp, e.g (Z12)* = [1,3,5,7,11]
  • Fermat's theorem Let p be a prime , V x <- 1="" an="" be="" can="" if="" in="" is="" number="" p-1="" p="" prime.="" test="" to="" used="" x="" zp="">
  • Dlog
    • Elliptic curve group

Public-Key Encryption

  • Asymmetric, public key (G,E,D) G: generate key pair, (sk, pk) E: enc(plaintext, public key) -> cyphered text; D: dec(cyphered text, private key) -> plain text
  • Using TDF
  • Using Diffie-Hellman: ElGamal

Trapdoor Function (TDF), or one-way function

  • TDF: a triple algorithms that X->Y, (G,F,F')
    • G: random algorithm generate key pair(sk, pk)
    • F: F(pk, X) -> Y
    • F': F'(sk, Y) -> X
  • Secure TDF, F is one way function, that is to say without sk, the probability of invert X correctly is negligible.

Public-Key Encryption using Secure TDF

  • we need three components, Iso standard
    • (G, F, F') : A secure TDF
    • (E, D) : A symmetric authenticated encryption defined over (K,M, C)
    • H: X -> k : A Hash
  • For the public encryption, we need (G,E,D)
    • G: Use the same G in TDF
    • E(pk, m) -> c
    • D(sk, c) -> m
  • E(pk, m)
    • random pick x from X
    • y = F(pk,x)
    • k = H(x) , c = E(k, m)
    • output [y, c]
 | F(pk,x)    |        E(H(x), m)                      |
header body
  • D(sk, [y,c])
    • x = F'(sk, y) , k = H(x), m = D(k, c)
  • Don't use TDF directly to the message! It is not secure. Instead, the TDF is used to encode/trapdoor the source of symmetric key that will be used to encrypt the message.


RSA in Practice, PKCS

  • used in practice, different from what we described above
  • key difference:
  key   ->   preprocess   ->   RSA
128bits 2048bits
  • RSA : Java API
// sender
// 1. generate private and public key
rasKeyG = KeyPairGenerator.getInstance("RSA");
privateKey = rasKeyG.getPrivate(); // secure you private key
publicKey = rasKeyG.getPublic();

// 2.share the public key, dont share the public key directly but share
// some parameters( e.g the modulus and exponent for RSA), so that the
// receiver can to reconstruct the public key.
RSAPublicKeySpec rsaPublicKeySpec =
MM,CX, (RSAPublicKeySpec)keyFactory.getKeySpec(
modulus = rsaPublicKeySpec.getModulus();
exponent =rsaPublicKeySpec.getPublicExponent();

// receiver
// 1. reconstruct the public key
// 2. generate the data and use the public key to encrypt the data
// 3. send the encrypted data and the other side will decrypt it using
// its private key

Public-Key Encryption using Diffie-Hallman

Public-Key Encryption Application

  • interactive
    • Used to exchange a symmetric key
  • non-interactive
    • Encrypted file system, and share the access enc. the message use enc. the sym. key with symmetric encryption public-key encryption | E(pk2, k) | <-- a="" b="" e="" enc.="" es="" k="" li="" m="" pk1="" pk="" s="" with="">

by Bin Chen ( at September 09, 2017 06:11

August 25, 2017

Steve McIntyre

Let's BBQ again, like we did last summer!

It's that time again! Another year, another OMGWTFBBQ! We're expecting 50 or so Debian folks at our place in Cambridge this weekend, ready to natter, geek, socialise and generally have a good time. Let's hope the weather stays nice, but if not we have gazebo technology... :-)

Many thanks to a number of awesome companies and people near and far who are sponsoring the important refreshments for the weekend:

I've even been working on the garden this week to improve it ready for the event. If you'd like to come and haven't already told us, please add yourself to the wiki page!

August 08, 2017 02:00

August 19, 2017

Leif Lindholm

OpenPlatformPkg is dead, long live edk2-platforms!

For a few years now, I have been working towards improving the availability of open source platform ports and device drivers for EDK2.

Initially, this began by setting up OpenPlatformPkg. This has been used both for platforms from Linaro members and external parties, and has already led to some amount of reduced code duplication, and moving common functionality to EDK2.

Now, the platforms that were in OpenPlatformPkg have been moved into the master branch of edk2-platforms, and OpenPlatformPkg itself has become a read-only archive.

So ... what changes?

Well, the first and most obvious change is that the repository now lives in the TianoCore area on github:

Like OpenPlatformPkg, this is not part of the main EDK2 repository. Unlike OpenPlatformPkg, there is an official way to work with this repository as part of the TianoCore group of projects. Code contributions to this repository are reviewed on the edk2-devel mailing list.

Secondly, the directory structure changes slightly. I will let you discover the specifics for yourself.

Thirdly, edk2-platforms is being kept license clean and source only. So binary-only content from OpenPlatformPkg was moved to a separate edk2-non-osi repository. We still want to enable platforms that have a number of non-open-source components to be able to share part of their code, but edk2-platforms will contain only free software.

At the same time, we change the build behavior from having OpenPlatformPkg nested under edk2 to building with edk2, edk2-platforms and (if needed) edk2-non-osi located "wherever" and individual packages located using PACKAGES_PATH.

Updates to uefi-tools

As before, I am way too lazy to keep figuring out the build command lines for each platform/toolchain combination, so I added support to uefi-tools for the new structure as well. Rather than breaking the compatibility of with OpenPlaformPkg, or making it more complex by making it support both, I added a new script called (which uses a new default platform configuration file called edk2-platforms.config).

Usage-wise, the most visible change is that the script no longer needs to be executed inside the edk2 directory; any directory it is executed from becomes the WORKSPACE, and build output, including intermediary stages, will be placed underneath it.

Secondly, the addition of new command line parameters to point out the locations of the various repositories involved in a build:

-e <edk2 directory>
-p <edk2-platforms directory>
-n <edk2-non-osi directory>

Release management

Well, the old strategies that could be used with edk2/OpenPlatformPkg to achieve a coherent commit on a single hash (git subrepos or submodules) are no longer much use. In order to make a tagged release over multiple repositories, a tool such as mr or repo will be necessary.

I will have to figure out which I pick for the Linaro Enterprise 17.10 release, but I have several weeks left for that :)

by Leif Lindholm at August 08, 2017 23:00

August 05, 2017

Leif Lindholm

Another new blog...

Well, I guess it's that time again. Much as I liked blosxom, it's not really maintained anymore, and the plugin architecture is ... archaic ... to say the least. So last year I started looking into pelican and found it would simplify my life a bit ... and then I started trying to move my existing blosxom theme over to pelican, and then I got bored of that and dropped everything.

However, I do need to be posting some more, and pelican has very nice and simple tags and drafts handling, as well as a lot more useful metadata functionality, whilst remaining as no-frills as I like (markdown support is good enough).

So here is a migration of all of the old content to the new architecture. Hopefully, I will get around to sorting the theme at some point, but at least I am functional again.

by Leif Lindholm at August 08, 2017 21:55

August 03, 2017

Siddhesh Poyarekar

Tunables story continued - glibc 2.26

Those of you tuned in to the wonderful world of system programming may have noticed that glibc 2.26 was released last night (or daytime if you live west of me or middle of the night/dawn if you live east of me, well you get the drift) and it came out with a host of new improvements, including the much awaited thread cache for malloc. The thread cache for malloc is truly a great step forward - it brings down latency of a bulk of allocations from hundreds of cycles to tens of cycles. The other major improvement that a bulk of users and developers will notice is the fact that glibc now detects when resolv.conf has changed and reloads the lookup configuration. Yes, this was long overdue but hey, it’s not like we were refusing patches for the past half a decade, so thank the nice soul (Florian Weimer) who actually got it done in the end.

We are not here to talk about the improvements mentioned in the NEWS. We are here to talk about an improvement that will likely have a long term impact on how optimizations are implemented in libraries. We are here to talk about…


Yes, I’m back with tunables, but this time I am not the one who did the work, it’s the wonderful people from Cavium and Intel who have started using tunables for a use case I had alluded to in my talk at Linaro Connect BKK 2016 and also in my previous blog post on tunables, which was the ability to influence IFUNCs.

IFUNCs? International functions? Intricate Functions? Impossibly ridiculous Functions?

There is a short introduction of the GNU Indirect Functions on the glibc wiki that should help you get started on this very powerful yet very complicated concept. In short, ifuncs extend the GOT/PLT mechanism of loading functions from dynamic libraries to loading different implementations of the same function depending on some simple selection criteria. Traditionally this has been based on querying the CPU for features that it supports and as a result we have had multiple variants of some very common functions such as memcpy_sse2 and memcpy_ssse3 for x86 processors that get executed based on the support declared by the processor the program is running on.

Tunables allow you to take this idea further because there are two ways to get performance benefits, (1) by utilizing all of the CPU features that help and (2) by catering to the workload. For example, you could have a workload that performs better with a supposedly sub-optimal memcpy variant for the CPU purely because of the way your data is structured or laid out. Tunables allow you to select that routine by pretending that the CPU has a different set of capabilities than it actually reports, by setting the glibc.tune.hwcaps tunable on x86 processors. Not only that, you can even tune cache sizes and non-temporal thresholds (i.e. threshold beyond which some routines use non-temporal instructions for loads and stores to optimize cache usage) to suit your workload. I won’t be surprised if some years down the line we see specialized implementations of these routines that cater to specific workloads, like memcpy_db for databases or memset_paranoid for a time invariant (or mostly invariant) implementation of memset.

Beyond x86

Here’s where another very important feature landed in glibc 2.26: multiarch support in aarch64. The ARMv8 spec is pretty standard and as a result the high level instruction set and feature set of vendor chips is pretty much the same with some minor trivial differences. However, even though the spec is standard, the underlying microarchitecture implementation could be very different and that meant that selection of instructions and scheduling differences could lead to sometimes very significant differences in performance and vendors obviously would like to take advantage of that.

The only way they could reliably (well, kind of, there should be a whole blog post for this) identify their processor variant (and hence deploy routines for their processors) was by reading the machine identification register or MIDR_EL1. If you’re familiar with aarch64 registers, you’ll notice that this register cannot be read by userspace, it can only be read by the kernel. The kernel thus had to trap and emulate this instruction, support for which is now available since Linux 4.11. In glibc 2.26, we now use MIDR_EL1 to identify which vendor processor the program is running on and deploy an optimal routine (in this case for the Cavium thunderxt88) for the processor.

But wait, what about earlier kernels, how do they take advantage of this? There’s a tunable for it! There’s glibc.tune.cpu for aarch64 that allows you to select the CPU variant you want to emulate. For some workloads you’ll find the generic memcpy actually works better and the tunable allows you to select that as well.

Finally due to tunables, the much needed cleanup of LD_HWCAP_MASK happened, giving rise to the tunable glibc.tune.hwcap_mask. Tunables also eliminated a lot of the inconsistency in environment variable behaviour due to the way static and dynamic executables are initialized, so you’ll see much less differences in the way your applications behave when they’re built dynamically vs when they’re built statically.

Wow, that sounds good, where do I sign up for your newsletter?

The full list of hardware capability tunables are documented in the glibc manual so take a look and feel free to hop on to the libc-help mailing list to discuss these tunables and suggest more ways in which you would like to tune the library for your workload. Remember that tunables don’t have any ABI/API guarantees for now, so they can be added or removed between releases as we deem fit. Also, your distribution may end up adding their own tunables too in future, so look out for those as well. Finally, system level tunables coming up real soon to allow system administrators to control how users use these tunables.

Happy hacking!

by Siddhesh at August 08, 2017 06:57

July 25, 2017

Rémi Duraffort

Using requests with xmlrpc

Using XML-RPC with Python3 is really simple. Calling system.version on http://localhost/RCP2 is as simple as:

import xmlrpc.client

proxy = xmlrpc.client.ServerProxy("http://localhost/RPC2")

However, the default client is missing many features, like handling proxies. Using requests for the underlying connection allows for greater control of the http request.

The xmlrpc client allows to change the underlying transport class by a custom class. In order to use requests, we create a simple Transport class:

import requests
import xmlrpc.client

class RequestsTransport(xmlrpc.client.Transport):

    def request(self, host, handler, data, verbose=False):
        # set the headers, including the user-agent
        headers = {"User-Agent": "my-user-agent",
                   "Content-Type": "text/xml",
                   "Accept-Encoding": "gzip"}
        url = "https://%s%s" % (host, handler)
            response = None
            response =, data=data, headers=headers)
            return self.parse_response(response)
        except requests.RequestException as e:
            if response is None:
                raise xmlrpc.client.ProtocolError(url, 500, str(e), "")
                raise xmlrpc.client.ProtocolError(url, response.status_code,
                                                  str(e), response.headers)

    def parse_response(self, resp):
        Parse the xmlrpc response.
        p, u = self.getparser()
        return u.close()

To use this Transport class, we should use:

proxy = xmlrpc.client.ServerProxy(uri, transport=RequestsTransport())

We can now use requests to:

  • use proxies
  • skip ssl verification (on a development server) or adding the right certificate chain
  • set the headers
  • set the timeouts
  • ...

See the documentation or an example for more information.

by Rémi Duraffort at July 07, 2017 08:33

July 24, 2017

Peter Maydell

Installing Debian on QEMU’s 64-bit ARM “virt” board

This post is a 64-bit companion to an earlier post of mine where I described how to get Debian running on QEMU emulating a 32-bit ARM “virt” board. Thanks to commenter snak3xe for reminding me that I’d said I’d write this up…

Why the “virt” board?

For 64-bit ARM QEMU emulates many fewer boards, so “virt” is almost the only choice, unless you specifically know that you want to emulate one of the 64-bit Xilinx boards. “virt” supports supports PCI, virtio, a recent ARM CPU and large amounts of RAM. The only thing it doesn’t have out of the box is graphics.

Prerequisites and assumptions

I’m going to assume you have a Linux host, and a recent version of QEMU (at least QEMU 2.8). I also use libguestfs to extract files from a QEMU disk image, but you could use a different tool for that step if you prefer.

I’m going to document how to set up a guest which directly boots the kernel. It should also be possible to have QEMU boot a UEFI image which then boots the kernel from a disk image, but that’s not something I’ve looked into doing myself. (There may be tutorials elsewhere on the web.)

Getting the installer files

I suggest creating a subdirectory for these and the other files we’re going to create.

wget -O installer-linux
wget -O installer-initrd.gz

Saving them locally as installer-linux and installer-initrd.gz means they won’t be confused with the final kernel and initrd that the installation process produces.

(If we were installing on real hardware we would also need a “device tree” file to tell the kernel the details of the exact hardware it’s running on. QEMU’s “virt” board automatically creates a device tree internally and passes it to the kernel, so we don’t need to provide one.)


First we need to create an empty disk drive to install onto. I picked a 5GB disk but you can make it larger if you like.

qemu-img create -f qcow2 hda.qcow2 5G

(Oops — an earlier version of this blogpost created a “qcow” format image, which will work but is less efficient. If you created a qcow image by mistake, you can convert it to qcow2 with mv hda.qcow2 old-hda.qcow && qemu-img convert -O qcow2 old-hda.qcow hda.qcow2. Don’t try it while the VM is running! You then need to update your QEMU command line to say “format=qcow2” rather than “format=qcow”. You can delete the old-hda.qcow once you’ve checked that the new qcow2 file works.)

Now we can run the installer:

qemu-system-aarch64 -M virt -m 1024 -cpu cortex-a53 \
  -kernel installer-linux \
  -initrd installer-initrd.gz \
  -drive if=none,file=hda.qcow2,format=qcow2,id=hd \
  -device virtio-blk-pci,drive=hd \
  -netdev user,id=mynet \
  -device virtio-net-pci,netdev=mynet \
  -nographic -no-reboot

The installer will display its messages on the text console (via an emulated serial port). Follow its instructions to install Debian to the virtual disk; it’s straightforward, but if you have any difficulty the Debian installation guide may help.

The actual install process will take a few hours as it downloads packages over the network and writes them to disk. It will occasionally stop to ask you questions.

Late in the process, the installer will print the following warning dialog:

   +-----------------| [!] Continue without boot loader |------------------+
   |                                                                       |
   |                       No boot loader installed                        |
   | No boot loader has been installed, either because you chose not to or |
   | because your specific architecture doesn't support a boot loader yet. |
   |                                                                       |
   | You will need to boot manually with the /vmlinuz kernel on partition  |
   | /dev/vda1 and root=/dev/vda2 passed as a kernel argument.             |
   |                                                                       |
   |                              <Continue>                               |
   |                                                                       |

Press continue for now, and we’ll sort this out later.

Eventually the installer will finish by rebooting — this should cause QEMU to exit (since we used the -no-reboot option).

At this point you might like to make a copy of the hard disk image file, to save the tedium of repeating the install later.

Extracting the kernel

The installer warned us that it didn’t know how to arrange to automatically boot the right kernel, so we need to do it manually. For QEMU that means we need to extract the kernel the installer put into the disk image so that we can pass it to QEMU on the command line.

There are various tools you can use for this, but I’m going to recommend libguestfs, because it’s the simplest to use. To check that it works, let’s look at the partitions in our virtual disk image:

$ virt-filesystems -a hda.qcow2 

If this doesn’t work, then you should sort that out first. A couple of common reasons I’ve seen:

  • if you’re on Ubuntu then your kernels in /boot are installed not-world-readable; you can fix this with sudo chmod 644 /boot/vmlinuz*
  • if you’re running Virtualbox on the same host it will interfere with libguestfs’s attempt to run KVM; you can fix that by exiting Virtualbox

Looking at what’s in our disk we can see the kernel and initrd in /boot:

$ virt-ls -a hda.qcow2 /boot/

and we can copy them out to the host filesystem:

virt-copy-out -a hda.qcow2 /boot/vmlinuz-4.9.0-3-arm64 /boot/initrd.img-4.9.0-3-arm64 .

(We want the longer filenames, because vmlinuz and initrd.img are just symlinks and virt-copy-out won’t copy them.)

An important warning about libguestfs, or any other tools for accessing disk images from the host system: do not try to use them while QEMU is running, or you will get disk corruption when both the guest OS inside QEMU and libguestfs try to update the same image.

If you subsequently upgrade the kernel inside the guest, you’ll need to repeat this step to extract the new kernel and initrd, and then update your QEMU command line appropriately.


To run the installed system we need a different command line which boots the installed kernel and initrd, and passes the kernel the command line arguments the installer told us we’d need:

qemu-system-aarch64 -M virt -m 1024 -cpu cortex-a53 \
  -kernel vmlinuz-4.9.0-3-arm64 \
  -initrd initrd.img-4.9.0-3-arm64 \
  -append 'root=/dev/vda2' \
  -drive if=none,file=hda.qcow2,format=qcow2,id=hd \
  -device virtio-blk-pci,drive=hd \
  -netdev user,id=mynet \
  -device virtio-net-pci,netdev=mynet \

This should boot to a login prompt, where you can log in with the user and password you set up during the install.

The installation has an SSH client, so one easy way to get files in and out is to use “scp” from inside the VM to talk to an SSH server outside it. Or you can use libguestfs to write files directly into the disk image (for instance using virt-copy-in) — but make sure you only use libguestfs when the VM is not running, or you will get disk corruption.

by pm215 at July 07, 2017 09:25

July 21, 2017

Gema Gomez

Acer Shawl

Last weekend I attended a class at The Sheep Shop. It was the Easy crochet lace class by Joanne Scrace. Just for attending the class, we got a copy of the Acer Shawl pattern by Joanne. It was easy to get into the rythm of it and well explained. This is the sample I managed to do during the three hours of the class:

Class sample

I have continued working on it this week, and I managed to finish two skeins of 50g each of Louisa Harding Yarn, Amitola, color Tinkerbell (134). I have bought a third skein to make it slightly bigger, but it is looking lovely:


Crochet hook used for this: 5.0mm.

This was the first time I work with a colour changing yarn on a project like this. I have been rather careful when changing skeins to match the tones of both ends of the yarn and the trick worked wonders for a very neat finish.

Thank you Joanne for such a lovely and simple pattern!

by Gema Gomez at July 07, 2017 23:00

July 15, 2017

Bin Chen

Booting Andriod with u-boot

Booting Andriod with u-boot

u-boot is an open source bootloader that you will find in lots of embedded devices, including Android, and that’s what we are going to talk about today - boot up Android with u-boot.

Andriod Boot Image

Andriod boot image usually contains the kernel image and rootfs, and sometime dtb, you can either conconact the dtb to the kernel image or put it into the 2ndloader section. We’ll explain in more detail later.
|             | Description
| -------- | ------------------------------------------------|
| Header | kernel cmdline, base/offset for kernel/ramdisk |
| kernel | kernel, may include dtb |
| ramdisk | roofs |
| 2ndloader | 2nd bootloader |
The following command (simplified for sake for simplicty) is what is used to create an Android boot image, using make bootimage.
mkbootimg --kernel zImage --ramdisk ramdisk.img.gz --cmdline 'xxxx' -o --boot.img
We’ll ignore the details regarding how the rootfs (ramdisk.img.gz) is created, which is indeed very intresting.

bootm overview

Bootm is an u-boot command that is used to booting a system from the memory, as the suffix mindicated. The full form of this command takes three parameters:
# bootm kernel ramfs dtb. 
The first parameter is the kernel address, the second one is the ram rootfs and the last one is the dtb. Only the first parameters is mandatory, and we call it boot image address generally without limiting it being kernel only.
u-boot support several boot image formats, such as uImage, which is u-boot defined image format. It is also called legacy image format but worth noting that uImage is not limited to be boot image. Instead, it is a generic container image format that can be recognized by u-boot. For example, you can package a raw ramdisk into the uImage format as well.
The other boot image format supported is Android bootimage format as we discussed above. Since Android bootimage contains the rootfs so there is no need to specify the second parameters but instead the uboot will extract the ramdisk out of the boot image and set up it correctly. For the dtb, there are approaches of concatenating it to the kernel but it requires the kernel’s awareness and ability to pull out of dtb. But not all the kernels are able to do that. For example, it is supported by aarch32 but not aarch64.
Nowadays, people are encouraged to use dtb and pass that explicitly to the kernel when booting. It is a good practice to have a dedicated partition for the dtb, so that it can be upgraded independently. But to retrieve the dtb (so as to pass to the bootm), it depends on the partition scheme you are using. Ideally, what we need is find_partition_by_name function (we have it in part.c::part_get_info_by_name). It is a breeze if gpt is used but in the case of mbrand/orerb there will be some hair scratch. As a comprise or solution, I proposed to use the underused section of Android boot image to hold the dtb, so that the Andriod boot image become a self-contained and self-sufficient boot image. And that is the way we use it for Poplar 96board.

bootm implementation

Down to essence, here are steps that are performed in the bootm.
  1. Find the kernel/os
  2. Find rootfs and dtb
  3. Relocate/decompress the kernel, if needed
  4. Relocate the rootfs if needed
  5. setup fdt or atag
  6. jump to the kernel
In the bootm implementations, those steps are called state and the entry function is do_bootm_states. You active specific step by passing corresponding enum values. say for bootm command, it is simply calling
 return do_bootm_states(
&images, 1);
We’ll go over those states and see what will be done in each steps:
It will set up the lmb. Isn’t particular interesting.
  1. BOOTM_STATE_FINDOS -> bootm_find_os -> boot_get_kernel
boot_get_kernel will set up images.os.image_start and images.os.image_len, which are the kernel location in the ram and its length.
Some other important fields and concept: os.load, ep, os.start
images.os.load = android_image_get_kload(os_hdr);
images.ep = images.os.load;
images.os.load is used as the relocation destination when it is of different value with os.image_start. And kernel relocation usually happens. Relocations happen in the BOOTM_STATE_LOADOS stage we’ll see later.
images.ep is an alias for os.load. They should be of same value and is used to as the kernel entry point after the relocation is done.
  kernel_entry = (void (*)(void *fdt_addr, void *res0, void *res1,
void *res2))images->ep;
images.os.start, for Android, it is the start of whole boot image (not the kernel inside of the boot image as os.image_start pointed to)
There are other fields not related to the addresses but more for information so that can be handled differently if required.
images.os.type = IH_TYPE_KERNEL;
images.os.comp = IH_COMP_NONE;
images.os.os = IH_OS_LINUX;
  1. BOOTM_STATE_FINDOTHER -> bootm_find_others -> bootm_find_images
Find ramdisk memory address, setting up images.rd_start and images.rd_end.
Find dtb memory address, setting up images.ft_addr and images.ft_len
copy the image, decompress if needed, from ram address (image.os.image_start) to its load address (image.load) and reserve that area from lmb.
iflag = bootm_disable_interrupts();
ret = bootm_load_os(images, &load_end, 0);
if (ret == 0)
lmb_reserve(&images->lmb, images->os.load,
(load_end - images->os.load));
Will this update the image.ep as well? NO. ep is fixed when os.load is fixed.
setup the ramdisk relocation address, i.e images.initrd_start and images.initrd_end.
initrd_start is the final ramdisk load address (as os.load is for the kernel). If initrd_start is different from rd_start, ramdisk relocation will happen. The source ramdisk address is determined in the bootmcommand line (directly or indirectly) and setup in the BOOTM_STATE_FINDOTHER stage mentioned above; the destination ramdisk address can be controlled by several factors, including compile options (CONFIG_SYS_BOOT_RAMDISK_HIGH), environment variables (initrd_high) and some field in the boot image, such as the ramdisk load address in the Android bootimage. And I wholehearted agree with you it is a huge headache for a beginner (like me) to sort this out.
 * boot_ramdisk_high() takes a relocation hint from "initrd_high" environment
* variable and if requested ramdisk data is moved to a specified location.
int boot_ramdisk_high(struct lmb *lmb, ulong rd_data, ulong rd_len,
ulong *initrd_start, // those are out
ulong *initrd_end) // out
Setup the kernel parameters using either atag or fdt, and the later take precedence over the former one. When fdt is used, it will fix up the kernel command line, ramdisk address if it is used, by amending the dtb.
int fdt_initrd(void *fdt, ulong initrd_start, ulong initrd_end) {
err = fdt_setprop_uxx(fdt, nodeoffset, "linux,initrd-start",
(uint64_t)initrd_start, is_u64);
Linux go!

by Bin Chen ( at July 07, 2017 10:37

July 05, 2017

Ard Biesheuvel

GHASH for low-end ARM cores

The Galois hash algorithm (GHASH) is a fairly straight-forward keyed hash algorithm based on finite field multiplication, using the field GF(2128) with characteristic polynomial x128 + x7 + x2 + x + 1. (An excellent treatment of Galois fields can be found here)

The significance of GHASH is that it is used as the authentication component in the GCM algorithm, which is an implementation of authenticated encryption with associated data (AEAD), a cryptographic mode that combines authentication of data sent in the clear with authentication of data that is sent in encrypted form at the same time. It is widely used these days, primarily in the networking domain (IPsec, IEEE 802.11)

ISA support

Both the Intel and ARMv8 instruction sets now contain support for carry-less multiplication (also known as polynomial multiplication), primarily to allow for accelerated implementations of GHASH to be created, which formerly had to rely on unwieldy and less secure table based implementations. (The Linux implementation pre-computes a 4 KB lookup table for each instance of the hash algorithm that is in use, i.e., for each session having a different key. 4 KB per IPsec connection does not sound too bad in terms of memory usage, but the D-cache footprint may become a bottleneck when serving lots of concurrent connections.) In contrast, implementations based on these special instructions are time invariant, and are significantly faster (around 16x on high end ARMv8 cores).

Unfortunately, though, while ARMv8 specifies a range of polynomial multiplication instructions with various operand sizes, the one we are most interested in, which performs carry-less multiplication on two 64-bit operands to produce a 128-bit result, is optional in the architecture. So on low-end cores such as the Cortex-A53 (as can be found in the Raspberry Pi 3), the accelerated driver is not available because this particular instruction is not implemented.

Using vmull.p8 to implement vmull.p64

The other day, I stumbled upon the paper Fast Software Polynomial Multiplication on ARM Processors Using the NEON Engine by Danilo Camara, Conrado Gouvea, Julio Lopez and Ricardo Dahab, which describes how 64×64 to 128 bit polynomial multiplication (vmull.p64) can be composed using 8×8 to 16 bit polynomial multiplication (vmull.p8) combined with other SIMD arithmetic instructions. The nice thing about vmull.p8 is that it is a standard NEON instruction, which means all NEON capable CPUs implement it, including the Cortex-A53 on the Raspberry Pi 3.

Transliterating 32-bit ARM code to the 64-bit ISA

The algorithm as described in the paper is based on the 32-bit instruction set (retroactively named AArch32), which deviates significantly from the new 64-bit ISA called AArch64. The primary difference is that the number of SIMD registers has increased to 32, which is nice, but which has a downside as well: it is no longer possible to directly use the top half of a 128-bit register as a 64-bit register, which is something the polynomial multiplication algorithm relies on.

The original code looks something like this (note the use of ‘high’ and ‘low’ registers in the same instruction)

.macro          vmull_p64, rq, ad, bd
vext.8          t0l, \ad, \ad, #1       @ A1
vmull.p8        t0q, t0l, \bd           @ F = A1*B
vext.8          \rq\()_L, \bd, \bd, #1  @ B1
vmull.p8        \rq, \ad, \rq\()_L      @ E = A*B1
vext.8          t1l, \ad, \ad, #2       @ A2
vmull.p8        t1q, t1l, \bd           @ H = A2*B
vext.8          t3l, \bd, \bd, #2       @ B2
vmull.p8        t3q, \ad, t3l           @ G = A*B2
vext.8          t2l, \ad, \ad, #3       @ A3
vmull.p8        t2q, t2l, \bd           @ J = A3*B
veor            t0q, t0q, \rq           @ L = E + F
vext.8          \rq\()_L, \bd, \bd, #3  @ B3
vmull.p8        \rq, \ad, \rq\()_L      @ I = A*B3
veor            t1q, t1q, t3q           @ M = G + H
vext.8          t3l, \bd, \bd, #4       @ B4
vmull.p8        t3q, \ad, t3l           @ K = A*B4
veor            t0l, t0l, t0h           @ t0 = (L) (P0 + P1) << 8
vand            t0h, t0h, k48
veor            t1l, t1l, t1h           @ t1 = (M) (P2 + P3) << 16
vand            t1h, t1h, k32
veor            t2q, t2q, \rq           @ N = I + J
veor            t0l, t0l, t0h
veor            t1l, t1l, t1h
veor            t2l, t2l, t2h           @ t2 = (N) (P4 + P5) << 24
vand            t2h, t2h, k16
veor            t3l, t3l, t3h           @ t3 = (K) (P6 + P7) << 32
vmov.i64        t3h, #0
vext.8          t0q, t0q, t0q, #15
veor            t2l, t2l, t2h
vext.8          t1q, t1q, t1q, #14
vmull.p8        \rq, \ad, \bd           @ D = A*B
vext.8          t2q, t2q, t2q, #13
vext.8          t3q, t3q, t3q, #12
veor            t0q, t0q, t1q
veor            t2q, t2q, t3q
veor            \rq, \rq, t0q
veor            \rq, \rq, t2q

However, things like veor t1l, t1l, t1h or using ext with upper halves of registers are not possible in AArch64, and so we need to transpose the contents of some of registers using the tbl and/or zip/unzip instructions. Also, the vmull.p8 instruction now exists in two variants: pmull operating on the lower halves and pmull2 operating on the upper halves of the input operands.

We end up with the following sequence, which is 3 instructions longer than the original:

.macro          __pmull_p8, rq, ad, bd, i
.ifb            \i
ext             t4.8b, \ad\().8b, \ad\().8b, #1         // A1
ext             t8.8b, \bd\().8b, \bd\().8b, #1         // B1
ext             t5.8b, \ad\().8b, \ad\().8b, #2         // A2
ext             t7.8b, \bd\().8b, \bd\().8b, #2         // B2
ext             t6.8b, \ad\().8b, \ad\().8b, #3         // A3
ext             t9.8b, \bd\().8b, \bd\().8b, #3         // B3
ext             t3.8b, \bd\().8b, \bd\().8b, #4         // B4

pmull           t4.8h, t4.8b, \bd\().8b                 // F = A1*B
pmull           t8.8h, \ad\().8b, t8.8b                 // E = A*B1
pmull           t5.8h, t5.8b, \bd\().8b                 // H = A2*B
pmull           t7.8h, \ad\().8b, t7.8b                 // G = A*B2
pmull           t6.8h, t6.8b, \bd\().8b                 // J = A3*B
pmull           t9.8h, \ad\().8b, t9.8b                 // I = A*B3
pmull           t3.8h, \ad\().8b, t3.8b                 // K = A*B4
pmull           \rq\().8h, \ad\().8b, \bd\().8b         // D = A*B
tbl             t4.16b, {\ad\().16b}, perm1.16b         // A1
tbl             t8.16b, {\bd\().16b}, perm1.16b         // B1
tbl             t5.16b, {\ad\().16b}, perm2.16b         // A2
tbl             t7.16b, {\bd\().16b}, perm2.16b         // B2
tbl             t6.16b, {\ad\().16b}, perm3.16b         // A3
tbl             t9.16b, {\bd\().16b}, perm3.16b         // B3
tbl             t3.16b, {\bd\().16b}, perm4.16b         // B4

pmull2          t4.8h, t4.16b, \bd\().16b               // F = A1*B
pmull2          t8.8h, \ad\().16b, t8.16b               // E = A*B1
pmull2          t5.8h, t5.16b, \bd\().16b               // H = A2*B
pmull2          t7.8h, \ad\().16b, t7.16b               // G = A*B2
pmull2          t6.8h, t6.16b, \bd\().16b               // J = A3*B
pmull2          t9.8h, \ad\().16b, t9.16b               // I = A*B3
pmull2          t3.8h, \ad\().16b, t3.16b               // K = A*B4
pmull2          \rq\().8h, \ad\().16b, \bd\().16b       // D = A*B

eor             t4.16b, t4.16b, t8.16b                  // L = E + F
eor             t5.16b, t5.16b, t7.16b                  // M = G + H
eor             t6.16b, t6.16b, t9.16b                  // N = I + J

uzp1            t8.2d, t4.2d, t5.2d
uzp2            t4.2d, t4.2d, t5.2d
uzp1            t7.2d, t6.2d, t3.2d
uzp2            t6.2d, t6.2d, t3.2d

// t4 = (L) (P0 + P1) << 8
// t5 = (M) (P2 + P3) << 16
eor             t8.16b, t8.16b, t4.16b
and             t4.16b, t4.16b, k32_48.16b

// t6 = (N) (P4 + P5) << 24
// t7 = (K) (P6 + P7) << 32
eor             t7.16b, t7.16b, t6.16b
and             t6.16b, t6.16b, k00_16.16b

eor             t8.16b, t8.16b, t4.16b
eor             t7.16b, t7.16b, t6.16b

zip2            t5.2d, t8.2d, t4.2d
zip1            t4.2d, t8.2d, t4.2d
zip2            t3.2d, t7.2d, t6.2d
zip1            t6.2d, t7.2d, t6.2d

ext             t4.16b, t4.16b, t4.16b, #15
ext             t5.16b, t5.16b, t5.16b, #14
ext             t6.16b, t6.16b, t6.16b, #13
ext             t3.16b, t3.16b, t3.16b, #12

eor             t4.16b, t4.16b, t5.16b
eor             t6.16b, t6.16b, t3.16b
eor             \rq\().16b, \rq\().16b, t4.16b
eor             \rq\().16b, \rq\().16b, t6.16b

On the Raspberry Pi 3, this code runs 2.8x faster than the generic, table based C code. This is a nice improvement, but we can do even better.

GHASH reduction

The accelerated GHASH implementation that uses the vmull.p64 instruction looks like this:

ext		T2.16b, XL.16b, XL.16b, #8
ext		IN1.16b, T1.16b, T1.16b, #8
eor		T1.16b, T1.16b, T2.16b
eor		XL.16b, XL.16b, IN1.16b

pmull2		XH.1q, XL.2d, SHASH.2d		// a1 * b1
eor		T1.16b, T1.16b, XL.16b
pmull	 	XL.1q, XL.1d, SHASH.1d		// a0 * b0
pmull		XM.1q, T1.1d, SHASH2.1d		// (a1 + a0)(b1 + b0)

eor		T2.16b, XL.16b, XH.16b
ext		T1.16b, XL.16b, XH.16b, #8
eor		XM.16b, XM.16b, T2.16b

pmull		T2.1q, XL.1d, MASK.1d
eor		XM.16b, XM.16b, T1.16b

mov		XH.d[0], XM.d[1]
mov		XM.d[1], XL.d[0]

eor		XL.16b, XM.16b, T2.16b
ext		T2.16b, XL.16b, XL.16b, #8
pmull		XL.1q, XL.1d, MASK.1d

eor		T2.16b, T2.16b, XH.16b
eor		XL.16b, XL.16b, T2.16b

What should be noted here is that the finite field multiplication consists of a multiplication step and a reduction step, where the latter essentially performs the modulo division involving the characteristic polynomial (which is how we normalize the result, i.e., ensure that it remains inside the field)

So while this sequence is optimal for cores that implement vmull.p64 natively, we can switch to a reduction step that does not involve polynomial multiplication at all, and avoid two copies of the fallback vmull.p64 sequence consisting of 40 instructions each.

ext		T2.16b, XL.16b, XL.16b, #8
ext		IN1.16b, T1.16b, T1.16b, #8
eor		T1.16b, T1.16b, T2.16b
eor		XL.16b, XL.16b, IN1.16b

__pmull_p8	XH, XL, SHASH, 2		// a1 * b1
eor		T1.16b, T1.16b, XL.16b
__pmull_p8 	XL, XL, SHASH			// a0 * b0
__pmull_p8	XM, T1, SHASH2			// (a1 + a0)(b1 + b0)

eor		T2.16b, XL.16b, XH.16b
ext		T1.16b, XL.16b, XH.16b, #8
eor		XM.16b, XM.16b, T2.16b

eor		XM.16b, XM.16b, T1.16b

mov		XL.d[1], XM.d[0]
mov		XH.d[0], XM.d[1]

shl		T1.2d, XL.2d, #57
shl		T2.2d, XL.2d, #62
eor		T2.16b, T2.16b, T1.16b
shl		T1.2d, XL.2d, #63
eor		T2.16b, T2.16b, T1.16b
ext		T1.16b, XL.16b, XH.16b, #8
eor		T2.16b, T2.16b, T1.16b

mov		XL.d[1], T2.d[0]
mov		XH.d[0], T2.d[1]

ushr		T2.2d, XL.2d, #1
eor		XH.16b, XH.16b, XL.16b
eor		XL.16b, XL.16b, T2.16b
ushr		T2.2d, T2.2d, #6
ushr		XL.2d, XL.2d, #1

eor		T2.16b, T2.16b, XH.16b
eor		XL.16b, XL.16b, T2.16b

Loop invariants

Another observation one can make when looking at this code is that the vmull.p64 calls that remain all involve right hand sides that are invariants during the execution of the loop. For the version that uses the native vmull.p64, this does not matter much, but for our fallback sequence, it means that some instructions essentially calculate the same value each time, and the computation can be taken out of the loop instead.

Since we have plenty of spare registers on AArch64, we can dedicate 8 of them to prerotated B1/B2/B3/B4 values of SHASH and SHASH2. With that optimization folded in as well, this implementation runs at 4x the speed of the generic GHASH driver. When combined with the bit-sliced AES driver, GCM performance on the Cortex-A53 increases twofold, from 58 to 29 cycles per byte.

The patches implementing this for AArch64 and for AArch32 can be found here.

by ardbiesheuvel at July 07, 2017 12:12

June 30, 2017

Gema Gomez

Stitching group

A couple of months ago we started a Stitch ‘n B*tch group at work. We meet every week on Thursdays at lunchtime in a meeting room for those of us in the office and via online conference for the rest.

We work in technology and most of our workforce is remote, so we decided to make the group inclusive and invite not only people that may be interested in the office, but also colleagues working from home. So far there is four of us regularly attending this once a week meetup at lunchtime and we are having lots of fun sharing stories from our respective areas of the company. We are all from different departments and if it weren’t due to this hobby we all share, we may have never gotten to know each other that much.

I cannot encourage crafters out there enough to get organised and do something other than sitting in front of the computer during the lunch hour. Stitching or walking are fun activities and help socialize with your colleagues, plus they are fun. And it makes us so much more productive afterwards!

We are currently planning to attend at the end of the month together :-)

by Gema Gomez at June 06, 2017 23:00

June 27, 2017

Gema Gomez


We went on a trip to Malta recently. There are a lot of beautiful places, we visited the three islands, here is a spot in Comino that I found particularly breathtaking, the Crystal Lagoon:

Crystal Lagoon

I tried to find a good yarn shop, as you do when you go on holidays somewhere nice and exotic. Malta has a lot of shops where they sell handmade items, crocheted or knitted, they seem to have a long tradition of the crafts. Finding a shop for yarn was, however, not possible.

Maltese Lace is beautiful. I was told that one of these shawls, 100% silk, take around 4 months to make and sells for 100 euros:

Maltese Shawls

They also make lovely jumpers:

Maltese Jumpers

by Gema Gomez at June 06, 2017 23:00

June 23, 2017

Riku Voipio

Cross-compiling with debian stretch

Debian stretch comes with cross-compiler packages for selected architectures:
 $ apt-cache search cross-build-essential
crossbuild-essential-arm64 - Informational list of cross-build-essential packages for
crossbuild-essential-armel - ...
crossbuild-essential-armhf - ...
crossbuild-essential-mipsel - ...
crossbuild-essential-powerpc - ...
crossbuild-essential-ppc64el - ...

Lets have a quick exact steps guide. But first - while you can use do all this in your desktop PC rootfs, it is more wise to contain yourself. Fortunately, Debian comes with a container tool out of box:

sudo debootstrap stretch /var/lib/container/stretch
echo "strech_cross" | sudo tee /var/lib/container/stretch/etc/debian_chroot
sudo systemd-nspawn -D /var/lib/container/stretch
Then we set up cross-building enviroment for arm64 inside the container:

# Tell dpkg we can install arm64
dpkg --add-architecture arm64
# Add src line to make "apt-get source" work
echo "deb-src stretch main" >> /etc/apt/sources.list
apt-get update
# Install cross-compiler and other essential build tools
apt install --no-install-recommends build-essential crossbuild-essential-arm64
Now we have a nice build enviroment, lets choose something more complicated than the usual kernel/BusyBox to cross-build, qemu:

# Get qemu sources from debian
apt-get source qemu
cd qemu-*
# New in stretch: build-dep works in unpacked source tree
apt-get build-dep -a arm64 .
# Cross-build Qemu for arm64
dpkg-buildpackage -aarm64 -j6 -b
Now that works perfectly for Qemu. For other packages, challenges may appear. For example you may have to se "nocheck" flag to skip build-time unit tests. Or some of the build-dependencies may not be multiarch-enabled. So work continues :)

by Riku Voipio ( at June 06, 2017 13:36

June 22, 2017

Steve McIntyre

-1, Trolling

Here's a nice comment I received by email this morning. I guess somebody was upset by my last post?

From: Tec Services <>
Date: Wed, 21 Jun 2017 22:30:26 -0700
Subject: its time for you to retire from debian...unbelievable..your
         the quality guy and fucked up the installer!

i cant ever remember in the hostory of computing someone releasing an installer
that does not work!!


you need to be retired...due to being retarded..

and that this was dedicated to ian...what a should be ashames..he is probably roling in his grave from shame
right now....

It's nice to be appreciated.

June 06, 2017 21:59

June 20, 2017

Steve McIntyre

So, Stretch happened...

Things mostly went very well, and we've released Debian 9 this weekend past. Many many people worked together to make this possible, and I'd like to extend my own thanks to all of them.

As a project, we decided to dedicate Stretch to our late founder Ian Murdock. He did much of the early work to get Debian going, and inspired many more to help him. I had the good fortune to meet up with Ian years ago at a meetup attached to a Usenix conference, and I remember clearly he was a genuinely nice guy with good ideas. We'll miss him.

For my part in the release process, again I was responsible for producing our official installation and live images. Release day itself went OK, but as is typical the process ran late into Saturday night / early Sunday morning. We made and tested lots of different images, although numbers were down from previous releases as we've stopped making the full CD sets now.

Sunday was the day for the release party in Cambridge. As is traditional, a group of us met up at a local hostelry for some revelry! We hid inside the pub to escape from the ridiculouly hot weather we're having at the moment.


Due to a combination of the lack of sleep and the heat, I nearly forgot to even take any photos - apologies to the extra folks who'd been around earlier whom I missed with the camera... :-(

June 06, 2017 22:21

June 18, 2017

Gema Gomez


Over the past two months I have been working on a tool to help my team provision ARM64 hardware with specific images so they can be tested. We currently have labs with OpenStack installed over a few servers, we have labs for testing that need to be reprovisioned regularly depending on what is being tested. We didn’t have a standard way to install across data centers, get repeatable and reliable test results, share and test systems across vendors and architectures. So this was the first step on the quest of setting up a reliable infrastructure lab that will help, amongst others with 3rd party CI for OpenStack.

The initial requirements were simple:

  • being able to assign servers to users for testing without having to share admin rights on the infrastructure (controlled test environment)
  • install servers with kernel/initrd of the engineer’s choosing
  • ability to preseed/kickstart the installs by the engineer and debug end to end
  • remote console access to the servers
  • asset management of the lab integrated, rather than an external spreadsheet
  • useful for manual (UI) and automated (API) testing
  • generic tool, not vendor specific
  • easy to use

Previously in the data center we were using tftp servers with manually uploaded images that some admin would upload at the engineer’s request, not very consistent versioning, manually updated grub configs with options that were not consistent.

Now we have a tool, under heavy development, that allows us to do all of the above and we have open sourced it for anyone to be able to use or contribute.

More information

Design documentation: readthedocs

Source code:

by Gema Gomez at June 06, 2017 23:00

June 10, 2017

Bin Chen

Linux Security: seccomp, and its usage in Android and Docker

seccomp is short for SECure COMPuting. It sounds like a quite broad techniques but actually its scope is quite narrow, but effective. Simply put, it is a default deny white-list firewall used by kernel to restricting what syscalls a process can make.
seccomp is widely used lots of popular systems to sandbox the processes and/to reduce the kernel attacking surface, notably Chromium, Android and Docker.

How it works

We mentioned previously seccomp fundamentally is a white-list that kernel will check again for each process where a particular process are allowed to call a certain system call.
Technically, the white-list is written using Berkeley Packet Filter (BPF) rules, which will then be passed to seccomp system call.
Writing the rules using BPF and isn't intuitively for most programmers, so there are different wrappers making it more user friendly. Android use minijail, which is actually come from Chromium. Docker has golang wrapper, where you can write the profile in json format.
We'll see how they are used in practice.

seccomp in Android

Each process or service will have a seccomp policy defined by Android. minijail is the helper library used to parse the policy file and pass it to the kernel.
Below we'll see in detail how seccomp is used for mediaextractor service. Let's jump directly to the code:
static const char kSeccompFilePath[] =
int MiniJail()
struct minijail *jail = minijail_new();
minijail_parse_seccomp_filters(jail, kSeccompFilePath);
return 0;
It is quite straightforward, thanks to the very self explanatory function name and the great analogy (minijail) used here.
We first create a minijail, parse policy (converting into the BPF filter), and finally enter the jail (calling seccomp system call) (so called enter the jail).
A peek of format/content of the mediaextractor-seccomp.policy makes things clearer - it lists all the syscalls that are allowed in the target process.
ioctl: 1
futex: 1
prctl: 1
write: 1
getpriority: 1
mmap2: 1
close: 1
10munmap: 1
dupe: 1
mprotect: 1
getuid32: 1
setpriority: 1

seccomp in Docker

seccomp was introduced to Docker after v1.0. A seccomp profile can be specified at docker run time using -security-opt seccomp=.jsonparameters, when docker create or docker create.
docker run -it --rm --security-opt seccomp=.json alpine sh ...
If no seccomp profile is not specified, a default profile will be used. With the default profile, 40+ system calls out of 300+ are disabled to ensure a moderate protection. The secure profile is in JSON format, which will be converted to the BPF filter by Docker daemon, and then apply to the created process/container.
The applications packaged in the Docker can only allowed to call the system calls listed in the seccomp profile you specified, giving you more power to control the security aspect of the container.


In this article, we discussed what is seccomp and how it used by Android and Docker to build a securer system.

by Bin Chen ( at June 06, 2017 04:33

May 26, 2017

Siddhesh Poyarekar

The story of tunables

This is long overdue and I have finally got around to writing this. Apologies to everyone who asked me to write about it and I responded with "Oh yeah, right away!" If you are not interested in the story bits, start with So what are tunables anyway below.

The story of tunables began in 2013 when I was a relatively fresh glibc engineer in the Red Hat toolchain team. We wanted to add an environment variable to allow users to set the default stack sizes for thread stacks and Carlos took that idea to the next level with the question: How do we make this more extensible so that we have full control over the kind of tuning parameters we accept in glibc but at the same time, allow distributions to add their own tuning parameters without affecting upstream code? He asked this question in the 2013 Cauldron in Mountain View, where the famous glibc BoF happened in a tiny meeting room which overflowed into an adjacent room, which also filled up quickly, and then the BoF overran its 45 minute slot by roughly a couple of hours! Carlos joined the BoF over Hangout (I think it was called Google Talk then) because he couldn’t make it and we had a lengthy back and forth about the pros and cons of having such tuning parameters. In principle, everybody agreed that such a thing would be desirable from a maintenance perspective. However the approach for doing it was something nobody seemed to agree on.

Thus the idea of tunables was born 4 years ago, except that Carlos wrote the first wiki page and called it ‘tunnables’. He consistently spelled it tunnables and I tunables. I won in the end because I wrote the patches ;)

Jokes aside, we were happy about the reception of the idea and we went about documenting it at length. However given that we were a two man army manning the glibc bunkers in Red Hat and the fact that upstream was still reviving itself from the post-Uli era meant that we would never come back to it for a while.

Then 2015 happened and it came with a memorable Cauldron in Prague. It was memorable because by then I had come up with a first draft of an API for the tunables framework. It was also memorable because it was my last month at Red Hat, something I never imagined would ever happen. I was leaving my dream team and I wasn’t sure if I would ever be as happy again. Those uncertainties were unfounded as I know now, but that’s a story for another post.

The struggle to write code

The first draft I presented at Cauldron in 2015 was really just a naive attempt at storing and initializing public values accessed across libraries in glibc and we had not even thought through everything we would end up fixing with tunables. It kinda worked, but it was never going to make the cut. A new employer meant that tunables will become a weekend project and as a result it missed the release deadline. And another, and then another. Towards the closing of every release I would whip out a patchset that would be poked holes into and then the change would be considered too risky to include.

Finally we set a deadline of 2.25 for tunables because by then quite a few devs had started maintaining their own list of tunables on top of my tree, frustratingly rebasing every time I completely changed my approach. We made it in the end, with Florian and I working through the year end holidays to get the whole patchset in before freeze.

So as of 2.25, tunables is firmly entrenched into glibc and as we speak, there are more tunables to come, especially to override IFUNC selections and to tune the processor capability mask.

So what are tunables anyway?

This is where you start if you want the technical description and are not interested in the story bits.

Tunables is an internal implementation detail in glibc. It is a way to manage ways in which we allow behaviour in glibc to be modified. As of now the only way to manage glibc is via environment variables and the way to do that was strewn all over the place in the source code. Tunables provide one place to add the tunable parameter with all of the characteristics it would have and then the framework will handle everything from there. The user of that tunable (e.g. malloc for MALLOC_MMAP_THRESHOLD_ or malloc.mmap.threshold in tunables parlance) would then simply access the tunable from the list and do what it wants to do, without bothering about where it came from.

The framework is implemented in elf/dl-tunables.c and all of the supporting code is named as elf/dl-tunable*. As is evident, tunables is linked into the dynamic linker, where it is initialized very early. In static binaries, the initialization is done in libc-start.c, again early enough to influence almost everything in the program. The list is initialized just once and is modifiable only in the dynamic linker before it relocates itself.

The main list of tunables is maintained in elf/dl-tunables.list. Architectures may define their own tunables in sysdeps/…/dl-tunables.list. There is a README.tunables that lists out the gory details of using tunables within glibc to access its values and if necessary, update it.

This gives us a number of advantages, some of them being the following:

Single Initialization

All environment variables used by glibc would be read in by a single double-nested loop which initializes all tunables. Accesses are then just a GOT away, so no more getenv loops in glibc code. This is not achieved yet since all of the environment variables are not yet ported to tunables (Hint: here’s a nice project for you, you aspiring glibc developer!)

All tunables are listed in a single file

The file elf/dl-tunables.list has a full list of tunables along with its properties such as type, value range, default value and its behaviour with setuid binaries. This caused us to introspect on each environment variable we ported into tunables and we ended up fixing a few bugs as well.

Very Early Initialization

Yes, very early, earlier than you would imagine, earlier than IFUNCs! *gasp*

Tunables get initialized very early so that they can influence almost every behaviour in glibc. The unreleased 2.26 makes this even earlier (or rather, delays CPU features initialization enough) so that tunables can impact selection of routines using IFUNCs. This fixes an important inconsistency in glibc, where LD_HWCAP_MASK was read in dynamically linked binaries but not in static binaries because it was not read in early enough.


The tunable list is read-only, so glibc reads from a list that cannot be tampered by malicious code that gets loaded after relocation.

What changes for me as a user?

The change in 2.25 is minimal enough that you won’t notice. In this release, only the malloc tuning environment variables have been ported to tunables and if you’ve been using those environment variables before, they will continue to work even now. In addition, you get to tune these parameters in a fancy way that doesn’t require the stupid trailing underscore, using the GLIBC_TUNABLES environment variable. The manual describes it extensively so I won’t go into details.

The major change is about to happen now. Intel is starting to push a number of tunables to allow you to tune your library to your liking, changing things like string routines that get selected for your program, cache parameters, etc. I believe PowerPC and S390 will see something simila too in the lock elision space and aarch64 multiarch will be tunable as well. All of this will hopefully come in 2.26 or latest by 2.27.

One thing to note though is that for now tunables are not covered by any ABI or API guarantees. That is to say, if you like a tunable that is in 2.26, we may well remove the tunable in 2.27 if we find that it either does not make sense to have that tunable exposed or exposing that tunable is somehow detrimental to user programs.

The big difference will likely come in when distributions start adding their own tunables into the mix. since it will allow them to add customizations to the library without having to maintain huge ugly patchsets.

The Road Ahead

The big advantage of collecting all tuning parameters under a single framework is the ability to then add new ways to influence those tuning parameters. We have environment variables now, but we could add other methods to tune the library. Some ideas discussed are as follows:

  • Have a systemwide configuration file (e.g. /etc/sysctl.user.conf) that sets different defaults for some tunables and limits the degree to which specific tunables are altered. This allows systems administrators to have more fine grained control over the processes on their system
  • Have user-specific configuration files (e.g. $HOME/.sysctl.user.conf) that does something similar but at a user level
  • Have some tunables modified during execution via some shared memory mechanism

All of this is still evolving, so if you have an idea or would like to work on any of these ideas, feel free to get in touch with me and we can find a way to get you contributing to one of the most critical parts of the operating system!

by Siddhesh at May 05, 2017 15:33

May 20, 2017

Neil Williams

Software, service, data and freedom

Free software, free services but what about your data?

I care a lot about free software, not only as a Debian Developer. The use of software as a service matters as well because my principle free software development is on just such a project, licensed under the GNU Affero General Public License version 3. The AGPL helps by allowing anyone who is suitably skilled to install their own copy of the software and run their own service on their own hardware. As a project, we are seeing increasing numbers of groups doing exactly this and these groups are actively contributing back to the project.

So what is the problem? We've got an active project, an active community and everything is under a free software licence and regularly uploaded to Debian main. We have open code review with anonymous access to our own source code CI and anonymous access to project planning, open mailing list archives as well as an open bug tracker and a very active IRC channel (#linaro-lava on OFTC). We develop in the open, we respond in the open and we publish frequently (monthly, approximately). The code we write defaults to public visibilty at runtime with restrictions available for certain use cases.

What else can we be doing? Well it was a simple question which started me thinking.

The lava documentation has various example test scripts e.g.

these have no licence information, we've adapted them for a Linux Foundation project, what licence should apply to these files?

Robert Marshall

Those are our own examples, contributed as part of the documentation and covered by the AGPL like the rest of the documentation and the software which it documents, so I replied with the same. However, what about all the other submissions received by the service?

Data Freedom

LAVA acts by providing a service to authenticated users. The software runs your test code on hardware which might not be available to the user or which is simply inconvenient for the test writer to setup themselves. The AGPL covers this nicely.

What about the data contributed by the users? We make this available to other users who will, naturally, copy and paste for their own tests. In most cases, because the software defaults to public access, anonymous users also get to learn from the contributions of other test writers. This is a good thing and to be encouraged. (One reason why we moved to YAML for all submissions was to allow comments to help other users understand why the submission does specific things.)

Writing a test job submission or a test shell definition from scratch is a non-trivial amount of work. We've written dozens of pages of documentation covering how and how not to do it but the detail of how a test job runs exactly what the test writer requires can involve substantial effort. (Our documentation recommends using version control for each of these works for exactly these reasons.)

At what point do these works become software? At what point do these need licensing? How could that be declared?

Perils of the Javascript Trap approach

When reading up on the AGPL, I also read about Service as a Software Substitute (SaaSS) and this led to The Javascript Trap.

I don't consider LAVA to be SaaSS although it is Software as a Service (SaaS). (Distinguishing between those is best left to the GNU document as it is an almighty tangle at times.)

I did look at the GNU ideas for licensing Javascript but it seems cumbersome and unnecessary - a protocol designed for the specific purposes of their own service rather than as a solution which could be readily adopted by all such services.

The same problems affect trying to untangle sharing the test job data within LAVA.

Adding Licence text

The traditional way, of course, is simply to add twenty lines or so of comments at the top of every file. This works nicely for source code because the comments are hidden from the final UI (unless an explicit reference is made in the --help output or similar). It is less nice for human readable submissions where the first thing someone has to do is scroll passed the comments to get to what they want to see. At that point, it starts to look like a popup or a nagging banner - blocking the requested content on a website to try and get the viewer to subscribe to a newsletter or pay for the rest of the content. Let's not actively annoy visitors who are trying to get things done.

Adding Licence files

This can be done in the remote version control repository - then a single line in the submitted file can point at the licence. This is how I'm seeking to solve the problem of our own repositories. If the reference URL is included in the metadata of the test job submission, it can even be linked into the test job metadata and made available to everyone through the results API.

  licence.text: http://mysite/lava/git/COPYING BSD 3 clause

Metadata in LAVA test job submissions is free-form but if the example was adopted as a convention for LAVA submissions, it would make it easy for someone to query LAVA for the licences of a range of test submissions.

Currently, LAVA does not store metadata from the test shell definitions except the URL of the git repo for the test shell definition but that may be enough in most cases for someone to find the relevant COPYING or LICENCE file.

Which licence?

This could be a problem too. If users contribute data under unfriendly licences, what is LAVA to do? I've used the BSD 3 clause in the above example as I expect it to be the most commonly used licence for these contributions. A copyleft licence could be used, although doing so would require additional metadata in the submission to declare how to contribute back to the original author (because that is usually not a member of the LAVA project).

Why not Creative Commons?

Although I'm referring to these contributions as data, these are not pieces of prose or images or audio. These are instructions (with comments) for a specific piece of software to execute on behalf of the user. As such, these objects must comply with the schema and syntax of the receiving service, so a code-based licence would seem correct.


Finally, a word about what comes back from your data submission - the results. This data cannot be restricted by any licence affecting either the submission or the software, it can be restricted using the API or left as the default of public access.

If the results and the submission data really are private, then the solution is to take advantage of the AGPL, take the source code of LAVA and run it internally where the entire service can be placed within a firewall.

What happens next?

  1. Please consider editing your own LAVA test job submissions to add licence metadata.
  2. Please use comments in your own LAVA test job submissions, especially if you are using some form of template engine to generate the submission. This data will be used by others, it is easier for everyone if those users do not have to ask us or you about why your test job does what it does.
  3. Add a file to your own repositories containing LAVA test shell definitions to declare how these files can be shared freely.
  4. Think about other services to which you submit data which is either only partially machine generated or which is entirely human created. Is that data free-form or are you essentially asking the service to do a precise task on your behalf as if you were programming that server directly? (Jenkins is a classic example, closely related to LAVA.)
    • Think about how much developer time was required to create that submission and how the service publishes that submission in ways that allow others to copy and paste it into their own submissions.
    • Some of those submissions can easily end up in documentation or other published sources which will need to know about how to licence and distribute that data in a new format (i.e. modification.) Do you intend for that useful purpose to be defeated by releasing your data under All Rights Reserved?


I don't enable comments on this blog but there are enough ways to contact me and the LAVA project in the body of this post, it really shouldn't be a problem for anyone to comment.

by Neil Williams at May 05, 2017 07:24

May 17, 2017

Gema Gomez

Yarn Bowl

When I started my current project with fingering yarn and realized how difficult it is to feed from the center of the ball, it was obvious that I needed a new strategy. Feeding from the outside, however, has its own issues and means that the ball of yarn rolls all over the place and crocheting becomes a chasing game also. That’s when the lightbulb went on and I decided to find a yarn bowl for home. I found this lovely one on etsy and got it shipped to the UK. It took a while to arrive but it was totally worth the wait:

Extermiknit Yarn Bowl

by Gema Gomez at May 05, 2017 23:00

May 12, 2017

Steve McIntyre

Fonts and presentations

When you're giving a presentation, the choice of font can matter a lot. Not just in terms of how pretty your slides look, but also in terms of whether the data you're presenting is actually properly legible. Unfortunately, far too many fonts are appallingly bad if you're trying to tell certain characters apart. Imagine if you're at the back of a room, trying to read information on a slide that's (typically) too small and (if you're unlucky) the presenter's speech is also unclear to you (noisy room, bad audio, different language). A good clear font is really important here.

To illustrate the problem, I've picked a few fonts available in Google Slides. I've written the characters "1lIoO0" (that's one, lower case L, upper case I, lower case o, upper case O, zero) in each of those fonts. Some of the sans-serif fonts in particular are comically bad for trying to distinguish between these characters.

font examples

It may not matter in all cases if your audience can read all the characters on your slides and tell them apart, put if you're trying to present scientific or numeric results it's critical. Please consider that before looking for a pretty font.

May 05, 2017 22:08

May 11, 2017

Gema Gomez

Birthday Yarn

Last month was my birthday and I received an email from my friend in Spain saying I had a voucher waiting for me at my local yarn shop, The Sheep Shop. We never discussed the shop that I can remember, yet she did her research and decided to go for it.

I know the shop and I know the owner, Sarah, and I was super-excited that my friend had thought of that as a present!

Today I finally found some time to go pick up yarn with my voucher. Sarah had some beautifully coloured superwash merino from and I had to get some:

Birthday yarn

I have not quite decided what to make with them yet, I have some vague ideas, but I am sure it’ll be awesome. I’d like to experiment with brioche knitting with those two colors.

One thing my friend told me is what a lovely customer service she had received over the phone, she was impressed. I have to agree with her, shopping at The Shep Shop is great. There is something for everyone and the customer service is second to none!

Getting older is ok, as long as it is with good friends and lots of yarn!

by Gema Gomez at May 05, 2017 23:00

May 05, 2017

Rémi Duraffort

A common mistake with jinja2

Jinja2 is a powerful templating engine for Python.

Inside LAVA, we use Jinja2 to generate configuration files for every boards that we support.

The configuration is generated from a template that does inherit from a base template.

For instance, for a beaglebone-black called bbb-01, the template inheritance tree is the following:

  • devices/bbb-01.jinja2
  • -> device-types/beaglebone-black.jinja2
  • --> device-types/base-uboot.jinja2
  • ---> device-types/base.jinja2

The first template (devices/bbb-01.jinja) is usually a list of variables with their corresponding values for this specific device.

{% extends 'beaglebone-black.jinja2' %}

{% set usb_uuid = 'usb-SanDisk_Ultra_20060775320F43006019-0:0' %}
{% set connection_command = "telnet localhost 6000" %}
{% set hard_reset_command = "/usr/bin/pduclient --daemon localhost --hostname pdu --command reboot --port 08" %}
{% set power_off_command = "/usr/bin/pduclient --daemon localhost --hostname pdu --command off --port 08" %}
{% set power_on_command = "/usr/bin/pduclient --daemon localhost --hostname pdu --command on --port 08" %}

In the base templates we where using:

host: {{ ssh_host|default(localhost) }}
port: {{ ssh_port|default(22) }}

This is in fact wrong. If the variables ssh_host and ssh_port are not defined, the resulting file will be:

port: 22

The default function in Jinja is expecting:

  • a python object (a string, an int, an array, ...)
  • a template variable name

In this case, localhost is interpreted as an undefined template variable name. Hence the result.

The correct template is:

host: {{ ssh_host|default('localhost') }}
port: {{ ssh_port|default(22) }}

That's a really simple mistake that can remain unnoticed for a long time.

by Rémi Duraffort at May 05, 2017 15:21

April 23, 2017

Mark Brown

Bronica Motor Drive SQ-i

I recently got a Bronica SQ-Ai medium format film camera which came with the Motor Drive SQ-i. Since I couldn’t find any documentation at all about it on the internet and had to figure it out for myself I figured I’d put what I figured out here. Hopefully this will help the next person trying to figure one out, or at least by virtue of being wrong on the internet I’ll be able to get someone who knows what they’re doing to tell me how the thing really works.

Bottom plate

The motor drive attaches to the camera using the tripod socket, a replacement tripod socket is provided on the base of plate. There’s also a metal plate with the bottom of the hand grip attached to it held on to the base plate with a thumb screw. When this is released it gives access to the screw holding in the battery compartment which (very conveniently) takes 6 AA batteries. This also provides power to the camera body when attached.

Bottom plate with battery compartment visible

On the back of the base of the camera there’s a button with a red LED next to it which illuminates slightly when the button is pressed (it’s visible in low light only). I’m not 100% sure what this is for, I’d have guessed a battery check if the light were easier to see.

Top of drive

On the top of the camera there is a hot shoe (with a plastic blanking plate, a nice touch), a mode selector and two buttons. The larger button on the front replicates the shutter release button on the body (which continues to function as well) while the smaller button to the rear of the camera controls the motor – depending on the current state of the camera it cocks the shutter, winds the film and resets the mirror when it is locked up. The mode dial offers three modes: off, S and C. S and C appear to correspond to the S and C modes of the main camera, single and continuous mirror lockup shots.

Overall with this grip fitted and a prism attached the camera operates very similarly to a 35mm SLR in terms of film winding and so on. It is of course heavier (the whole setup weighs in at 2.5kg) but balanced very well and the grip is very comfortable to use.

by broonie at April 04, 2017 13:17

April 16, 2017

Gema Gomez


I have been doing none of the crafts I love so much lately, and I wonder why. Several things come to mind when I think about it:

  • Lack of spare time
  • Lack of inspiration or mental energy to do anything useful

Do I lack spare time? I work every day from 9am to 6pm give or take half an hour depending on meetings. In the mornings, before I go to work I have been trying to fit an hour workout in during the work week, this means going to the gym, working out, coming home, having a shower, which on average takes 2 hours end to end. That leaves lunchtime (which I mostly work or waste my time online), evenings and weekends. This means, in theory, I have 2 hours each evening plus weekends of spare time. That is 10 plus 24 hours roughly. That would be a lot of knitting and crochet if I put my mind to it.

So, is the problem that I lack inspiration? Mental energy? I think one of my main problems is that I don’t disconnect. When work finishes I continue to stare at one screen or another. Whether it is email, facebook, twitter or the news. They are, probably, my biggest sink of time and it is worth reclaiming. All the time I spend staring at a screen mindlessly is time I don’t spend doing what I love. Whether it is spending time in meaningful conversations or crochetting/knitting or designing something. I also like to read and I haven’t read for quite a while. My attention span has suffered dearly over the past few years. I won’t blame social sites but my inability to ignore them, rather.

Why do I invest so much time on something so shallow? Why do others do this? I think it is time to change. No time like the present to make some adjustments to the way I invest my time. At the end of the day social media would be just fine for spare time if it made me feel better somehow or helped me unwind, but it doesn’t.

by Gema Gomez at April 04, 2017 23:00

April 11, 2017

Riku Voipio

Deploying OBS

Open Build Service from SuSE is web service building deb/rpm packages. It has recently been added to Debian, so finally there is relatively easy way to set up PPA style repositories in Debian. Relative as in "there is a learning curve, but nowhere near the complexity of replicating Debian's internal infrastructure". OBS will give you both repositories and build infrastructure with a clickety web UI and command line client (osc) to manage. See Hectors blog for quickstart instructions.

Things to learned while setting up OBS

Me coming from Debian background, and OBS coming from SuSE/RPM world, there are some quirks that can take by surprise.

Well done packaging

Usually web services are a tough fit for Distros. The cascade of weird dependencies and build systems where the only practical way to build an "open source" web service is by replicating the upstream CI scripts. Not in case of OBS. Being done by distro people shows.

OBS does automatic rebuilds of reverse dependencies

Aka automatic binNMUs when you update a library. This however means you need lots of build power around. OBS has it's own dependency resolver on the server that recalculate what packages need rebuilding when - workers just get a list of packages to install for build-depends. This a major divergence from Debian, where sbuild handles dependencies client side. The OBS dependency handler doesn't handle virtual packages* / alternative build-deps like Debian - you may have to add a specific "Prefer: foo-dev" into the OBS project config to solve alternative choices.

OBS server and worker do http requests in both directions

On startup workers connect to OBS server, open a TCP port and wait requests coming OBS. Having connections both directions is a bit of hassle firewall-wise. On the bright side, no need to setup uploads via FTP here..

Signing repositories is complicated

With Debian 9.0 making signed repositories pretty much mandatory, OBS makes signing rather complicated. obs-signd isn't included in Debian, since it depends on gnupg patch that hasn't been upstreamed. Fortunately I found a workaround. OBS signs release files with /usr/bin/sign -d /path/to/release. Where replacing the obs-signd provided sign command your own script is easy ;)

Git integration is rather bolted-on than integrated

OBS provides a method to integrate with git using services. - There is no clickety UI to link to git repo, instead you make an xml file called _service with osc. There is no way to have debian/ tree in git.

The upstream community is friendly

Including the happiest thanks from an upstream I've seen recently.


All in all rather satisfied with OBS. If you have a home-grown jenkins etc based solution for building DEB/RPM packages, you should definitely consider OBS. For simpler uses, no need to install OBS yourself, openSUSE public OBS will happily build Debian packages for you.

*How useful are virtual packages anymore? "foo-defaults" packages seem to be the go-to solution for most real usecases anyways.

by Riku Voipio ( at April 04, 2017 20:14

March 23, 2017

Ard Biesheuvel

Project dogfood: my arm64 desktop

As a developer who gets paid to work on improving ARM support in various open source projects, including the Linux kernel, I am used to things like cross compiling, accessing development boards over serial wires and other stuff that is quite common in the embedded world. However, as a LEG engineer, I actually work on systems that are much more powerful, and involve firmware layers and other system software components that are typically associated with a desktop or server PC, and not with a NAS box or a mobile phone. So why am I still using my x86 box to do the actual work?

The reality is that the desktop PC market is not a very appealing market to try and conquer with a new CPU architecture, and conquering the appealing ones is already proving to be hard work. So if the ARM development community wants ARM based workstations, it appears we will have to take matters into our own hands.

Please, do try this at home!

Due to my involvement with the UEFI port of the Celloboard (which is due to ship any day now), I was given a AMD Overdrive B1 development board last year, which is based on the same AMD Seattle SoC (aka Opteron A1100), but has a ATX form factor, a standard ATX power supply connector, two [working] PCIe slots, and onboard SATA (14 ports!) and networking, all of which are fully supported in the upstream Linux kernel.

So what would I need to turn this into a desktop system that is good enough for my day to day work?

The fan

The most annoying thing about switching from embedded/mobile dev boards to ‘server’ dev boards is the bloody fans!! To anyone reading this who is in charge of putting together such systems: a development board is quite likely to spend most of its lifetime within earshot of a developer, rather than in a data center rack. So could we please have quieter fans?!?


OK, so the first thing I did was replace the fan with a less noisy one. Do note that the AMD Seattle SoC uses a custom design for the heatsink, so this replacement fan will fit Cello and Overdrive, but not other arm64 based dev boards.

The case

Due to the ATX form factor and ATX power supply connector, there are lots of nice cases to choose from. I chose the smallest one I could find that would still fit a full size ATX board, so I ended up with the Antec Minuet 350, which takes low-profile PCIe cards.

The peripherals

My Overdrive board came with RAM installed, and has networking and SATA built in. So what’s lacking in terms of connectivity for use as a workstation is graphics and USB.

The AMD Seattle SoC has one peculiarity compared to x86 that complicates matters a little here: the RAM is mapped at physical address 0x80_0000_0000 (yes, that’s 9 zeroes), which means there is no 32-bit addressable RAM for PCI DMA. This is something that we could work around using the SMMU (IOMMU in ARM speak), but this is currently not implemented in the UEFI firmware or the Linux kernel, and so we need PCI peripherals that are capable of 64-bit DMA addressing.

For USB, I ended up selecting the SilverStone SST-EC04-P, which ships with a low-profile bracket, and has an onboard connector that can be used to wire up the two USB ports on the front of the case.

For graphics, I looked for a passively cooled, not too recent (for driver support, see below) card with HDMI output, and ended up with the Geforce 210 based MSI N-210, which has a nice, big heatsink (and no fan) and ships with a low profile bracket as well.

Kernel support

The lack of 32-bit addressable RAM for PCI DMA breaks assumptions in quite a few kernel drivers. For the Realtek 8169 Gig-E chip on the CelloBoard, we upstreamed patches that enable 64-bit DMA addressing by default on PCIe versions of the chip.

Much in the same way, I had to fix the nouveau and the ALSA drivers for the Geforce 210. Note that the proprietary, closed source NVidia driver is only available for x86, and so cards that are well supported by the open nouveau driver are strongly preferred.

All these patches have been in mainline since v4.10.

Userland support

‘Userland’ is the word kernel hackers use to refer to everything that executes outside of the kernel. My userland of choice is the Gnome3 desktop, which works quite well on the upcoming Ubuntu version (17.04), but older releases suffer from an annoying SpiderMonkey bug, which is caused by the incorrect assumption on the part of the SpiderMonkey developers that pointers never use more than 47 bits, and that bits 48 and up can be used for whatever you like, as long as you clear them again when trying to dereference the pointer value.

However, the arm64 kernel can be configured to use only 39 bits for virtual addressing, which still leaves plenty of address space and sidesteps the SpiderMonkey bug. This way, older Ubuntu versions are usable as well. I am currently using 16.10.

Firmware support

Now this is where it gets interesting. And I am not just saying that because I work on firmware.

So far, we have enabled everything we need to run an ordinary Ubuntu desktop environment on the Overdrive board. But interacting with the UEFI firmware at boot time still requires a serial cable, and a PC on the other end.

The problem here is driver support. Unlike SATA and USB, which are usually supported by class drivers, network interfaces and graphics cards require UEFI drivers that are specific to the particular chip. For the network interface on my Overdrive, this is a solved problem, given that it is integrated with the SoC, and supported by a driver that AMD have contributed. However, for plug-in peripherals such as my Geforce 210, the driver problem is usually addressed by putting a driver in a so-called option ROM on the card, and exposing it to the firmware in a way that is standardized by the PCI spec.

EFI Byte Code

Back when Intel was conquering the world with Itanium, they foresaw the problem that is affecting the ARM ecosystem today: an x86 driver can only run on a x86 CPU, and an ARM driver can only run on an ARM CPU, but option ROMs do not have unlimited space. Intel solved this by inventing an instruction set called EBC (for EFI byte code), and adding an interpreter for it to the UEFI reference code base. In theory, this allows expansion card vendors to recompile their code using an EBC compiler, and flash it into the option ROMs, so that the cards can be used on any architecture.

In reality, though, EBC is not widely used, is not promoted anymore by Intel (now that Itanium is dead), and even if expansion card vendors could get their hands on the compiler (which is not offered for sale anymore), recompiling source code that is riddled with x86 architecture (or rather, PC platform) based assumptions is not guaranteed to produce a driver that works on other architectures, especially ones with weakly ordered memory that does not start at address 0x0. For graphics cards in particular, the situation is even worse, given that many cards ship with a legacy VBIOS ROM (which requires legacy PC-BIOS compatibility in the x86 UEFI firmware) rather than a UEFI driver built for x86.

And indeed, it turned out that my nice low profile passively cooled Geforce 210 card did not have a UEFI driver in the option ROM, but only a legacy VBIOS driver.

X86 emulation in UEFI

Fortunately, I have another GeForce 210 card that does have a UEFI driver in its option ROM. So I dumped the ROM and extracted the driver, only to find out – not entirely unexpectedly, given the above – that it was a x86 driver, not a EBC driver, and so it is not supported on UEFI for 64-bit ARM.

So when Alexander Graf (of Suse) approached me at Linaro Connect two weeks ago, to discuss the possibilities of running x86 option ROMs on an emulator inside UEFI, I was skeptical at first, but after some more thought and discussion, I decided it was worth a try. Over the past ten days, we have collaborated online, and managed to implement an X86 emulator inside UEFI, based on an old version of QEMU (which is still LGPL licensed) combined with the more recent AArch64 support (whose copyright is owned by HiSilicon)

While this does not solve the problem of crappy drivers that make PC platform based assumptions, it works quite reliably for some network drivers we have tried, and even performs a lot better than EBC (which is a straight interpreter rather than a JIT).

And of course, it allows me to boot my Overdrive B1 in graphical mode.

by ardbiesheuvel at March 03, 2017 18:07

March 19, 2017

Siddhesh Poyarekar

Hello FOSSASIA: Revisiting the event *and* the first program we write in C

I was at FOSSAsia this weekend to deliver a workshop on the very basics of programming. It ended a pretty rough couple of weeks for me, with travel to Budapest (for Linaro Connect) followed immediately by the travel to Singapore. It seems like I don’t travel east in the timezone very well and the effects were visible with me napping at odd hours and generally looking groggy through the weekend at Singapore. It was however all worth it because despite a number of glitches, I had some real positives to take back from the conference.

The conference

FOSSAsia had been on my list of conferences to visit due to Kushal Das telling me time and again that I’d meet interesting people there. I had proposed a talk (since I can’t justify the travel just to attend) a couple of years ago but dropped out since I could not find sponsors for my talk and FOSSAsia was not interested in sponsoring me either. Last year I met Hong at SHD Belgaum and she invited me to speak at FOSSAsia. I gladly accepted since Nisha was going to volunteer anyway. However as things turned out in the end, my talk got accepted and I found sponsorship for travel and stay (courtesy Linaro), but Nisha could not attend.

I came (I’m still in SG, waiting for my flight) half-heartedly since Nisha did not accompany me, but the travel seemed worth it in the end. I met some very interesting people and was able to deliver a workshop that I was satisfied with.

Speaking of the workshop…

I was scheduled to talk on the last day (Sunday) first thing in the morning and I was pretty sure I was going to be the only person standing with nobody in their right minds waking up early on a Sunday for a workshop. A Sunday workshop also meant that I knew the venue and its deficiencies - the “Scientist for a Day” part of the Science Center was a disaster since it was completely open and noisy, with lunch being served right next to the room on the first day. I was wary of that, but the Sunday morning slot protected me from that and my workshop personally without such glitches.

The workshop content itself was based on an impromptu ‘workshop’ I did at FUDCon Pune 2015, but a little more organized. Here’s a blow by blow account of the talk for those who missed it, and also a reference for those who attended and would like a reference to go back to in future.

Hell Oh World

It all starts with this program. Hello World is what we all say when we are looking to learn a new language. However, after Hello World, we move up to learn the syntax of the language and then try to solve more complex user problems, ignoring the wonderful things that happened underneath Hello World to make it all happen. This session is an attempt to take a brief look into these depths. Since I am a bit of a cynic, my Hello World program is slightly different:

#include <stdio.h>

main (void)
  printf ("Hell Oh World!\n");
  return 0;

We compile this program:

$ gcc -o helloworld helloworld.c

We can see that the program prints the result just fine:

$ ./helloworld 
Hell Oh World!

But then there is so much that went into making that program. Lets take a look at the binary by using a process called disassembling, which prints the binary program into a human-readable format - well at least readable to humans that know assembly language programming.

$ objdump -d helloworld

We wrote only one function: main, so we should see only that. Instead however, we see so many functions that are present in the binary In fact, you you were lied to when they told back in college that main() is the entry point of the program! The entry point is the function called _start, which calls a function in the GNU C Library called __libc_start_main, which in turn calls the main function. When you invoke the compiler to build the helloworld program, you’re actually running a number of commands in sequence. In general, you do the following steps:

  • Preprocess the source code to expand macros and includes
  • Compile the source to assembly code
  • Assemble the assembly source to binary object code
  • Link the code against its dependencies to produce the final binary program

let us look at these steps one by one.


gcc -E -o helloworld.i helloworld.c

Run this command instead of the first one to produce a pre-processed file. You’ll see that the resultant file has hundreds of lines of code and among those hundreds of lines, is this one line that we need: the prototype for printf so that the compiler identifies the call printf:

extern int printf (const char *__restrict __format, ...);

It is possible to just use this extern decl and avoid including the entire header file, but it is not good practice. The overhead of maintaining something like this is unnecessary, especially when the compiler can do the job of eliminating the unused bits anyway. We are better off just including a couple of headers and getting all declarations.

Compiling the preprocessed source

Contrary to popular belief, the compiler does not compile into binary .o - it only generates assembly code. It then calls the assembler in the binutils project to convert the assembly into object code.

$ gcc -S -o helloworld.s helloworld.i

The assembly code is now just this:

    .file   "helloworld.i"
    .section    .rodata
    .string "Hell Oh World!"
    .globl  main
    .type   main, @function
    pushq   %rbp
    .cfi_def_cfa_offset 16
    .cfi_offset 6, -16
    movq    %rsp, %rbp
    .cfi_def_cfa_register 6
    movl    $.LC0, %edi
    call    puts
    movl    $0, %eax
    popq    %rbp
    .cfi_def_cfa 7, 8
    .size   main, .-main
    .ident  "GCC: (GNU) 6.3.1 20161221 (Red Hat 6.3.1-1)"
    .section    .note.GNU-stack,"",@progbits

which is just the main function and nothing else. The interesting thing there though is that the printf function call is replaced with puts because the input to printf is just a string without any format and puts is much faster than printf in such cases. This is an optimization by gcc to make code run faster. In fact, the code runs close to 200 optimization passes to attempt to improve the quality of the generated assembly code. However, it does not add all of those additional functions.

So does the assembler add the rest of the gunk?

Assembling the assembly

gcc -c -o helloworld.o helloworld.s

Here is how we assemble the generated assembly source into an object file. The generated assembly can again be disassembled using objdump and we see this:

helloworld.o:     file format elf64-x86-64

Disassembly of section .text:

0000000000000000 :
   0:   55                      push   %rbp
   1:   48 89 e5                mov    %rsp,%rbp
   4:   bf 00 00 00 00          mov    $0x0,%edi
   9:   e8 00 00 00 00          callq  e 
   e:   b8 00 00 00 00          mov    $0x0,%eax
  13:   5d                      pop    %rbp
  14:   c3                      retq   

which is no more than what we saw with the compiler, just in binary format. So it surely is the linker adding all of the gunk.

Putting it all together

Now that we know that it is the linker adding all of the additional stuff into helloworld, lets look at how gcc invokes the linker. To do this, we need to add a -v to the gcc command. You’ll get a lot of output, but the relevant bit is this:

$ gcc -v -o helloworld helloworld.c

/usr/libexec/gcc/x86_64-redhat-linux/6.3.1/collect2 -plugin /usr/libexec/gcc/x86_64-redhat-linux/6.3.1/ -plugin-opt=/usr/libexec/gcc/x86_64-redhat-linux/6.3.1/lto-wrapper -plugin-opt=-fresolution=/tmp/ccEdWzG5.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --no-add-needed --eh-frame-hdr --hash-style=gnu -m elf_x86_64 -dynamic-linker /lib64/ -o helloworld /usr/lib/gcc/x86_64-redhat-linux/6.3.1/../../../../lib64/crt1.o /usr/lib/gcc/x86_64-redhat-linux/6.3.1/../../../../lib64/crti.o /usr/lib/gcc/x86_64-redhat-linux/6.3.1/crtbegin.o -L/usr/lib/gcc/x86_64-redhat-linux/6.3.1 -L/usr/lib/gcc/x86_64-redhat-linux/6.3.1/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/6.3.1/../../.. /tmp/cc3m0We9.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-redhat-linux/6.3.1/crtend.o /usr/lib/gcc/x86_64-redhat-linux/6.3.1/../../../../lib64/crtn.o
COLLECT_GCC_OPTIONS='-v' '-o' 'helloworld' '-mtune=generic' '-march=x86-64'

This is a long command, but the main points of interest are all of the object files (*.o) that get linked in because the linker concatenates those and then resolves dependencies of unresolved references to functions (only puts in this case) among those and all of the libraries ( via -lc, via -lgcc, etc.). To find out which of the object code files have the definition of a specific function, say, _start, disassemble each of them. You’ll find that crt1.o has the definition.

Static linking

Another interesting thing to note in the generated assembly is that the call is to puts@plt, which is not exactly puts. It is in reality a construct called a trampoline, which helps the code jump to the actual printf function during runtime. We need this because printf is actually present in, which the binary simply claims to need by encoding it in the binary. To see this, disassemble the binary using the -x flag:

$ objdump -x helloworld

helloworld:     file format elf64-x86-64
architecture: i386:x86-64, flags 0x00000112:
start address 0x0000000000400430
Dynamic Section:

This is dynamic linking. When a program is executed, what is actually called first is the dynamic linker (, which then opens all dependent libraries, maps them into memory, and then calls the _start function in the program. During mapping, it also fills in a table of data called the Global Offset Table with offsets of all of the external references (puts in our case) to help the trampoline jump to the correct location.

If you want to be independent of the dynamic linker, then you can link the program statically:

$ gcc -static -o helloworld helloworld.c

This will however result in bloating of the program and also has a number of other disadvantages, like having to rebuild for every update of its dependent libraries and sub-optimal performance since the kernel can no longer share pages among processes for common code.

BONUS: Writing the smallest program

The basics were done with about 10 minutes to spare, so I showed how one could write the smallest program ever. In principle, the smallest program in C is:

main (void)
  return 42;

As is evident though, this pulls in everything from the C and gcc libraries, so it is clearly hard to do this in C, so lets try it in assembly. We already know that _start is the main entry point of the program, so we need to implement that function. To exit the program, we need to tell the kernel to exit by invoking the exit_group syscall, which has syscall number 231. Here is what the function looks like:

.globl _start
    mov $0xe7, %rax
    mov $0x42, %rdi

We can build this with gcc to get a very small binary but to do this, we need to specify that we don’t want to use the standard libraries:

gcc -o min -nostdlib min.s

The resultant file is 864 bytes, as opposed to the 8.5K binary from the C program. We can reduce this further by invoking the assembler and linker directly:

$ as -o min.o min.s
$ ld -o min min.o

This results in an even smaller binary, at 664 bytes! This is because gcc puts some extra meta information in the binary to identify its builds.


At this point we ran out of time and we had to cut things short. It was a fun interaction because there were even a couple of people with Macbooks and we spotted a couple of differences in the way the linker ran due to differences in the libc, despite having the same gcc installed. I wasn’t able to focus too much on the specifics of these differences and I hope they weren’t a problem for the attendees using Macs. In all it was a satisfying session because the audience seemed happy to learn about all of this. It looked like many of them had more questions (and wonderment, as I had when I learned these things for the first time) in their mind than they came in with and I hope they follow up and eventually participate in Open Source projects to fulfill their curiosity and learn further.

by Siddhesh at March 03, 2017 17:15

February 28, 2017

Gema Gomez

Atlanta PTG, OpenStack

Last week was all about OpenStack and making sure my team from Linaro/ARM was present on all the relevant sessions. There was only two of us there and with so many sessions happening at the same time, it became obvious that we would only be able to cover so much ground, so we decided to focus on the topics that are important for us this cycle.

Some of the most interesting conversations for me happened outside the sessions themselves. I found out about the 3rd party CI team and decided to join them on their weekly meetings from now on. This is important for Linaro because one of our main priorities is to get 3rd party CI for OpenStack set up on AArch64 hardware.

We were also working on the automatic set up of tempest for RefStack users. Right now the set up of tempest has a very steep learning curve. We want to make this as low as possible so that people can start testing without having a very deep understanding of tempest. See point 3 of action items on this etherpad. Having changed test environments and test clouds my fair share of times, this is a topic very close to my heart. There is a lot of engineering time going into configuring tempest properly on all sort of different environments, so trying to minimise this should save time in the long run to many teams.

Vertical teams

As discussed on the previous post, the first two days were about Kolla and ramping up on the project’s priorities for us. The Kolla meetings were well organised and driven, making us feel like we were using our time wisely by being there.

Some interesting topics from these discussions:

Horizontal teams

During the horizontal teams days, I attended mostly Ironic sessions. This is a comprehensive summary of the discussions (

One of the main issues in Ironic is to get good reviewers/core developers involved in the project. They were having very interesting discussions around how they may or may not have enough time to review things that are not in their current roadmap, but they feel like they should make the time for this. Becoming core developer of any OpenStack project does require a lot of time and dedication. Reviews of code from others, not only the code that you may be interested in is required. Also, Ironic has a variety of core developers that may review code from very different angles, all those are valuable, but there is a feeling that people contributing may find them a bit of a hit and miss when trying to get a patch landed.

The Ironic team is going to work on a list of recommendations for new contributors to be able to join the efforts in a more seamless way.

There were also discussions about how to deprecate the Ironic client and move to the new OpenStack client over the coming cycles.

The Ironic UI is a good place for new contributors to make a positive impact in the project. There is a list of features on a google doc that are there for new contributors to work on, the person to coordinate with on irc is TheJulia.

I also attended a couple of Nova sessions. My take from those was about quotas and how quotas may be breaking compatibility on Pike, for more information see etherpad. another interesting topic was the Nova REST API discussion, see etherpad.


Overall I am quite pleased with how the PTG was organised and run. It was up to the different PTLs to decide how to run the sessions and the ones I attended were mostly productive.

One problem I had, during the horizontal days, is that I could only really focus on one project. Other years I have attended the midcycles for Interop WG and separately for Infra/QA, and that gave me time to be part of both conversations, whilst now I have to choose just one horizontal team, so my involvement with Interop and Infra/QA was minor this cycle, due to the need to focus on Kolla. Funnily enough, the weekly IRC meetings for Interop and Kolla also coincide, so I have been having to choose between the two for a few weeks now. Having to choose between horizontal teams is not my preferred choice, I would have preferred to be able to attend Interop and Infra/QA even if it meant travelling an extra time.

I was however able to attend the Ironic meetings, which I wouldn’t have if I hadn’t been in Atlanta last week, as our involvement with that projec is not big enough to justify going to a midcycle for it.

Another lesson learnt, I didn’t need to be there on Friday, since the sessions I was interested in pretty much winded down on Thursday.

I would have liked to have a t-shirt from this event, but we got project mascot stickers instead. My laptop surely liked this, as it doesn’t wear t-shirts well.

It was overall a great week and I got a voucher for ODS in Boston that I intend to use.

by Gema Gomez at February 02, 2017 00:00

February 22, 2017

Gema Gomez

Project Teams Gathering OpenStack - First thoughts

I am in Atlanta this week at the first OpenStack PTG meetings. Since this is the first meeting of this kind, I didn’t really know what to expect. We had a schedule with a lot of project meetings happening at the same time. The first two days have been all about horizontal teams, this is teams that interact with all the other teams in one way or another.

I have been busy at the OpenStack Kolla meetings this time around. I tried to attend a few discussions of the Interop/Refstack meeting, but I couldn’t really keep up with both, so I decided to focus in Kolla, which is going to be where our contributions are going to go mainly for OpenStack Pike. Kolla is a project that produces Docker containers and scripts to be able to easily install OpenStack. Up until now we were producing our own packages for Debian and CentOS, but this has become difficult to maintain and it doesn’t scale very well. Contributing AArch64 containers to Kolla and helping the project become truly multiarch seems to be the way forward.

My team has a blueprint that we are working towards. The Kolla team have welcomed us and our contributions and are being very helpful getting us up to speed to be able to contribute effectively.

The first two days have been very productive, here is a link to the etherpad with the conversations we have been having:

Now ready to start attending the vertical team’s meetings. This should be easier for me as I don’t need to be everywhere at once.

Here is a picture of the Interop Working Group, taken on Tuesday of the PTG: Interop WG

by Gema Gomez at February 02, 2017 00:00

February 16, 2017

Ard Biesheuvel

Time invariant AES

Rule #1 of crypto club: don’t roll your own

Kernel hackers are usually self-righteous bastards who think that they are smarter than everyone else (and I am no exception). Sometimes, it’s hard to fight the urge to reimplement something simply because you think you would have done a slightly better job. On the enterprise scale, this is usually referred to as not invented here syndrome (NIH), and in the past, I have worked for companies where this resulted in entire remote procedure call (RPC) protocol stacks being developed, including a home cooked interface definition language (IDL) and a compiler (yes, I am looking at you, TomTom).

The EDK2 open source project is also full of reinvented wheels, where everything from the build tools to string libraries have been implemented and even invented from scratch. But when it came to incorporating crypto into the code base, they did the right thing, and picked the OpenSSL library, even if this meant putting the burden on the developer to go and find the correct tarball and unpack it in the right place. (Due to license incompatibilities, merging the OpenSSL code into the EDK2 tree would render it undistributable.)

The bottom line, of course, is that you are not smarter than everyone else, and in fact, that there are very smart people out there whose livelihood depends on breaking your supposedly secure system. So instead of reimplementing existing crypto algorithms, or, god forbid, inventing ‘better’ ones, you can spend your time more wisely and learn about existing algorithms and how to use them correctly.

Rule #2 of crypto club: read the manual

Not all encryption modes are suitable for all purposes. For instance, symmetric stream ciphers such as RC4, or AES in CTR mode, should never reuse the same combination of key and initialization vector (IV). This makes stream ciphers mostly unsuitable for disk encryption, which typically derives its IV from the sector number, and sectors are typically written to more than once. (The reason is that, since the key stream is xor’ed with the plaintext to obtain the ciphertext, two ciphertexts encrypted with the same key and IV xor’ed with each other will produce the same value as the two plaintexts xor’ed together, which means updates to disk blocks are essentially visible in the clear. Ouch.)

Many other algorithms have similar limitations: DES had its weak keys, RSA needs padding to be safe, and DSA (as well as ElGamal encryption) should not reuse its k parameter, or its key can be trivially factored out.

Algorithm versus implementation

Unfortunately, we are not there yet. Even after having ticked all the boxes, we may still end up with a system that is insecure. One notable example is AES, which is superb in all other aspects, but, as Daniel J. Bernstein claimed in this paper in 2005, its implementation may be vulnerable to attacks.

In a nutshell, Daniel J. Bernstein’s paper shows that there is an exploitable correlation between the key and the response time of a network service that involves AES encryption, but only when the plaintext is known. This is due to the fact that the implementation performs data dependent lookups in precomputed tables, which are typically 4 – 8 KB in size (i.e., much larger than a typical cacheline), resulting in a variance in the response time.

This may sound peculiar, i.e., if the plaintext is known, what is there to attack, right? But the key itself is also confidential, and AES is also used in a number of MAC algorithms where the plaintext is usually not secret to begin with. Also, the underlying structure of the network protocol may allow the plaintext to be predicted with a reasonable degree of certainty.

For this reason, OpenSSL (which was the implementation under attack in the paper), has switched to time invariant AES implementations as much as possible.

Time invariant AES

On 64-bit ARM, we now have three separate time invariant implementations of AES, one based on the ARMv8 Crypto Extensions and two that are NEON based. On 32-bit ARM, however, the only time invariant AES implementation is the bit sliced NEON one, which is very inefficient when operating in sequential modes such as CBC encryption or CCM/CMAC authentication. (There is an ARMv8 Crypto Extensions implementation for 32-bit ARM as well, but that is currently only relevant for 32-bit kernels running on 64-bit hardware.)

So for Linux v4.11, I have implemented a generic, [mostly] time invariant AES cipher, that should eliminate variances in AES processing time that are correlated with the key. It achieves this by choosing a slightly slower algorithm that is equivalent to the table based AES, but uses only 256 bytes of lookup data (the actual AES S-box), and mixes some S-box values at fixed offsets with the first round key. Every time the key is used, these values need to be xor’ed again, which will pull the entire S-box into the D-cache, hiding the lookup latency of subsequent data dependent accesses.

So if you care more about security than about performance when it comes to networking, for instance, for unmonitored IoT devices that listen for incoming network connections all day, my recommendation is to disable the table based AES, and use the fixed time flavour instead.


The priority based selection rules will still select the much faster NEON code when possible (provided that the CPU has a NEON unit), but this is dependent on the choice of chaining mode.

Algorithm Resolved as
Disk encryption xts(aes) xts-aes-neonbs
mac80211 CMAC cmac(aes) cmac(aes-fixed-time)
VPN ccm(aes) ccm_base(ctr-aes-neonbs,cbcmac(aes-fixed-time))

by ardbiesheuvel at February 02, 2017 09:33

February 14, 2017

Steve McIntyre

Start the fans please!

This probably won't mean much to people outside the UK, I'm guessing. Sorry! :-)

The Crystal Maze was an awesome fun game show on TV in the UK in the 1990s. Teams would travel through differently-themed zones, taking on challenges to earn crystals for later rewards in the Crystal Dome. I really enjoyed it, as did just about everybody my age that I know of...

A group have started up a new Crystal Maze attraction in London and Manchester, giving some of us a chance of indulging our nostalgia directly in a replica of the show's setup! Neil NcGovern booked a load of tickets and arranged for a large group of people to go along this weekend.

It was amazing! (Sorry!) I ended up captaining one of the 4 teams, and our team ("Failure is always an option!") scored highest in the final game - catching bits of gold foil flying around in the Dome. It was really, really fun and I'd heartily recommend it to other folks who like action games and puzzle solving.

I just missed the biting scorn of the original show presenter, Richard O'Brien, but our "Maze Master" Boudica was great fun and got us all pumped up and working together.

February 02, 2017 23:32

February 11, 2017

Ard Biesheuvel

UEFI on the Pi

Zen and the art of UEFI development

UEFI is an acquired taste. The EDK2 reference implementation has a very steep learning curve, and everything about it, from its build tools to the coding style, is eerily different, in a Twilight Zone kind of way.

UEFI as a firmware specification, however, has huge value: it defines abstractions for the interactions that occur between the OS and the firmware, which means [in theory] that development at either side can occur against the specification rather than against one of the many implementations. This allows things like universal OS installers, which is so common on x86 that people are sometimes surprised that this has always been a cause for headaches on ARM.

UEFI on the Pi

A prime example of this is the Raspberry Pi. It has a very peculiar hardware architecture, consisting of a Broadcom VideoCore 4 GPU, which is the primary core on the SoC, combined with one or more ARM CPUs. Revision 3 of the Raspberry Pi combines this GPU with 4 Cortex-A53 cores, which are low end 64-bit cores designed by ARM. The boot architecture matches the hardware architecture, in the sense that the GPU boots first, and reads a configuration file off a FAT partition on the SD card that describes how to proceed with booting the OS. A typical installation has a 32-bit ARM Linux kernel in the FAT partition, and a Raspbian installation (a variant of the Debian GNU/Linux distro compiled specially for the Raspberry Pi) on another SD partition, formatted as EXT4.

Using a standard Linux distro is impossible, which is unfortunate, given the recent effort in upstreaming SoC support for the Raspberry Pi 3. If we could only run UEFI on this board, we could boot a bog standard Ubuntu installer ISO and have an ordinary installation that boots via GRUB.

So over the past couple of months, I have been spending some of my spare cycles to porting EDK2 to the Raspberry Pi 3. This also involves a port of ARM Trusted Firmware, given that the Raspberry Pi 3 boots all its ARM cores in EL3 when configured to boot in 64-bit mode. It is a work in progress, and at the moment, it does little useful beyond booting the board into the UEFI Shell.

Building the secure firmware

Follow this link to my Raspberry Pi 3 branch of ARM Trusted Firmware, and build it using the following commands:

export CROSS_COMPILE=aarch64-linux-gnu-
export PRELOADED_BL33_BASE=0x20000
make PLAT=rpi3 fip all 

# add this so we can find the resulting image in the next step
export ATF_BUILD_DIR=$(pwd)/build/rpi3/release

This port is a minimal implementation of ARM Trusted Firmware, which pens up the secondary cores until the OS is ready to boot them via PSCI. It also implements PSCI System Reset via the SoC watchdog. Beyond that, it does the usual initialization of the secure world, and drops into EL2 to boot UEFI.

Building the UEFI firmware

Clone this repository and symlink it into an existing EDK2 working environment. Build it as follows:

build -a AARCH64 -t GCC5 -b DEBUG \
      -p OpenPlatformPkg/Platforms/RaspberryPi/RaspberryPi.dsc \

The resulting bootable image, containing both the secure and UEFI firmware, can now be found in the following file


Copy it to the FAT partition on the SD card, and update config.txt so it contains the following lines:


The DEBUG build of EDK2 is very noisy, so after inserting the SD and powering the device, you should see lots of output from the serial port (115200n8) in a matter of seconds.


For this UEFI port to do anything useful, we need driver support. In order of relevance, we need

  1. USB host mode support
  2. Graphics Output Protocol (GOP) support
  3. wired Ethernet support (USB based)
  4. SDHCI support
  5. Random Number Generator (RNG) protocol [for KASLR]

The first two items would allow booting and installing a distro without use of the serial port at all, which would be a huge improvement to the user experience.

Contributions welcome!

by ardbiesheuvel at February 02, 2017 09:19

February 09, 2017

Alex Bennée


A couple of weeks ago I mused that I should really collect together the various hacks to integrate checkpatch into my workflow into a consistent mode. Having a quick look around I couldn’t find any other implementations and went to create the said mode. It turns out I’d created the directory and done the initial commit 3 years ago. Anyway I polished it up a bit and you can now get it here. I hope it’s useful to the wider community and as ever patches welcome 😉

by Alex at February 02, 2017 15:30

February 08, 2017

Bin Chen

Android Security: A walk-through of SELinux

In DAC, each process has an owner and belong to one or several groups, and each resource will be assigned different access permission for its owner and group members. It is useful and simple. The problem is once a program gain root privileged, it can do anything. It has only three permissions which you can control, which is very coarse.
SELinux is to fix that.
It is much fine-grained. It has lots of permissions defined for different type of resources. It is based on the principle of default denial. We need to write rules explicitly state what a process, or a type of process (called domain in SELinux), are allowed to do. That means even root processes are contained. A malicious process belongs to no domain actually end up can do nothing at all. This is a great enhancement to the DAC based security module, and hence the name Security-Enhanced Linux, aka SELinux.
In this article, we'll have a walk-through of various components of SELinux on Android. We will take a top-down approach. The reason is SELinux has lots of concepts to understand, and files to manipulate, so quite likely, you will won't get the big picture if I were to start with the details first.
However, it is also possible that this approach will create more confusion. But let's try anyway.

How SElinux works

The core idea of SELinux is to label every resources and process, and on the base of default denial, to craft explicit rules granting a process certain permissions to access the resources.
Let's break it down.
  1. Define resource types, and label the resources
  2. Define process domains, and label the processes
  3. Write rules that grant a process the permissions
  4. Rules in action.

Define resource types, and label the resources

Define resource types

Use the type keyword and declare it in a .te file. The suffix te stands for Type Enforcement and no surprise, type are defined in this file.
Specifically, in Android:
  • Types for (normal) files are defined in the /system/sepolicy/file.te.
  • Types for devices files are defined in the /system/sepolicy/device.te.
  • Types for executables files are defined in individual domain files, e.g /system/sepolicy/mediaserver.te for mediasever domain.
Here is the example for each type of te files:
Code snippet of file.te:
# Filesystem types
type labeledfs, fs_type;
type pipefs, fs_type;
type sockfs, fs_type;
type rootfs, fs_type;

# proc, sysfs, or other nodes that permit configuration of kernel
type proc_bluetooth_writable, fs_type;
type proc_cpuinfo, fs_type;
type proc_iomem, fs_type;
type proc_meminfo, fs_type;
type proc_net, fs_type;

type adb_data_file, file_type, data_file_type;
Code snippet of device.te:
type audio_device, dev_type;
type binder_device, dev_type, mlstrustedobject;
type block_device, dev_type;
type camera_device, dev_type;
Code snippet of mediaserver.te, an example domain te file. (Domain te files also include other important things but at this moment, we'll focus on the type definition only.)
type mediaserver_exec, exec_type, file_type;
A few takeaways:
  • It is really a very fine-grained type system. Take a look at those different types of proc_xxx. Because of this fine-grained labeling, we can write accurate rules that will only allow a process to access a very narrow subset of resources, or even a single file, when that type labels only a single file.
  • It is targeted and designed for Android. See the adb_data_file.

Label the resources

To label a resource is also called to create a Security Contexts for a resource, or file contexts. The file contexts common to all devices are put in system/sepolicy/file_contexts. Vendor specific file contexts should reside in device/vendor/device/sepolicy. They will be combined together to generate the final file context during the build process.
Let's take a look at the common file_contexts.
# system/sepolicy/file_contexts
# truncated to show an overall view

# Root
/fstab\..* u:object_r:rootfs:s0
/init\..* u:object_r:rootfs:s0
/ueventd\..* u:object_r:rootfs:s0

# Dev
/dev/audio.* u:object_r:audio_device:s0
/dev/binder u:object_r:binder_device:s0

# System
/system/bin/mediaserver u:object_r:mediaserver_exec:s0
/system/bin/servicemanager u:object_r:servicemanager_exec:s0
/system/bin/surfaceflinger u:object_r:surfaceflinger_exec:s0

# Vendor
/vendor(/.*)? u:object_r:system_file:s0
/vendor/bin/gpsd u:object_r:gpsd_exec:s0

/odm(/.*)? u:object_r:system_file:s0
/oem(/.*)? u:object_r:oemfs:s0

# Data
/data/security(/.*)? u:object_r:security_file:s0
/data/drm(/.*)? u:object_r:drm_data_file:s0
/data/gps(/.*)? u:object_r:gps_data_file:s0
A few takeaways:
  • It labels everything : normal file, device file, executables.
  • It labels everywhere : rootfs, system, vendor, data partitions.
  • It is tailed for Android, sometime its called SEAndroid.
The label follows form of user:role:type:sensitivity. From the example shown above, you can tell type is the most important part, since the other part are same in Android at the moment.
To check the context for files use ls -Z
$ ls -Z /system/bin/mediaserver
u:object_r:mediaserver_exec:s0 /system/bin/mediaserver

Define process domains, and label processes

Domain is similar to Type but used to label a process, instead of a file. To define a domain, use type keyword as well, but the second parameter is domain instead of xxx_type. See line (1) in following code snippet.
# mediaserver - multimedia daemon
type mediaserver, domain, domain_deprecated; (1)
type mediaserver_exec, exec_type, file_type; (2)

init_daemon_domain(mediaserver) (3)
However, labeling a process with a Domain is different from labeling the corresponding executable file (with a Type). For example, we know from preceding introduction that the /system/bin/mediaserver is labeled as u:object_r:mediaserver_exec:s0, and that's down to the mediaserver_exec type.
But, the domain of the process mediaserver of is mediaserver. And, we'll see late, when writing the rules, we will use domain mediasever, instead of mediaserver_exec.
$ ps -Z | grep mediaserver
u:r:mediaserver:s0 media 1662 1 44340 10456 /system/bin/mediaserver
It's not hard to imagine there is something done under the hood creating this association (or transition) between the exec type and process domain. That's done through the macro in line (3). For now, we can just pretend we know what it is.

Write rules that will apply to a process and resources.

A rule will roughly say "allow somebody do something on something". Or better, "allow a subject take some actions on an object". The subject here is the process; the object here are the resources, such files, sockets; and the actions are the permissions. We have already know how to label the subject and object, in order to write a rules, we also need to define the permission.


In DAC, we have three access permissions, read, write, execute, for all type of files. In SELinux, things are much more complex, for a good reason.
SELinux (or SEAndroid) defines a list of types, called security classes. Its includes File, Directory, File System, Socket, Processes, security and capability. A full list can be found at system/sepolicy/security_class.
For different classes, there are different permissions, or called access vectors. A full list can be found at system/sepolicy/access_vectors.
An simple explanation of the permissions defined by SELinux can be found here.
Among those security classes, there is a class called Process, which defines the permission a process itself can do, for example, whether the process is allowed to change the security context of its own and the child processes. This constraint can be done in DAC based model.

Final, write the policies.

Policy files is the place to put all the pieces together and do the ultimate job of imposing access control.
The policy file ends with .te suffix as well, same as the type definition files. They are organized by domains, for example bluetooth.te for bluetooth domain/service/daemon, mediaserver.te for mediaserver domain/service/daemon.
The format of the rule is:
rule_name source_type target_type : class perm_set;
Below is truncated code of mediaserver.te. It shows an example of how to control the access of different type of resources: files, directories, devices, sockets, process, and binder services. Most of the rules are (hopefully) self explanatory, if you had followed the discussion (and congratulation on that). What worth note is the ability to control the out going bind calls. It means you have add that explicitly in the policy file, if you want to use a new binder service, or call a new API of a binder service. This is really tough, and some may consider it is cumbersome. However, it makes the system more secure. Everything vital permission must be explicated allowed. That is the core idea of SELinux, or Mandatory Access Control (aka MAC) in general.
# /system/sepolicy/mediaserver.te:
# truncated, an example for different type of permissions:

# files
allow mediaserver media_data_file:file create_file_perms;
# dirs
allow mediaserver oemfs:dir search;
# devices
allow mediaserver gpu_device:chr_file rw_file_perms;
allow mediaserver video_device:dir r_dir_perms;
# socket
allow mediaserver rild:unix_stream_socket { connectto read write setopt };
# process
allow mediaserver self:process ptrace;
# bind service
allow mediaserver activity_service:service_manager find;
allow mediaserver drmserver:drmservice { setPlaybackStatus openDecryptSession }
allow is the most used rule, and that is conform to the selinux's default denial model. There is also other rules, such as neverallow. This rule specifies that an allow rule must not be generated for the operation, even if it has been previously allowed.
For example, app.te states that app are never allowed to access the hardware device files. It is also a requirement in CTS, the neverallow rules in system/sepolicy are never allowed to be modified.
# Access to any of the following character devices.
neverallow appdomain {
}:chr_file { read write };

Rules in actions

sepolicy(.bin), file_context.bin and policy.conf

We have discussed a bunch of different files that are used to define the sepolicy. .te files are used either to define type , or rules; file_contexts are used to label the resources. Those are all human readable files, for obvious reason. And, not surprise, a binary representation will be generated from those file for efficiency. Among the binary files, two important ones are sepolicy(.bin) and file_contexts.bin, which are all end up in the root file system during the build process. And they will be used by selinux to enforce the rules.
The seplicy(.bin) is the output of checkpolicy, with an intermediate file called policy.conf as the input. In turn, the policy.conf is an aggregation of all the *.tefiles, among others. And, a good news is that policy.conf is text based, so we can diff that to diagnose policy issues between two version change.
Apart from the sepolicy.bin and, there are a few other files will be installed in either the rootfs or system partition during the build process, such as seapp_contexts, service_contexts and mac_permissions.xml. But we won't go details in this tutorial.

init and selinux in kernel

We have all the labels and rules in place but we haven't really set up anything yet. Say, apply a label for a file. That is taken care by the init program.
Apart from the well know functionality such as read the init.rc, mount the partitions, and start the services, init also responsible for the initiation of SELinux. It includes set up the labels for all the files according to the file_contexts.bin and pass the sepolicy to the kernel, among other things. Grepselinux_ in system/core/init/init.cpp for all the gory details.
How the SELinux policy is actually enforced by the kernel is outside of the scope of this tutorial, but at least make sure the relevant configuration is enabled.


In this article, we start with why SELinux is introduced and the core idea of it - label every thing, default denial, and explicit rules to allow anything. Then we look at the various components and steps need to implement that strategy, in Android.
Hopefully at the end of the introduction, you are convinced that SELinux is a very effective way, if not the most effective way, to protect the device security. And I think SELinux should be mandatory on all the devices, be it mobile devices, IoT devices or servers. Of course, that means they have to run Linux first, which is nice :)
For other stuff such as enable/disable SElinux, how to interpret and fix the SELinux warning (svc message), you can check this official documentation.

by Bin Chen ( at February 02, 2017 05:11

January 30, 2017

Gema Gomez

LDAP Authentication for OpenStack

One of the things we needed to do to be able to manage the Linaro Developer Cloud sensibly was to use LDAP for user authentication. It is interesting to note that our cloud being ARM64 didn’t make any difference in terms on how to configure LDAP or the dependencies that were required. Documentation for LDAP configuration is scattered and unclear in terms of what is or what is not up to date, so I reached out to the Keystone PTL for guidance on where to find the right documentation. He was very helpful and pointed me to his blog post “Configuring Keystone with IBM’s Bluepages LDAP”.

When I started this work I had a few questions:

  • How will we manage projects?
  • Will we be able to have users outside LDAP?
  • How will we disable users when they no longer require access?

In the case of the Linaro Developer Cloud, we are producing/using/testing our own build of OpenStack for ARM64, this build is based on Newton and was installed on Debian.

Searching for the right docs I found a page that seemed to have old instructions, I did get deprecation messages when trying to follow these instructions.

Our cloud is deployed using our own ansible scripts, so a few changes were required for LDAP to be integrated with our deployment. There are two projects that were affected by this change, Keystone and Horizon.

One of the most important things to take away is that there is still need for a services domain, so you will need to store the services accounts in SQL. LDAP will take care of user authentication and however you decide to manage projects for those users is independent of the initial set up of LDAP.

I followed Steve’s instructions and created a domain for us and a project to test the set up:

# openstack domain create your_domain
| Field       | Value                            |
| description |                                  |
| enabled     | True                             |
| id          | c99102f5d72b45b380b2fda60eb59a36 |
| name        | your_domain                      |

# openstack project create your_project --domain your_domain
| Field       | Value                            |
| description |                                  |
| domain_id   | c99102f5d72b45b380b2fda60eb59a36 |
| enabled     | True                             |
| id          | ecde336d467a49928add5ee13b8f1b9d |
| is_domain   | False                            |
| name        | your_project                     |
| parent_id   | c99102f5d72b45b380b2fda60eb59a36 |

Then added the three lines that tell Keystone to enable domain specific identity drivers to our keystone.conf file:

domain_specific_drivers_enabled = True
domain_config_dir = /etc/keystone/domains

And then I added an /etc/keystone/domains/keystone.linaro.conf and populated it with the specifics from our LDAP directory. Here is our template, make sure to use ldapsearch to figure out what values are needed for your specific case:

driver = ldap

url = {{ ldap_url }}
user = {{ldap_user}}
password = "{{ldap_password}}"
suffix = {{ldap_suffix}}
query_scope = sub

user_tree_dn = {{ldap_user_tree_dn}}
user_objectclass = {{ldap_user_objectclass}}
user_id_attribute = {{ldap_user_id_attribute}}

user_name_attribute = {{ldap_user_name_attribute}}
user_mail_attribute = {{ldap_user_mail_attribute}}
user_pass_attribute = {{user_pass_attribute}}
user_enabled_attribute = {{ldap_user_enabled_attribute}}

group_tree_dn = {{ldap_group_tree_dn}}
group_objectclass = {{ldap_group_objectclass}}
group_id_attribute = {{ldap_group_id_attribute}}
group_name_attribute = {{ldap_group_name_attribute}}
group_member_attribute = {{ldap_group_member_attribute}}
group_desc_attribute = {{ldap_group_desc_attribute}}

user_filter = {{ldap_user_filter}}

After this we assigned a role of ‘member’ to users from the dev-cloud-users group.

openstack role add member --group dev-cloud-users
                          --group-domain your_group_domain
			  --project your_project
			  --project-domain your_domain

After the configuration was changed, we restarted Keystone and went on to test whether we could get a token or list images. Both useful tests to figure out if the configuration is sound. To be able to use the command line, a new OS_USER_DOMAIN_NAME and OS_PROJECT_DOMAIN_NAME need to be added to the user’s rc file:


One last detail to be able to log into Horizon with the new domain, some changes need to be made to Horizon’s config file:

    "data-processing": 1.1,
    "identity": 3,
    "volume": 2,

Note these instructions have been tested with OpenStack Newton. The following patch summarizes the changes we had to do our reference architecture ansible scripts to integrate LDAP support to it:


  1. Steve Martinelli’s Blog Post
  2. Keystone Documentation

by Gema Gomez at January 01, 2017 00:00

January 23, 2017

Rémi Duraffort

LAVA events

For some releases now, LAVA server is publishing events every time:

  • a Job status change (Submitted, Running, ...)
  • a Device status change (Idle, Reserved, ...)

The stream of events is public and anyone can subscribe and receive updates.

A dump of the streams for the production and the staging instances is available on ReactOWeb. For instance, here is a (partial) extract of the stream:

topic: org.linaro.validation.testjob, data: {"status": "Submitted", "priority": 50, "job": 1318796, "health_check": true, "device": "aa9-01", "description": "lab-health-aa9", "submit_time": "2017-01-19T13:31:17.995716+00:00", "submitter": "lava-health", "pipeline": false, "visibility": "Publicly visible"}
topic: org.linaro.validation.device, data: {"status": "Reserved", "pipeline": false, "job": "1318429.1", "health_status": "Pass", "device_type": "x15", "device": "x15-01"}
topic: org.linaro.validation.device, data: {"status": "Reserved", "pipeline": false, "job": "1318455.0", "health_status": "Pass", "device_type": "kvm", "device": "kvm21"}

Use cases

A typical use case of the stream of event would be to know when:

  • a job is finished without polling the database
  • jobs are running on a specific board
  • a given device is going offline/online/...

For instance a user might want to build a website that aggregate all the results of the jobs on a given device type. Thanks to the event stream, it's possible to build and update such website in real time and without any polling.

Listening to the stream

The stream of event is published by a ZMQ PUB socket. In order to connect you should use the ZMQ library or a program like ReactOBus.

In Python, printing all events from the production instance is as simple as:

import zmq

ctx = zmq.Context()
sub = ctx.socket(zmq.SUB)
sub.setsockopt(zmq.SUBSCRIBE, b"")

while True:
    msg = sub.recv_multipart()

For the official Linaro LAVA instances the streams are available at:

Each event is an array made of:

  1. topic
  2. uuid (as generated by uuid.uuid1() in Python)
  3. datetime when the message was generated (isoformat)
  4. username of the sending process or user
  5. data as JSON

LAVA configuration

If you want to use the stream of events for your own instance of LAVA, you should add to /etc/lava-server/settings.conf:

  • EVENT_NOTIFICATION: should be true to allow to generate events
  • EVENT_TOPIC: the base topic for each events

Read the documentation for more information.

by Rémi Duraffort at January 01, 2017 08:25