3fx.ch

Why Wordle is so successful

Mon, 14 Feb 2022 12:00:00 +0100

The game Wordle exploded in popularity recently, to the point that it was bought by the New York Times because of its popularity. But why is Wordle such a resounding success? I will take a swing at an explanation in this post.

Simple rules with a clever twist

Wordle is a modified variant of the board game Mastermind. In Mastermind, in which one player constructs a code word consisting of four colored pegs, while the other player’s objective is to determine said code word in a limited number of tries. Any combination of pegs is admissible, making it mostly a game of deduction¹.

The cleverness of Wordle is the fact that, instead of colored pegs, the game uses the letters A through Z and a restricted set of admissible code words². Instead of requiring the player to reason about code words in abstract terms, five-letter English words are much more familiar to the player. This gives Wordle a tangibility that Mastermind cannot quite achieve.

Perfect game length

Another important factor to Wordle’s success lies in its format: a single Wordle can be played per day, and everybody guesses the same code word. Playing a round of Wordle takes a few minutes at most, and can be played anywhere: in bed after waking up, on the way to work, during a work break or in the evening to relax. There is no endless optimization or retrying for a better score. Once a game is done it is done, and the next game can only be played the next morning. This prevents people from burning out on Wordle after playing too much, and leaves us excited for the next day’s Wordle.

Casual and competitive, with just the right amount of luck

Wordle has two competitive aspects: correctly guessing the word, and doing it in fewer guesses than your friends and family. I wager most English-speaking people can guess almost every code word in the alotted six tries, but it isn’t a trivial task: finding a good guess feels very rewarding. With sufficient skill the game’s code word choice is insignificant, thus ensuring that good players don’t feel punished by unfair word choices³.

Competing with other players is very simple: the player to guess the word in fewer tries wins, and if both players require the same amount of tries, a draw is declared. This competition is interesting in the sense that it is possible to avoid bad scores with sufficient skill, but a player’s initial guess plays an important role in a day’s score: there is a fair amount of random chance and/or luck involved in getting very good scores. Thus, it is impossible for a player to consistently dominate the game. I would describe Wordle as similar to Mario Kart in this instance: in Mario Kart, skill is an important factor to not lose, but in the end it comes down to luck and chance who places first.

The perfect package

Wordle has it all: a solid foundation, the perfect length to keep up engagement over a longer time frame, both for casual and more competitive gamers. Coupled with a very clean user interface and a satisfying guess revelation animation it is not surprising that it has become as successful as it has. It is an instant classic, on par with other puzzle-type games such as crosswords and sudokus.

I’ve spent some time thinking about which other games I can think of that fit in the above category, but I could not come up with a comparable game. There’s always something odd to it such that doesn’t quite measure up to Wordle. If you have an idea, let me know!

It has been shown that the classic Mastermind can always be solved in five tries. ↩︎
The full list of admissible guesses and possible code words is directly exposed in the game’s JavaScript code. ↩︎
The only way to ensure that every word is fair is by hand-picking the entire word list. ↩︎

CSS: Avoid horizontal scrolling

Sun, 25 Jul 2021 10:30:40 +0200

This piece is a bit of a rant and a bit of advice. If you are not in the mood to read this, there is a TL;DR at the bottom.

If you had told me last year that I’d write not one, but two blog posts about web design, I would not have believed you. Alas here we are, and I’m going to tell you about another pet peeve of mine and how to fix it.

I view at least half the written content I consume via my mobile phone, which belongs to the class of phones with a slightly smaller than average screen. In fact, it’s width as reported by the Firefox and Chrome viewport simulators is 375 pixels wide, small but not tiny. And yet many websites I visit hinder my mobile viewing experience because of a rather simple issue: horizontal scrolling.

There I am, happily reading a blog post or article from the front page of $NEWSAGGREGATOR. But suddenly I’m not scrolling down, but down-right and constantly have to scroll back to the left edge in order to read full sentences. My experience is even worse if the site for some reason has a minimum width larger than my screen is wide.

Reading articles on those sites really cumbersome and more often than not I decide that it’s not worth the hassle to continue reading and move on to the next article. I find that quite sad, since the author invested time and effort to produce it, and the only reason I’m not reading is because of its subpar layouting.

Two types of scrolling madness

The smartphone browsers I have used in the past had two scrolling modes when scrolling is possible in both horizontal and vertical directions.

One of them is to always allow the user to scroll in both directions. This makes thumb-scrolling a tedious task, as the natural scrolling motion is not completely vertical, and leads to horizontal displacement¹.

The other one locks a scroll direction at the start of the scroll and restricts movement to that axis. This seems like a good idea in practice, but it does not work well for small adjustments. A slight horizontal movement on the initial scroll can lock the wrong axis, and there is a small cooldown time until the lock releases.

Finally, there’s the problem that horizontal scrolling is already used for navigation, at least on iOS Safari: scrolling horizontally triggers history navigation, which makes for a smooth user experience. Except when I unintentionally trigger the movement options when navigating a horizontally scrolling site. Most sites handle accidental navigation cleanly and return to where I left from, but some² sites don’t, and that’s another source of annoyance for me.

My conclusion is that both scrolling behaviours suck, so it’s best to avoid them as much as possible. So let’s fix it! Here’s a bunch of tips that hopefully help you understand and prevent horizontal scrolling. I will concern myself with mobile design tips, as this issue is not as prevalent on laptops and desktops.

Use viewport scaling

Some sites do not have viewport scaling enabled. This is a browser feature which accounts for pixel density and screen size differences between low and high resolution screens. For example, even though my phone actually has a screen that is 750px wide, displaying the site for a 750px wide screen causes the content to become tiny and unreadable. Readers then have to use zoom and use horizontal scrolling to read the content.

Using viewport scaling accounts for this discrepancy, and ensures that the page is rendered at a sane size.

Vertical design is superior

When designing a site, make sure that the mobile version is designed with verticality in mind. Smartphone screens are narrow so information rarely, if ever, fits nicely side-by-side. Instead use the infinite vertical space you have available and stack it on top of each other.

Sometimes it might be better to outright remove some features of the full-width site. Is a navbar really necessary when viewing a blog post on a mobile screen? Probably not. There should still be some navigation which allows you to go back to the index, for example. It is important to strike the right balance of feature and layout here, and some experimenting might be required.

Be conservative about minimum width

When I was still in high school a few years back my best friend used to have an interesting metric for selecting his mobile phones: more durability equals better. He was, in fact, quite clumsy³ when handling his phone so it made a lot of sense to him to choose a cheap and durable phone. I do not quite recall what the actual model was, but it was an LG phone similar to the Optimus L3 E400, with a whopping 256MB RAM and a 320x240 display. He used it as his daily driver, as it could do everything he needed: play music and browse the web.

I do not know whether he was as annoyed as I am about horizontal scrolling, but every time I work on mobile web design I think of him and his phone, and that he and others with really narrow screens would like to enjoy my content as well. Thus I try to design everything to be as usable as possible, even on a 320x240 screen⁴. Starting at 360px is fine for most purposes as well – it is definitely an improvement over doing nothing.

Dealing with wide content

Paragraphs of prose have the neat property that they are easily compressed horizontally, and still look nice on small screen sizes. Unfortunately, this is not possible for very long words, as the default CSS behaviour is to let them overflow to the side. The CSS property word-break can help here: setting it to break-word allows the browser to break very long words. It is not a very nice way of breaking words (no hyphen is inserted to indicate breakage), but it makes the page fit nicely without horizontal scrolling, which is more important. You can see an example of this happening here: Donaudampfschifffahrtsgesellschaftskapitän. Try viewing it in 240px or 360px width⁵ and note how only the offending word is broken apart.

There are two other content types where I frequently encounter overflow: images and source code.

Images are easily handled: make sure they are always at most 100% wide. Sometimes the image loses detail on very small screens, but that is totally fine. Make sure users can still access the raw image, for example by its URL.

Source code poses a bigger problem: line wrapping for code straight up sucks, so it has to be presented as is. Here we cannot nicely avoid horizontal scrolling, but we can ensure that we only scroll what is necessary, and not the entire page. There are 1001 different ways to achieve this, on this blog I set the overflow-x property on the <pre> tags.

This is the thing most blogs or articles I read get wrong: Most of the content neatly fits into the screen width, but overflow of one code block spills over and ruins the scrolling experience on the entire page.

Test it!

Finally, all theoretical discussion is useless if you do not test your design! All major modern browsers have some responsive design testing mode which allows you to set the viewport’s dimensions and see how your site behaves. On Firefox and Chrome it is part of the developer tools, which can be accessed by the keyboard shortcut Ctrl+Shift+I.

And if you have some ~~unwilling test subjects~~ friends, ask them to check out your site on mobile and give feedback! As an added bonus you get a wider browser coverage this way. There are a lot of subtle compatibility issues between web browsers, and the only way to figure them out is to test everything.

Conclusion

Hopefully this post explained why I think we should abolish horizontal scrolling, and give you practical tips on how to avoid it. If you have any questions or comments feel free to reach out to me on Mastodon or via my public inbox.

TL;DR

Horizontal scrolling sucks
Vertical composition is superior
Be conservative about minimum width, support at least 360px, if not 240px
If you need wider content then scroll locally
Test your site using the devtools supported by all major modern browsers

This is worse for left-handed people, as their natural thumb scrolling motion goes down-right. ↩︎
Especially the JS-heavy sites ↩︎
You know who you are ;) ↩︎
Who knows, one day you might want to browse the web on your smart fridge. ↩︎
On iOS devices you can achieve this via the page zoom setting. ↩︎

CSS: prefers-color-scheme

Wed, 27 Jan 2021 16:45:30 +0100

When developing tools that interface with users in a visual sense, I believe it is important to cater to the users’ wants (and needs) as much as possible, whilst retaining a recognizable product identity. Probably the brightest decision in this regard is the background color: light or dark?

Specifically regarding webpages, a variety of solutions have been developed: JavaScript triggers, cookie-based theming, user stylesheets and time-based theming. All of these solution falls short in one important aspect: they do not cover 100% of a user’s interaction with a site. The first two solutions have no safe default - they may always show the user the “wrong” theme first, and require active engagement to fix, which is a burden on the user. User stylesheets do work on first visit, however one often has to deal with broken sites, since they were not designed for theming. Finally, time-based theming is silly in its own right¹.

The solution

In 2017, the CSS Media Queries Level 5 specification was released, and it proposes the best solution to date: a media query named prefers-color-scheme. It is set by your browser, and can be set to either light or dark - more values may be supported in the future. Instead of requiring a user to interact with each site they visit, the decision is only made once at the browser, and can be applied to every site, even on the first visit! And it is supported by all major desktop and mobile browsers, simply marvelous.

Using this feature is as simple as adding a media query to your styling. Using CSS custom properties (“variables”), all you need to do is define the appropriate colors and you’re good to go.

@media (prefers-color-scheme: light) {
 // light styling
 // the default if no preference set
}

@media (prefers-color-scheme: dark) {
 // dark styling
}

The prefers-color-scheme still has one drawback: correct page rendering must be delayed until CSS is downloaded and parsed, and might cause flickering on slow connections, when the browser defaults to a white background but the user prefers dark mode for example. Using the color-scheme meta tag, the webpage can immediately signal which themes it supports, and the browser can react by styling the background dark immediately, before applying CSS, which prevents the aforementioned flickering. The first named theme is the author’s preferred default, which the browser can respect if the user has not given an explicit preference.

<!-- supports light and dark mode, defaults to light mode-->
<meta name="color-scheme" content="light dark">

Trying it out

This website fully supports theme selection based prefers-color-scheme, try it out! It even inverts images² to perfectly match your preferred style, a feature I’m very proud of, even if it is virtually invisible (since almost nobody looks at both styles). See the website’s source to see the actual SCSS source files.

On desktop browsers you can test it using the developer tools (Firefox, Chrome, Safari). On mobile, modern smartphones (both iOS and Android) set your preferred color scheme based on the system theme, so changing that should also change the appearance of this site.

Sadly, many of the sites I interact with daily do not support prefers-color-scheme. Especially people who rarely do frontend work (such as myself) may not know of this feature, and I hope to raise awareness for user-friendly theme support.

If you have a personal blog, product or company website or any other side, consider adding support for prefers-color-scheme to your theme. Even though it has no impact on the majority of people, we ultimately want design our products to be as user-friendly as possible. Right?

Bonus question: server time or client time? One is inaccurate for large parts of your visitors, the other one requires JavaScript. ↩︎
when it makes sense, so photos for example are left as-is ↩︎

Bundlewrap: First Impressions

Mon, 25 Jan 2021 00:00:00 +0100

I have spent quite some time with configuration management for my home infra setup, and I have recently come across a new tool that I’m excited to share with you. It’s called Bundlewrap, and it is a flexible, small-scale¹ configuration management solution. Bundlewrap is written in Python 3², and is unique in the sense that the infrastructure configuration is also written in Python. I was first introduced to Bundlewrap by @kunsi in December 2020, who gave a brief presentation at a cozy conference.

My current setup uses Ansible and the oldest parts are about two years old, so it’s mostly written for Ansible 2.4 and later. Most comparisons done are against Ansible.

Brief overview

Bundlewrap manages nodes via so-called bundles, which roughly correspond to Ansible roles. A bundle describes a desired state on the target nodes, comprising of one or more items, which would be tasks in Ansible.

Nodes are defined in nodes.py. Each node has a separate dictionary called metadata associated with it, which holds all the node’s configuration data and can be read and written by the bundles. Machines can be grouped and group metadata can be applied to all members by specifying it in groups.py.

Let’s look at an example: the bundle ssh-server should install and configure an SSH server on our node test. To do so we have to specify the items we want to use, here pkg_apt, file and svc_systemd. An entire repository could look something like this:

.
├── bundles # All bundles are in the directory bundles/
│   └── ssh-server # Our ssh-server bundle
│   ├── files # File templates for the ssh-server bundle
│   │   └── sshd_config # SSH server configuration template (omitted)
│   ├── items.py # Item definition file, Ansible: tasks/main.yml
│   └── metadata.py # Metadata definition file, Ansible: defaults/main.yml
├── groups.py # Group configuration
└── nodes.py # Node configuration

# nodes.py
nodes = {
 'test': {
 'hostname': '198.51.100.1',
 'bundles': {'ssh-server'},
 'metadata': {'nodevar': 'string'},
 },
}

# groups.py
groups = {
 'group': {
 'members': {'test'},
 'metadata': {'groupvar': 5},
 },
}

# items.py for bundle ssh-server
pkg_apt = {
 'openssh-server': {'installed': True},
}

files = {
 '/etc/ssh/sshd_config': {
 'source': 'sshd_config',
 'triggers': {'svc_systemd:sshd:restart'},
 },
}

svc_systemd = {
 'sshd': {
 'enabled': True,
 'running': True,
 'needs': {'pkg_apt:openssh-server'},
 },
}

# metadata.py for bundle ssh-server
defaults = {
 'bundlevar': ['abc', 'def'],
}

Configuration is done via Python dictionaries, and since the files are literal Python code, you can embed arbitrary logic in these files. Bundlewrap is well-documented, and I encourage you to read the docs if you want to figure out what an item does.

Now that we have seen the general structure of bundles, let us come to the feature comparison.

The Good

Agentless, push-based, no Python required on guest

Just like Ansible, Bundlewrap is both agentless and push-based. Managed nodes are accessed via SSH, which is my preferred way. Contrary to Ansible though, Bundlewrap does not require Python to be installed on the managed hosts, and instead relies on common tools.

Privilege escalation must work noninteractively. Since I anyway dislike entering passwords, this not a problem for me. The exact privilege escalation method used is configurable, so BSDs for example have the option to use doas instead of sudo (the default).

Bundlewrap does not do SSH multiplexing by default, but it is possible to pass arbitrary arguments to the underlying ssh invocation via an environment variable.

Automatic metadata merging, metadata generation

In the example above you might have noticed that we have defined metadata for both the node and the group it belongs to. Generally in Bundlewrap, non-collection metadata follows a strict hierarchy: node metadata overrides group metadata overrides bundle metadata. Collections are merged recursively, which is one of the best features Bundlewrap has. We can instruct Bundlewrap to display the metadata associated with test. The output is color-coded³ according to where the key comes from (group/node/bundle), which is very helpful.

% bw metadata test
{
"bundlevar": [ # Colored blue = from bundle
"abc",
"def"
],
"groupvar": 5, # Colored yellow = from group
"nodevar": "string" # Colored red = from node
}

This is one of the features I miss most from Ansible. I have a ton of roles which would like to have their variables merged. One example is my Prometheus setup: My monitoring server has to know about every exporter that a node has installed in order to scrape all of them. Ideally I’d just have a list for each node which has (exporter, port) pairs and each exporter role appends a pair to this node, thus allowing the monitioring role to work independently of the available exporters. However, since Ansible does not allow appending to an existing variable, I am stuck hardcoding every possible exporter into the main prometheus role.

Bundlewrap also allows for generating new metadata from existing metadata, using a concept called metadata reactors. These are defined at the bundle level and are extremely powerful. You can, for example, ensure that every virtual host automatically also gets issued a letsencrypt certificate, while still separating the webhost and letsencrypt bundles.

Secret derivation

Ansible has secrets, which allow you to store encrypted data and decrypt it with a static key. Bundlewrap can also do this, but additionally it allows you to generate secrets dynamically, which you can extract on demand. This is especially useful for automatic password generation for user accounts or when connecting a service to a DB user account: In both cases I don’t really care what the secret is, only that 1. it is a secret known only to the correct parties and 2. I can recover it if needed. Additionally, the secrets can easily be rotated by replacing the key used for secret derivation! Of course, now anyone in possession of the Bundlewrap master secret can derive all your passwords, so be sure to secure it well.

Offline testing

This one’s huge: Bundlewrap supports sensible offline testing. Bundlewrap tests involve assembling all metadata for all nodes, checking that all items are well-formed, all templates instantiate without errors, and so on. This is a feature I sorely miss from Ansible. While Ansible has the --check parameter, it still simulates each step by connecting to the target node, which is really slow compared to local execution. Plus, you can run Bundlewrap tests as part of your CI pipeline (even works for secrets without the decryption/generation keys!).

% bw test
✓ No reactors violated their declared keys
✓ group has no subgroup loops
✓ test has no metadata conflicts
✓ test ssh-server file:/etc/ssh/sshd_config
✓ test ssh-server pkg_apt:openssh-server
✓ test ssh-server svc_systemd:sshd
✓ test ssh-server svc_systemd:sshd:restart
✓ test ssh-server svc_systemd:sshd:reload

Small core

Bundlewrap has an extremely small “standard library” of items, and prides itself on staying that way. Personally, I value scope-restriction a lot in projects, so this is a good thing. On the other hand it means that, more often than not, you have to write the code for new items yourself, e.g. support for a new package manager. Fortunately, the code is quite accessible, and the methods you need to implement are well-documented.

The Neutral

Python dicts

Python dictionaries look much more like JSON than YAML, however in my opinion this does not impact readability. Writing Python dicts is slightly more pleasant than raw JSON, since it allows the use of single quotes for string identifiers⁴. Formatting is taken care of by any linter, which is nicer than YAML, where indentation cannot be automatically inferred. Of course, this is true for raw Python code as well.

Statistics and dependency graphs

This is undoubtedly a cool feature: Bundlewrap can output graphs (in graphviz format) visualizing the item/bundle dependencies on a node, or your repositories’ group relationships. And it also keeps track of statistics such as the number of items, nodes, groups, bundles and so on. These features don’t have a downside, however I also haven’t (yet) discovered clear upsides other than “ooh, shiny”.

Output of bw plot test

The Bad

Python

I really, really, really dislike Python. Mainly because it is interpreted and dynamically typed, which means that most errors will occur at runtime when it is too late to fix stuff. Working with Bundlewrap snippets is even worse, since some variables are passed automagically, which confuses my poor language server (and mypy as well), so any possibility of static type checking is chucked right out of the window.

Bundlewraps excellent local testing feature alleviates this issue somewhat.

Turing-complete config language

Having all the flexibility and power of Python also means having more footguns available to shoot yourself with. Bundlewrap relies much more on the user for constraining the bundle complexity. Personally, I think that for small infrastructures (such as what I run at home) this is fine, however I would be wary of this power for bigger deployments.

Conclusion

I’ve spent the last month thinking about and testing configuration management systems, and believe I have found a hidden gem in Bundlewrap. The only other notable mention I tried was cdist, however it has its fair share of oddities, most notably being a 100% sh-based solution. Of course, this doesn’t mean that it’s not good for you! Go check it out if that premise excites you.

I hope you got a brief overview of Bundlewrap and it’s features. Personally, I think it’s a better solution for my usecase than Ansible, and I’m going to slowly port my Ansible roles to Bundlewrap this year. If you’re interested and want to see more configurations/examples check out my and (especially) Franziska’s repositories.

If you have any questions or comments feel free to reach out to me via my public inbox or toot @bfiedler on Mastodon.

think <=1000 nodes/machines ↩︎
Quick reminder that Python 2 went EOL in January 2020. ↩︎
on your terminal, at least ↩︎
Laugh at me all you want, that is my biggest gripe when writing JSON by hand. ↩︎

On Decidability and the MU puzzle

Sun, 13 Dec 2020 00:00:00 +0100

Gödel, Escher, Bach takes the reader on a journey through mind, music, machines and self-reference. In the first few chapters, Hofstadter introduces a formal system called the MIU-system. The MIU-system consists of four simple rules for manipulating strings consisting of the characters M, I and U.

xI -> xIU, where x matches the rest of the string
Mx -> Mxx, where x matches the rest of the string
xIIIy -> xUy, where x and y match the rest of the string
xUUy -> xy, where x and y match the rest of the string

Note that the placeholders x and y must always match the entire string, i.e. the application MII -> MIII, choosing x = I, is not valid. The correct application is MII -> MIIII.

Then Hofstadter asks the reader to answer the MU puzzle:

Given the initial string MI, is it possible to construct the string MU using only the four above rules?

Take a few minutes and try for yourself. Many people quickly suspect that it is impossible, but why?

The solution

Let us add an additional, imaginary rule.

Each string Mx in the new system now has a form using only an M followed by Is, constructed by expanding all Us.

 MUIU
=> MIIIIIII
MU
=> MIII
MI
=> MI

Define the value of a string to be the number of Is in it, after transforming it to its MIIIIII...III form.

value(MUIU) = 7
value(MU) = 3
value(MI) = 1

The value of our target MU is 3, which is divisible by 3, while the value of our starting string MI is 1, which is not divisible by 3. If we can show that, starting with a string of value not divisible by 3, every rule application cannot create a string with value divisible 3, then it is also impossible to get MU by starting with MI.

So take a string Mx whose value is not divisible by 3. Rules 1, 3 and 4 preserve the value of Mx modulo 3, so by assumption the resulting string also has value not divisible by 3.

Rule 2 doubles the value of a string. However, by doubling any number which is not a multiple of 3 we can never create a number divisible by 3: a number n is divisible by 3 iff 3 is one of its prime factors. When doubling n the only prime factor we add is 2, hence the resulting number also cannot have 3 as a prime factor. We can express this more succinctly as

∀x. x ≠ 0 (mod 3) --> 2x ≠ 0 (mod 3)

Thus 2*n is not divisible by 3, and we can never construct the string MU, starting from MI.

Characterizing all generatable strings

Not only have we shown that MU is not constructible, starting from MI, but also any other string with a value divisible by 3. The question remains: which strings can we generate? Is it possible to generate all other strings, i.e. all strings Mx such that value(Mx) != 0 (mod 3)?

The answer turns out to be yes, using the following algorithm.

Generate My = MIIIIII...III by applying rule 1 to MI, such that the following holds: the value of My is larger than Mx and value(My) = value(Mx) (mod 3).
Append U if value(My) != value(Mx) (mod 6).
Merge IIIIII to UU and delete until value(My) == value(Mx).
Replace the MIIII...III with Mx by applications of rules 2 and 3.

It is always possible to apply step 1: the infinite sequence of strings generated by repeatedly applying rule 1 to MI has values 1, 2, 4, 8, 16, 32, ..., generating all powers of 2. Taking these values modulo 3 we get 1, 2, 1, 2, 1, 2, 1, 2, ..., i.e. 2^i (mod 3) is 1 if i is even, and 2 otherwise. Since value(Mx) != 0 by assumption, there always exists a longer string My such that value(My) = value(Mx) (mod 3).

In step 3 we need to delete U pairs until we have that value(My) = value(Mx). Unfortunately, we can only decrease value(My) in steps of six, since we can only remove Us in pairs. This is where rule 2 comes into play: if value(My) != value(Mx) (mod 6), then there would always be one U left over. (Note: since these values are congruent modulo 3, the only possible case is that value(Mx) == value(My) + 3 (modulo 6)). Appending an additional U before deleting UUs, increases value(My) by 3, and everything works out.

Step 4 is simple: Mx has the same value as My and we can use rule 2 to convert IIIs to Us, in the right positions. Thus we have shown that the MIU system lets us generate precisely the strings which have a value not divisible by 3.

The MIU-system and decidability

The MIU-system isn’t just a neat puzzle to solve: Hofstadter shows the reader that some questions about formal systems cannot be answered solely from within. Rather, we had to step outside the restrictions placed upon us by the four rules to successfully answer the question.

Given infinite time we could have concluded this ourselves, by generating all possible strings: however, in this case there exists a solution which is finite. We have constructed a decision procedure which solves not only the MU-problem, but any decision problem of the form “Does candidate string Mx belong to the MIU-system, starting from MI?”.

It is not always possible to find a finite decision procedure. Take for example all strings which are valid C programs (or choose any other sufficiently powerful language, it doesn’t matter). The decision problem “Does a given C program terminate at some point?” is not solvable in finite time, as shown by Alan Turing (1936). This problem is also known as the halting problem, and is one of the most famous undecidable problems.

These abstract problems can even have real world consequences: this year it was shown that type-checking a Swift program is also an undecidable problem. The author shows that in order to type-check a program, the compiler must solve the word problem for finitely generated groups. I like how this example shows us that some abstract problems pop up in unexpected places, and why seemingly purely theoretical knowledge matters, even for applied problems such as building compilers.

If you have any questions or comments feel free to reach out to me via my public inbox. If you are interested in undecidability in other programming languages, you might also like this website I made: typing-is-hard.ch.

Composition in Trick-Taking Card Games

Thu, 03 Dec 2020 00:00:00 +0900

This article was written as part of the Advent of Haskell 2020, be sure to check it out for other cool Haskell content!

A trick-taking game is a genre of (primarily) card games. Play revolves around rounds, in which each player plays exactly one card. The player who played the highest card wins the round, or trick.

On the surface these games seem to possess imperative core, processing well-defined actions once after the other. In this blog post we dive a bit deeper into the implementation of determining trick winners, and discover a neat application of composition to solve this problem, using Haskell and its standard library.

Hearts

We consider concrete games, increasing in complexity, starting with Hearts. Hearts is played with a 52-card deck of four suits (Clubs, Spades, Hearts, Diamonds) and thirteen ranks (from highest to lowest: A-K-Q-J-10-9-…-3-2).

data Suit = Clubs | Spades | Hearts | Diamonds
 deriving (Show, Eq)

data Rank = Two | Three | Four | .. | Jack | Queen | King | Ace
 deriving (Show, Eq, Ord)

data Card = Card { getSuit :: Suit, getRank :: Rank }
 deriving (Show, Eq)

The deriving clauses cause Haskell to automatically derive some boilerplate functions for us. This includes the show function for converting Cards, Suits and Ranks to Strings, equality predicates for all three types and, in case of Ranks, an ordering such that we have Two < Three < .. < Ace. We can verify that this works using ghci.

Main> :l Cards.hs
*Cards> aceOfSpades = Card Spades Ace
*Cards> aceOfSpades
Card Spades Ace
*Cards> twoOfDiamonds = Card Diamonds Two
*Cards> aceOfSpades == twoOfDiamonds
False
*Cards> aceOfSpades == aceOfSpades
True
*Cards> Two < Three
True

To determine the winner of a trick, we first inspect the suit which was led, i.e. played first. The highest card of that suit wins. Programatically this corresponds to choosing the maximum of a list, for which Haskell has the aptly named maximum function in its standard library. The maximum function requires that the type of elements of the list has Ord instance, i.e. is orderable. Unfortunately this is not the case: the ordering we choose depends on the suit that led, there is no global ordering we can apply to every Hearts trick. Haskell allows us to use a custom ordering function in this case, and the function maximumBy ordFunc computes the maximum using ordFunc as underlying ordering.

The comparison function ordFunc must take two list elements and return a value of type Ordering, where Ordering can be LT (for less), EQ (for equal) or GT (for greater).

import Data.Foldable
import Data.Ord

heartsCompare :: Suit -> Card -> Card -> Ordering
heartsCompare lead (Card s1 r1) (Card s2 r2)
 | s1 == lead && s2 == lead = r1 `compare` r2
 | s1 == lead && s2 != lead = GT
 | s1 != lead && s2 == lead = LT
 | otherwise = undefined -- neither c1 nor c2 have the leading suit,
 -- we cannot compare these two cards

trickWinner :: [Card] -> Card
trickWinner (lead:rest) =
 maximumBy (heartsCompare $ getSuit lead) $ lead:rest

Mathematically inclined readers will notice two things: Firstly, the Haskell Ord type class models a total order, which means that it assumes that any two cards are comparable. The Hearts ordering function we constructed however is actually a partial order, since we cannot compare two cards of different suits. This is not a problem however: Our ordering is always defined if the leading suit is played in a trick, and this is, by definition of leading suit, always the case.

Secondly, the ordering suspiciously looks like a lexicographic (partial) ordering: first we compare suits, and the “higher” suit wins¹. If the suits are equal then we move on to ranks. Lexical composition on orderings returns the first decisive (i.e. non EQ) ordering, or EQ if both are equal.

lexicalComp :: Ordering -> Ordering -> Ordering
lexicalComp EQ o2 = o2
lexicalComp o1 _ = o1

Looking a bit closer we can see two interesting properties of the lexicalComp function: It is associative, and it has a neutral element (EQ).

-- the function is associative
> lexicalComp (lexicalComp a b) c) == lexicalComp a (lexicalComp b c)

-- the function has a left and right identity element
> lexicalComp EQ x == x == lexicalComp x EQ

Thus lexical composition of Orderings forms a Monoid with identity element EQ. This instance is available in Haskell’s standard library, so we can shorten our definition somewhat.

import Data.Monoid ((<>))

lexicalComp = (<>) -- used infix like a <> b

Using this idea we can simplify our definition of heartsCompare. Note that the cases which previously were undefined are now defined, however as argued before it does not matter much how we decide in these cases.

suitCompare :: Suit -> Suit -> Suit -> Ordering
suitCompare lead = comparing (== lead)

rankCompare :: Rank -> Rank -> Ordering
rankCompare = compare

heartsCompare :: Suit -> Card -> Card -> Ordering
heartsCompare lead c1 c2 = suitCompare lead (getSuit c1) (getSuit c2)
 <> rankCompare (getRank c1) (getRank c2)

We have separated comparison of suit and rank and combine the results using abstractions provided by the Haskell standard library. Instead of combining the results we can also combine the comparison functions themselves, by defining a newtype for Comparison functions.

newtype Comparison a = Comparison { getComparison :: a -> a -> Ordering }

-- we can combine any two ordering functions, as long as
-- they compare the same elements
instance Monoid (Comparison a) where
 mappend f g = Comparison $ \x y -> (getComparison f) x y <> (getComparison g) x y
 mempty = Comparison $ \x y -> EQ

We would like to use this comparison on the above definition of heartsComparison.

suitComparison :: Suit -> Comparison Suit
suitComparison lead = Comparison $ comparing (== lead)

rankComparison :: Comparison Rank
rankComparison = Comparison compare

-- type error! `suitCompare` is of type `Comparison Suit` and
-- `rankCompare` is of type `Comparison Rank`
badHeartsComparison :: Suit -> Comparison Card
badHeartsComparison lead c1 c2 = suitComparison lead <> rankComparison

This, however, is a type error: The comparisons have different types, so we cannot combine them. Instead, we have to ensure that both Comparisons act on Cards.

suitCardComparison :: Suit -> Comparison Card
suitCardComparison lead = Comparison $ \c1 c2 ->
 comparing (== lead) (getSuit c1) (getSuit c2)

rankCardComparison :: Comparison Card
rankCardComparison = Comparison $ \c1 c2 ->
 compare (getRank c1) (getRank c2)

verboseHeartsComparison :: Suit -> Comparison Card
verboseHeartsComparison lead c1 c2 = suitCardComparison lead
 <> rankCardComparison

These definitions are verbose and repetitive, since all we did was to apply an extra function prior to applying the comparison function. Also both definitions look similar, so let us extract the functionality into a new function mystery.

mystery :: (Card -> a) -> Comparison a -> Comparison Card
mystery f c = Comparison $ \c1 c2 -> (getComparison c) (f c1) (f c2)

-- now we can express suitCardComparison and rankCardComparison more succinctly
suitCardComparison lead = mystery getSuit suitComparison
rankCardComparison = mystery getRank rankComparison

We can abstract mystery even further, by replacing Card with a type parameter b. The resulting type looks almost like the definition of a Functor, with flipped arrows!

fmap :: (a -> b) -> f a -> f b
mystery :: (b -> a) -> Comparison a -> Comparison b
mystery f c = Comparison $ \c1 c2 -> (getComparison c) (f c1) (f c2)

Indeed, Comparison is what the literature calls a contravariant functor, which is related to the (covariant) Functor. The terms co- and contravariant come from the definitions of the mapping function: A type is called covariant when it preserves the direction of the mapping function (going from a -> b to f a -> f b), while a contravariant reverses the direction of the arrow (going from b -> a to f a -> f b. There is a lot of literature available² for the theoretical background for interested readers, we will not go into further detail here.

class Functor f where
 fmap :: (a -> b) -> f a -> f b

class Contravariant f where
 contramap :: (b -> a) -> f a -> f b

The abstractions Covariant and Comparison we invented before are part of Haskell’s standard library, including the Monoid instance of Comparison. They can found in the Data.Functor.Contravariant module.

import Data.Functor.Contravariant

-- Data.Functor.Contravariant defines defaultComparison for us
--defaultComparison = Comparison compare

suitComparison :: Suit -> Comparison Suit
suitComparison lead = Comparison $ comparing (== lead)

heartsComparison :: Suit -> Comparison Card
heartsComparison lead = contramap getSuit $ suitComparison lead
 <> contramap getRank $ defaultComparison

trickWinner :: [Card] -> Card
trickWinner (lead:rest) =
 maximumBy (getComparison . heartsComparison . getSuit $ lead) $ lead:rest

When I first read about contravariant functors I had troubles imagining where these might come in handy, but hopefully this example demonstrates their usefulness.

Other Trick-taking games

This seems like an overly complicated solution if we are only interested in implementing the Hearts card game. However, this abstraction proves very useful if we turn to other trick-taking games, which have more complicated rules. Let’s look at Skat next, one of my favorite trick-taking games. Skat is played with three players and a 32-card deck, consisting of the same suits as Hearts, however only using cards of rank seven and up.

There is a variety of possible game modes, each with different orderings to score the tricks. The typical the so-called Farbspiel, in which all Buben (jacks) and all colors of a specific suit are trump cards. The jacks have an ordering based on suits, from highest to lowest: Clubs-Spades-Hearts-Diamonds. Trump cards always win a trick, and if multiple trumps are played in the same trick then the highest one wins. If no trump is played, then the trick is scored as it is in Hearts: The highest card of the leading suit wins the trick.

Notably, the suit cards are ranked A-10-K-Q-9-8-7, with 10 between ace and king. The jacks are omitted since they belong to the trump cards. Let’s first define the skatRank comparison.

import Data.List (elemIndex)

-- If `a` is an instance of the `Ord` typeclass,
-- then we can also order elements of type `Maybe a`
-- using
-- compare Nothing (Just x) = LT
highTen :: Comparison Rank
highTen = contramap (`elemIndex` skatRanks) defaultComparison
 where
 skatRanks = [Seven, Eight, Nine, Queen, King, Ten, Ace]

compareSkatRank :: Comparison Card
compareSkatRank = contramap getRank highTen

The function highTen captures the anomaly in the Skat ranking, namely that Ten is the second-highest ranked card of a suit. We can use the same technique as above to rank the suits when comparing the suits of two jacks.

suitOrder :: Comparison Suit
suitOrder = contramap (`elemIndex` rankedSuits) defaultComparison
 where
 rankedSuits = [Diamonds, Hearts, Spades, Clubs]

compareSuits :: Comparison Card
compareSuits = contramap getSuit suitOrder

Finally we can combine the two Comparisons above to determine the winner of a trick in a Farbspiel. To do this we consider a lexicographical composition of three orderings: First we compare both cards according to the jack-comparison rules. A jack wins against a non-jack and if both cards are jacks we compare using the defined suitOrder. Then we compare the trumps on both sides. Again trump always wins against non-trump, and if both are trumps then we need to consider the skatRank order. Finally if neither of these is true then we use the same algorithm as in the Hearts game, awarding the trick to the highest card of the leading suit.

-- generalization of suitComparison, useful when we only care
-- about a particular subset of values (e.g. one suit, one rank, etc.)
when :: (Card -> Bool) -> Comparison Card -> Comparison Card
when f cmp = Comparison aux
 where
 aux x y
 | f x && f y = getComparison cmp x y
 | otherwise = comparing f x y

whenSuit s cmp = when ((==) s . getSuit) cmp
whenRank r cmp = when ((==) r . getRank) cmp

farbspiel :: Suit -> Suit -> Comparison Card
farbspiel trump lead =
 (whenRank Jack compareSuits)
 <> (whenSuit trump compareSkatRank)
 <> (whenSuit lead compareSkatRank)

Now we can easily use our powerful combinators to implement the Grand variant, in which there is no trump suit, only the jacks are trumps.

-- in a grand only the four jacks are trumps
grand :: Suit -> Comparison Card
grand lead =
 (whenRank Jack compareSuits)
 <> (whenSuit lead compareSkatRank)

Even different rankings are not a problem: In the so called Nullspiel no trumps exist, and the cards are sorted as they are in the Hearts game. The player’s goal is to ensure that he does not win a trick.

nullspiel :: Suit -> Comparison Card
nullspiel = heartsComparison

This example highlights the use of composition very well: Instead of writing one large comparison function per card game we can combine four simple ones using the powerful abstractions in the Haskell standard library.

Bonus game: Doppelkopf

Lastly, let’s take a short look at another one of my favorite games: Doppelkopf. Doppelkopf is special since it is not played with one, but two sets of cards, with the same suits as before and ranks 9-J-Q-K-10-A³. Most Doppelkopf games are scored similar to a Farbspiel where Diamonds are trump, with some extra trumps: all jacks and all queens are trumps, with the queens ranked higher than the jacks, and both queens and jacks being ordered by the same suitOrder as before. The two tens of hearts are the two highest trumps, and are called Dullen (sg. Dulle⁴).

To uniquely determine a trick winner now, we also need to know who played which card, since the same card may be present twice. The official tournament rules state: If the highest card is present twice, then the first card played wins.

firstWins = Comparison $ \x y -> GT

dokoTournamentRules :: Suit -> Comparison Card
dokoTournamentRules lead =
 (when (== Card Hearts Ten) firstWins)
 <> (whenRank Queen $ compareSuits <> firstWins)
 <> (whenRank Jack $ compareSuits <> firstWins)
 <> (whenSuit Diamonds $ compareSkatRank <> firstWins)
 <> (whenSuit lead $ compareSkatRank <> firstWins)

dokoTrickWinner :: [(Card, Player)] -> (Card, Player)
dokoTrickWinner (c@(lead, player):rest) =
 maximumBy (getComparison $ contramap fst (dokoTournamentRules $ getSuit lead)) $ c:rest

Doppelkopf has more variants than any other card game I know of, ranging from slight alterations to total game changers. The most common variant we play with concerns the ten of hearts, in which the second ten of hearts beats the first. This sounds like a complete headache to model, however using the abstractions we developed for Skat already modeling Doppelkopf is child’s play.

firstWins = Comparison $ \x y -> GT
secondWins = Comparison $ \x y -> LT

secondHeartsTenWins :: Suit -> Comparison Card
secondHeartsTenWins lead =
 (when (== Card Hearts Ten) secondWins)
 <> (whenRank Queen $ compareSuits <> firstWins)
 <> (whenRank Jack $ compareSuits <> firstWins)
 <> (whenSuit Diamonds $ compareSkatRank <> firstWins)
 <> (whenSuit lead $ compareSkatRank <> firstWins)

Another variant is called Schweinchen⁵, which the player having both Diamond Aces must announce. Both Diamond Aces then become the highest trumps. Even playing with the piglet rule, some games may not feature piglets at all, in which case the Diamond Aces are not special.

withSchweinchen :: Comparison Card -> Comparison Card
withSchweinchen normalComparison =
 when (== Card Diamond Ace) firstWins
 <> normalComparison

Drawbacks

I want to point out some drawbacks of this approach. The Ord typeclass models total orderings, and this ordering is not total, only partial. The comparisons we created may not work with minimums or other ordering functions, for example. Arguably, a new type class PartialOrd would be cleaner, including an “incomparible” comparison result in PartialOrdering. To my knowledge, these are not available in the standard library sadly, although there exists an implementation in the lattices package.

Conclusion

We have developed an abstraction for modelling trick-taking games. In the process, we (re)discovered concepts from the Haskell standard library. Hopefully, this post helped you in understanding these concepts and their applications. If you have any questions or comments feel free to reach out to me via my public inbox.

Thanks for sticking with me, now go play some card games!

Where the leading suit is higher than the other suits, and the other suites are incomparable. ↩︎
This StackOverflow question gives a good overview and has links to great resources. ↩︎
Local rules may even omit nines. ↩︎
I do not know the etymology of the German word Dulle, so I cannot translate it. ↩︎
German for piglets. ↩︎

Build Your Own Wi-Fi Antenna

Sat, 28 Nov 2020 16:00:00 +0100

Building a wireless antenna is not hard. While the underlying physics and protocols are relatively complex, an antenna’s job is very simple: it captures or creates electromagnetic waves. We built a basic parabolic receiving antenna using a 3D printer and a female-female N-connector, available in any well-stocked electronic parts store.

There exist a multitude of different antenna designs, each with different characteristics. The simplest type is the isotropic antenna, which sends or receives from all (three dimensional) directions approximately equally well. Closely related is the omnidirectional antenna, which receives equally well from all compass directions, but is sensitive to the vertical angle of incoming electromagnetic waves. The WiFi antennas used in laptops and smartphones are often (close to) omnidirectional, as it would be really impractical if orientation had an effect on signal strength!

Antennas which are sensitive to the angles of incoming radio waves are called directional. Common uses of directional antennas are at space observatories, but also satellite TV antennas or military and civilian radars. They allow for efficient communication over much greater lengths than omnidirectional or isotropic antennas. Directional antennas come in different forms, a well-known one being the parabolic antenna, which uses a parabolic dish to focus the electromagnetic waves on a single point.

The build

The most important part of a parabolic antenna is the parabola: it is described by the equation $y = x^2 / 4a$, where $a$ is the focal length, i.e. the distance from the center of the antenna to the focal point, at which the signal is concentrated. Of course it is infeasible to build an infinitely large antenna, so the parabola is cut off at some point, resulting in a diameter $d$ and height $h$. These parameters are important when printing the design, since they impact the stability and feasibility of the print.

Optimal parabolic antenna equation

Such a parabolic antenna has a theoretical gain of $η * (π * d / λ)²$, where $λ$ is the wavelength of the measured signal and $η$ is the so-called aperture efficiency, commonly between 0.5 and 0.7. It is a catch-all for uneven dish surface, poor antenna placement and other errors.

Balz, a friend of mine, helped us model and print the antenna using Fusion360¹ and his Prusa i3 MK3S+. The 3D model can be downloaded here. Of course, any decent 3D modeling software and printer will do.

Rendered antenna model

Due to the layer-by-layer operation of 3D printing, shallow slopes often end up with an uneven surface², which lowers the effectiveness of the antenna. We attempted to fix this by taping a high-density rubber foam sheet in the dish, and covering it with aluminium tape. This better reflects the electromagnetic waves that hit the inside of the dish. Both materials can be bought cheaply in a hardware store.

The completed build

And all set! In order to measure how good our DIY antenna is we compared it against a professional so-called cantenna, another type of directional antenna which can be built easily at home. We used a TP-Link WN722N USB wifi adapter as sending station, and the Alfa AWUS036NH USB adapter as receiver. Getting the driver of the TP-Link stick to work was quite the pain, but we managed and set out to measure the different antennas. For our test setup we created a Wi-Fi network on the TP-Link adapter and connected to it from the Alfa adapter. We measured the signal strength using iwconfig.

There are many effects which influence the signal strength at the receiver, such as the transmission power, sending and receiving antenna gains, distance between sender and receiver, the carrier medium, and so on. Except for the receiving antenna all other properties stay the same. Thus it suffices to measure the received signal strength only - we can infer the relative gain between the antennas that way.

We are interested in determining the recipient’s antenna gain, i.e. how much the antenna amplifies a signal when receiving. This is generally measured in decibels-isotropic (dBi), which is the factor between this antenna’s gain and an idealized isotropic antenna.

Signal strength is measured in decibel-milliwatts (dBm), which expresses the change in signal power level per milliwatt increase. Both antenna gain and signal strength are logarithmically scaled, which simplifies calculations: the signal strength measured using an antenna is the actual signal strength at the receiver (in dBm) plus the antenna gain (in dBi). It is important to consider the right environment when measuring antenna gain: there should be no reflective surfaces such as building walls, bridges or similar close by. Also be sure to have line-of-sight between both antennas, as any objects between them disturb the measurements. We chose an open field near our city.

We measured the signal strength once for increasing distances between 1m and 100m, and another time in a 360° radius at 20m, using 20° increments using three antennas: Our DIY parabolic antenna, the professional cantenna, and the omnidirectional antenna which came included with the Alfa adapter, which is rated at 5dBi. Based on the omnidirectional antenna we can estimate the gain of the other two antennas.

Distance measurements

Signal strength (dBm) relative to distance (m)

Surprisingly, the omnidirectional antenna outperforms not only our antenna (which was probably to be expected), but also the professional cantenna.

We have a few theories regarding the comparatively poor performance of our antenna: during transport I accidentally bent the copper wire a bit, and we did not manage to bend it back perfectly, potentially shifting wire from the focus. Furthermore, our antenna dish is not a perfect parabola, but contains blemishes from the underlying sponge rubber, diffracting signals instead of focusing them.

Directionality measurements

Signal strength (dBm) relative to angle (°)

Unsurprisingly, the omnidirectional antenna is not sensitive to orientation, while the other two antennas are. The cantenna displays a beautiful profile of directionality: Good signal strength when pointing directly towards the source, then decreasing until hitting the minimum around 180°. Our parabolic antenna is also directional, albeit less consistently. In addition to the reasons listed above the length of our copper wire may also be responsible: since it extends beyond the focal point in both directions, signals which are reflected close to the focal point hit the copper wire as well, reducing directionality.

Conclusion

Our DIY antenna is a success: while it is not as good as an off-the-shelf omnidirectional antenna it is certainly good enough to receive Wi-Fi signals over a range of 100m, comparable to a non-DIY cantenna. It displays some form of directionality, if not a very good one. Coming back to antenna gain it seems that our antenna’s gain is not even close to its theoretical maximum of 14 dBi, we estimate it to be closer to 2-3 dBi.

There are many possible improvements to this design. The easiest way to increase antenna gain (and directionality) is to increase its diameter. While 200mm is close to the maximum of the Prusa i3, it is possible to split the design into quarter dishes, assembling a dish twice as large. Using a smoother material to cover the inside of the dish may also help. Finally, replacing the copper wire with a receiver only at the focal length of 100mm should increase directionality, at cost of a significantly more complex model.

A variety of other DIY antenna models also exist, such as the cantenna model we compared against, Wok-Fi (using woks or similar kitchenware as antenna dish), and many more. Hopefully this post showed you that building a Wi-Fi antenna is doable using only cans or a 3D printer, as well as some inexpensive hardware store/electronics. And can be a fun side project!

If you have any questions or comments feel free to reach out to me via my public inbox.

See his reddit thread about parabolic antennas in Fusion360. ↩︎
also called the “staircase effect” ↩︎

The Online Knapsack Problem

Thu, 12 Nov 2020 00:00:00 +0100

Last semester I took a course on online and approximation algorithms, and I want to present some really neat proofs related to the online knapsack problem. The course was taught by Hans-Joachim Bökenhauer and Dennis Komm, wo incidentally also first discovered the results I will present below. All results mentioned (and more) can be found in the original paper.

An online algorithm is given some input sequence $x1, x2, …, xn$, and must respond to each input $xi$ with some output $yi$, which may depend both on the current (and previous) inputs and the algorithm’s previous outputs. The algorithm does not know in advance how long the input sequence is, which presents an additional challenge.

Online algorithms are judged by their competitive ratio, which is the ratio between the score of an optimal solution of offline variant and the score of the solution produced by the algorithm. If this ratio is less than some number $k$, then we call the algorithm $k$-competitive. If the competitive ratio in unbounded, then we say the algorithm is not competitive.

Normally an offline algorithm is deterministic, which means it cannot access any other information than the inputs its given. A randomized online algorithm can additionally access an infinite stream of bits, each independently either $0$ or $1$ with probability exactly $1/2$. In this case we score the algorithm by its expected competitive factor, since its exact score may vary based on the randomness. Finally, an online algorithm with advice is permitted to access an infinite stream of advice bits, which an oracle has computed based on the input it is given. When analyzing randomized online algorithms or online algorithms with advice we are interested in minimizing the number of random or advice bits used to achieve specific (expected) competitive ratios.

We will focus on the simple online knapsack problem, which is the following: Given a backpack of capacity $1$ and $n$ objects with weights $w1, …, wn$, fill the backpack as much as possible, without exceeding the capacity. We process the weights in order, and for each weight we have to immediately decide whether we pack it or not, and we cannot change our choice at a later point in time. Restricting ourselves to capacity $1$ seems arbitrary, but we can easily adjust for any backpack capacity $b$ by scaling the weights accordingly.

First of all we notice that any deterministic algorithm for the online knapsack problem is not competitive. We can “trick” any algorithm $A$ using the following procedure: First we offer an object of weight $eps$, where $eps$ can be arbirarily small. If $A$ takes this object, we offer an object of weight $1$ and no further objects. Clearly an optimal solution can entirely fill the back, while $A$ only achieved a fill of $eps$. Since $eps$ can be arbirarily small, the competitive ratio is unbounded, and $A$ is not competitive. If on the other hand $A$ refuses the element with weight $eps$ then we do not offer any other objects, and again the competitive ratio is unbounded.

Clearly, some help is needed. How much help do we need to be optimal? Using $n$ advice bits the online algorithm is trivially optimal: we simply encode for each item one bit whether to take it or not. It turns out that we also need at least $n-1$ advice bits to be optimal: Imagine the weight sequence $1/2, 1/4, …, 1/2^(n-1), wb$. Let $b$ be a bit string of length $n-1$, and define the weight $wb$ as follows:

wb = 1 - sum (i = 1 to n-1) b_i * 1/2^i

For each of the $2^(n-1)$ possible bitstrings, the weight of $wb$ uniquely determines¹ which of the previous $n-1$ weights must be taken in order to achieve the optimal fill of $1$. Thus any optimal algorithm must need at least $n-1$ advice bits to distinguish the $2^(n-1)$ identical prefixes.

Being optimal however is quite an ambitious goal. Can we do better using fewer advice bits? It turns out that we can: There exists a 2-competitive algorithm using only one advice bit. What information do we encode in this one bit? Take a few minutes to think about this if you wish.

What we need to know is whether there exists an object with weight at least $1/2$. If yes, then we simply wait until the first such object occurs and pack it. Since the optimal solution can pack at most $1$ and we pack at least $1/2$ we are 2-competitive. Now what if there does not exist such an object, and all objects weigh less than $1/2$? Either the total sum of weights is less than $1$, then we can pack all items into the bag greedily and are even optimal. If the total weight is larger than $1$, then we claim that greedy is 2-competitive. There must exist at least one element that greedy cannot pack since it overfills the backpack. But by our assumption this object must weigh less than $1/2$, meaning that our backpack must contain at least objects of total weight $1/2$, which is 2-competitive by the same reasoning as in the first case.

Until now we have only considered advice algorithms, it seems natural to ask how a randomized algorithm performs. Naturally we expect a randomized algorithm to perform a bit worse than an advice algorithm given the same number of bits. Indeed, if we execute the same algorithm as above and guess the advice bit (by sampling a random bit), then in half the cases we are 2-competitive, which implies that we are 4-competitive overall.

However it turns out that there exists a randomized algorithm which is 2-competitive using only one advice bit. I consider this algorithm and its correctness proof to be one of the most elegant constructions I have seen to date. Again I urge you to take some time to think about this yourself.

The 2-competitive randomized online algorithm $A$ works as follows: We have two algorithms $A1$ and $A2$, and $A$ simply chooses one of them uniformly at random. Algorithm $A1$ uses a greedy approach: Pack every item that fits until the backpack is full. We have already learned that this strategy by itself is not competitive, the beauty comes in combination of $A2$: algorithm $A2$ simulates $A1$, but does not pack any items. Once $A1$ cannot pack an item for the first time, $A2$ begins to pack items greedily into its backpack.

This algorithm is 2-competitive: If the sum of all weights are less than $1$, then $A1$ is optimal and $A2$ never packs an item - in expectation $A$ achieves half of the optimal solution and we are done. If the total weight is larger than $1$, then the sum of weights packed by both algorithms must be greater than $1$, and again on average we will have packed at least $1/2$ total weight, which makes $A$ 2-competitive. I find this algorithm and it’s correctness proof both amazingly simple and exceptionally clever, a beautiful combination that awakes a motivation to delve deeper into this subject matter.

If you want to learn more about the online knapsack problem then I recommend you read the original paper by Bökenhauer, Komm, Královič and Rossmanith, the section on the simple² knapsack problem is quite accessible even with little theoretical background. We have barely scratched the surface, and the paper deals with further interesting problems. If you are interested in online algorithms in general then I recommend Komm’s An Introduction to Online Computation, which holds its promise and is also quite accessible with basic theoretical computer science knowledge.

I hope I could convey some of the appreciation I have for this topic and you learned a little bit about online algorithms. If you have any questions or comments feel free to reach out to me via my public inbox.

Proving this is a simple but fun exercise in discrete mathematics. ↩︎
In the paper this is called the “unweighted case”. ↩︎

Long monotone trails in graphs

Tue, 10 Dec 2019 10:10:10 +0100

During a recent dinner discussion I was presented with a beautiful proof to the following problem: Given an arbitrary graph $G = (V, E)$, and an ordering $\Phi$ of $E$, what is the longest monotone trail in $G$ that is guaranteed to exist?

A trail in a graph is a sequence of edges such that

adjacent edges share an incident vertex and
no edge is traversed twice

Take a few moments to think about this problem before reading the solution, as it such a simple seeming problem is quite tricky: Clearly there are graphs where there exists no trail of length greater than zero, namely the empty graph. Thus the solution must somehow correlate with some graph property.

The solution

It turns out that the longest monotone trail has length equal to the average degree $d$ of $G$. The proof is truly astouding:

Let $n = |V|$ and $m = |E|$ the number of edges of $G$. Put a person on each vertex of $G$, and for each edge $e$ in $E$, traversed in order of $\Phi$, exchange the two people standing at vertices incident to $e$. Each person will walk a monotone trail on $G$.

All people together traversed a distance of $2m$ units, and since there are $n$ people the average distance traveled is $2m / n$, which is exactly the average degree of $G$. By the pigeonhole principle at least one person must have traveled $2m / n$ units, which concludes our proof.

It is not difficult to see that this result is best possible. If $G$ is a matching on $2k$ vertices, then the average degree $d$ in $G$ is $1$, and the longest monotone trail also has length $1$.

'changeme' is valid base64

Mon, 09 Dec 2019 00:10:58 +0100

Base64-encoding is ubiquitous in our modern world. Many programs communicate in base64-encoded messages, since these have nice properties: They consist of a very limited subset of ASCII characters, and can thus be displayed as text. E-Mails for example are commonly encoded in base64 in transfer, which can be seen in the Content-Transfer-Encoding header.

Where I volunteer we have a tradition of putting changeme as a placeholder for many template variables indicating that they should be overwritten later. Thus, if we later still see changeme anywhere we know that somewhere somebody must have forgotten to specify a variable¹.

Sometimes these template variables should correspond to base64-encoded content, i.e. the base64-decoding of said variable should produce meaningful content. But what happens if somebody forgets to define said variable, and it is substituted with changeme instead? I always thought that it would fail to decode the string since the probability that changeme is actually valid base64 encoding must be very low. But lo and behold, this is not the case:

% echo changeme | base64 -d
r�[1m

Running hexdump on the output we can identify the characters:

% echo changeme | base64 -d | hexdump -C
00000000 72 16 a7 81 e9 9e |r.....|
00000006

Mostly nonsense, but the second byte is interesting: It is an ASCII control character ², meaning trips up all sorts of programs which expect to read printable ASCII characters, such as JSON deserializers.

This bug actually occurred when I was migrating some applications between different Kubernetes clusters: In the old cluster the application ran without problems, but I forgot to specify some variable in the new cluster, and it lead to Go complaining about non-ASCII characters in a JSON string.

I guess the moral of the story is: If you use dummy values be sure that they cannot be misinterpreted.

this is not the ideal way to solve this, but we have our historical reasons for choosing this solution. ↩︎
specifically Ctrl+V ↩︎

Solving graph problems in bash

Fri, 22 Nov 2019 18:30:40 +0100

Many interesting problems in computer science are expressible as problems on graphs: Finding the shortest path between two vertices, calculating a spanning tree, finding a large independent set, etc. Often in Theoretical Computer Science we only look for a theoretical solution - once we know how to model a specific problem as graph we give an algorithm in pseudocode and leave it at that. We will try an unconventional method practically solving these problems using bash.

At ETH, Computer Science Master students learn how to efficiently (and practically) solve graph problems in the Algorithms Lab, using C++ and BGL¹. In this course it is not enough to come up with a theoretical algorithm. One also have to implement and pass testcases as part of his solution. The BGL offers many sophisticated graph algorithms, each one cleverly optimized in order to make the generated code as fast as possible. It does its job well: It makes it (relatively) easy to use sophisticated graph theoretic algorithms in practice. But it is huge and complicated. Is there maybe an easier way to solve graph problems on a computer, without having to implement most of the boring algorithms and without installing a bazillion dependencies?

It turns out that we can (ab)use just about any Linux system for this, and we do not even need a C++ compiler, or any other compiler for that matter. All we need is any run-of-the-mill shell (bash will do just fine), and some standard system utilities such as find(1), tr(1), paste(1), cut(1), etc.², which (for GNU/Linux systems) are packaged in the so-called GNU coreutils. Most other systems provide similar, if not the same, commands - they may be packaged differently however.

We will start with a simple problem: Given an undirected, acyclic and connected graph, commonly known as tree, find the unique path between two vertices i and j of said tree.

The first problem we have to solve is representing our graph. Recall from that there are may possible ways to represent graphs: Adjacency lists, (for each vertex list all its neighbors), Adjacency matrices (entry (i, j) is 1 iff edge (i, j) exists), and many more. The most obvious solution would be to put all of that information in a text file and then operate on that, but then we would be no further than C++, since we would still have to implement most of the algorithms by hand. So let us go up a step in the hierarchy: If files are bad, can we maybe use directories instead?

It turns out that this works nicely, but only when our graph is a rooted tree. In a rooted tree we choose an arbitrary vertex as root and then list all vertices ordered by their distance from the root: First comes the root itself, then the root’s neighbors, then the root’s neighbors’ neighbors, etc.

A tree with 7 vertices, rooted at vertex 0

We can easily translate this tree into a directory structure:

mkdir -p 0/1/3/{4,5}
mkdir -p 0/2/6

How do we display our tree? The best utility I know for this task is aptly named tree(1). Sadly, it is not part of the GNU coreutils and you most likely have to install it with your favorite package manager. Running tree then shows us the tree we constructed:

% tree
.
└── 0
├── 1
│   └── 3
│   ├── 4
│   └── 5
└── 2
└── 6

Now we can answer questions such as “How can I get from 0 to i?”, where i is an arbitrary vertex other³ of our tree using the find(1) command. The find command traverses all files in a given directory, and can optionally search for specific criteria, for example directory names. If we wanted to know how to get from vertex 0 to vertex 3 we can run the following command:

% find 0 -name 3
0/1/3

which tells us that we can go from 0 to 3 via vertex 1. In fact, we can generalize this to arbitrary vertices i and j: First we check how to get from 0 to i, and then from 0 to j, and finally we have to find the common ancestor of both paths. Then we reverse the path from 0 to i (now it is a path from i to 0), walk along it until we hit the common ancestor of i and j, and then walk down the path from the common ancestor to j.

The most difficult part of this procedure is finding the common ancestor. Lucky for us the package diffutils provides the cmp(1) command, which can tell us the first byte at which two files differ. The output of cmp is a bit verbose, but using cut(1) and tr(1) from the coreutils we can extract the relevant information:

% cmp <(echo "testa") <(echo "testb")
/proc/self/fd/11 /proc/self/fd/12 differ: byte 5, line 1
% cmp <(echo "testa") <(echo "testb") | cut -d " " -f 5 | tr -cd "[:digit:]"
5

How does this work? The program cut takes a delimiter (-d) and some position number (-f) and returns the string at the given position (1-indexed) when splitting on the delimiter specified by -d. We use tr to delete (-d) all non-digits (-c stands for the complement of the character set) to ensure that we get 5 as result, and not 5,. So far so good, let us look where we are now:

# In Bash arguments are passed implicitly and referenced
# using $1, $2, etc. We use $1 as i and $2 as j
path_between() {
 path_to_i=$(find -name $1)
 path_to_j=$(find -name $2)

 first_diff_byte=$(cmp <(echo $path_to_i) <(echo $path_to_j) \
 | cut -d " " -f 5 \
 | tr -c -d '[:digit:]')

It remains to find the common ancestor of i and j. Since we know where the paths differ, we know up to which character they are identical. Using bash variable expansion we can identify the common prefix, and using cut extract the vertex they have in common. The rev command from the util-linux package reverses a string of characters:

 common_prefix=${path_to_i:0:(( $first_diff_byte - 1 ))}
 common_ancestor=$(echo -n $common_prefix | rev | cut -d / -f 2 | rev)

The part in the double parenthesis is subject to the expr(1) command⁴, which and evaluates the expression as arithmetic expression. Now all we have left to do is to stick all of it together:

 # if we omit local here we set $PATH...
 local path=$(echo ${path_to_i#"$common_prefix"} \
 | tr '/' $'\n' \
 | tac \
 | paste -s -d '/')
 path=$path/$common_ancestor/${path_to_j#"$common_prefix"}

 echo $path
}

Using the tr, tac and paste commands (all part of coreutils) we can reverse the path from i to the common ancestor of i and j. We do this by changing the slashes which delimit our vertices to newlines and using tac⁵ to reverse the order of our vertices. Finally we can use paste to put the reversed vertices back together in a single string and then append the path from the common ancestor to j.

Testing it on our input graph gives the following output:

% path_between 4 5
4/3/5
% path_between 5 4
5/3/4
% path_between 0 5
/.//1/3/5
% path_between 3 4
/1//4

It seems like it fails when the common ancestor is either i or j. We can remedy this by handling these cases separately: If i is the common ancestor, we can simply take the path from i to j minus the path from 0 to the ancestor of i. Else if j is the common ancestor then we have to first reverse the path from j to i (so we get the path from i to j) and the remove the path to j’s ancestor.

if [[ $common_prefix == $path_to_i ]]; then
 local path=$1${path_to_j#"$common_prefix"}
elif [[ $common_prefix == $path_to_j ]]; then
 local path=$(echo ${path_to_i#"$common_prefix"} \
 | tr '/' $'\n' \
 | tac \
 | paste -s -d /)
 path=$path$2
else
 # Do what we did before
fi

Now it gives correct results for any query:

% path_between 3 4
3/4
% path_between 4 3
4/3
% path_between 6 4
6/2/0/1/3/4
% path_between 0 1
0/1
% path_between 1 0
1/0

That’s it. In 20 lines of bash we wrote a pathfinding algorithm for trees. Not only that, but we did it without needing to compile any code, only using tools available on most Linux systems! I find that quite impressive.

Pathfinding in general graphs

Restricting ourselves to rooted trees sounds pretty limiting doesn’t it? Can we generalize our model to arbitrary graphs? At first it seems like we cannot, since then our file system (which is where we store our graph) does not support cycles. After all, no directory can be a (grand-)parent of itself. But it turns out that using symbolic links we can “point” to other directories, which can in turn point to other directories, which can point to other directories, etc. We can create symlinks using the ln command (incidentally also found in coreutils), by writing ln -s <directory point to> <name of the link>. For each vertex we want to create a directory and in that directory create symbolic links to all of its neighbors. If you recall the different forms of graph representation before this corresponds to adjacency lists. Let us try this with a simple example.

A triangle with a three-armed star, connected via one arm

We can create this graph as follows:

% mkdir `seq 0 5`
% ln -s ../0 1
% ln -s ../1 0
% ln -s ../2 0
% ln -s ../0 2
% <continue for each edge>
% tree
.
├── 0
│   ├── 1 -> ../1
│   └── 2 -> ../2
├── 1
│   ├── 0 -> ../0
│   └── 2 -> ../2
├── 2
│   ├── 0 -> ../0
│   ├── 1 -> ../1
│   └── 4 -> ../4
├── 3
│   └── 4 -> ../4
├── 4
│   ├── 2 -> ../2
│   ├── 3 -> ../3
│   └── 5 -> ../5
└── 5   
└── 4 -> ../4

Now let us look at the same problem as before: We want to specify two vertices i and j, and want to find a path between them (if it exists). Last time we used the find(1) utility to find these. This time we will explicitly tell find in which vertex i we would like to start by specifying it as the root of our search:

% find 0/ -name 1
0/1
% find 0/ -name 3
<nothing>

Hmm, it seems as if find does not follow the symbolic links. Looking at find’s manual page (man find) we see the following

OPTIONS The -H, -L and -P options control the treatment of symbolic links.
[...]
-P Never follow symbolic links. This is the default behaviour. When
find examines or prints information a file, and the file is a sym‐
bolic link, the information used shall be taken from the properties
of the symbolic link itself.
-L Follow symbolic links. When find examines or prints information
about files, the information used shall be taken from the properties
of the file to which the link points, not from the link itself (un‐
less it is a broken symbolic link or find is unable to examine the
file to which the link points). Use of this option implies -noleaf.
If you later use the -P option, -noleaf will still be in effect. If
-L is in effect and find discovers a symbolic link to a subdirectory
during its search, the subdirectory pointed to by the symbolic link
will be searched.

Aha, we have to specify find -L if we want to follow symlinks. But our symlinks induce cycles in the file system, what happens if find encounters a cycle? Again the manpage has the answer for us:

The POSIX standard requires that find detects loops:
The find utility shall detect infinite loops; that is, entering a
previously visited directory that is an ancestor of the last file en‐
countered. When it detects an infinite loop, find shall write a di‐
agnostic message to standard error and shall either recover its posi‐
tion in the hierarchy or terminate.

So we are good to go!

% find -L 0/ -name 3
find: File system loop detected; ‘0/2/4/5/4’ is part of the same file system loop as ‘0/2/4’.
0/2/4/3
find: File system loop detected; ‘0/2/4/3/4’ is part of the same file system loop as ‘0/2/4’.
find: File system loop detected; ‘0/2/4/2’ is part of the same file system loop as ‘0/2’.
find: File system loop detected; ‘0/2/1/2’ is part of the same file system loop as ‘0/2’.
find: File system loop detected; ‘0/2/1/0’ is part of the same file system loop as ‘0/’.
find: File system loop detected; ‘0/2/0’ is part of the same file system loop as ‘0/’.
find: File system loop detected; ‘0/1/2/4/5/4’ is part of the same file system loop as ‘0/1/2/4’.
0/1/2/4/3
find: File system loop detected; ‘0/1/2/4/3/4’ is part of the same file system loop as ‘0/1/2/4’.
find: File system loop detected; ‘0/1/2/4/2’ is part of the same file system loop as ‘0/1/2’.
find: File system loop detected; ‘0/1/2/1’ is part of the same file system loop as ‘0/1’.
find: File system loop detected; ‘0/1/2/0’ is part of the same file system loop as ‘0/’.
find: File system loop detected; ‘0/1/0’ is part of the same file system loop as ‘0/’.

We get two paths and a whole lot of errors that find detected a loop. We can just ignore these:

% find 0/ -name 3 2>/dev/null
0/2/4/3
0/1/2/4/3

Perfect! Now the find command returns all the paths between 0 and 3 in the graph. And it is much simpler than our previous solution: find does all the work for us.

This approach can easily be extended to directed graphs by only symlinking in the direction the edge goes, which means that we can find all paths between vertices i and j in any graph in a single line of bash. That’s pretty amazing, don’t you think? We can even improve the solution a bit: What if we only want a path, and not all paths? The manual page of find once again has the answer:

-quit Exit immediately. No child processes will be left running, but no
more paths specified on the command line will be processed. For ex‐
ample, find /tmp/foo /tmp/bar -print -quit will print only /tmp/foo.
Any command lines which have been built up with -execdir ... {} +
will be invoked before find exits. The exit status may or may not be
zero, depending on whether an error has already occurred.
[...]
-print True; print the full file name on the standard output, followed by a
newline.

If we specify -print and -quit the find command will immediately exit when it finds the first path:

% find -L 0/ -name 3 -print -quit 2>/dev/null
0/2/4/3

This works on any graph, directed or undirected, and will print a path if it finds one. Since find will never continue running if it found a loop this is successful even if no paths exist between the given vertices:

% find -L 0/ -name 6 -print -quit 2>/dev/null
<nothing>

Note that find operates by depth first search, and thus might not find the shortest path between two vertices. The last thing we will show is how to (ab)use the mindepth and maxdepth options in order to force find to do a breadth first search, which always finds the shortest path in unweighted graphs:

find_shortest() {
 local i=0
 local p=""
 while [[ -z $p ]]; do
 p=$(find -L $1 -mindepth $i -maxdepth $i -name $2 -print -quit 2>/dev/null)
 i=$(( i+1 ))
 done
 echo $p
}

# Output
% find_shortest 0 3
0/2/4/3

While we have not found a path, we tell find to extend its search radius by one, and once we found our desired path we output it. This approach is guaranteed to always return the shortest path, although it is computationally more expensive since we often traverse the same parts, and it also won’t detect if no path exists between these vertices. We could extend this program to add termination detection, however then it would lose its elegance, which is what I wanted to show in this post.

Closing remarks

Interestingly enough, somebody actually implemented bfs(1), which behaves like find and internally uses breadth-first search, but it is not (and probably will never) be part of coreutils.

I did use this algorithm in one of my courses to solve small pathfinding problems⁶. Implementing this was quite fun and not as much pain as I’d feared. Sometimes these `clever’ solutions have their merit - although for more sophicated problems I’d rather stick to C++ and BGL.

short for Boost Graph Library ↩︎
The numbers in parenthesis correspond to the man page section of the programs. Most programs have their manpage in section 1, which is for user commands, If you are interested in more details or the other sections man man gives great insight. ↩︎
Actually the case where we input the same number twice is tedious since our solution depends on the paths being different, as you will see later. ↩︎
Also a part of the coreutils package ↩︎
Like cat but reverses the order of the lines ↩︎
and so far I haven’t been kicked out ↩︎