Intercontinental Kubernetes Cluster: Lessons Learned

Two years ago, I started building a Kubernetes cluster. Because I wanted it to be highly available, I decided to make sure the nodes were on different internet connections. Because I like owning my data, I wanted these locations to be my home or the homes of trusted family members.

This means I ended up with nodes:

  • At my home in Germany
  • At my mother’s home in the US
  • At my fiancée’s mother’s home in the US

all connected to residential internet connections from different providers.

And then, because I wasn’t content to deal with availability zones and data replication, I made this all one cluster. For those of you who aren’t already looking at me like I’m nuts: This is nuts.

In standard production deployments, one usually has services replicated both within availability zones and between availability zones. In theory, this means that you can arbitrarily restart running pods within a zone (eg: to upgrade something to a new version) and are resilient …

Continue reading

Smarter Cat Feeder Part 1: Intro & Hardware Checkout

This post is the first part in my series on making a smarter cat feeder. When it’s complete, I’ll add a link to the “how-to” here.

Motivation

I have two cats who I love very dearly. I also occasionally travel for short enough times that it does not make sense to leave the cats with someone. While someone has to stop by to do some tasks (like emptying the litterbox) asking someone to come by to feed them four times a day would be excessive.

So I bought some automated feeders. Specifically the Trixie TX7 (page in German, because I’m not aware of them being sold on english sites). At the time of writing, this feeder is listed at €85. I bought them fo €57. Honestly, for the current price, I’d pay the €5 extra to get something better-designed and already app-controllable. But I bought these years ago, and it was a €100 euro jump to something similar…

Anyway. I have these feeders. Mechanically they work pretty well, but they’re basically …

Continue reading

Per-client nsupdate/RFC2136 DNS Updates when dnsmasq Issues a Lease

I have a fairly complicated home network setup. I use bind to provide DNS to the network. I also like to have each host on the network have both forward- and backward-resolvable DNS. The best way to do this - that I found - was to run BIND and configure it to take DNS updates via nsupdate/RFC2136.

OpenWRT used to use ISC DHCP, which natively provides a way to have updates sent when a DHCP lease is issued to a client. Unfortunately, they no longer do (as of the 23.x release series, I believe). Instead, ISC DHCP has been replaced with the very-capable dnsmasq.

Unfortunately, dnsmasq doesn’t have support for nsupdate/RFC2136. what it does have is a way to call a script each time a DHCP lease is issued. So let’s use that to perform the DNS updates.

Below is the quick script I wrote to do this. The script can be installed by:

  1. Writing it to /usr/local/bin/dnsmasq-nsupdate.sh.
  2. Running chmod +x /usr/local/bin/dnsmasq-nsupdate.sh to make the script executable.
  3. Appending …
Continue reading

Lenovo 300e Gen 2, Part 4: Laptop/slate Driver Investigation

Recap

In Part 3, we dug through the ACPI definition of the Laptop/slate indicator. While we don’t understand absolutely everything about how it works, we’re confident that the ACPI definitions look relatively normal.

This time, let’s start looking at driver integrations.

Identifiers Recap

We got the following identifiers out of Windows in part 1:

  • BIOS Name: \_SB.CIND
  • Compatible IDs: ACPI\PNP0C60 and PNP0C60
  • Hardware IDs: ACPI\VEN_AMDI&DEV_0081, ACPI\AMDI0081, *AMDI0081
  • Device Instance Path: ACPI\AMDI0081\0

In part 3, we saw that:

So let’s dig into these identifiers and see if we can turn up anything useful online.

Searching… Searching…

AMDI0081

No meaningful results. But result 4 points to this series of posts, so… yay?

PNP0C60

Lots of results!

The most useful ones are from Microsoft:

Continue reading

Lenovo 300e Gen 2, Part 3: Digging in to the Laptop/slate switch and ACPI

Recap

Last time, we determined that simply asking Linux to identify as Windows didn’t make a difference. That’s okay, it was a long shot anyway. However, we also gathered a bunch of useful general information about the system.

Now we’ll get started with specific tasks to determine why the laptop/slate mode switch doesn’t work.

Identifiers Recap

We got the following identifiers out of Windows in part 1:

  • BIOS Name: \_SB.CIND
  • Compatible IDs: ACPI\PNP0C60 and PNP0C60
  • Hardware IDs: ACPI\VEN_AMDI&DEV_0081, ACPI\AMDI0081, *AMDI0081
  • Device Instance Path: ACPI\AMDI0081\0

From these identifiers, we know the Laptop/slate mode switch is an ACPI device - mostly from Device Instance Path, but also because ACPI appears in a bunch of places in identifiers.

In part 2, we gathered the full dmesg output. That’s rather long, so let’s find all the dmesg output that mentions ACPI:

[steve@fedora ~]$ sudo dmesg | grep -i acpi
[sudo] password for steve: 
[    0.000000] …
Continue reading

Lenovo 300e Gen 2, Part 2: lshw, lspci, dmesg, and some initial investigation

Recap

Last time we determined what devices don’t work under Linux - the touchpad and the laptop/tablet mode sensor. We also went in and gathered some useful hardware identifiers for these devices.

Next, we’ll reboot to Linux and do some early exploratory work.

A Word on Distros

For various reasons, I ended up installing Fedora Workstation 37. I’m going to use it as the “working” linux distro while I explore and fix – and from there, potentially try others.

Why Fedora? Normally on new hardware, I try something that keeps the newest software possible. In the past, I’ve used Manjaro Linux, as it strikes an excellent balance between “easy to boot/install” and “up-to-date”. For whatever reason, I couldn’t get it to boot - the system refused to recognize the USB drive.

On my “stable” systems, I tend to run Debian Linux. As someone who often works with bleeding edge systems, it’s nice when I don’t …

Continue reading

Fixing Linux Unsupported Devices on the Lenovo 300e Gen 2

I got a Lenovo 300e (with Windows) from a Lenovo sale (for $126!). It seems to be a neat little machine - but it doesn’t fully work under Linux. As far as I know, cheap machines like this tend to be built with cheap parts, which tend not to be new, but instead minor additions or changes to new ones. Manufacturers often give these new device IDs, which sometimes means drivers don’t support them. I’m hoping that’s the case here.

Lenovo 300e AMD laptop, shown in four different positions. It’s a laptop with a 360 degree hinge. Shown top-right, as a laptop. Bottom-right as a tablet, with the keyboard on the table. Bottom-left as a tent, with the hinge at the top of the system. And top-left as a tablet-stand, with the keyboard on the table, effectively halfway to tablet mode.

I tried this with an older Lenovo Yoga machine and it turned out the sensor for laptop/tablet mode did some weird accelerometer-based stuff. I figured out how it worked, but was never able to get the system fully working. So let’s hope this one goes better!

The Plan

My plan with this device is pretty simple, at least at a high level:

  1. Identify non-working devices
  2. Gather identifiers
  3. Determine if existing drivers work for those devices
  4. If yes, add support to those drivers
  5. If not, is there anything close we can borrow from?
  6. If yes, …
Continue reading

Publications and Appearances Update

I did a bunch of public speaking this year. I have finally got all the material here on my website and the various conferences have released the video. So, here’s updates from all my speaking this year.

IEEE Cluster HPCMASPA

I presented the first work towards my PhD at the HPCMASPA workshop of IEEE Cluster. Slides and preprint paper are online. No video was made, so no video will be uploaded.

A New HOPE

HOPE is always a favorite of mine to attend and this year I got to present! I presented a significantly revised and extended version of my hCaptcha work. Video, slides, and FAQ are now online.

Enigma 2022

Way back in January, I gave a first talk on hCaptcha at Enigma 2022. The video of this went up a couple months ago and I missed it. It’s now been added to the page for that appearance.

Continue reading

Website hosting changed

So I just changed the hosting for my website. It’s now hosted using Hugo and on GitHub pages. Long story short, if anything seems broken, Contact me

This is because Django2 support is coming to an end and Zinnia (the blogging engine I used with Django) never got support to move it to Django3. So after at least 6 years, it’s time for a new site.

Honestly, the technical of this is not that interesting.

Some improvements

I took the opportunity to make some improvements to the theme I’m using. It’s now more responsive and (hopefully) more helpfully responsive as screen size shifts.

Why Hugo?

Mostly because this is a static site. I don’t use any non-static features on my website, so why use (and pay for) something more complicated than I need?

As a side effect, this reduces the amount of software I need to keep up to date, removes two daily cron job emails, and shuts down a web server.

I also like writing in Markdown. I keep my personal notes in Markdown when …

Continue reading

Enigma 2022

I appeared at Enigma 2022 to give a talk called “Broken CAPTCHAs and Fractured Equity: Privacy and Security in hCaptcha’s Accessibility Workflow”. That talk came with a fair amount of accompanying material, including videos, slides, and code. That material can all be found on the Enigma 2022 page.

There will eventually be video.

Continue reading