Talk about Separation of Concerns

README.md

@@ -9,11 +9,14 @@ in the nick of time by [Andres Freund][freund], most of our planet's SSH
 servers would have begun granting root access to the party behind this
 attack.
 
-Unfortunately, too much analysis has focused on how [malicious code][JiaT75] and
-made its way into the xz-utils repo.  Instead, I'd like to argue that two
-longstanding design decisions in critical open source software are what made
-this attack possible: [linking OpenSSH against SystemD][biebl], and the
-existence of [GNU IFUNC][sourceware].
+Unfortunately, too much analysis has focused on how [malicious
+code][JiaT75] made its way into the xz-utils repo.  Instead, I'd like to
+argue that two longstanding design decisions in critical open source
+software are what made this attack possible: [linking OpenSSH against
+SystemD][biebl], and the existence of [GNU IFUNC][sourceware].
+
+
+
 
 ## Quick Recap of CVE-2024-3094
 There are tons of good writeups outlining the high level details of the xz-utils
@@ -43,13 +46,17 @@ flowchart TD
     G --> D
 ```
 
+
+
+
 ## Why do Linux Distros modify OpenSSH?
-The short answer is that they have to. OpenSSH is developed by the OpenBSD
-community, for the OpenBSD community, and they do not give one flying shit about
-Linux.  The [OpenSSH Portable][mindrot] project is a best-effort collection of
-patches which replace OpenBSD-specific components with generic POSIX components,
-and some platform-specific code where applicable. The software supply-chain for
-SSH ends up looking something like this in practice:
+The short answer is that they have to. OpenSSH is developed by the
+OpenBSD community, for the OpenBSD community, and they do not give a
+flying shit about Linux.  The [OpenSSH Portable][mindrot] project is a
+best-effort collection of patches which replace OpenBSD-specific
+components with generic POSIX components, and some platform-specific
+code where applicable. The software supply-chain for SSH ends up looking
+something like this in practice:
 
 ```mermaid
 flowchart TD
@@ -88,10 +95,10 @@ downstream to the OpenSSH Portable project, which attempts to re-implement new
 features in ways that aren't specific to OpenBSD. This is what allows SSH to
 work on platforms like Linux, macOS, FreeBSD, and even Windows. 
 
-But it doesn't stop there. Some operating systems apply further customization
-beyond what OpenSSH Portable provides. For example, Apple adds the
-[`--apple-use-keychain`][github] flag to `ssh-add` to help it integrate with the
-macOS password manager.
+But it doesn't stop there. Some operating systems apply further
+customization beyond what OpenSSH Portable provides. For example, Apple
+adds the [`--apple-use-keychain`][keith] flag to `ssh-add` to help it
+integrate with the macOS password manager.
 
 In the case of CVE-2024-3094, Fedora and Debian maintained their own
 [SystemD patches][biebl] for their forks of OpenSSH in order to fix a
@@ -111,34 +118,45 @@ flowchart TD
 ```
 
 These patches never went into OpenSSH Portable, because the OpenSSH
-Portable folks have explicitly stated ["we're not interested in taking a
-dependency on libsystemd"][djmdjm]. And they never went into upstream OpenSSH,
-because OpenBSD doesn't have anything that resembles SystemD.
+Portable folks were ["not interested in taking a dependency on
+libsystemd"][djmdjm]. And they never went into upstream OpenSSH, because
+OpenBSD doesn't have any need to support SystemD.
 
+
+
+### Concerns about "Separation of Concerns"
 This seems harmless enough, but it's an example of a much large problem in Open
 Source, particularly in Linux: critical components of the operating system are
 developed by people who don't know each other, and don't talk to each other. 
 
-* The folks patching OpenSSH for SystemD might not have known that SystemD
-  depends on xz-utils.
-* The SystemD folks might not have known that xz-utils began leveraging ifunc.
-* The folks designing OpenSSH do so on a platform that doesn't even *have*
-  ifunc, or anything that resembles it.
+* Did the folks who patched OpenSSH for SystemD know (or care) that
+  libsystemd depends on xz-utils?
+* Did the SystemD folks know (or care) that xz-utils had begun using ifunc?
+* Did the OpenSSH folks know (or care) that ifunc was a thing? It's
+  certainly not a thing on OpenBSD.
 
 In some sense, this breakdown in communication is a feature of open source: I
 can adapt your work to my needs without having to bother you about it. But it
 can also lead to a degree of indirection that prevents critical design
 assumptions (such as a traditional dynamic linking process) from being upheld.
 
+The obvious corollary to [Conway's Law][conway] is that if you are
+shipping your org chart, you're also shipping the bugs that live in the
+cracks of your org chart. No one person or team really made a mistake
+here, but with the benefit of hindsight it's clear the attackers
+perceived that the left hand of Debian/Fedora SSH did not know what the
+right hand of xz-utils was doing.
+
 
-## What does GNU IFUNC even do?
-It allows you to determine, at runtime, which version of some function you'd
-like to use. It does this by allowing you to run **arbitrary code** to
-influence how the linker resolves symbols.
+
+## What is GNU IFUNC *supposed* do?
+It allows you to determine, at runtime, which version of some function
+you'd like to use. It does this by giving you to an opportunity to run
+**arbitrary code** to influence how the linker resolves symbols.
 
 ![](memes/ifunc_hard_right.png)
 
-Suppose you have one application that must run on a wide variety of x86 CPUs.
+Suppose you have an application that must run on a wide variety of x86 CPUs.
 Depending on the specific features of the current CPU, you may prefer to use
 different algorithms for the same task. The idea behind IFUNC was to allow
 programs to check for CPU features the first time a function is called, and
@@ -160,6 +178,7 @@ uses IFUNC to load the appropriate implementation when the program
 starts.
 
 
+
 ### Isn't that just function pointers?
 Yes, it's like function pointers, but slower:
 
@@ -184,6 +203,7 @@ document, but for now just understand that using GNU IFUNC incurs a little extra
 overhead, even though it exists for the sake of performance optimizations.
 
 
+
 ### Can't I accomplish the same thing with `LD_PRELOAD`?
 Sortof! GNU IFUNC allows developers to make runtime decisions about which
 version of a function is best to use. If you know what decisions need to be
@@ -203,6 +223,8 @@ fi
 (If you are unfamiliar with `LD_PRELOAD`, check out catonmat's ["A Simple
 `LD_PRELOAD` Tutorial"][catonmat].)
 
+
+
 ### How does GNU IFUNC work?
 There are three things at play here:
 * PLT
@@ -243,6 +265,9 @@ RELRO           STACK CANARY      NX            PIE             RPATH      RUNPA
 Partial RELRO   No canary found   NX enabled    No PIE          No RPATH   No RUNPATH   36 Symbols        No    0               0               ./plt_example.exe
 ```
 
+
+
+
 ## IFUNC is Probably a Bad Idea
 ![](memes/ifunc_change_my_mind.png)
 
@@ -261,6 +286,7 @@ an internal interface for glibc, and limit its use in other
 applications.
 
 
+
 ### It's too Confusing to Use Safely
 ifunc is entirely too difficult to use. There are too many [corner cases][nagy], and the
 [official documentation][gnu-cfa] is [scant][sourceware]. This gives users the
@@ -277,6 +303,7 @@ It isn't just IFUNC either. Apple Mach-O has a similar feature called
 `.symbol_resolver` which they ["regret adding"][rjmccall].
 
 
+
 ### It isn't Much Faster than Alternatives
 Given that the usual justification for ifunc is performance-related, I wanted to
 see how much overhead *ifunc itself* causes. After all, any function worth
@@ -324,6 +351,7 @@ optimizing are much more expensive than the "increment by one" functions that we
 are analyzing here. It is interesting because GNU IFUNC claims to be a boon for
 performance.
 
+
 #### Performance of Other Techniques
 There are other techniques which are slower than ifunc. Take a look at the
 `super_rigorous_speed_demo`, which brings to other experiments into play:
@@ -355,21 +383,24 @@ single CPU feature to check.
 | always  | 10.07| 10.56 | 10.2333 |
 
 
+
+
 ## Recap
 ![Yes, all shared libraries](memes/brain.png)
 
 [agner]: https://www.agner.org/optimize/blog/read.php?i=167
 [biebl]: https://salsa.debian.org/ssh-team/openssh/-/commit/818791ef8edf087481bd49eb32335c8d7e1953d6
 [catonmat]: https://catonmat.net/simple-ld-preload-tutorial
 [checksec]: https://man.archlinux.org/man/checksec.1.en
+[conway]: https://en.wikipedia.org/wiki/Conway's_law
 [djmdjm]: https://github.com/openssh/openssh-portable/pull/251#issuecomment-2027935208
 [fr0gger]: https://infosec.exchange/@fr0gger/112189232773640259
 [freund]: https://www.openwall.com/lists/oss-security/2024/03/29/4
-[github]: https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent?platform=mac#adding-your-ssh-key-to-the-ssh-agent
 [gnu-cfa]: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-ifunc-function-attribute
 [goodin1]: https://arstechnica.com/security/2024/04/what-we-know-about-the-xz-utils-backdoor-that-almost-infected-the-world/
 [jasoncc]: https://jasoncc.github.io/gnu_gcc_glibc/gnu-ifunc.html#relocations-and-pic
 [JiaT75]: https://github.com/tukaani-project/xz/commit/cf44e4b7f5dfdbf8c78aef377c10f71e274f63c0
+[keith]: https://keith.github.io/xcode-man-pages/ssh-add.1.html#apple-use-keychain
 [mindrot]: https://anongit.mindrot.org/openssh.git
 [nagy]: https://sourceware.org/legacy-ml/libc-alpha/2015-11/msg00108.html
 [nvd]: https://nvd.nist.gov/vuln/detail/CVE-2024-3094