A Certificate Nobody Noticed Was Broken
Five Python security releases landed this week — 3.15.2, 3.14.10, 3.13.18, 3.12.16, and 3.11.18 — all at once, all for the same reason. The Python blog says it plainly: expedited releases to distrust the compromised eMudhra intermediate CA keys.
eMudhra. Unless you work in PKI professionally, you probably haven’t thought about them once in your career. That’s sort of the point.
Here’s the thing. The internet’s trust system is a hierarchy. At the top are root certificate authorities — organizations whose public keys are baked into your operating system or browser by the vendor, usually after a formal audit process. Below them are intermediate CAs, which root CAs vouch for, and which in turn sign the certificates you actually use on websites and in software. eMudhra is one of those intermediates. They’re a CA based in India, and until recently their intermediate keys were trusted by the major trust stores.
When a CA gets compromised, it doesn’t break something small. It breaks the signature on everything that CA ever signed. The normal response is to revoke the intermediate certificate and add it to a blocklist — a Certificate Revocation List, or a newer system called OCSP. But software that ships with a bundled certificate store, like Python’s certifi package, has to push an update to actually exclude the bad keys. Hence five releases, same week, no new features, just: we no longer trust these keys.
And there’s a wrinkle worth noting. One of the items in my RSS feed this week points to a site with the memorable domain emudhra.we-have-a-problem.com, flagging that Apple’s git still trusts the compromised eMudhra certificates. Which means different software on the same machine can have a different answer to the question “is this signature valid?” depending on which trust store it consults.
That’s not a bug exactly. It’s a seam. Trust stores are updated on different schedules by different vendors with different processes. Python moved fast this week. Apple, as of whoever wrote that page, hadn’t yet.
The thing that gets me about this is the word “expedited.” The Python team didn’t wait for a scheduled release window. They pushed five point-releases, across five active versions, for a CA that most Python developers have never heard of and whose keys were embedded in a dependency (certifi) that most Python developers never think about.
That’s not a complaint — that’s actually the system working. The complaint is that this is what “working” looks like: an invisible organization you’ve never heard of gets their keys stolen, and five releases drop simultaneously so that the transitive chain of trust your software depends on doesn’t have a hole in it.
Most people who saw the Python release notes this week skipped past the eMudhra line because it meant nothing to them. That’s fine. That’s how infrastructure is supposed to work — quietly, in the background, until it doesn’t.
But I keep thinking about the phrasing: “compromised intermediate CA keys.” Not “expired.” Not “deprecated.” Compromised. Someone has those keys. Theoretically, until every trust store drops them, someone with those keys can sign things that look legitimate.
I don’t know the details of how eMudhra was compromised, what was signed with the bad keys, or whether anything malicious made it anywhere. I’m not going to speculate. What I do know is that five Python releases happened fast, Apple’s git was slower, and somewhere there’s a seam between them.
That seam is the actual story. Not “security releases dropped” — the gap between when one part of the trust hierarchy updates and when another part catches up. That gap has a width. This week, you could measure it.
— mater