Well, that escalated quickly
Bunfight between developers or matters of (inter-) national security?
Despite the best efforts of the security community, there is still an understanding gap between the various tools and formats related to code signing and “software supply chain security”. To an outsider, it’s hard even to determine what the nouns and verbs are. Even to someone with extensive knowledge of the Cloud Native space as a whole, I get lost as to the point of various projects. I remember in-toto being something to do with orange juice? Thankfully, the maintainers are a friendly lot, often academic or academia-adjacent. Maybe Stuxnet changed the vibe? Reading the tone of the security space is hard enough when you do know what the nouns are.
The Update Framework (TUF) is an academic project dating back to 2009. From its website:
The Update Framework (TUF) helps developers maintain the security of software update systems, providing protection even against attackers that compromise the repository or signing keys. TUF provides a flexible framework and specification that developers can adopt into any software update system.
In 2015, Docker announced an implementation of TUF for validating delivery of containers, called Notary. (It’s worth pointing out this was one month before the formation of the CNCF.) From that point, TUF and Notary were so tied together that they joined the CNCF as a pair in October 2017 and were announced in the same blog post. Importantly, the code for both was hosted in the same GitHub organization, https://github.com/theupdateframework/.
By mid 2019, TUF was on the way to becoming a graduated project (a level it reached in December 2019) and Notary was at risk of being removed from the CNCF due to inactivity.
At the end of 2019, planning began on a version 2 of Notary, starting with a KubeCon talk from Docker security engineer Justin Cormack. The talk stated that in retrospect, Docker didn’t think the TUF model was fully suitable for containers. A team representing many container registry vendors — with one representative from TUF — met a month later.
February 20211 saw the first commit to Cosign, which was announced in May as part of the new Sigstore initiative — itself launched in March. Dan Lorenc from Sigstore would later go on to write up why and how the Cosign project differs from Notary v2.
With all that history out of the way, we move to the saga at hand.
The saga at hand
The TUF project is led by Professor Justin Cappos from the NYU Tandon School of Engineering, who is also one of the tech leads of CNCF’s TAG Security.
In July 2021, on behalf of a number of TUF, Sigstore and TAG Security contributors, Cappos wrote to the CNCF’s Technical Oversight Committee (TOC). His message raised concerns about the governance, security, and branding of the Notary project — notably that the project had effectively launched a net new thing (Notary v2) under the same name, without any of the review that would normally have been done on a CNCF incubating project.
Soon after, we learn some of the reasons for the creation of Cosign as a separate entity: Lorenc says he and others had tried to join the Notary community in 2020, finding it “impossible to navigate and very unwelcoming”.
Cappos’ outreach to the TOC caused the Notary project to be asked to make some changes, including splitting into two distinct GitHub organizations and electing official maintainers. No members of the TUF community were added, even though this was challenged at the time; guidance was given that other maintainers could be added later2. Notably, the TOC also stated, via precedent, that it is OK to completely replace the code in a project and keep the same name and level.
Given no substantive progress on the main issues, Cappos raised the issue publicly in December 2022.
By the powers vested in me
At the start of February, the TOC’s second response was returned: it’s up to the project to fix things, and it should be able to do so following its own guidelines. (Oh, and please get a security audit.)
The dissent was swift and loud, though (in my opinion) respectful. On one hand, it was a story of big companies vs. students working on their dissertations. On the other, the TOC doesn’t crown kings3.
Mention was made of the fact that the TUF maintainers didn’t qualify for a seat at the table, as they were not maintainers of the Notary project. Cappos pointed out that the ways that the projects had always operated as one, and how they were not invited as maintainers when the separation happened. Code of conduct complaints were discussed as having been ignored, with relevant parties having not been consulted4. A public meeting on the topic was due to be held, and then canceled. Further commentary implied the TOC had no further levers to pull regarding this issue5. (This issue has bled the TOC of time to process the 7 projects on their graduation backlog, among other responsibilities.)
The heat is on
The heat turned up to 11 with TAG Security tech lead Andres Vega claiming the Notary v2 project had weakened the security of the project at the request of the Chinese government. Two weeks later, the American government released a new National Cybersecurity Strategy, which explicitly called out the People’s Republic of China as “the broadest, most active, and most persistent threat to both government and private security networks”, among other things.
Vega jumped straight to a hypothetical exploit scenario6, and asked “who bears responsibility, given we had the opportunity to prevent it but did not?”
Nic Chaillan, the former Chief Software Office of the US Air Force and Space Force was equally displeased. He reserved stronger condemnation for LinkedIn, claiming “[the] CCP is infiltrating [Notary] and it seems that [the LF and CNCF] are just sitting there watching and doing a whole bunch of nothing!”.
The next piece of nothing was the closing of the issue as “too heated”.
Someone may have to do some actual journalism to learn what happens next. In the immortal words of OMC: “Wanna know the rest: hey, buy the rights”.
New and newsworthy
After a very public divorce from the cloud, 37signals are now breaking up with Kubernetes too. CTO David Heinemeier Hansson has released MRSK7. MRSK is perhaps best explained as “the authors of Capistrano pricked their finger on a thorn and fell asleep for 15 years”, or as Justin Garrison put it on deprecated social network Twitter. It’s unashamedly imperative, eschewing the declarative model that Kubernetes and even Docker Swarm have made popular.
The key to understanding Google’s open source projects is that very few are “what we are currently running” and a lot are “what we wish we could migrate to one day”. (Does it feel weird to think of Google as a legacy company?)
Please meet Service Weaver, a framework for writing distributed applications. Or, in their words, “write your application as a modular monolith and deploy it as a set of microservices.
Now, as an ex-Googler with no knowledge of this project, is it destined to become Kubernetes or Stadia?8 Well, it was announced on the Open Source blog, not the Cloud blog, and it looks to have come out of the MapReduce team9. Until you see Google Cloud Service Weaver10, don’t assume that Google wants to make any money on it, but do imagine it’s full of interesting ideas. (I assume, given it appears there are some “greybeards” involved, they are not destined to reimplement CORBA.)
The Kubernetes project really, really wants you to know that k8s.gcr.io is going away and you have to use registry.k8s.io now. There will be redirects, but they will look like a man in the middle attack and may not help you if your security infrastructure is not aware.
A lot of planning has gone into this, but does it all turn on the fact the domain used from the start was “gcr.io”? Could Google not have redirected k8s.gcr.io to some other proxy service?11 Do they not, to put it in a nutshell, peer?
The rest
Like Talos Linux, the OS designed to be managed by Kubernetes, but worried it’s too free? There’s now a SaaS management platform for it: meet Omni, from Talos creators Sidero.
Like Kubernetes, the platform designed to manage nodes and containers, but worried there are too many nodes and containers? Kwok, Kubernetes without Kubelet12, is a toolkit for simulating the lifecycle of fake nodes, pods, and other Kubernetes API resources. Use it to test how your controller might work on a cluster with thousands of nodes, without, you know, paying for that.
Postgres-based database provider Timescale is getting out of the Prometheus backend game, discontinuing their Promscale open source utility.
Moving up the stack: a new release of Cilium adds support for the Gateway API, and lays the groundwork for promised future of mTLS support on the datapath.
Moving down the stack: “we’ve had that for years!” says Istio, and merges the new ambient mode into the main branch, along with a Rust-based implementation of the zero-trust tunnel.
Mirantis has launched a new major release of its “can’t call it Docker” container runtime, the first in two years. The release also reconnects with the upstream Moby Project, which you may remember was created to separate “Docker the open source project” from “Docker the brand name”13. The announcement also claims to have rebooted Moby, but not so much that their website doesn’t still recommend “Docker EE”, a project that hasn’t existed since 2020.
Beware of blindly using Alpine Linux; it might be a bad choice for you, especially if you don’t like debugging DNS. (Martin Heinz)
Backend Service-based external Network load balancers are now generally available with GKE. The overloading of all of those words (c.f. load balancer and LoadBalancer) makes that sentence really hard to parse, but I guess progress is good even if naming is hard.
CNCF news
The IoT Edge Working Group has published an Edge Native Application Principles Whitepaper, launched in draft last October. It largely spells out the differences between what they are terming “edge native” and cloud native, boiling them down into five principles. Meaningful advice on how to actually meet these principles is a topic for another day.
Kubevela, an application delivery engine, and OpenKruise, an open source automation engine, have been promoted to incubating projects. Both were created at Alibaba.
Inspektor Gadget (eBPF toolkit) and Clusternet (multi-cluster management tool) have been voted into the CNCF Sandbox.
containerd and Cilium audits were published.
Seeeeeee-curity
ARMO has added support for the CIS EKS benchmark to ARMO Platform and Kubescape. Aside from the regular14 Kubernetes benchmark, CIS publishes variants for the hosted versions. Kubescape can check for 50 of the 53 recommendations, which is a lot more than the 16 that the tool AWS recommended when they launched! (Some handsome bloke on the ARMO Blog)
Following the recent GA of the Sigstore project, version 2.0 of Cosign has been released. The tool will interact with Sigstore services by default, fetch an identity-based certificate from Fulcio and uploading the signature and signing key to Rekor15.
In the other sandbox: Azure Kubernetes Service now supports pod sandboxing in preview. Microsoft brought in Kata Containers16 to solve this problem, and thus your nodes will require Nested Virtualization for it to work.
gVisor had a “slow mode” (PTrace sandbox) and a “fast mode”, which was saved for Google-internal use cases. The fast mode was recently open sourced. A new root overlay feature for runsc can also show performance improvements of up to 38%. (Lucas Manning/Twitter)
Tigera has added runtime threat defense to their commercial Calico product, promising to combine signature and behaviour-based techniques to detect both known and zero-day threats.
A release candidate for SLSA v1.0 has been announced. A major change, the RC divides the SLSA spec into multiple tracks, which are separate sets of levels that measure a particular aspect of software supply chain security. The new “Build” track mostly matches the levels from the 0.1 spec, with source requirements removed. A “Source” track is planned for the future, as is Level 4 for Build.
Was this change set about because few people actually reached the higher levels of SLSA? KubeEdge proudly claims to be the only CNCF project so far to have attained Level 3 compliance.
The Sysdig Threat Research Team discovered an attack they named SCARLETEEL, which started poking around their customer’s AWS environment and ended up stealing proprietary software running on Lambda. The exploit started with a vulnerable public service running on Kubernetes, mined some crypto just to show off, and then went looking for Terraform state files in S3 buckets. Sysdig’s Nigel Douglas also showed us how to use the OWASP Top 10 to do a risk assessment of your Kubernetes environment, and recommends some tools to help here. They make some. I do too.
Business and finance
That thing with the bank happened, and hopefully has settled down by the time you read this.
Red Kubes announced another $1.2 million in seed funding for their Otomi Kubernetes platform, along with the appointment of ex-VMware sales exec Rouven Bester as CEO. (Silicon Canals)
Edgeless Systems has taken a €5 million seed round to build out Confidential Kubernetes and its open source Constellation platform to provide it.
Kubernetes storage vendor Ondat, formerly (and officially) known as StorageOS, was acquired by Akamai, after layoffs last July. No purchase price was announced. (Blocks and Files).
Akamai also hit the news by raising prices across its portfolio, including the Linode cloud.
In case you were worried you can’t raise money without the K-word in your pitch deck, Temporal took another $75M investment to sit at a $1.5B valuation. (TechCrunch)
That was too many nouns. Talk about something else please
No major upsets in the Oscars. Haven’t seen any of the films. Watching films is one of the first things to go, but it’ll come back. Would probably enjoy “Everything Everywhere All At Once”.
Made sure to listen to Dark Side of the Moon for its recent 50th anniversary, though. I may only have been familiar with it for slightly over half its life, but you really do get something new from every listen. This time, for whatever reason, I ended up focusing on the drums in the song “Breathe”, and how the hi-hat pattern changes from bar to bar. That makes you wonder, was it planned like that, or was it just Nick Mason being Nick Mason17?
I am still not sure where 2020 went.
Narrator: they never were.
Did anyone else see that the CNCF now publishes a Code of Conduct Transparency Report? I don’t recall this being announced.
The only action the TOC believes it can take is to archive a project, and that has never been done against the wishes of a project.
The kind with its own logo and domain name.
A naughty acronym as yet unexplained, or just a thinly-veiled reference to the shipping company?
Sanjay Ghemawat’s name is on some recent commits!
Please let them codename it “Sigourney”.
To save Amazon money?
See also Garfield Minus Garfield.
You may also remember Docker the brand name was separated from Mirantis a couple of years ago, with all their acquired-from-Docker products renamed.
Not “vanilla”. I like vanilla; it’s the finest of the flavours.
Did I say I there was an understanding gap? There are a lot of new nouns here.
Did you forget that the OpenStack Foundation was renamed the Open Infrastructure Foundation? I did.
Roger Waters once described him as “a metronome that eats”, which is an anecdote that was probably more enjoyable a few months ago.