It started with the frustration of going through an information security compliance audit. I had no issue with the standards - they build trust between organisations. But the process was laborious and irritating.

I just couldn’t understand why it would be so hard to prove that you do as you say, in a software company where every single interaction happens on a SaaS platform.

Why can’t more of this process be automated? The data is right there.

The idea

Working with cloud infra, I was already a fan of Terraform and HCL.

Basically, Terraform lets you declare the infrastructure you want with a domain-specific language (DSL), and then makes it so by building a dependency graph and calling cloud APIs in the right order to create the resources you need.

Could we do a similar thing for work data?

I mean, a structured, declarative representation of a business that’s accessible to both humans and machines.

If you could do that, you’d close the loop that audits close manually today. And probably a lot more.

The experiment

In autumn of 2025, I decided to go my own way with 42futures - a so-called “lifestyle business” focused on building production-ready software pilots.

The context is entirely different as a small business owner. But it was a good time to put my idea into practice. See if there’s anything to it.

I didn’t need it to succeed, but I did need to satisfy my curiosity.

So I built Firm which I describe as a plain-text work management tool. I published it on GitHub and was positively surprised by the amount of interest from technologists out there thinking similar thoughts.

What is Firm?

Firm is a set of tools and libraries to organise a “Business-as-Code”.

Technically, Firm is:

• A DSL to define and organise business entities in plain text

• A library to build and use graphs of business entities

• A CLI for humans and machines to interact with business graphs

• An MCP server for chatbots to do the same

The Firm DSL is what makes it code. I like code, because it’s structured, lightweight, automatable, easily stored and easily versioned. As it turns out, AI agents like code, too.

There’s more to say about how Firm works, but I’d rather show how I use it. Here are my experiences from the last 4 months.

How I use Firm

I like to start small and focus on immediate needs. This is a story of how I’ve progressively introduced the “business-as-code” concept in 42futures.

Stage 1: Rolodex

The first thing I needed was a structured way of doing outreach. I wanted to be able to keep track of contacts, early convos and potential leads. Something like a lightweight CRM or a Rolodex.

So I created a git repo and started adding my immediate network into .firm files. I would note a few things about each contact:

contact john_doe {
    name = "John Doe"
    role = "Senior software engineer"
    status = enum"network"
    notes = "We used to build cool stuff together"
}

interaction coffee_chat_with_john {
    interaction_date = 2025-10-02 at 11:00 UTC+1
    initiator_ref = person.daniel_rothmann
    primary_contact_ref = contact.john_doe
    type = enum"in-person meeting"
    
    subject = "Coffee chat"
    outcome = """
        Always nice to connect! 
        John showed me how to set up Rust workspaces.
    """
}

At this point, I’m not much better off than free-form notes. But with references, I build up relationships between entities. And with the Firm CLI, I can quickly recall related data before a meeting:

firm related contact john_doe
> Found 1 relationship for 'contact' entity with ID 'john_doe'

interaction.coffee_chat_with_john

Interaction Date = 2025-10-02 at 11:00 UTC+1
Subject: Coffee chat
<etc...>

Stage 2: Project management

Some of the early conversations presented opportunities, and some project work trickled in. Nice!

Still using the DSL, I evolved my Firm workspace to track opportunities, projects and tasks.

opportunity app_pilot {
    name = "Potential app pilot"
    source_ref = interaction.coffee_chat_with_john
    status = enum"closed won"
    value = 4200 EUR

    notes = """
        Discussed at the coffee meeting. 
        This will be an exciting first project. 
        Thanks John!
    """
}

project app_pilot {
    name = "App pilot"
    status = enum"in progress"
    description = "Build an app to do a thing"
    
    due_date = 2026-01-01 at 12:00 UTC+1
    owner_ref = person.daniel_rothmann
}

task app_pilot_platform {
    name = "Figure out cross-platform or native"
    description = "John wants to know which is better"
    source_ref = project.app_pilot
    assignee_ref = person.daniel_rothmann
    
    is_completed = false
}

My pipeline from contact -> interaction -> opportunity -> project -> work started taking shape. It’s all linked together in the business entity graph.

Here’s what that graph looks like for my workspace:

Every entity is a node. Every reference is an edge. As the workspace grows, so does the graph.

Stage 3: Query engine

At this point I started using the Firm CLI more to read and write entities instead of editing DSL directly.

With a bigger workspace, wrangling individual entities got annoying. So I added query capabilities.

This query gets my unfinished tasks:

firm query 'from tasks | where assignee_ref == person.me and is_completed == false'

And this query gets new leads indirectly related to tech companies:

firm query 'from organization | where industry == "tech" | related(2) lead | where status == "qualified" | order created_at | limit 10'

With the workspace growing, the query engine enabled me to do more complex searches without trolling through too much data.

Stage 4: Coding agents

AI agents have gotten pretty good at writing code and using tools. The Firm DSL is code. The CLI is a tool. Maybe there’s something there?

So I hooked up Claude Code to my Firm workspace and was positively surprised.

The coding agent was great at taking high-level, ambiguous questions and turning them into concrete queries for the Firm CLI and direct interactions with the Firm DSL.

That felt like a huge unlock! I hooked up MacWhisper on my machine to dictate inputs, and suddenly I found myself more often “talking” to my workspace.

I started to absorb functionality I would previously use other tools for. Just because it was easier this way.

For example, I replaced my time tracker with a custom time_entry entity type. After a billable session, I would tell the agent what I did and for how long. The entries pool on an invoice entity for billing.

Stage 5: MCP on the move

I spend some (yes, some!) time away from my computer. When I get an idea or think of something I need to do, I want to access the Firm workspace.

So I added an MCP server. You run it locally with:

firm mcp

The MCP server provides validation, ensuring changes are valid and safe.

I put it in the cloud and added it as a custom connector in the Claude app. It connects to a remote git repository and makes changes on a dedicated branch. Now I can tell my bot secretary what I need, and it takes care of it.

I can also delegate: “Research potential clients in my area and log them as leads for my review”. Bot secretary does a decent job.

When I’m back at the PC, I review and merge to main. Every change is tracked and versioned.

So… does the idea work?

To be honest, it’s hard to say for a bigger org. I’m trying not to get overly attached. I wanted to try it, iterate on it, and abandon if it turned out a waste of time.

But for me, the tool has progressively provided more value - as the data model expanded, as I customised workflows, and as I layered automations on top.

With scripting and bots, Firm is becoming a backbone of my business.

What I like about it is that this is enough. I don’t need as many tools. Firm enables this sort of “second brain” - from low-level note-taking to a holistic picture of my priorities, readily available to me and my bot.

I should also say: where it is now, Firm is for technologists. It works for me because I like the command line.

But I’ve noticed that as I use MCP more, I’m less often in the terminal and more often just blabbering away. Chat as the interface.

That feels like a sci-fi dream coming true - not because of Firm specifically, but because of where AI is going. Voice transcription is good, language models are good. Tool calling is getting there. Firm fits right into that.

Why it works

Most structured data is machine-friendly but human-opaque. SQL tables, JSON blobs, protocol buffers. You need dashboards to make sense of it.

Most human-friendly formats are the opposite. Notes, documents, scribbles on a napkin. An LLM can parse them, but there’s no structure to verify against. Hard to know if the machine understood it correctly.

Firm sits in a sweet spot. The DSL is structured enough to be verifiable - things can be right or wrong, and agents can self-correct. It’s the same reason that language models are good at coding. Firm brings that property to work management.

But the underlying data is just text. The bot can read it, but so can I. To catch mistakes and provide feedback. Version control gives me safety around automation.

Essentially:

Firm creates a shared surface between bots and humans. Structure and verifiability for machines. Readable plain text and version control for me.

I think it’s the thing that makes this approach work.

What’s next?

I increasingly see Firm as a data layer. I don’t want to add AI features to it. It’s a tool, not a platform. The value is in how it’s used together with other things.

I already log invoices in Firm. Could I integrate that with plain text accounting? I write reports with Typst and track them as file assets in Firm. What else can you connect?

When your business data is right there, structured and queryable, many workflows that usually require monolithic SaaS tools become a script or a prompt away.

I think that’s where this gets exciting: Firm as one tool in the “agentic toolbox”, where the prompt is your interface and structured data keeps the agents honest. Not adding intelligence to the data layer, but letting the intelligence sit on top, and giving it something good to work with.

As for the compliance question that started all of this - I’m working in a different context now, but the itch remains.

A year ago, this was an abstract idea. Now it’s a central component in how I run my business.

It works for me.

Daniel Rothmann runs 42futures, where he helps technical leaders validate high-stakes technical decisions through production-ready software pilots.

Figure it out first, then execute.

Explore, then exploit!

But when you’re creating new products, that line doesn’t exist. You learn by building. You build on learning. The learn/build ratio shifts over time, sure, but they never really separate.

Many would agree that Waterfall failed because it separated planning from execution. But we keep making that same distinction, only in different ways.

3 ways to draw a line

Most software initiatives share a common goal:

1. Figure out a great thing to build

2. Build said great thing

3. Release the thing

4. Profit!

After you try it a few times, you realise that the process isn’t quite that linear. There’s uncertainty to grapple with.

We try to groom uncertainty by drawing boundaries. Through the stack, through phases, through scope. But these lines, though well-intentioned, tend to separate learning from doing.

That’s a problem, because learning and doing aren’t separable activities.

Stack boundaries

You can draw lines through the stack.

Horizontal: layer by layer, bottom-up. Get the data layer right, then business logic, then UI. “Software is like building a house”, so you build the foundations first.

But unless you can anticipate what the final thing looks like, you end up with great foundations for a house you’re not building.

Vertical: feature by feature, left to right. A thin slice through the whole stack: One complete thing you can test and ship.

Vertical is better than horizontal when facing uncertainty. But it doesn’t tell you what to build first. You could slice any feature - which one? And why? Without some ordering principle you end up wasting time on slices that miss the point entirely.

Phase boundaries

You can draw lines through time.

First we learn, then we build, then we test. That’s POCs for you: build a prototype, see if it works, then build the real thing.

POCs often fail in two ways:

1. It’s too far from real product. The POC environment doesn’t match reality, so the lessons don’t transfer. You throw it away and learn new lessons.

2. It’s too close to real product. The POC ships, and now you’re maintaining cowboy code in production. That’s no fun either.

You can’t really front-load learning like that. Building real product is how you find out what works.

Scope boundaries

You can draw lines through scope.

What’s in scope, what’s out of scope. An MVP: define the “minimum viable” thing, build that, ship it.

But “minimum” according to whom? “Viable” how? Technically, commercially? Viable enough to… uh, learn from?

Too often, the “minimum viable” scope becomes whatever’s in budget. So we reverse engineer the “MVP” from constraints: we have X time and Y €, so the MVP is just whatever fits in that box.

This is deciding how much to build before clarifying what it is.

There is only product

Product is the only tangible thing. You can have more of it, or less of it. Its content can be more or less central to the product idea.

And building real product teaches real lessons.

All three boundaries (stack, phase, scope) try to draw lines through a continuum. But the lines don’t map onto something real. There’s no such a thing as complete knowledge - typically, we’re partially ignorant throughout the entire process. That’s just the reality of creating something new.

So instead of pretending that we can separate learning from building, why not embrace it? Set up the process to maximise learning. Order the work by what teaches the best lessons.

If that sounds like a rehash of what agile practitioners were already saying, well, yeah, it is. Learning fosters adaptation.

The goal is to gain insight for sound software investment decisions. Are we building the right thing? Should we do it differently? Should we stop?

Here’s how I put these concepts into practice.

How I run pilots

Critically, I’ve stopped asking “what’s the scope?” and started asking “what should we do first?”

Start with the big question

The first thing to figure out is what we want to learn. That’s just a question. A good question is answerable just like a scientific hypothesis is falsifiable.

Then we can list out high-level components that mighthelp answer that question.

Order matters

Next, we order the work by:

1. Centrality: What is most likely to answer the question?

2. Dependency: What depends on what?

By centrality, I mean: how likely is this work to answer the big question? The most central work is what the whole premise stands on. If it doesn’t work, nothing else matters. If it does, we’ve validated something essential.

By dependency, I mean: some features don’t work without others. We can’t build recommendations without something to base recommendations on.

The first thing we build is the most central component that doesn’t depend on anything else. Then we work outward. What’s the next-most central thing we can do now that we have that? And the next?

We start with the highest learning potential, then expand.

There is no minimum

If we spend all our time on the core premise because that’s where the danger is, that’s time well spent. Either it works out, or we find fundamental flaws. That’s more helpful than building peripheral features.

In this line of thinking, minimum doesn’t exist. Only centrality matters.

Fix effort, not scope

But planning matters. Budgets matter. You can’t just say “give me infinite time and I’ll build whatever you need.”

A straightforward way to manage that risk is to fix the effort [1].

Fix the time and price. Order by centrality and dependency, then let the scope expand as you go. Build production software from day one [2], and you’ll be happy at any stopping point.

What we learn

Sometimes the solution is hard to find - a signal to stop.

Sometimes it proves viable quickly - a signal to continue.

If we maximise learning about the big question [3], we can make that decision quickly and confidently. Abandon, pivot, or expand.

The planets model

To illustrate this thinking, I use a metaphor: planets.

Think of the Earth’s core. The molten center that sustains life on the crust. That’s a central component.

I work in concentric circles from the middle out. Each ring extends outwards from the last. We keep expanding until we’ve covered enough to answer the question.

Coverage is the ground we’ve covered so far. Not a fixed target, but a way to see progress. It grows organically as you move outward.

Adjacent features become “moons”: Smaller ideas orbiting the work. Unrelated questions become other “planets”: work you might tackle later.

This is a way to communicate:

• What I’m focusing on

• How I order the work

• How components relate

• How we’re doing on progress

Planets are not a fancy new methodology. They’re just a communication tool; A way I can discuss priorities without pretending to have a fixed plan.

Planets in practice

Let me share a real example to make things concrete.

The project: EnergyMate, an app for post-cancer cognitive rehabilitation.

The question: Can mobile technology help with cognitive rehab by making energy accounting more accessible?

Mapping components: The client had already done a lot of thinking into what the app might do: energy logging, assessments, statistics, community, AI coaching, psychoeducation. Lots of ideas!

Ordering by centrality: Where do we start? With energy logging. That’s the central component. Without energy logging, we can’t test if energy accounting helps and if mobile technology makes it easier.

From there, we expanded outward:

• Assessments (to measure progress)

• Statistics (to see patterns)

• Recommendations (to increase accessibility)

• Educational resources (to explain the method)

We focused on iOS, local-first, and left AI features for later. This increased optionality [4] while allowing us to maximise learning about the big question: does energy accounting with a mobile app actually help?

The result: In 6 weeks, we had a working app ready for closed group testing. Real users, data and insights into whether the idea works.

It covers about half of the original vision, but it’s the important half. Complete enough to test with real people. Ready to release and extend: not just a demo.

We managed it in a short timeframe by changing the shape of the problem - ordering the work by centrality.

When it works

I find this way of working effective under uncertainty [5].

You’re working under uncertainty when:

• You have incomplete knowledge (don’t know the best solution)

• The solution space is unknown (new technology or unproven markets)

• System complexity makes outcomes hard to predict

This describes many new software projects.

Working from centrality, you create the conditions to fail fast or succeed quickly. You maximise learning per unit of effort. The work you do is high-impact and unlikely to be thrown away.

More specifically, you create real options to:

• Abandon: Filter out bad ideas early

• Pivot: Adjust based on evidence

• Expand: Invest in working software

If you’re interested in the financial mechanics of real options, I’ve written way more here: https://blog.42futures.com/p/code-economics

When it doesn’t work

If you have complete knowledge, fixed requirements, or you’re working with commodity solutions (all situations with low uncertainty) then just make a plan and stick to it. No need to complicate things.

But lots of software projects aren’t like that.

That’s why I think this mindset matters.

Conclusion

All these words to pitch for a shift.

A shift from asking:

• What’s minimum viable?

• Should we explore or build?

• What foundations do we need?

, to asking:

• What’s the big question?

• What’s more central to the question?

• What depends on what?

I’ve stopped drawing lines through scope, phases, or the stack. Instead, I work in circles: centrality, dependency and coverage.

The planet model helps me do that, and I think it’s a neat way to pritise work in software pilots that maximises learning to minimise risk.

Daniel Rothmann runs 42futures, where he helps technical leaders validate high-stakes technical decisions through structured software pilots.

Notes

[1] Fixing effort while varying scope isn’t a new idea - that’s common sense agile. But it works well when you order by centrality. You front-loaded the essential work, so any stopping point gives you something worthwhile. Compare that to pre-scoping an MVP: if you got the scope right, you’ve done the same thing with more planning overhead. If you got it wrong, you overspend or ship the wrong stuff. Fixing effort keeps all those options open: stop early, continue, or reassess.

[2] By “production software” I mean reliable, maintainable, robust code. You could deploy it and it works. I’m not saying you check off every non-functional requirement from day one. If you have specific needs (compliance, integrations, etc.), you may work them in as you go. The point is to avoid the POC trap of code that’s intended to be throwaway but either ships because “it kind of works” or fails to transfer because it only worked in isolation.

[3] This is like validated learning from Eric Ries’s Lean Startup, but applied to technical product. Lean Startup focused on commercial validation. Do people want this? You might test that with a landing page. For technical product validation we need to go further. Does this actually work? Does the concept hold up? To answer those questions, we need to build working software.

[4] Deferring decisions creates options. By not committing to things that aren’t central to the main question, you keep future paths open. The ability to change direction based on what you learn is valuable, especially early on when you know the least.

[5] Complex systems create uncertainty. Cause and effect is hard to link, so long-term planning doesn’t work that well. Optimization is a better approach: instead of trying to predict the best outcome, you step deliberately in a direction that looks good, assess, and step again. Think of it like navigating terrain: you can’t see the other side of the mountain, so you look at what’s around you and move based on that. Ordering by centrality is a way of picking those first steps.

A new frontier in data security is becoming production-ready. It’s called confidential computing, and it’s jumped straight up my list of “exciting boring technologies”.

In this post, we’ll explore confidential computing - what it is, how it works and if it’s worth your time.

TL;DR

Confidential computing closes a major security gap by protecting data while it’s in-use. With hardware-based “trusted execution environments”, it seals applications in a “black box” that is inaccessible to the host system or cloud provider. This enables verifiably private processing of sensitive data.

The technology is now practical: historical barriers of performance, cost and availability have largely been overcome with modern approaches. This makes it a strategic imperative for anyone handling regulated or highly sensitive data. For everyone else, it positions confidential computing as a future secure-by-default foundation of the cloud, making it a technology worth understanding today.

In the near future, “confidential computing” will just be “computing.”
Mark Russinovich, CTO at Azure

The case for confidential computing

All modern businesses process sensitive data. Personal, financial, trade secrets. Keeping that stuff safe is pretty important, and one of the ways we do that is by using encryption.

Traditionally, we protect data:

• In-transit: where we encrypt it as it travels from one system to another.

• At-rest: where we encrypt it before we store it.

These measures help prevent unauthorized access through man-in-the-middle attacks, data breaches and physical theft. But when that data is decrypted to be processed, it’s exposed to anyone with insider access. The security of your application alone isn’t enough: vulnerabilities in the underlying operating system or hypervisor is part of the attack surface, too.

Confidential computing is a hardware-based technology that protects data and code in-use. It achieves this by having the processor create and enforce a dedicated processing environment, isolating your app from the host [1].

This solves big problems of trust. If an app is truly isolated, you don’t need to trust the operating system, the administrator, or your cloud provider. Just the hardware vendor. I’ve written previously on the risks of hosting EU data with US public cloud providers and this kind of technology can mitigate some of those risks. As it happens, the US tech industry is pushing pretty hard for confidential computing these days [2].

There’s more to it, but let’s get some context first.

Privacy-preserving computation

The core idea isn’t new. It stretches all the way back to the late 70’s with Rivest, Adleman, and Dertouzos work on “private homomorphism” [3].

In the 90’s, the idea of “Trusted Computing” started to take hold. This popularised the concept of the “Trusted Platform Module”, which made its way into consumer technology in the 00’s, enabling use cases like disk encryption [4]. As of Windows 11, Microsoft requires it [5].

The next big thing is the invention of “Fully Homomorphic Encryption” in 2009, which enables computation on encrypted data [6]. That’s a huge leap forward (!), but also extremely slow [7].

At this point, practical hardware-based security solutions pick up in popularity. Apple introduces its “Secure Enclave” in 2013 [8] and in 2015, Intel launches “Software Guard Extensions” (SGX), designed for secure remote computation and DRM [9]. This causes a growth spurt in practical use cases for confidential computing, leading to the formation of the Confidential Computing Consortium (CCC) under the Linux Foundation in 2019 [10].

Fast-forward to now.

In hardware, the big dogs are in: AMD SEV-SNP, Arm CCA, Intel SGX + TDX, and NVIDIA Confidential Computing.

In open source, we have projects like Contrast for Kubernetes, OpenHCL for VMs, and ManaTEEfor data analysis.

The service providers are in, too. Azure has built a decent offering in confidential computing, like VMs, containers and managed services. European STACKIT provides confidential Kubernetes and VMs. We also have smaller vendors like Edgeless Systems with focused products like Privatemode AI.

How it works

There’s no doubt that the momentum is building for confidential computing. The real appeal is that it pushes trust down the stack: from trusting the software environment to trusting the computing hardware. That opens for new possibilities, like “data clean rooms”, privacy-preserving AI and bringing more sensitive workloads in the cloud.

But how does confidential computing establish that trust?

Trusted execution environments

It all begins with secure hardware. This is our “root of trust”. This, along with code designed to run within it, constitutes a “Trusted Execution Environment” (TEE).

Let’s take Intel SGX as an example:

1. An SGX-enabled processor (like Intel Xeon 6 [12]) reserves a region of main memory. This is where the processor will host “enclaves” which we can think of as “secure containers” for code and data.

2. When we create an enclave, our initial program is loaded from unprotected memory into enclave memory, passing through an encryption module using a key known only to the hardware itself.

3. When processor loads that data into its cache, it passes it through the same module to decrypt it.

4. When it writes data back, that is also encrypted. So on, and so forth.

5. The programmer of the enclaved process decides what is ultimately committed to unprotected memory: some aggregate plaintext result, something encrypted or nothing at all [13].

The data is only ever plaintext inside the physical chip. Everything outside that boundary, including main memory, is encrypted. This creates a hardware-enforced black box that isolates the enclave from the host system.

While SGX pioneered this approach, newer technologies like Intel TDX and AMD SEV-SNP apply the same ideas of hardware-enforced memory encryption to entire virtual machines and containers, making it easier to adopt for existing apps.

Remote attestation

If you have the required hardware, you can compute confidentially. Provided you trust the manufacturer to have designed it properly. But how can you know the hardware is legit? That the vendor didn’t swap it? Or the host isn’t lying to you?

If we’re thinking in principles of “zero-trust”, a cloud operator can’t be implicitly trusted. “Trust me bro, I’ve got TEEs” isn’t enough. We need assurances that our code and data is secure [14].

That’s where attestation comes in. Attestation is the process that lets the hardware prove that it’s a geniuine TEE running your software. Think of it like certification. Here’s how it works:

1. When an enclave is created, the processor fingerprints its initial code and data. This is called a measurement. Any change to the code will result in a different measurement.

2. The processor creates a document called a quote (or report) that contains the measurement and maybe other data you will want to verify.

3. The report is signed with a private key that was embedded in the processor at the factory. It’s like the processor’s identity.

4. Your remote system receives the signed report. It verifies the signature using public certificates from the hardware vendor. Then, it checks that the measurement inside the report matches the fingerprint of the software it expected to run [15].

If these checks succeed, you have strong evidence that you’re interacting with your own, unmodified software inside a genuine trusted execution environment.

Subscribe now

Levels of isolation

Confidential computing isn’t one single thing - it’s a variety of technologies and solutions with different approaches and tradeoffs.

One of the central factors is how different technologies approach memory isolation. We have 3 primary models [16].

Code

This is the isolation model introduced by Intel SGX. It wraps individual applications or even specific functions in a secure enclave. This offers great protection, as the “trusted computing base” is only your application code. But it’s not plug-and-play.

Virtual Machine

Intel TDX and AMD SEV-SNP focus on isolating an entire VM, protecting its memory from the host hypervisor. This model works for “lift-and-shift” migrations of existing workloads to a confidential environment, no code changes needed. The trade-off is a larger “trusted computing base” because you must trust the guest OS inside the VM.

Container

Container-level isolation builds on the hardware features of VM isolation (like TDX and SEV-SNP) to create a solution that works for cloud-native setups. Instead of running a full OS, it uses a minimal, specialized runtime to launch the container within the hardware-protected memory boundary.

This approach is easier to adopt than code-level enclaves (no code changes needed) but offers a smaller attack surface than a full confidential VM, since you don’t need to trust an entire guest OS.

Downsides

Okay, if confidential computing is so great, why isn’t it everywhere yet? Where’s that hype train?

Security

While confidential computing raises the bar for security, it’s not invincible.

Researchers have demonstrated physical side-channel attacks. A recent example is TEE.Fail, where a custom hardware device physically intercepts memory traffic to extract secrets [17]. Other attacks have successfully inferred data by looking at CPU power consumption patterns [18].

This shows that even with memory encryption, the physical hardware is still an attack vector.

Availability

The big players are in on confidential computing, but the hardware isn’t exactly everywhere.

Not all cloud providers offer confidential computing, and those that do, offer it for a subset of machines and services.

At this time, Microsoft Azure probably has the most expansive catalog.

Performance

Memory encryption isn’t free.

For code-level isolation with Intel SGX, the overhead can be significant. For some operations, enclaved code can be >5x slower than native [19].

But when we look at VM/container-level isolation, the performance is much better. Just a few percentage points of overhead compared to equivalent unprotected cloud setups [20].

A 5x slowdown is a dealbreaker for most, but the minimal overhead of newer approaches is a far more acceptable compromise for the security it provides.

Costs

With the performance overhead described above, we can expect a slightly higher baseline cost per unit of work. Additionally, cloud providers tend to price confidential instances at a premium. This varies between vendors.

We also need to factor in, that development for confidential computing can be more complex. If we make use of code-level isolation, we’ll require special SDKs and a different way of thinking about app architecture.

At this time, I would expect to spend 10-20% more on confidential workloads, depending on how you host.

Is it worth your time?

It depends on what you’re building. You should actively explore it today if you:

Handle highly sensitive data:

If you’re working with medical records, financial data, proprietary AI models, the extra layer of hardware-enforced protection could be a big win.

Need to collaborate on private data:

If you need to involve multiple parties to analyse a shared dataset without revealing raw data between parties, confidential computing enables this with “data clean rooms” and collaborative AI training.

Are subject to strict data regulation:

If you’re navigating complex regulation like GDPR/CCPA/HIPAA/PCI DSS, confidential computing lets you do more while staying compliant. It helps you prove that not even a cloud provider can access your data.

Want to build true zero-trust architecture:

If your goal is to minimize trust in your infrastructure and provider, confidential computing is the next frontier. It allows you to “assume breach” of the host environment and still protect your workload.

For everyone else...

The benefit will probably arrive silently. As the technology becomes more widespread and costs decrease, it is on a solid trajectory to become the default, secure foundation for the cloud.

And while “encrypting data in-use” is a good shorthand, the real innovation is the practical approach - a hardware-sealed black box, where data is only ever plaintext for the briefest of moments inside the box itself.

This is the idea that finally allows us to protect data through its entire lifecycle: at-rest, in-transit, and now, in-use. It might not be long before, as Russinovich said, “confidential computing” will just be “computing”.

Daniel Rothmann runs 42futures, where he helps technical leaders validate high-stakes technical decisions through structured software pilots.

Sources

[1] https://www.linuxfoundation.org/hubfs/Research%20Reports/TheCaseforConfidentialComputing_071124.pdf

[2] https://www.abiresearch.com/press/the-advent-of-gpu-based-confidential-compute-is-set-to-revolutionize-the-confidential-computing-market

[3] https://luca-giuzzi.unibs.it/corsi/Support/papers-cryptography/RAD78.pdf

[4] Trusted Computing by Denggue Feng, Yu Qin, Xiaobo Chu, Shijun Zhao, 2017

[5] https://support.microsoft.com/en-us/windows/enable-tpm-2-0-on-your-pc-1fd5a332-360d-4f46-a1e7-ae6b0c90645c

[6] https://en.wikipedia.org/wiki/Homomorphic_encryption

[7] https://arxiv.org/pdf/2501.18371

[8] https://eclecticlight.co/2025/08/30/a-brief-history-of-the-secure-enclave/

[9] https://en.wikipedia.org/wiki/Software_Guard_Extensions

[10] https://www.linuxfoundation.org/press/press-release/confidential-computing-consortium-establishes-formation-with-founding-members-and-open-governance-structure-2

[11] https://dualitytech.com/blog/data-clean-room/

[12] https://www.intel.com/content/www/us/en/architecture-and-technology/software-guard-extensions-processors.html

[13] https://eprint.iacr.org/2016/086.pdf

[14] Zero Trust Networks by Razi Rais, Christina Morillo, Evan Gilman, Doug Barth, 2024

[15] https://edera.dev/stories/remote-attestation-in-confidential-computing-explained

[16] Azure Confidential Computing and Zero Trust by Razi Rais, Jeff Birnbaum, Graham Bury, Vikas Bhatia, 2023

[17] https://thehackernews.com/2025/10/new-teefail-side-channel-attack.html

[18] https://arxiv.org/pdf/2405.06242v1

[19] https://www.ibr.cs.tu-bs.de/users/weichbr/papers/middleware2018.pdf

[20] https://dl.acm.org/doi/10.1145/3696843.3696848

It’s a big deal and it’s not. Sometimes, cloud providers have outages. Doesn’t matter where you host. But when something bad happens in a US public cloud, it’s jarring to see the degree of trouble that causes in the EU as well.

That’s what I want to focus on here: How European digital infrastructure relies on a handful of US companies, the risks it poses, and what we can do about it.

To clarify, we need to answer three questions:

• Is hosting EU services in US clouds still a good idea?

• What are the options for EU-sovereign hosting?

• How should an EU business choose its hosting strategy?

Let’s get into it.

The cloud race is over: US hyperscalers won

This probably won’t surprise you. Today, most businesses have adopted the cloud. As many 90% of them, according to a 2021 O’Reilly survey [1]. The vast majority of that cloud capacity is provided by the “Big Three”: AWS, Azure and Google Cloud, in that order [2].

The advent of cloud offered many benefits: faster iteration, scalability and cost savings through consumption-based billing plus economies of scale. Although businesses migrating to the cloud had varied cost savings success (–144% to +500%) [3], I think for a long time the Big Three delivered on their core promise.

So they won hard, and they won globally. To a point where they now hold massive leverage over today’s internet businesses. That’s when the dynamic starts shifting.

Squeezing the lemon

That’s a Danish saying, by the way. You use it when someone exploits a situation fully, to their own benefit. But if you squeeze a lemon too hard, it runs out of juice.

Global cloud spend is going up. Of course it is, because more businesses are using more cloud. At the same time, ~80% of businesses report that their cloud spend is higher than expected, according to a 2024 CloudZero survey [4]. That number has been going up too.

Wasn’t cloud supposed to be cheaper? Well. We have seen the Big Three reduce prices over time, but they’re not tracking with effeciency gains in available hardware. In 2014, Google Cloud reported that hardware costs improved 20-30% annually while public cloud prices only fell by 8% a year [5]. Google’s cloud platform pricing would start following the Moore’s Law curve, announcing price reductions on various core offerings.

In 2022, Google announced a 50% price hike for multi-region nearline storage and a doubling of storage network fees [6]. AWS has hiked prices too, for example through the introduction of the Public IPv4 Charge [7]. I have my own anecdotes for Azure, like when they deprecated Azure CDN in favour of Front Door, forcing us to take on new idle fees and an effective downgrade to our Web Application Firewall if we didn’t pay up for the “Premium” SKU.

These days, AWS operating margins are approaching 40% [8], quite the step up from 28% back in 2015 [9]. You have to wonder how these cloud companies are getting rich so much quicker than the rest of us.

US-EU relationship strain

Putting aside how we each might feel about the current US administration, it’s clear that the situtation is posing a risk to European businesses.

First, there’s the US CLOUD Act, signed into law by Donald Trump in 2018 [10].

The CLOUD Act lets US law enforcement compel American companies to provide access to data stored abroad, even if that data is in a European data center and belongs to a non-US entity.

It gives the US government the ability to access data controlled by American-owned clouds without the cooperation of (or disclosure to) European authorities. This conflicts with Europe’s GDPR, which seeks to protect the privacy of EU citizens by preventing their personal information to be transfered without legal basis [11].

Then we have the “kill switch” story. Following the arrest warrants for Isreali officials issued by the International Criminal Court, the ICC chief prosecutor lost access to his Microsoft email account as sanctioned by the US government [12]. While there is contention over the details, this has sparked serious discussions about over-reliance on US digital infrastructure by EU authorities.

Concentration risk

Time for a reality check.

We must expect that corporations will maximise shareholder profit. Corporations are supposed to do that.

We must expect that authoritarian governments will maximise their reach. In their frame of mind, that’s rational.

But the over-concentration on three US-owned cloud providers is leaving EU consumers exposed. We see that reflected in growing cloud spend, shrinking control over European data, and an increasing barrier to exit due to vendor lock-in and borderline anti-competitive behaviour [13].

That doesn’t mean that everyone should stop using the Big Three, though. It also doesn’t mean we, in the EU, need to get rid of all US technology services. But I think it would be healthy for European business leaders to ponder:

Do you think this trajectory is headed for better or for worse?

If you think worse, figuring out how to reduce your dependency on the Big Three could be a smart strategy. Let’s explore the alternatives so you can make an informed choice about what to do next.

EU-sovereign public clouds

As we focus on the Big Three, we sometimes forget that European public cloud offerings exist. They generally have less features, fewer datacenters and modest service level objectives. But there are solid, (actually) EU-sovereign clouds - good enough for many use cases.

Hetzner Cloud is is a popular choice for EU and US businesses looking to optimise cloud cost. They’re based out of Germany, and offer IaaS through datacenters in Central Europe, South-East Asia and the US [14]. Managed services for Kubernetes and databases might be coming [15], but for now, you would have to roll your own or add a PaaS layer with something like Coolify [16]. On the positive, Hetzner is cheaper than most competitors by quite a margin [17].

If you’re looking for managed services, you could consider Scaleway. They’re based in France, and their offerings span wider, including serverless [18] and managed services for Kubernetes [19], Postgres, MySQL, MongoDB, Redis [20]. On some parameters, Scaleway is cheaper than AWS, but it depends how you compare them [21]. A limiting factor is that they only have datacenters in the EU [22].

Finally, I’ll mention OVHcloud. Another French cloud provider. Of the options I’ve listed, OVHcloud has the broadest regional coverage with 43 datacenters across Europe, North America and Asia Pacific [23]. Like Scaleway, they provide managed Kubernetes [24], databases [25] and more. OVHcloud is typically cheaper than AWS, but again, it depends [26].

As you can see, the feature set and regional coverage varies by provider, but they have this in common:

• They can probably host your apps (with some migration effort)

• They are European-owned and comply with EU regulations

• They are generally cheaper to use

Something to keep in mind here is vendor lock. EU public cloud providers appear more open than US ones. But if we migrate from the Big Three to one of these, we ultimately swap one proprietary dependency for another. Each business will have to assess if that’s an acceptable risk.

Subscribe now

Your own cloud

You don’t have to be a renter to use cloud technologies. Over the years, open source has been catching up on cloud software. If you can put up the CapEx and deal with the operational complexity, running cloud-native workloads on your own hardware is a viable option.

Enter OpenStack, an open source cloud software project which was co-founded by NASA and RackSpace in 2010 [27]. OpenStack is popular among large European organizations such as CERN, Deutsche Telekom and Banco Santander [28]. It’s modular platform provides what you need for elastic storage, compute, networking and auth.

OpenStack isn’t the only option, either. There’s CloudStack, OpenNebula and proxmox, each with their own characteristics that suit them to different use cases [29].

On the hardware side, we have Oxide. They build specialised cloud computing hardware that ships with cloud platform software, ready to go [30]. Buy Oxide racks and start building.

Though all these approaches have matured significantly over the years, they’re not at feature parity with the US public clouds for managed services. Your engineering team will need to fill in some blanks if you adopt these technologies.

Bare metal

Another way to reduce your dependency on cloud providers is by leaving the cloud altogether. In 2022, 37signals (creators of Basecamp and HEY) made waves by announcing they would do that: exit the cloud and revert to the “old ways” of hosting with owned hardware [31].

Buy servers, rent a rack, put them in it and run your applications on them. If your traffic patterns are predictable and your application architecture isn’t too complex, running on your own hardware offers long-term cost savings and vendor independence. The tradeoff is a higher capital cost and a bigger (or at least different) operational burden.

David Heinemeier Hansson’s chronicle on the 37signals cloud exit is worth a read. It was a bold, but appropriate choice for them. Their application architecture was well-suited [32], they had scale with predictable growth [33], they had $600k to spend on hardware [34] and an ops team that could pull it off [35].

At scale, there is a strong financial case to be made for bringing home your cloud workloads. Sarah Wang and Martin Casado of a16z put it nicely: “You’re crazy if you don’t start in the cloud; you’re crazy if you stay on it” [36].

But how do you decide whether to make the leap?

Weighing the tradeoffs

Choosing a technology platform is a sticky decision with long-term consequences. Each option presents a trade-off: a benefit in one dimension costs you in another.

Here are some factors to weigh against your business constraints.

Short-term cost-effectiveness

How important is minimising initial spend? Pay-as-you-go and managed services of public clouds reduce the barrier to entry when cash is tight.

Long-term cost-effectiveness

Are we optimising for the next five years? As you scale, rental costs add up. Self-hosting requires upfront investment but can lower your total cost of ownership.

Data sovereignty

Who controls your data? A US-owned cloud is subject to the US CLOUD Act. EU-sovereign infrastructure keeps your data under European law.

Vendor independence

How much control do you want to retain? Building on proprietary services creates dependency. Self-hosting or open source solutions offer independence at the cost of operational complexity.

Workload elasticity

Is your traffic predictable or spiky? The Big Three excel at on-demand elasticity for unpredictable workloads. If your growth is stable, other options are more cost-effective.

Availability

What uptime do you actually need? Hyperscalers sell high-availability guarantees, but they’re expensive. Before paying for a 99.99% SLA, weigh the premium against the actual cost of downtime. Is it a crisis or an inconvenience?

Regional coverage

Where are your users? Performance means being physically close to them. The Big Three have a global footprint, while European providers are well-suited to customer bases primarily in the EU.

Putting these into your own business context is important:

A global hypergrowth startup with unpredictable traffic will benefit from hosting with the Big Three.

An non-tech B2B business with European customers will benefit from hosting with EU-sovereign public clouds.

An established tech business where margins matter might benefit from hosting on bare metal.

An EU enterprise that values elasticity with predictable spend could benefit from running their own open-source cloud.

There is no silver bullet.

Conclusion

At the beginning I asked three questions. Let’s see if we can answer them now.

Is hosting EU services in US clouds still a good idea?

For EU businesses, the answer is increasingly no. The current concentration on the Big Three is creating unfavourable market conditions, and the current political landscape poses a risk to European data security.

What are the options for EU-sovereign hosting?

There are several EU-sovereign public cloud offerings to choose from. These are typically the most direct migration path. But as we have seen, self-hosting with bare metal or even your own cloud is also a possibility.

How should an EU business choose its hosting strategy?

It’s a multivariate problem. You must weigh each option based on its fit to your business context and strategy. I highlighted concrete factors to consider, like short/long-term cost, sovereignty, independence, elasticity, availability and regional coverage needs.

As you will have gathered by now, there is a lot of nuance to the EU-US cloud dependency conundrum. I hope this post has provided you with tools for figuring out what you want to do about it.

Daniel Rothmann runs 42futures, where he helps technical leaders validate high-stakes technical decisions through structured software pilots.

Sources

[1] https://www.oreilly.com/radar/the-cloud-in-2021-adoption-continues

[2] https://www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers

[3] Caesar Wu, Rajkumar Buyya, Cloud Data Centers and Cost Modeling (Part 1, Chapter 1)

[4] https://www.cloudzero.com/state-of-cloud-cost/

[5] https://developers.googleblog.com/en/google-cloud-platform-live-blending-iaas-and-paas-moores-law-for-the-cloud/

[6] https://cloud.google.com/storage/pricing-announce

[7] https://aws.amazon.com/blogs/aws/new-aws-public-ipv4-address-charge-public-ip-insights/

[8] https://www.cnbc.com/2024/10/31/amazons-cloud-unit-records-highest-profit-margin-in-at-least-a-decade.html

[9] https://en.wikipedia.org/wiki/Amazon_Web_Services

[10] https://en.wikipedia.org/wiki/CLOUD_Act

[11] https://wire.com/en/blog/cloud-act-eu-data-sovereignty

[12] https://www.politico.eu/article/donald-trump-eu-internet-europe-us-trade-war-data-cyber/

[13] https://www.sdxcentral.com/news/anti-competitive-cloud-practices-threaten-13tn-eu-economic-impact-lobbying-group-claims/

[14] https://docs.hetzner.com/cloud/general/locations/

[15] https://www.reddit.com/r/hetzner/comments/18yhy89/seems_like_we_are_finally_getting_managed/

[16] https://coolify.io

[17] https://dev.to/dkechag/cloud-provider-comparison-2024-vm-performance-price-3h4l#performance-price-on-demand-pay-as-you-go

[18] https://www.scaleway.com/en/serverless/

[19] https://www.scaleway.com/en/kubernetes-kapsule/

[20] https://www.scaleway.com/en/cloud-database-solutions/

[21] https://getdeploying.com/aws-vs-scaleway

[22] https://www.scaleway.com/en/docs/account/reference-content/products-availability/

[23] https://www.ovhcloud.com/en/datacenters-ovhcloud/

[24] https://www.ovhcloud.com/en/public-cloud/kubernetes/

[25] https://www.ovhcloud.com/en/public-cloud/databases/

[26] https://getdeploying.com/aws-vs-ovh

[27] https://openmetal.io/resources/blog/who-is-using-openstack/

[28] https://www.openstack.org/use-cases/#location=europe

[29] https://storpool.com/blog/4-leading-cloud-management-platforms-you-need-to-consider

[30] https://oxide.computer

[31] https://basecamp.com/cloud-exit

[32] https://medium.com/signal-v-noise/the-majestic-monolith-29166d022228

[33] https://world.hey.com/dhh/don-t-be-fooled-by-serverless-776cd730

[34] https://world.hey.com/dhh/we-stand-to-save-7m-over-five-years-from-our-cloud-exit-53996caa

[35] https://world.hey.com/dhh/the-big-cloud-exit-faq-20274010

[36] https://a16z.com/the-cost-of-cloud-a-trillion-dollar-paradox/

Over time, software folks come to appreciate adaptability. Building something once, and having it keep working: elastic, scalable, elegant, serving any which need.

That’s an intuition for the value of platforms and learning. But justifying that intuition in the business context can be difficult.

When we build software, we invest. Time and money spent on this, not the other. So we want to pick good investments. Software with “a good ROI”.

Financial methods can help us value software investments in a way that ties directly business outcomes. But traditional valuation methods underestimate the power of platforms and learning because they can’t capture the value of strategic flexibility and uncertainty management.

In this post, I want to dive into the financial mechanics of software investment to highlight where traditional analysis falls short, and show you how to measure the value of adaptability.

Three types of software investment

Let’s get this out of the way first: A business wants to make money, save money, or do both later. Commercial software exists to serve these goals.

At a macro level, we can think about software investment in three categories:

Features create value now

Build a checkout flow -> people can buy -> number goes up. The path from investment to return is quite direct.

Platforms create future options

Build an SDK -> enable new features -> build some of those features -> number goes up. The value is in the flexibility of future investments.

R&D manages uncertainty

Run a pilot -> learn what works -> make better investments -> number goes up. The value is in discovering good investments.

Most businesses benefit from a healthy mix of these three things.

“Feature factory” gets thrown around like a derogatory term, but actually: Features are solid short-term investments because they make money faster. Making €1 today is better than making €1 tomorrow. Where feature factories go wrong is when they fail to balance that with options in the long term. Going only for quick wins, you’ll miss a better long play.

Part of the problem is that we, in software, struggle to show how platforms and learning create business value. The value of features is easily understood (build this, get that), but the rest is a bit harder to reason about. To do that, we need the right financial tools.

When we’re through, I hope to have shown you that adaptability has value, and with the right inputs, we can put a number to it.

A software portfolio

Consider a (fictional) vacation rental platform. The platform manages bookings for property owners and processes €10M in monthly bookings, earning a 10% commission (€1M monthly, €12M annually).

The team is arguing over product roadmap. They have engineering capacity for a few new projects, but several good ideas are on the table. Each idea promises growth, retention or new opportunities, but resources are limited.

Here’s what they’re considering:

Feature: Instant booking

Let guests book immediately without owner approval, for a higher conversion rate.

Feature: Auto-reply

Pre-written templates for check-in, house rules, and checkout, saving property managers time, improving retention.

Platform: Public booking API

Enable property managers to add booking on their own platforms. Could attract more and larger property managers.

R&D: Dynamic pricing

Test if ML-based pricing can beat simple rules. This could increase revenue, but the feasibility of the idea is unknown.

How should they prioritize the work? We can evaluate the portfolio using progressively strategic financial methods.

A note on inputs

Models are only as good as their inputs. Garbage in = garbage out.

When we look at custom software investment, we are mostly thinking about labor cost and impact to ongoing revenue. In other words, we look at the effort required to build the thing, and the money you could make as a result by selling more of it, keeping users for longer or making it cheaper to deliver.

Software estimation is notoriously hard. Some say we shouldn’t bother trying, and I sometimes agree. We can still use estimates to compare investments, though. Not to predict the final outcome, but to get a sense of which investments are more likely to be good. If the estimates are all wrong, hopefully they’re equally wrong. (Side-note: It’s possible to anticipate estimate uncertainty in the cost model itself.)

Rather than front-loading all these inputs, I’ll try to sprinkle them in as we go along.

Round 1: Return on Investment (ROI)

You know this one. Return on Investment (ROI)measures the efficiency of an investment by looking at the ratio between benefits and costs.

# Benefits: What we expect to earn
benefits = 120

# Costs: What we expect to spend
costs = 100

# ROI: The expected return per unit invested
roi = benefits / costs

> ROI = 1.2

An ROI over 1 is good. To make ROI more fair, it’s common to combine it with Total Cost of Ownership (ROI/TCO), where we consider all the costs we can think of to build and run the software.

Let’s put some numbers to the investment ideas at hand. Bear with me as we work these out - it will be handy later!

Instant booking

Benefits: Based on industry benchmarks, we expect this will lead to a 10% higher conversion rate. If we assume that 20% of properties adopt this feature over time, it could lead to ~€240k in added revenue yearly.

Costs: The booking team estimates 10 weeks to implement, our initial labor costs being ~€100k and ongoing support cost of ~€20k yearly.

ROI (year 1): €240k / €120k = 2.0

Auto-reply

Benefits: Exit surveys show that half of churned property managers cite “too much time spent on guest communication”. With auto-reply for guests, we expect to quickly reduce churn amounting to ~€120k in retained value a year.

Costs: The frontend team estimates 4 weeks to implement, our initial labor costs being ~€60k and ongoing support cost of ~€10k a year.

ROI (year 1): €120k / €70k = 1.7

Booking API

Benefits: A public API could attract larger property management companies who want to embed booking on their own sites. Let’s optimistically assume 10 new big customers, generating ~€380k a year. This is pure speculation.

Costs: The platform team estimates 16 weeks to implement, our initial labor costs being ~€180k and ongoing support cost of ~€30k a year.

ROI (year 1): €380k / €210k = 1.8

Dynamic pricing

Benefits: The pilot itself generates no revenue. It tests whether a dynamic pricing system is even feasible.

Costs: The data team estimates 6 weeks to prototype, our labor cost being ~€50k.

ROI (year 1): €0 / €50k = 0.0

ROI Scoreboard

If we rank the projects by ROI goodness, instant booking comes out as a clear winner, public API coming in 2nd. Dynamic pricing isn’t even worth talking about.

End of story? Well, no!

There are several things wrong here:

• We are not considering timing factors

• Projected benefits for the API are highly speculative

• We’re ignoring the strategic value of learning

Round 2: Net Present Value (NPV)

Let’s say you take a contract to build an app. Would you rather get paid before or after the work is done?

Before, right?

If you get paid earlier, you can spend earlier: to pay your bills, re-invest or go grocery shopping. There’s also the element of risk. If you get paid before, you’ll definitely have the money. If you wait till later, maybe the client won’t pay. That makes intuitive sense.

Net Present Value (NPV) helps us compare investment decisions in terms of what they’re worth to us today, factoring in the time value of money by “discounting” future cash flows. The discount rate is set to match your expectations of risk, interest and opportunity cost. The influence of discount on cash flow grows exponentially with time.

# Cash flow: The net in/out cash flow at each timestep
cash = [-100, 10, 20, 40, 80]

# Discount rate: How to “discount” future cash flows
discount = 0.1

# NPV: The sum of discounted cash flows
npv = 0
for t in range(len(cash)):
    npv += cash[t] / (1 + discount)**t

> NPV = ~10.3

A positive number is good. When we compare investments through the lens of NPV, we are optimising for positive cash flows sooner and negative cash flows later. It also lets us compare investments that play out on different time-scales.

Let’s calculate the NPV for our investment options and see what happens. In these examples I choose a discount rate of 2.5% per quarter, reflecting adoption risk and opportunity cost.

Instant booking

Cash flow: We will be paying for initial labor in the first quarter, then ongoing support. We expect the revenue added to grow slowly thereafter, as users progressively adopt.

NPV (year 1): €17.3k

Auto-reply

Cash flow: Fast 4-week build means we can launch in Q1. Retention benefits kick in quickly, from Q2 onward.

NPV (year 1): €22.8k

Booking API

Cash flow: 16-week build drags into Q2. Customer adoption is slow and uncertain. We assume gradual onboarding starting Q3.

NPV (year 1): €-69.5k

Dynamic pricing

Cash flow: The pilot costs €50k in Q1 but generates no revenue without further investment.

NPV (year 1): €-50k

NPV Scoreboard

Let’s rank the projects by Net Present Value. Now that we account for cash flow timings, priorities have changed!

Auto-reply wins on NPV because it pays back quicker. That’s interesting. The API and R&D projects now look quite dire, both projected at a loss. What we see, is that NPV captures the value of timing but not the value of adaptability.

Round 3: Real Options Analysis (ROA)

Traditional valuation methods assume a static world. But our world isn’t static. When the work is complex and the environment is dynamic, it’s good to be adaptable.

There’s a modern approach that can help us here: Real Options Analysis (ROA), quantifying the value of optionality. To understand how it works, we should start with financial options.

Understanding options

Think of a refundable airline ticket. Assume you paid €100 extra for the ability to cancel. If you get sick or change your plans, you can cancel the ticket and only lose the €100 premium. If you end up using the ticket, you’ll have paid extra for flexibility you didn’t need.

The €100 premium buys you an option: the right to refund the ticket later, when you know more.

Financial options are kind of like that. A financial option is the right (but not obligation) to buy or sell an asset at a pre-agreed price and time. Stock options are one example, often used in tech startups as compensation. Take less salary (that’s the premium), get the option to buy stock at a good price later.

Real options build on this idea. The core insight is that you can think of growth investment opportunities as “call options”. When an investment has options, that adds value, because options let us change our approach in response to new information. We can analyse a business strategy as a series of real options.

Using options in software

Building a platform gives you the option to expandinto new territory later. Running an R&D pilot gives you the option to abandon a big project if its outcome isn’t good. These options have value because they let you invest more when things are looking good and cut your losses when they’re not.

# The cost of learning
learning = 30
# The probability of success (40% chance)
chance = 0.4
# The expected benefits (if the project succeeds)
benefits = 200
# The costs to implement the project
costs = 100

# With option: weighting our success and failure branches
branch_success = benefits - costs
branch_failure = 0
expected_with_option = (-learning
                        + chance * branch_success
                        + (1 - chance) * branch_failure)

# Without option: commit costs regardless of success
expected_no_option = (-costs
                      + chance * benefits)

> With learning: €10k
> Without learning: €-20k

The code above demonstrates the value of the option to abandon. It may seem counterintuitive: learning adds €30k in upfront cost. But without it, you commit €100k regardless of outcome, losing that investment 60% of the time. With learning, you only invest the €100k when the learning validates the idea (40% of the time). The option to abandon the project before full investment creates €30k of expected value.

More broadly, we’re happy to see investments that have:

• The option to expand (let’s do more!)

• The option to contract (let’s do less…)

• The option to switch (let’s do something else instead?)

• The option to defer (let’s wait and see…)

• The option to abandon (let’s drop it and sell for parts.)

The total value of an investment can then be viewed as the value of its options weighted by the probability that we’ll use them.

There are many ways to model real options, binomial trees being quite common. Think decision trees. The underlying mechanics of these models are a bit complicated, which is probably why it has taken some time for real options to gain traction in corporate finance.

Let’s apply it to our software portfolio. If we call the “static NPV” we calculated earlier sNPV, let’s now determine the “expanded NPV” with real options included and call that eNPV. We’ll focus on the API and R&D since these are investments that have options

Booking API

If we see strong enterprise adoption in Q3, we realize that we have the option to expand by investing an additional €80k in Q3 to build premium enterprise features. This will justify a high-value enterprise tier that could generate €400k in additional Q4 revenue.

In the diagram below, we’re looking at two possible futures. In the first, we build the API and see weak adoption: this gives us the original NPV of €-69.5k. In the second, we see strong adoption and capitalize on that by expanding with premium features, reaping a new NPV of €116.4k driven by enterprise license sales.

Weighing these possible futures by their probabilities, the eNPV flips the API from a loss to a gain through the ability to scale based on market signals.

eNPV (year 1): €23.4k

Dynamic pricing

Finally (!) we can explain the value of the dynamic pricing pilot. It creates an option to abandon before a bigger investment.

We spend €50k in Q1 to see if it works. If the pilot fails (70% risk), we abandon and lose only €50k. If it succeeds (30% chance), we invest €200k in Q2-Q3 to build the full thing, expecting to generate €450k in Q4 from dynamic pricing benefits.

Without a pilot, we would either commit €200k on blind faith (risking a total loss) or miss the opportunity altogether. In other words, the pilot limits the downside while preserving potential upside.

eNPV (year 1): €17.5k

eNPV Scoreboard

Now let’s see how accounting for real options changes our rankings:

Our original ROI winner has moved to the bottom because it doesn’t make money quickly and it doesn’t have strategic options. The API is at the top, because we realized the potential of introducing an enterprise pricing tier.

Making the call

What is the rational next step for our vacation rental platform? As with most things in software, it depends.

If we believe and agree on the assumptions presented, we should build the API first because of its strategic potential. The current revenue looks like it could support that. If the focus is instead on short-term survival, we should probably build the auto-reply feature first.

Are these valuations objective and true? Not really!There’s a bunch of assumptions, expectations and guesses involved here. But as we worked our way up the ladder of strategic valuation methods, we became increasingly explicit about the stakes involved.

At the end of the day, we make our best educated guess. But thinking in NPV and real options, we ground our decision-making in structured thinking about uncertainty, timing and adaptation.

Conclusion

I hope you found this helpful (and not boring). Figuring out how to value new software initiatives is a bit of work, but I invite you to try it.

My hope is that you come away with a clearer sense of the financial value of adaptability. Traditional ROI and NPV undervalue platforms and R&D because they don’t capture optionality. Real options analysis gives you the tools to quantify what engineers intuitively know: building for adaptability has real, measurable value.

Knowing this won’t eliminate uncertainty in decision-making. But it will make your assumptions explicit, tradeoffs visible, and priorities defensible.

Whether you’re advocating for piloting a new idea or weighing features against platforms, you now have financial frameworks that match the complexity of the decisions you’re actually making.

Daniel Rothmann runs 42futures, where he helps technical leaders validate high-stakes technical decisions through structured software pilots.

Sources

Here are my primary sources for this post. They have lots more great insights, and this write-up would’ve been way too long if I tried to include them all. I can very much recommend checking them out if this topic interests you!

• Kent Beck: Tidy First?

• Kent Beck: The Good News Factory

• Caesar Wu, Rajkumar Buyya: Cloud Data Centers and Cost Modeling

• Timothy A. Luehrman (in HBR): Strategy as a Portfolio of Real Options

The demos look cool. I’ve found LLMs helpful in some scenarios, but is “vibe coding” really the future of software engineering, as some AI enthusiasts claim?

Let’s find out. I’ll vibe code an app from scratch as a novice and then review the code as an experienced software engineer.

Weapon of choice

Some of the tools I’ve heard about in this context are Cursor, Claude Code and Lovable. Initially, I was excited to try out Cursor, but unfortunately, Anysphere did not believe I was a human.

So, I decided to try Claude Code instead. I’ve enjoyed using Claude for search, docs and test stubs in the past, so it seemed a decent choice.

I was ready to go after a quick registration process with Anthropic and upgrading my NodeJS installation.

Mission statement

Vibe coding aims to make software development accessible to non-programmers. I tested this by acting as a beginner, giving minimal technical input and letting the AI guide the process.

Still, I wanted a slightly more advanced use case to review the capabilities better later.

The prompt:

Build a progressive web app with C# that allows users to create and share playlists. Users should be able to subscribe to other users playlist for a custom monthly fee.

The first thing that happened was impressive: Claude ran commands to create and design a scaffold for a Blazor frontend and API, including reasonable data models, DTOs, database integration and a Swagger UI.

The next thing that happened was more frustrating: It didn’t work. When registering my user via /api/Auth/register, I got a 403 Forbidden error. Naturally, I let Claude know about this, which launched a 45-minute process of building, rebuilding and testing the project.

I could register my user after disabling HTTPS redirection and vibe debugging some weird .NET compatibility issues.

At this point, I am still running on complete auto-pilot. I have no idea how the code works, but it does appear to work.

I want a front-end, too, so I ask for it. Halfway through building it, Claude exceeds its context window and restarts the conversation, passing a summary to itself.

I ask how to test the frontend and get a satisfactory response. Now, I can see a webpage, although it does not integrate with the API in any way. We have:

• A home page with a menu, a blue box and some text

• An “About” button that links Microsoft’s ASP.NET docs

• A popup about an unhandled error

• A Playlists page which says: Sorry, there's nothing at this address.

I then asked for the front end to be expanded with capabilities from the API, after which Claude realised the entire project was wrongly structured and needed to be reorganised to compile.

Now, both the API and the frontend build again, but uh-oh:

Naturally, I asked for help to get this fixed. Claude was happy to oblige. And the webpage now loads. Except there is now nothing here to do with playlists.

Claude has populated the page with “hello world” placeholders, a counter, a link to a Blazor survey and a broken weather forecast. The unhandled error is still unhandled.

I tried to request a proper integration, but now the builds fail. After another hour of vibe debugging, we now have more correct-looking content, but the user registration doesn’t work again.

Oh my god. That’s it. I’ve spent 4 hours and $20 Claude-bucks - I’m done.

I’m tired and frustrated with acting as QA for a bot. Claude Code occasionally makes heroic progress and then chases its tail for hours on random trivial issues.

Every prompt feels like one step forward, two steps back. I made some fast initial progress, but now I pay the price with a sluggish, frustrating debugging flow.

I struggle to fix issues without investing more effort into reading and understanding Claude's code. And if I can do that, I might as well write the code myself.

Code review

Up until this point, I’ve been acting as a novice. By “vibe coding” for 4 hours, I could spin up an API with authentication and a frontend that is nearly functional.

Frustrations aside, this is a neat tool that anyone can use to make working programs. But is it more than a toy? Can these programs be used in production? Let’s review.

⚠️ Request changes: run.sh doesn’t do anything

This is a small but silly one. run.sh doesn’t do anything but echo to your console that you should go ahead and use dotnet run. The frontend port note is wrong.

⚠️ Request changes: Duplicated projects

While reorganising the project, Claude has created not one, not two, but three duplicate API folders. As far as I can tell, only PlaylistPro/ApiProject is in use. While it doesn’t break anything, it’s a mess (!) and will be hard to maintain.

⚠️ Request changes: Weather forecast

Why do the API and client apps implement a weather forecast? This was not part of the requirements or even tangentially related to playlists. The result returned by the weather forecast is random and arbitrary.

🟥 Decline: Mocked auth implementation

The first auth implementation was not too bad (I reviewed it early). However, after the confusion with structure and integration, Claude has opted for a mocked auth implementation that lets everyone in. This is not acceptable for a public application.

Declined: Do not merge

As a reviewer, I would not accept a pull request like this into my repo and would certainly not release it to end-users.

I have been impressed with the tool's capabilities, though. Maybe an innovation that’ll make these tools “the future of software engineering” is around the corner. I doubt it, but I am open to being wrong.

“Vibe coding” is making software engineering available to a new group of people. It makes prototypes and “personal software” cheaper to produce. But I don’t think it’ll displace the traditional software engineering industry in its current form.

Instead, I’m guessing we are looking at a Jevons Paradox situation, where lowering the cost of creating simple software will lead to more simple software being created.

Personally, I found “vibe coding” a frustrating experience, and I felt less productive. Someone without coding experience might find learning the tech equally a barrier to building what they want. Each to their own.

You can “vibe your way” to decent prototypes for apps where quality and security are unimportant. But please, don’t deploy these apps to production without serious scrutiny. Otherwise, you may need to prepare for some “vibe firefighting” as well.

Daniel Rothmann runs 42futures, where he helps technical leaders validate high-stakes technical decisions through structured software pilots.

Software engineers sometimes need to be like Elsa—minus the ice magic, of course. Finding solutions to novel problems requires that we brave the fog of uncertainty. We need to get things done with incomplete knowledge. But when does strategic ignorance cross the line into reckless risk-taking?

In this post, I explore systematic approaches to handling uncertainty in software projects.

A theory of knowledge

Engineering decisions require balancing what we know against what we don't. If perfect knowledge were possible, we'd always implement the optimal solution. But reality is messy.

Categorising knowledge isn't an academic exercise—it's a tool for deciding where to invest our limited research time and when to start building. With this in mind, we can categorise information into three groups:

1. Known: Facts and constraints we're confident about.

2. Knowable: Information we could discover with some effort.

3. Unknowable: Uncertainties that cannot be resolved in advance.

The first two categories are straightforward—what we know and what we could learn. The third category is trickier: things we can't reasonably know until we're in the weeds of building the software.

In history, “unknowables” are what Nassim Nicholas Taleb would call “Black Swan” events. In his research, Taleb demonstrates how our planning consistently underestimates unknowable events—not because we're careless, but because they are outside our predictive capabilities.

In tech, as in most domains, some things are unknowable—like how markets and cultures will evolve in relation to what we're building. In the structured world of software, many things are theoretically knowable. Practically though, some information requires so much effort to obtain that it is very nearly unknowable. It’s a continuum ranging from easy-to-know to practically unknowable.

When venturing into something new, whether it's a greenfield project or major changes to existing systems, it can be helpful to map the information that matters to us most (knowable or not) so we can increase our chances of building the right thing in the right way.

The knowledge quadrant

Every project begins with decisions that require learning. How else can we make informed choices in a complex, vast universe if we’re not first obtaining knowledge?

To keep things practical, I use a 2x2 matrix to prioritise what I need to learn:

It’s like Eisenhower’s classic “Urgent-Important” matrix, except I’m not concerned with a temporal dimension, but instead trying to prioritise learning according to impact and effort.

When making decisions under uncertainty, it can help to create a short-list of “things you’d like to know” and organise it to maximise certainty with minimal effort. At the end of the day, we gotta ship products.

You can use the knowledge quadrant to do that:

Important + Easy

These are no-brainers. Do quick research, build a prototype, resolve the uncertainty.

Consider payment processing in a web app—deciding whether to build your own system or use an established provider like Stripe. Basic research into security requirements, PCI compliance, and implementation time will quickly reveal that using an existing payment provider is a good idea. This high-impact decision can be made with a small investment in upfront research.

Important + Hard

This is where agile development methods shine. Start working, target the uncertainty, and create feedback loops to learn as you build.

When designing a real-time collaboration feature (like multiple users editing the same document), the exact conflict resolution strategy that will provide the best user experience isn't something that’s easy to find out. Interactions between concurrency, latency, and user expectations make that complicated. An answer may theoretically be knowable, but probably not without significant effort. Instead of trying to design a perfect system upfront, start small, test it with real people, and improve it based on feedback and your own observations.

Not Important + Easy

Make assumptions, pick something and move on. If it's not important, it doesn't matter if you're wrong.

When choosing a logging library or deciding on code formatting standards, just pick something reasonable based on quick research or personal preference. These decisions rarely make or break a project.

Not Important + Hard

Don't waste precious engineering time here.

Like trying to predict exactly how your application might need to scale three years from now if you succeed beyond all expectations. Instead of over-engineering for hypothetical future scale, build a solid architecture that handles current needs well with reasonable extension points, knowing you might change parts of the system as you learn more.

The solution landscape

Think of your journey into the unknown as navigating an unfamiliar mountainous landscape. Your goal is to plant a flag on the highest mountain but you don’t have a map.

You have two activities at your disposal:

1. Scouting the terrain (exploration): Climbing nearby high points to see what’s ahead, sending small parties in different directions to discover paths, obstacles and resources. This is your research, prototypes and user interviews.

2. Traveling the path (exploitation): Committing to a direction and making efficient progress along it. This is your focused development, optimisation, and refinement work.

Both activities are important, but they compete for your effort. Every day spent scouting is a day not traveled towards a destination. Every day traveling without terrain awareness could lead you down a wrong path, forcing you to backtrack.

The landscape metaphor isn't just helpful imagery—it connects to optimisation theory in computer science. When building software, we're solving what mathematicians would call an "optimisation problem with incomplete information"—finding the best solution without seeing the whole map. A journey through this imaginary landscape reveals relevant optimisation constraints:

• Local / global optima: You might climb the nearest hill (local optimum) not realising a much taller mountain (global optimum) was just outside your line of sight. This is the same challenge that optimisation algorithms like gradient descent face—getting stuck on a good solution while missing the best one.

• Gradient locality: The further ahead you look, the less accurate your predictions become. Plans made for distant terrain will be inaccurate. In optimisation theory, this mirrors how gradients provide only local information—they tell you which direction to climb at your current position, but can't reliably predict the landscape beyond your immediate vicinity.

• Path dependence: The cost of changing direction increases dramatically the further you travel along a given path. Optimisation theorists would recognise this as a form of "hysteresis", where your history constrains your future options.

• Unknown unknowns: No matter how thoroughly you scout, some features of the terrain only reveal themselves when you encounter them directly. This connects to the "No Free Lunch Theorem" in optimisation, which tells us that no strategy can discover all properties of an unknown space without direct sampling.

Software engineers often face this dilemma: do we invest time exploring potential solutions (scouting) or commit to a path and start building (traveling)? This challenge is particularly complex because software landscapes, unlike physical terrain, exist in multiple dimensions, evolve continuously, and shift in response to our own actions.

Lost in the woods

I've seen it happen before (and done it myself, too): teams get stuck in analysis paralysis, debating theoretical concerns while competitors ship products. Analysis paralysis is like spending all your time on the nearest vantage point, mapping every possible route but never actually traveling. The quest for perfect knowledge is understandable—but it's ultimately self-defeating.

Behavioural economists have studied this fact. In their research on decision-making under uncertainty, Daniel Kahneman and Amos Tversky identified "ambiguity aversion"—our irrational preference for known risks over unknown ones, even when they're mathematically equivalent. Ambiguity aversion makes us prefer well-mapped routes with known obstacles over potentially shorter but uncharted paths.

I’ve also seen the opposite: teams charging ahead with blind confidence, only to build something nobody wants or that collapses under real-world conditions.

Kahneman and Tversky identifies this cognitive bias as well—"planning fallacy"—where we systematically underestimate complexities and overestimate our understanding. The planning fallacy has us underestimating how difficult certain terrain will be to traverse, even when similar journeys have taken longer in the past.

As such, the magic is in the middle—enough exploration to set a clear direction, followed by decisive action with built-in feedback.

Conclusion

Uncertainty isn't a bug in software engineering—it's a feature. It's what makes our field challenging, dynamic, and fun. Mastery in software isn't about memorising APIs or frameworks (hello LLMs). It's about developing judgment under uncertainty. At least, that’s part of it.

I've provided a few ideas to help you tackle uncertainties more systematically. The rest is up to you—mastering this skill requires artistic intuition and scientific rigor. Though I’m far from perfecting this myself, I'm committed to developing this skill because I think it's essential for successful software projects.

Like Elsa venturing into the unknown, we must move forward with courage, humility, and a willingness to adapt as we learn. The path reveals itself to those in motion.

Daniel Rothmann runs 42futures, where he helps technical leaders validate high-stakes technical decisions through structured software pilots.

Here we are, a software company, with nearly all of our operations running in interconnected digital systems, yet the core of our business—our policies, procedures, and organisational structure—is a basic collection of documents.

It just felt ironic. We use advanced tools to automate compliance checks, store our code in version-controlled repositories, and manage our infrastructure as code. However, when describing and managing our company, we resort to digital paper and tidbits of info distributed across people in the building.

The disconnect became increasingly apparent as I reflected on our day-to-day process: 90% of our products, documents, communication, and decision-making live in digital channels. That’s data. It lives in the cloud, spread over SaaS solutions that specialise in handling individual work processes—all systems with robust APIs and programmatic access.

At the centre of it all sits a lonely island of documents: our ambitions, goals, policies and formal structures. And I think those are pretty important.

Our security posture was solid before we even considered ISO 27001 because we’d already worked hard to comply with our customer’s requirements. Between collecting evidence for controls, arguing about and updating policy wording, document review, and the actual audit, we spent hundreds of additional person-hours that could’ve otherwise been spent creating great products for our users.

A missing link

If we desire operational data to be so rich, why do we accept organisational data to be so sparse? We’ve revolutionised how we handle infrastructure with Infrastructure as Code (IaC), how we manage deployments with GitOps, and how we handle security with Policy as Code.

We see the benefit.

But when representing our organisation (the beating heart of our operations), we apply old-school methods.

Imagine if we could represent our entire organisational structure programmatically instead—not a static picture, but a living, breathing digital representation of our company that can be versioned, queried, tested, and automatically verified. A system where policy changes could be tracked as code changes, where compliance could be continuously monitored, and where the relationships between people, processes and technology could be explicitly mapped and understood.

Thinking bigger

Existing solutions like HRIS systems manage people data but struggle with policy relationships. GRC tools track compliance but rarely connect meaningfully to organisational structure. I’m proposing we think bigger. It’s about creating a holistic, programmatic representation of the entire organisation: a “company manifest” that serves as a single source of truth for organisational structure, policy, and operations.

Consider how helpful this might be for:

• Compliance audits: Instead of manually piecing together evidence from various systems, auditors could query the company manifest directly, with clear traceability between policies and their implementations.

• Policy changes: Updates could be version-controlled, and automated impact analysis could show which teams and processes would be affected.

• Organisational design: Leaders could model structural changes in a “staging environment” before implementing them, gaining a better understanding of how changes cause ripple effects throughout the company.

Why isn’t this a thing?

As this idea swirls around in my mind, I find more questions than answers.

Has anyone tried this yet? If not, why not?

Is it because organisations are inherently too complex and dynamic to be represented as code? If so, that seems at odds with regulation and standards, where we expect corporate activities to be so uniform and procedural that we can reliably stamp them as compliant, non-compliant, legal, or illegal.

Is it because we haven’t yet found the right abstraction level—the equivalent of what Infrastructure as Code did for system administration?

The tools and concepts exist: Graph databases for representing organisational relationships, domain-specific languages for describing business rules, and API-first architectures for integration.

Maybe what’s missing is a framework to bring these ideas together in a way that’s powerful enough to be useful and simple enough to be used.

Putting it into practice

Let’s think about how this could work.

In this section, I’ll fantasise about what I would want from “Company as Code.” Then, I’ll map that to some system components that could address those needs.

As an engineer and business stakeholder, I would want a company model to be:

Queryable

The system must trace relationships between people, policies, and systems—similar to code dependency tracking. Users should easily see the organisation from different angles, such as which people are affected by a policy and who owns it.

Versionable

Explicit tracking of organisational changes, including who made them, what changed, and why. This is crucial for audits and understanding organisational evolution.

Integrated

Seamless data exchange with existing tools (Azure, Slack, etc.) to maintain an up-to-date organisational picture and enforce tool configurations based on policy.

Testable

A “staging environment” where organisational changes can be modeled before implementation, supporting automated tests for individual rules and controls.

Accessible

Though powered by code, the interface should be intuitive enough for non-technical leaders to use effectively.

Bringing this vision to life requires several puzzle pieces. Each component must address specific requirements to approximate something that can be powerful while being relatively simple to use.

A declarative language for organisations

Drawing inspiration from Infrastructure as Code tools like Terraform, let’s imagine a declarative Domain Specific Language (DSL) that reads naturally while expressing a formal structure.

The basic syntax follows a clear pattern:

EntityType "Identifier" {
    References = AnotherEntity.Identifier
    Attribute = Value
    ListAttribute = [
        "Item One",
        "Item Two"
    ]
}

A type, unique identifier, and set of attributes define each entity. Entities can reference each other using dot notation, creating a web of relationships that forms our organisational graph.

Specifying an organisation

Let's walk through how you could define a small engineering team in this language:

1. First, let’s define the roles that exist in the organisation:

Role "SoftwareEngineer" {
    Responsibilities = [
        "Write clean, maintainable code",
        "Participate in code reviews",
        "Document technical decisions"
    ]
}

Role "EngineeringManager" {
    Responsibilities = [
        "Provide technical leadership",
        "Conduct performance reviews",
        "Manage team resources"
    ]
}

2. Next, we’ll create an organisational unit for our team:

OrganisationalUnit "EngineeringTeam" {
    Department = "Engineering"
    CostCenter = "ENG-001"
}

3. With these structures in place, we can define the actual people and their relationships:

Person "AliceSmith" {
    FullName = "Alice Smith"
    Role = Role.EngineeringManager
    Unit = OrganisationalUnit.EngineeringTeam
    Email = "alice.smith@company.com"
}

Person "BobJohnson" {
    FullName = "Bob Johnson"
    Role = Role.SoftwareEngineer
    Unit = OrganisationalUnit.EngineeringTeam
    Manager = Person.AliceSmith
    Email = "bob.johnson@company.com"
}

4. Policy definitions create a framework for compliance:

PolicyGroup "SecurityPolicies" {
    Owner = Person.AliceSmith
}

PolicyRule "MFARequired" {
    Group = PolicyGroup.SecurityPolicies
    Enforcement = "Mandatory"
    ComplianceLevel = "Critical"
}

5. Finally, we can map these policies to external requirements:

ExternalRequirement "ISO27001_A9_4_1" {
    Standard = "ISO 27001:2013"
    Control = "A.9.4.1"
    ComplianceLevel = "Mandatory"
}

ComplianceMapping "MFACompliance" {
    Requirement = ExternalRequirement.ISO27001_A9_4_1
    ImplementingPolicies = [PolicyRule.MFARequired]
}

This declarative approach allows us to build a complete picture of our organisation, from high-level structures to individual policies and their regulatory implications. Each definition is clear and self-documenting, while the references between entities create a graph of relationships that can be analysed, validated, and used to automate organisational processes.

With a representation like this, we benefit from organising definitions into logical files and directories, treating organisational changes like code changes: versioned, reviewed, and validated before application.

This enables practices like reviewing changes through pull requests, testing policy modifications before rollout, and automatically generating compliance documentation while tracking the evolution of organisational structures over time.

Building a model

While Infrastructure as Code tools like Terraform work with directed acyclic graphs (DAGs) to determine deployment order, an organisation's structure is inherently more interconnected. People manage other people, policies reference activities that refer back to policies, and teams have complex interdependencies. This calls for an undirected cyclic graph model to represent these rich relationships.

public record Node(
    string Id,          // e.g. "Person.AliceSmith"
    string Type,        // e.g. "Person", "PolicyRule"
    List<Edge> Relations = null
);

public record Edge(
    string FromId, 
    string ToId,
    string RelationType, // e.g. "ManagedBy"
);

public class CompanyGraph
{
    private Dictionary<string, Node> _nodes = new();
    private List<Edge> _edges = new();

    public void AddNode(string id, string type) =>
        _nodes[id] = new Node(id, type);

    public void AddRelation(string fromId, string toId, string type)
    {
        var edge = new Edge(fromId, toId, type);
        _edges.Add(edge);
        (_nodes[fromId].Relations ??= new()).Add(edge);
        (_nodes[toId].Relations ??= new()).Add(edge);
    }

    // Example: Find all requirements impacted by changing a policy
    public IEnumerable<Node> GetImpactedRequirements(string policyId) =>
        _nodes[policyId].Relations
            .Where(e => e.RelationType == "ImplementsRequirement")
            .Select(e => _nodes[e.FromId == policyId ? e.ToId : e.FromId])
            .Where(req => req.Type == "ExternalRequirement");
}

The example above shows a naive implementation of a queryable graph structure based on a declarative DSL. This representation makes it easier to answer questions requiring multiple references in the structure, such as "Which external requirements would be affected if we changed our MFA policy?"

Bridging models and reality

A DSL can define an organisation's structure and rules, but the model must be connected to real-world data to be useful. This requires a storage strategy that can:

• Persist the organisational graph itself

• Store data associated with graph entities (like evidence)

• Enable validation of considered changes

A way to deliver this at scale would be to combine several data stores: A graph database for the organisational model, a relational database for associated data, and an event store for audit trail and change tracking. If you wanted to start small, you could serialise all this information to your local filesystem instead.

To activate external data, we would need an integration framework with a plug-in architecture to bridge the organisational model and production systems. This would handle three key aspects: (1) data collection from integrated systems like GitHub or Azure, mapping evidence to nodes in our graph; (2) policy validation that programmatically checks collected evidence against rules; and (3) policy enforcement where organisational changes automatically trigger updates across connected systems—from employee provisioning to access permission changes.

The DSL would supply this functionality using built-in modules or user-supplied plug-ins to perform the work needed to interact with other systems. Let’s imagine a Control entity in the DSL which uses a custom script to perform routine checks on MFA usage and links results to the compliance mapping:

Control "MFAMonitoring" {
    Implements = ComplianceMapping.MFACompliance
    
    Verify {
        Script = "Security/mfa-checks.js"
        Methods = [
            "allUsersHaveMfaEnabled"
        ]
        Frequency = "Daily"
    }
}

Then in security/mfa-checks.js:

export async function allUsersHaveMfaEnabled() {
    const users = await myAuthClient.listUsers()
    return users.every(user => user.mfaEnabled)
}

The solution should be open and extendable to maximise its usefulness, allowing for custom integrations, validations, and automation. Terraform does a good job of this with its “Providers” plug-in architecture, although integrations to inspect and modify data in corporate systems might need to support more custom scripting.

For organisations wanting to test this approach, you could start small:

1. Model just your organisational structure and reporting lines.

2. Add key policies and compliance mappings.

3. Begin connecting to your most critical systems.

Making it shine

A code-based declaration of an organisation offers powerful capabilities for technologists, but we also need to acknowledge that many business stakeholders don’t “think in code”.

The elegance of a DSL doesn’t have to be limited to those most comfortable with text editors and version control software. A genuinely accessible “Company as Code” solution must connect a programmatic representation and the business users who make daily decisions about organisation structure, policy and compliance.

This could be solved with a low-code / no-code interface where:

• Business leaders drag and drop organisational entities and reporting structures while the system generates code declarations behind the scenes.

• Compliance officers use simple forms to define policies and instantly visualise which parts of the business they affect. When regulations change, they can quickly identify gaps or conflicts through visual highlighting.

• Technologists keep the codebase organised and provide data integrations and tools to implement policy in external systems.

With an approach like this, we can maintain the rigour of the code model while making it accessible to everyone. Changes through the interface update the underlying code, preserving that single source of truth that can be versioned and validated.

Final notes

Beyond the technical feasibility, the real promise lies in the organisational benefits: faster audits, clearer decision-making, and better understanding of changes' impact before implementing them. Hundreds of hours spent on compliance documentation could instead be invested in creating value.

In this post, I hoped to establish that a codification of organisational structure is missing and that it’s buildable. Practical? Don’t know. Viable? I don’t have the answer. Buildable, though? Yes. I believe so.

Let me know what you think about it.

Daniel Rothmann runs 42futures, where he helps technical leaders validate high-stakes technical decisions through structured software pilots.

In this article, I'll introduce Corporate Agile as a way to label dark patterns that drive the rising negativity about agile working methods. As a path forward, I will make the case for a return to principles with Basic Agile.

The Heart of Agile

I welcome a change of attitude - too many Agile implementations seem to create significant worker frustration and corporate overhead. It inevitably leads to debates about whether insert-your-favourite-flavour-of-agile is inherently broken or poorly implemented.

Nevertheless, I don't think the agile principles are wrong. An agile mindset is fantastic when working under complexity and uncertainty. In circumstances like these, a plan-driven approach isn't helpful because the outcomes of the process are too hard to predict.

Consider questions like these, common-place in start-ups and software shops:

• What is the best way to do something youve never done before?

• What do customers want, what do they need, and what will they pay for?

• How could markets and funding impact our capacity to do the thing?

• What might competitors do, and how should we react?

As workers in tech, we participate in hard-to-predict Complex Systems because people, markets and new inventions are involved. How can we thrive in such conditions? Well, in an agile spirit, we make headway by:

• Collaborating closely and communicating often to come up with good starting ideas.

• Working in small increments, doing the important parts first to test our ideas against reality.

• Continually reflecting and improving from our experiences, changing course when needed.

Dead simple. Thats just my rough summary of Heart of Agile, by the way - a response to the overly complex state of the Agile industry launched by Alastair Cockburn, an agile-manifesto-original.

Theres no need to evangelise or make a religion of it: Working Agile boils down to prioritising adaptability over predictability. We minimise risk by reacting quickly to unforeseen troubles, and we maximise impact by reprioritising based on the opportunities we see. In times when the future is hard to predict, an agile way of working is remarkably efficient.

Corporate Agile

The straightforward and practical mantra of Agile is getting lost in complicated processes, pedantry and productivity theatre.

Reputable Agile voices have highlighted this problem before: Agile, as implemented in many organisations, is bloated, inefficient and, frankly, missing the point.

• Martin Fowler calls it faux-agile, perpetuated by the agile industrial complex.

• Ron Jeffries talks about Dark Scrum.

• Andy Hunt said, "Agile now means we do half of Scrum poorly and use Jira".

Some even advocate for leaving the Agile label behind to escape its widespread misimpression. I think that would be throwing the baby out with the bathwater, but that's just me. At least, we should continue to shed light on the dysfunction and figure out how to avoid it. I'll call it Corporate Agile.

Corporate Agile attempts to prioritise adaptation AND predictability, not realising that you must prefer one over the other. It maintains a strict hierarchical decision-making process. It assumes general managers are qualified to decide how specialists should do their work.

Corporate Agile reckons splitting work into arbitrary deadlines (like sprints) improves meaningful outcomes. It believes estimates are accurate and uses them to plan delivery timelines. It uses prescriptive metrics like velocity and burndown, prompting teams to do work that fits in the box, regardless of whether its valuable.

Corporate Agile needs coaches, consultants and middle managers in abundance to keep workers in line. It supposes that a 2-day course prepares someone without technical or domain expertise to run the process. It does not show concern that all involved attend lengthy mandatory meetings to conform to the process. It does not admit to the astounding amount of time and money wasted, which couldve been spent working on tangible outcomes for real customers.

In summary, Corporate Agile practices emerge when businesses attempt to implement Agile as a "canned process" while continuing to apply traditional project management thinking to the process, unwilling to pass control to the workers.

The price of planning

To get an indication of the price we pay to do Corporate Agile, lets review the time spent to perform a typical process. Ill take a Scrum team as an example, making a few simplifications to make measures easy to follow.

Our hypothetical team consists of 7 Developers doing 1-week sprints. They have four team meetings each sprint: Refinement, Planning, Retrospective and Review. Well assume each meeting takes one hour, totalling four hours a week per person.

That's 28 person-hours spent each week doing Scrum instead of doing work that directly benefits customers, and were not even counting the Daily. Now add the overhead of a professional scrum master, dedicated product owner, and layers of management between the team and its real stakeholders.

I think it's fair to say this team uses the equivalent of a full-time role (or more!) in meetings and management to keep the gears turning. Think about that. What did they gain? In my experience, efforts toward backlog grooming, task refinement, and sprint planning rarely yield noticeable benefits except to make work fit in a box.

For those currently in Scrum teams, ask yourself which would make your products more awesome: These meetings? Another engineer, designer, artist or domain expert? Budget for tools, services or runway? A few hours to relax and recharge?

Of course, coordinating a group of co-workers costs something. My point here is that Corporate Agile encourages wasteful behaviour - large groups of clever people sitting around in lengthy sessions to coordinate, align, and plan work that is likely to change before it's even ready to be started. These meetings are typically run by managers, and only a few participants contribute. We can do better than that.

The blame game

Techies often point to the business as the perpetrator when an organisation devolves into Corporate Agile. But I've also noticed that some technical types are quick to forfeit responsibility, like taking the role of a code monkey working on tickets and waiting for payday. Compared to speaking up, taking ownership and collaborating intently, working tickets is a simple and comfortable life (but only half the fun).

Zooming out, I do think multiple groups contribute to the problem:

• Leaders keen to exert control and influence may disempower the teams, denying their ability to reflect and adapt.

• Managers who seek self-preservation may unnecessarily insert themselves into the work, making processes complicated and bureaucratic.

• Consultants and trainers who hunger for sales may tell corporate clients what they want to hear: Sure, Agile methods are compatible with a top-down, plan-driven approach. It's called SAFe. Yes, you can have your cake and eat it, too!

• Workers who choose the path of least resistance may not speak up about barriers to work that they face for fear of conflict or apprehension to change.

• Stakeholders and customers who feel unsettled by uncertainty may push toward fixed-scope solutions that require a more plan-driven approach.

Yeah, I'm firing shots at everybody here. If you agree with me, you can see that a goal of being agile concerns the entire organisational stack, not just a single group.

Every individual is unique. Some struggle without the predictability of a plan-driven approach, while others thrive. It's important to be empathetic to that, understanding that agility does not come naturally to all.

Basic Agile

If Corporate Agile is so bad, how can we do better?

I think we need to strip away the broken processes and go back to basics. I'll call it Basic Agile, but it has many names: (lowercase) agile, Heart of Agile, the Agile principles and being agile.

Basic Agile is about returning to the principles of agility, allowing teams to self-organise to be efficient and under uncertainty. It's about rejecting the complicated processes that spawn from corporate bureaucracy. It's not something you can easily buy or roll out as a one-size-fits-all ruleset. It's about adopting a no-nonsense way of thinking about work that lowers the barriers to prioritising things that make customer's lives better.

Basic Agile empowers those closest to the work to decide how the job gets done. It flips the control flow of the organisation from top-down to bottom-up: Managers support the teams to do their best work rather than telling them what to do. That leaves less to manage, and naturally, the management layer shrinks as a result.

Basic Agile is about embracing uncertainty. It's being humble to the fact that we simply don't know enough for big plans to be realistic. It's about climbing a mountain one step at a time, finding the route as we go, and making our way to the peak at a sustainable pace.

Basic Agile has certain prerequisites. You need:

• Strong teams that can get great work done without intervention.

• Managers who support their teams without controlling or sheltering them.

• Leaders who can set a goal without dictating how it should be achieved.

• Contracts that are flexible in scope. If youre working under fixed-price-fixed-scope terms, agility is impossible.

• A readiness among all to face uncertainty, make changes and figure things out along the way.

If you can't satisfy these requirements, you may be better off dropping Agile entirely and embracing a plan-based approach like Waterfall despite its well-known disadvantages.

How do teams practice Basic Agile? The short answer is that it's up to them. People, projects and priorities differ, so it makes sense that a team can figure out a workflow that suits them instead of just following the Scrum Bible.

Tools? You can use Jira, Monday, Trello, Tuesday, Atera, Wednesday, post-its, or whatever works for you. The critical thing is that everyone involved can:

• Collaborate closely and communicate often.

• Work in small increments.

• Continually reflect and improve.

I'm starting to sound like a broken record here. By focusing on principles over process, Basic Agile encourages teams to learn and develop the best approach for their problem space.

Basic Agile is less like executing a plan and more like a process of evolution that drives towards more optimal solutions. However, we can still understand our progress by projecting past work into the future. These projections are like weather forecasts where results are consistent in the short term and get more uncertain as you look further out. Forecasts are much better than estimates because they're based on empirical data instead of guesswork. If you're interested in this, I recommend you watch Allen Holub's talk on #NoEstimates.

A Basic example

To make things more practical, I'll share some highlights from the process weve developed in my current team. For context, we work on spatial training simulators.

Our daily routine starts with a 15-minute "App Bash" for testing the latest build - optional and open to all. A focused 15-minute Daily follows this to hash out findings and today's blockers. For deep dives, we turn to Slack and ad-hoc huddles.

Sprint planning stays short and flexible, focusing on weekly themes and potential dependencies between people. We break features into tasks when we're actually ready to work on them, and we don't spend time on estimates.

Immediate concerns? We tackle them on the spot and hold a Retro every few months for a deeper look.

For those interested in our progress on long-term objectives, we make rough forecasts based on past work. Between forecasts, open builds and Slack updates, we don't need formal Review meetings.

Compared to the Corporate Scrum example from before, we get by just fine with less than a quarter of the weekly person-hours dedicated to team meetings. We don't have a product owner or project lead, as we collaborate directly with our stakeholders.

We've worked this way throughout 2023, and things have not descended into chaos. To the contrary: The team is healthy, and we keep delivering great work in good time.

Your mileage may vary. The point isn't that you should replicate our process in your teams, but it's worthwhile allowing your teams to find their own.

Now what?

If you just read this and thought, That made sense to me, then maybe you should try making things more Basic and observe the outcome. If things get better, try doing more of that. Let me know how it goes. If you're looking to shake things up, you could quit doing Scrum and re-design your process with adaptability and efficiency in mind.

Tell your friends - the extent to which the organisational stack understands agile principles is vital to the success of a Basic Agile approach.

Walking away from established processes takes some courage, but remember: Nothing is stopping you from going back if things don't work out. You might learn something new, and if you can adapt and improve from it, that is an agile way to be.

Daniel Rothmann runs 42futures, where he helps technical leaders validate high-stakes technical decisions through structured software pilots.

In the Unity project graveyard, many ventures meet their demise in silence. Unsuccessful initiatives are plenty, often abandoned due to a slow buildup in complexity that brings development to a painful, grinding halt.

Over the years, I've noticed that many of us (myself included) fall into bad habits while designing software with Unity. Today, I want to share these anti-patterns with you so you can avoid them in your projects.

In this article, I'll focus on Unity's MonoBehaviour paradigm.

Excessive use of global state

The Hierarchy is a central concept in Unity, and it makes it extremely easy to access shared state. For instance, any MonoBehaviour can find and use any GameObject and its corresponding behaviors. Each MonoBehaviour can manage them as needed, including making changes or removing them.

It's a straightforward programming model: A GameObject represents a game entity that combines individual behaviors to perform complex tasks in a modular way.

The MonoBehaviour concept blurs the lines between software and data. When we add behaviors to a GameObject, we are composing functionality, but also configuration, such as which other GameObjects our behaviors will interact with.

This is easily overlooked - how all this configuration, all this data, is exposed to other entities in the Hierarchy. Unfortunately, this exposure can lead to breaking encapsulation principles.

What's wrong with that?

Let's take an example. In your Unity project, you're developing a user interface with nested GameObjects, each representing individual UI elements.

Some of these nested objects have MonoBehaviours with public fields like Width or FontColor. These are intended to be configured by their parent UI window.

Unity Hierachy
├── MyUserInterface
│   ├── FirstUiWindow
│   │   ├── APanel
│   │   ├── AButton
│   ├── SecondUiWindow
│   │   ├── SomeText
│   │   ├── AnAnimation

As the UI implementation grows, you notice that certain elements directly access public fields on other nested objects in the Hierarchy. For instance, when you press a button within one UI window, it directly modifies the FontColor property on a text element in a different UI window.

Unity Hierachy
├── MyUserInterface
│   ├── FirstUiWindow
│   │   ├── APanel
│   │   ├── AButton       >>| OnPress()
│   ├── SecondUiWindow      |
│   │   ├── SomeText      <<| FontColor = Color.Green
│   │   ├── AnAnimation

The button and text elements are now directly related, even though you would not expect them to be so. We could easily imagine how you could end up with several other UI elements modifying this same text element.

Direct access between nested objects creates tight coupling within the system, and unintended consequences start to emerge. The UI elements become harder to maintain, as any one component may be modified by many others, causing race conditions and making bug detection and resolution a challenging task.

As developers venture into the labyrinth of tightly coupled UI complexities, stress and frustration rear their mischievous heads 😡 The absence of clear separation and encapsulation casts a shadow on the code, slowing down progress to a snail's pace.

Tips to limit global state exposure

1. If you're intending to expose a class field for setting in the Unity Inspector, but not for other behaviors to modify, consider making it [SerializeField] private instead of public.

2. If changing a class field has side effects, consider exposing it through setter methods or properties that can validate the new value and apply side effects immediately, providing better observability when things go wrong.

3. When a group of GameObjects are directly related, encapsulate them in a Prefab. If you are feeling particularly defensive, instantiate children at runtime, so they cannot be referenced by other entities at edit-time.

4. Try to point Inspector references laterally or downward in the Hierarchy. Avoid referencing entities and behaviors that are not siblings or children of the GameObject you're working on.

5. Avoid global search methods like FindObjectOfType when possible. Instead, opt for GetComponent or GetComponentInChildren to limit the search to the current entity or its children in the Hierarchy.

Weak dependency management

If you've worked with .NET's dependency injection features in other contexts, you'll likely share my frustration with the absence of robust dependency management in most Unity codebases.

Unity offers a helpful attribute called [RequireComponent], allowing us to state dependencies explicitly. By using this attribute, when we add a new component to a GameObject in the Editor, all the required dependencies are automatically included.

However, the real challenge lies in handling unstated dependencies that lurk beneath the surface. These hidden dependencies may not be immediately apparent, but they can wreak havoc at runtime, leading to perplexing and unexpected issues. A repeat offender is the use of GetComponent to find a MonoBehaviour, which introduces implicit dependencies that may cause problems down the road.

To complicate things further, we cannot make assumptions on the execution order of game loop callbacks for MonoBehaviours unless we manually manage them using the Script Execution Order settings or the less-known [DefaultExecutionOrder] attribute. For this reason, each behavior may need to double-check if its dependencies are ready before trying to use them.

So, what's the problem?

Let's expand upon the previous example to illustrate the scenario further. Imagine you have a UI window dedicated to displaying a player's inventory, organized in the Hierarchy as follows:

Unity Hierachy
├── MyUserInterface
│   ├── InventoryWindow
│   │   ├── AllItemsPanel
│   │   ├── SelectedItemImage
│   │   ├── EquipItemButton

Similar to before, the InventoryWindow contains several UI elements, each designed for specific purposes. To bring it together, we add a new script called InventoryWindowManager to the InventoryWindow entity:

public class InventoryWindowManager : MonoBehaviour
{
    private void Awake()
    {
        // Get the button component and subscribe to press events
        var button = GetComponentInChildren<MyButton>();
        button.OnPressed += OnEquipItemButtonPressed();
    }

    private void OnEquipItemButtonPressed()
    {
        // TODO: Equip the selected item from inventory...
    }
}

Initially, using GetComponentInChildren in InventoryWindowManager seems promising. It finds the MyButton component on EquipItemButton within the InventoryWindow hierarchy and handles its OnPressed event smoothly.

However, as the project evolves, challenges arise from this implicit dependency. UI changes, like altering the InventoryWindow layout or using a different button interaction can break the implicit link established by GetComponentInChildren. This may lead to null references or unexpected behavior during runtime.

Moreover, the current implementation tightly binds the InventoryWindowManager script to the specific hierarchy of the InventoryWindow, limiting its flexibility and reusability.

As the codebase expands, dealing with implicit dependencies becomes an exasperating task, hindering progress and creating a sense of burden in the development process.

Tips to manage your dependencies

1. If it makes sense to store dependent behaviors on the same GameObject, use the [RequireComponent] attribute to automatically enforce the dependency.

2. Instead of using search methods like GetComponentInChildren, consider if you can take a dependency explicitly in the Editor with a serialized private field instead.

3. Gather and validate all your behavior's dependencies in Awake or Start. This makes it easier to debug problems caused by bad dependencies down the road.

4. In situations where a dependency could be uninitialized or null, Assertions can be used to validate it in a performant way, as these are automatically stripped from production builds.

5. Try out a dependency injection framework like Reflex or VContainer, or make your own IoC container and learn something new!

Hard-to-verify behavior

Many developers, including myself, tend to lean towards manual testing when working with Unity. However, I think it's worth questioning whether automated testing would be more appealing if our components weren't so tightly coupled.

Personally, I have found the most value in targeting the lower end of the Testing Pyramid, focusing on component tests that validate individual behaviors from the "outside-in" through their public interfaces. Oftentimes, MonoBehaviour methods (like Update) don't return a result. Instead, they produce side effects in the Hierarchy or their dependencies. These can be verified with automated tests.

Yet, the path to effective testing is often obstructed by coupling and dependency management issues. Additionally, most Unity apps rely heavily on Unity engine features, and these are often accessed through static methods or classes without interfaces. This makes it hard to test a behavior independently of the engine.

Without careful consideration, individual scripts become entangled in specific contexts, resulting in what I call "hard-to-verify behavior." Such behaviors heavily depend on their surroundings, breaking when they are moved away from certain components, Hierarchy objects, or if they are instantiated at the wrong time. As the number of such behaviors grows, manual testing becomes the only way to ensure app functionality, leading to an error-prone process as the app's complexity expands.

A concrete example

Imagine you are developing a 2D platformer game with a player character controlled by a MonoBehaviour script called PlayerController. The script handles essential functionalities such as movement, jumping, and collision detection:

[RequireComponent(typof(RigidBody))]
public class PlayerController : MonoBehaviour
{
    public delegate void PlayerEvent();
    public event PlayerEvent OnPlayerDied;

    private bool _isInvincible;

    public void SetInvincibility(bool shouldBeInvincible)
    {
        _isInvincible = shouldBeInvincible;
        // TODO: Do more stuff while invincible (e.g., visual effects)
    }

    private void OnCollisionEnter(Collision collision)
    {
        // Trigger death event on collission if not invincible
        if (!_isInvincible)
        {
            OnPlayerDied();
        }
    }

    // TODO: Movement, jumping, etc.
}

Initially, everything seems to work well, and manual testing confirms that the game behaves as expected. However, as the development progresses and more features are added, the codebase starts to become more complex.

Now, you want to introduce a new power-up that gives the player temporary invincibility. To achieve this, you create a PowerUpController script that manages the power-up behavior:

[RequireComponent(typof(PlayerController))]
public class PowerUpController : MonoBehaviour
{
    private PlayerController _playerController;

    private void Awake()
    {
        _playerController = GetComponent<PlayerController>();
    }

    public async Task ApplyInvincibilityPowerUp()
    {
        // Make player invincible for 5 seconds
        _playerController.SetInvincibility(true);
        await Task.Delay(5000);
        _playerController.SetInvincibility(false);
    }
}

However, you soon discover that interacting with the existing PlayerController script is not as straightforward as you thought. This script depends on Unity's physics system, making it challenging to test in isolation. When trying to verify the behavior of the PowerUpController through automated testing, difficulties arise again due to these interactions with physics, which cannot be easily mocked or simulated.

As more features are added, and the game becomes more intricate, manual testing becomes increasingly time-consuming and prone to human error. Bugs related to the interactions between different scripts, such as the PowerUpController and PlayerController may go unnoticed until playtesting, leading to a frustrating development experience and potentially jeopardizing the project's success.

Tips to make your logic more testable

• Instead of an all-or-nothing mindset, make a habit of writing automated tests for critical things. This will help nudge your designs towards testable behaviors, and these can be tested more extensively later if necessary.

• If a class relies heavily on Unity engine features, consider creating wrappers with interfaces that you can replace for test doubles when needed.

• Consider using the Humble Object pattern to segregate your business logic from engine code, making it a lot simpler to set up in tests.

• Avoid singletons. If you are using dependency injection, consider registering a service instead, so it can be replaced with a test double when needed.

• Check out Test Desiderata for a great overview of tradeoffs and valuable properties in automated tests.

In conclusion

Navigating the Unity software design landscape can be a difficult task, but understanding and avoiding common anti-patterns will lead to faster development progress and more maintainable projects.

Excessive use of global state leads to coupling and race conditions, making it hard to maintain and extend your app. By limiting global state exposure and embracing encapsulation principles, you can achieve a more modular and robust design.

Weak dependency management can introduce hidden links, causing runtime issues and restricting the flexibility of your codebase. Embracing explicit dependency declarations or utilizing dependency injection frameworks can help untangle these complexities.

Hard-to-verify behavior, arising from tightly coupled scripts and a heavy reliance on platform features, can become a testing nightmare. By prioritizing automated testing, segregating business logic from engine code, and employing designs like the Humble Object pattern, you can make your components more testable and easier to maintain.

In my view, understanding and addressing these anti-patterns is key to maintaining a well-organized and stable project. By employing best practices, embracing modular design, and prioritizing automated testing, you can create harmony in your Unity software designs, setting the stage for successful and enjoyable development experiences.

Earlier this year, I read David Epstein's "Range." It's a great book that presents a lot of evidence favoring a generalist approach to complex work. Since reading it, thoughts have kept circulating in my mind about how its ideas apply in software, data, and technology — Here's my take.

David Epstein examined the world’s most successful athletes, artists, musicians, inventors, forecasters and scientists. He discovered that in most fields — especially those that are complex and unpredictable — generalists, not specialists, are primed to excel. Generalists often find their path late, and they juggle many interests rather than focusing on one. They’re also more creative, more agile, and able to make connections their more specialized peers can’t see.

A sports analogy

We've all heard tales of child prodigies. This type of story is the ultimate archetype of early specialization and its benefits. In the world of sports, we have Tiger Woods, who showcased his golf skills on national TV at two! Safe to say, Tiger was on his road to mastery early and, by now, he's had his 10000 hours of deliberate practice.

With this model in mind, does it follow that the path to greatness in a tech career boils down to "pick something good (ASAP) and practice a lot"?

According to Epstein, not so. The story of tennis professional Roger Federer reveals a different strategy for excellence. As a young child, Roger did not focus on tennis. Instead, he would try many sports, including skateboarding, badminton, swimming, ping pong, and basketball. Roger went on to become a tennis world champion, crediting his hand-eye coordination to the wide range of sports played as a child.

Building skills for complex environments

At first glance, the success stories of Woods and Federer seem at odds with each other. To explain the difference, Epstein refers to psychologist Robin Hogarth and his work on "kind" and "wicked" learning environments.

In kind environments, where the goal is to re-create prior performance with as little deviation as possible, teams of specialists work superbly.

Golf is a kind learning environment. To succeed, you must be able to repeat learned procedures with minimal deviation — Choosing the right swing for a specific situation and executing it precisely. Tennis, however, is more wicked. Due to the game's speed and being real-time rather than turn-based, players must spontaneously anticipate their opponent's moves.

In wicked domains, the rules of the game are often unclear or incomplete, there may or may not be repetitive patterns and they may not be obvious, and feedback is often delayed, inaccurate, or both.

The world of technology is a wicked learning environment. It moves rapidly with no shortage of disruptive innovations; those who do not keep up the pace risk being left behind.

Connecting the dots

As a society, we are becoming increasingly reliant on knowledge workers (such as IT professionals) who can operate in wicked environments.

However, I still feel there is a cultural bias in favor of narrow specialization. Maybe it's an industrial way of thinking that carried over into our post-industrial society. You've probably heard the saying: "jack of all trades, master of none." In my experience, this saying does not ring true. A more accurate (and boring) reinterpretation might be: "jack of some trades, master of some."

The fact is that most generalists in tech stay within specific domains, focusing their energy on a few topics at a time while dabbling in related areas. More often than not, these experiences overlap to some extent. You can combine them to achieve great results, especially in areas where innovation and complex problem solving are required.

I dislike the "generalist" label for technologists because the way it's used is usually inaccurate. What we're talking about are "multi-skilled" profiles or polymaths who have cultivated a broad general knowledge in addition to deep expertise on one or more subjects.

Polymath superstars

In his 2020 review of Range, Bill Gates recognizes his path to success in Epstein's findings. This came as a surprise to me, as I had always thought of Bill as this kid who got obsessed with computers early and created Microsoft.

My own career fits the generalist model pretty well. As a kid, I used to sneak out of my basement bedroom to do late-night coding at the University of Washington, but my passion for computers was always mixed with many other interests. I spent a lot of time reading books on a wide range of topics.

On the topic of generalists in tech, we must not fail to mention Elon Musk. He is one of the most visible and influential personalities in tech today. Cited as an "expert generalist," Elon has navigated many domains such as fintech, public transportation, solar energy, electric cars, and space exploration.

It is important to view knowledge as sort of a semantic tree — make sure you understand the fundamental principles, i.e., the trunk and big branches, before you get into the leaves/details or there is nothing for them to hang on to.

Complex problem solvers

Throughout the history of computers, generalists have made exceptional contributions to the field. Consider John Von Neumann, who is often cited as the "father of the modern computer." A mathematician by trade, John pioneered new ideas in mathematics, physics, economics, computing, and statistics. Notably, he invented one of the world's most popular computer architectures, laying the foundation for many innovations to come.

If people do not believe that mathematics is simple, it is only because they do not realize how complicated life is.

Another example comes from the origin of the artificial neural network (ANN), one of my favorite topics in machine learning. Warren Sturgis McCulloch and Walter Pitts initially conceived the ANN.

Warren was a student of philosophy and psychology. He went on to become a professor of psychiatry and contributor to the cybernetics movement. In his spare time, he wrote poetry and engineered buildings for his farm. Walter was a self-taught logician who would later dive into physics, electrical engineering, and computer science.

Together, they sowed the seeds for a paradigm shift in artificial intelligence, connectivism, and computational neuroscience fueled by the neural network.

It's good to have a hobby

In "Range", Epstein presents an exciting finding: Nobel Prize winners are significantly more likely than their peers to have an aesthetic hobby like acting, dancing, music, and photography. For honored scientists, the finding persists: They also cultivate more hobbies than the general population.

Let's qualify this finding with some current names in technology:

Mark Russinovich

At the moment and since 2014, he's been the CTO at Azure.

What you might not know about Mark is that, in addition to being a software engineer and technology leader, he's written multiple science fiction books.

Andy Hunt

Co-founder of the Agile Manifesto and co-author of The Pragmatic Programmer. When he's not writing books on software development, Andy is playing various horns, producing electronica, writing fiction, or doing woodworking.

Rachel Andrew

Staff technical writer at Google, member of the CSS Working Group, and author of 22 books on web development. Rachel used to be a trained dancer working in West End theatre. Now, she enjoys religious studies and learning to fly airplanes.

John Resig

Creator of JQuery and chief software architect at Khan Academy. In his spare time, John has combined an interest in art history and computer vision engineering with his programming skills to improve digital humanities study.

In conclusion

Don't get me wrong: A singular drive is an excellent thing to have under the right circumstances. However, the examples I have highlighted in this article show us that our role models in technology can also be multi-skilled.

If you are an aspiring polymath, dabbler, or generalist in the world of tech, this article was for you. I hope to have convinced you that your broad interests should not be neglected in favor of a clean and straightforward career timeline. In fact, you can use your breadth as a superpower to navigate complex and uncertain environments with ease as long as you manage to connect the dots and combine your knowledge.

As a software engineer with a broad set of interests, I’ve found Epstein’s ideas empowering and energizing. I hope you will too.

Who are some of your favorite generalists in software and tech?

My work as a data engineer at Kanda has shifted from being focused on projects to products as we scale up our platforms. This has caused me to reflect on my early days of coding, and my experience getting familiar with the products I have worked on in different organizations.

To be honest, I’ve wasted too much energy feeling frustrated. It would bother me when I had to work on a system where I didn’t agree with the design or when I inherited code that didn’t have great docs. Just a few examples.

It took me a while to accept that software development is always a work-in-progress.

Despite everyone’s best intentions, it won’t ever be perfect. In that way, code is like art (except you keep refining it even after it's released). Because of this, I now try to approach software engineering with an “upgrader mindset” — Focusing my energy on improving what’s there and figuring out what’s next.

There’s Always Scope To Improve

You’re excited.

It’s your first commercial software project, and you’re ready to work on a clean, well-architected codebase for a big project. Hitting it big time! You quickly realize that the codebase is not perfect. There might be documentation missing or you might find some questionable designs and awkward workarounds. “What would Uncle Bob say?”, you wonder.

It works, but…

If you voice the issues you’ve found, your senior colleagues will likely agree with you. The product works, but there are definitely a few kinks in the codebase that they’d like to iron out. Also, there are some manual processes that they’d love to automate. Also, that system component could really use a rewrite. Ah right, and they forgot to tell you about this helpful tool that you’ll need for debugging.

This is normal.

We’ll explore why that is. But realizing that production software can always be improved will start you off on the right foot. Reality is messy, so expect to find a bit of mess. This is where CI/CD practices come in handy, enabling the team to accelerate the delivery of those crucial improvements to the customer.

Sometimes Decisions Get Made With Incomplete Knowledge

Nobody wants to make a bad decision. I like to think that, as a team, we want to always do our best work and make great decisions. Here’s the thing: When we make a software decision, we need to balance many factors. Accounting for uncertainty is just one of those factors.

Optimal technical decisions always seem more clear in hindsight. However, in the moment when a decision is made, the future is unknowable. We can lean into that uncertainty by favoring dynamic, extendable designs. On the other hand, if we put too much focus on uncertainty, we won’t get our stuff done on time. Worse yet, we might waste months building something very dynamic that doesn’t solve our customer’s most pressing problems.

Software teams balance many factors when making a decision, such as:

• What’s our budget?

• How fast does this need to be delivered?

• Are other teams affected by this decision?

• How likely is it that the requirements for this will change in the future?

• Are there any limitations in our dependencies that could impact this?

For these reasons, I try to avoid questioning the intent behind a software decision. Always assume the best intentions and try to focus on the content of the decision and how it could be tweaked to work for the new circumstances.

Adopting An Upgrader Mindset

Accepting that a decision made sense at the time, does that mean we should just leave it alone if it’s not a critical issue right now?

No, of course not! A “leave-it-be” attitude is going to pile up technical debt which can ultimately sink your software project.

Let’s try an upgrader mindset instead.

Start on the assumption that we all want to create a great product. If you find a problem, the team has an upgrade opportunity (and that’s great).

Identifying a problem isn’t enough. To make an impact as an upgrader, you need to find a path to improve the situation. Sometimes you can just go ahead and fix it yourself. You know, leave the campsite cleaner than you found it. Larger problems aren’t usually solved with quick fixes, so in those situations, you’ll probably need some teammates to put their heads together.

When you are working on a software product as a new developer and you encounter something questionable, here are five points to consider:

• Be respectful. Assume that the person who worked on this before made the best decision they could, given the circumstances.

• Don’t hesitate to (politely) ask: “Is this intentional?”. Your teammates might be able to provide additional context.

• If you understand the context and feel that the problem is fixable, go ahead and fix it.

• If the problem is larger or structural in nature, raise it to the team. Think of a solution ahead of time, but be open-minded to ideas from the team.

• Don’t feel bad if the team isn’t ready to fix a large issue right away. Some improvements are not time-critical and prioritizations must be made.

Sometimes you need to champion an improvement for a while. A respectful tone goes a long way to get the technical team on board. For example:

“I had an issue implementing Feature X in System Y. I got some context from Person Z and I think we can avoid this issue in the future by reworking System Y. Here’s an idea for how we could do it, but what do you think?”

If management needs convincing, keep it short and focus on the impact to budgets and time spent by the team. For example:

“Every week, our team is losing 5 hours while working on Feature X because of a technical issue. The team agrees that, by reworking System Y, we can save that time going forward. We estimate it will take 20 hours to do. Can we plan this in?”

Conclusion

I believe that an upgrader mindset is a highly productive way of approaching old code, and it’ll make you feel less frustrated too. I may not always succeed at being this kind of coder, but it’s the kind of coder I strive to be. After all, there’s always something to improve.

So, let’s get to work.

We often hear that Big Data is the key to building successful machine learning projects.

This is a major problem: Many organizations won’t have the data you need.

How can we prototype and validate machine learning ideas without the most essential raw material? How can we efficiently obtain and create value with data when resources are sparse?

At my workplace, we produce a lot of functional prototypes for our clients. Because of this, I often need to make Small Data go a long way. In this article, I’ll share 7 tips to improve your results when prototyping with small datasets.

1: Realize that your model won’t generalize that well.

This should be the first order of business. You’re building a model whose knowledge is based on a tiny fraction of the universe, and that should be the only place or situation where it can be expected to work well.

If you’re building a computer vision prototype based on a selection of indoor photos, don’t expect it to work well outdoors. If you have a language model based on chatroom banter, don’t expect it to work for a fantasy novel.

Make sure this is understood by your manager or your client. This way, everyone can align on a realistic expectation of the results that your model should deliver. It also creates an opportunity to come up with useful new KPIs to quantify model performance both inside and outside the prototype scope.

2: Build good data infrastructure.

In many situations, the client won’t have the data you need, and public data won’t be an option. If part of your prototype requires collecting and labeling new data, make sure that your infrastructure to do this creates as little friction as possible.

You’ll want to make sure data labeling is very easy so it’s approachable for non-techies as well. We have started using Prodigy, which I think is a good tool: Both accessible and extendable. Depending on the size of the project, you might also want to set up an automated data ingest which can take in new data and automatically feed it to the labeling system.

If getting new data into your system is quick and easy, you will get more data.

3: Do some data augmentation.

You can often extend your dataset by augmenting the data that you have. It’s about making slight changes to the data that should not significantly change the model output. For example, an image of a cat is still an image of a cat if it’s rotated 40 degrees.

In most cases, augmentation techniques allow you to produce many more “semi-unique” data points for training your model. You could, as a start, try adding small amounts of Gaussian noise to your data.

For computer vision, there are lots of neat ways to augment your images. I have had positive experiences with the Albumentations library which does many useful image transformations while keeping your labels unharmed.

Another augmentation technique that many have found useful is Mixup. This technique literally takes two input images, mixes them together and combines their labels.

When augmenting other input data types, some consideration needs to be given to which transformations would change a label and which wouldn’t.

4: Generate some synthetic data.

If you have exhausted your options for augmenting real data, you could start thinking about creating some fake data. Generating synthetic data can also be a great way to cover some edge cases that your real dataset does not.

An an example, many reinforcement learning systems for robotics (like OpenAI’s Dactyl) are trained in simulated 3D environments before they are deployed to real robots. For image recognition systems, you can similarly build 3D scenes that could provide you with thousands of new data points.

There are many approaches to creating synthetic data. At Kanda, we are developing a turntable-based solution to create data for object detection. If you have a very high data requirement, you could consider using Generative Adverserial Networks to create synthetic data. Be aware that GANs are notorious for being hard to train, so make sure that it’ll be worth it first.

Sometimes you can combine approaches: Apple had a very clever way of using a GAN to process images of 3D modeled faces to look more photo-realistic. Awesome technique to extend your dataset, if you have the time.

5. Beware of lucky splits.

When training machine learning models, it is quite common to randomly split the dataset into train and test sets according to some ratio. Usually, this is fine. But when working with small datasets, there is a high risk of noise due to the low volume of training examples.

In this case, you may accidentally get a lucky split: A particular dataset split where your model will perform and generalize really well to the test set. In reality though, this might just be because the test set (by coincidence) contained no difficult examples.

In this scenario, k-fold Cross-Validation is a better choice. Essentially, you split the dataset into k “folds” and train a new model for each k where one fold is used for the test set and the rest is used for training. This controls that the test performance you see is not simply due to a lucky (or unlucky) split.

6. Use transfer learning.

If you’re working with a somewhat standardized data format like text, images, video or sound, you can leverage all the previous work others have put into these domains with transfer learning. It’s like standing on the shoulders of giants.

When you do transfer learning, you take models that others have built (usually, “others” being Google, Facebook or a major university) and fine-tune them to suit your particular needs.

Transfer learning works because most tasks to do with language, images or sound share many common characteristics. For computer vision, it could be detecting certain types of shapes, colors or patterns, for example.

Recently, I worked on an object detection prototype for a client with a high accuracy requirement. I was able to speed up development enormously by fine-tuning a MobileNet Single Shot Detector which had been trained on Google’s Open Images v4 dataset (~9 million labeled images!). After a day of training, I was able to produce a fairly robust object detection model with a test mAP of 0.85 using ~1500 labeled images.

7. Try an ensemble of “weak learners”.

Sometimes, you just have to face the fact that you do not have enough data to do anything fancy. Luckily, there are many traditional machine learning algorithms you can fall back to which are less sensitive to the size of your dataset.

Algorithms like the Support Vector Machine are a good choice when the dataset is small and the dimensionality of the data points is high.

Unfortunately, these algorithms are not always as accurate as state-of-the-art methods. This is why they can be called “weak learners”, at least in comparison to highly parameterized neural networks.

One way to improve the performance is to combine several of these “weak learners” (this could be an array of Support Vector Machines or Decision Trees) so that they “work together” to produce a prediction. This is what Ensemble Learning is all about.

When machine learning techniques are used in “mission critical” applications, the acceptable margin of error becomes significantly lower.

Imagine that your model is driving a car, assisting a doctor or even just interacting directly with an (perhaps easily annoyed) end user. In these cases, you’ll want to ensure that you can be confident in the predictions your model makes before acting on them.

Measuring prediction uncertainty grows more important by the day, as fuzzy systems become an increasing part of our fuzzy lives.

Here’s the good news: There are several techniques for measuring uncertainty in neural networks and some of them are very easy to implement! First, let’s get a feel for what we’re about to measure.

Putting a number on uncertainty.

When you make models of the world, your models cannot always provide accurate answers.

This is partly due to that fact that models are simplifications of a seriously complicated world. Since some information is unknown, the predictions from your model are subject to some degree of uncertainty.

Parts of our world (and the ways we measure it) are simply chaotic. Some things happen randomly, and this randomness is also a source of uncertainty in your model’s predictions.

Prediction uncertainty can be divided into 3 categories:

1. Model uncertainty.

Model uncertainty comes from “ignorance” of the problem. That is, model uncertainty quantifies the things that could be correctly captured by the model but isn’t.

Yoel and Inbar from Taboola provide a fun example:

Sometimes it is also referred to as epistemic or structural uncertainty. Measuring model uncertainty is an area of statistics which is considered to be particularly challenging. One reason for this, is that principled techniques like Bayesian model averaging become very costly as models grow more complex.

2. Model misspecification.

If your model produces good predictions during training and validation but not during evaluation (or in production), it might be misspecified.

Model misspecification uncertainty captures scenarios where your model is making predictions on new data with very different patterns from the training data.

3. Inherent noise.

This is uncertainty produced by noise present in the dataset. It could be attributed to imperfect measurement techniques or an inherent randomness in the thing being measured.

Inherent noise is also sometimes called aleatoric or statistical uncertainty. The amount of inherent noise is linked to the Bayes error rate which the lowest achievable error rate of a given classifier. As you can imagine, the lowest possible error rate that a model can achieve is tightly linked to the amount of error produced by noise in the data itself.

These concepts lean heavily on Bayesian statistics. I have outlined these ideas in a simple way, but that’s just scratching the surface on these deep topics.

To learn more about uncertainty measures in Bayesian neural networks, I recommend taking a look at this article by Felix Laumann. For an in-depth explanation of Bayesian statistics in the context of data science, Ankit Rathi has written a series of great articles on the subject.

Implementing uncertainty.

At this point, you may be thinking: “That sounds good, but how do I implement uncertainty in my model?”.

The Bayesian neural network integrates uncertainty by default in addition to generally being more robust to overfitting and handling smaller datasets. However, the toolchain for building Bayesian neural networks is still emerging and the models tend to be more computationally costly, both during training and when making predictions.

Also, migrating your work to a probabilistic model (like a Bayesian neural network) is going to be annoying.

In the long run, probabilistic deep learning will likely become the default. For now though, practical techniques to integrate the probabilistic perspective in our existing work is a good first step!

Monte Carlo dropout.

A couple of years ago, Yarin and Zoubin from University of Cambridge found a way to approximate model uncertainty without changing the structure or optimization techniques of the neural network.

Here’s the short version: By using dropout before each weight layer at test time and running your predictions for several iterations, you can approximate Bayesian uncertainty. They call this process Monte Carlo dropout:

1. You feed an input to your model.

2. You predict for several iterations on that single input, each time disabling small parts of the neural network randomly.

3. You take the mean output value. This is your prediction. Finally, you measure the variance between iterations. This is the model uncertainty.

Intuitively, I think of it like this: The more your prediction fluctuates with tiny structural changes to the model, the more uncertain that prediction is.

Implementing Monte Carlo dropout is really easy. Here, I start with a simple dense network for the MNIST problem built with Keras. By default, dropout layers are only enabled during training. To enable the dropout layers at test time, set training=True for each layer.

img_dim = 28

x_in = Input(shape=(img_dim, img_dim,))
x = Dropout(0.2)(x_in, training=True)
x = Dense(500, activation="relu")(x)
x = Dropout(0.2)(x, training=True)
x = Dense(250, activation="relu")(x)
x = Dropout(0.2)(x, training=True)
x_out = Dense(10, activation="softmax")(x)

model = Model(inputs=x_in, outputs=x_out)
model.compile(loss="categorical_crossentropy", optimizer="rmsprop")

Next, we need a custom prediction function which can predict iteratively and return the mean and variance of those iterations. In this example, we measure the standard deviation instead of the variance, because it’s expressed in the same units as the mean.

def predict_with_uncertainty(model, 
                             X, 
                             batch_size=None, 
                             num_iterations=100):

    last_layer = model.layers[-1]

    results = np.zeros((num_iterations,
                       X.shape[0],
                       last_layer.output_shape[1]), 
                       dtype="float")

    for i in range(num_iterations):
        results[i] = model.predict(X, batch_size=batch_size)
    
    predictions = results.mean(axis=0)
    uncertainty = results.std(axis=0)
    
    return predictions, uncertainty

Now you’re ready to predict with approximated uncertainty:

predictions, uncertainty = predict_with_uncertainty(model,your_data)

Problems and critique.

As we saw, using Monte Carlo dropout is really easy. Maybe even too easy. The technique was critiqued by Ian Osband from Deepmind who noted that the predictive uncertainty of a simple model with Monte Carlo dropout did not decrease with more data. This raises the question of whether it is an inaccurate approximation of Bayesian uncertainty or if there are any underlying assumptions that need to be made more clear.

For more that issue, Sebastian Schöner has written a great blog post summarizing the critique.

In my workplace at Kanda, we have had mixed experiences with the effectiveness of the Monte Carlo dropout.

For a simple fully connected model trained on MNIST like my previous example, the uncertainty approximations behaved as expected: When presented with noise instead of a handwritten digit, the approximated uncertainty was higher. We found that 50–100 iterations of Monte Carlo dropout produced satisfactory results.

Later, we had a scenario, where we needed to run an image recognition task locally on a smartphone as part of an AR application. We used transfer learning, building a classifier on top of the NASNet Mobile architecture.

Running 100 iterations of NASNet on a smartphone is not a good idea.

Even with heavy parallelization, we were only realistically able to run ~20 iterations on the device in order to provide a prediction in good time.

Secondly, the uncertainty estimates were inaccurate. When fed a picture of random noise, the uncertainty was surprisingly low. It is worth noting that we only implemented dropout in the densely connected part of the classifier which sat on top of NASNet. If you’d like to share intuitions about what went wrong here, I’d be very interested to read a response from you!

Conclusion.

First, we had a look at why it’s important to quantify uncertainty in machine learning models. Then, I introduced you to 3 different ways of thinking about prediction uncertainty: Model uncertainty, model misspecification and inherent noise.

Monte Carlo dropout is an easy-to-implement technique to approximate Bayesian uncertainty, but there is some disagreement to whether the approximation is indeed accurate. Practically, I have found Monte Carlo dropout effective for simpler models, but have had some problems with the approach for complex models, both in terms of accuracy and performance.

Integrating Bayesian probability in machine learning will only become more important in the future, and I look forward to seeing more probabilistic techniques become part of the toolchain.

Results and perspectives.

This is the last part of my article series on “Human-Like” Machine Hearing: Modeling parts of human hearing to do audio signal processing with AI.

This last part of the series will provide:

• A concluding summary of the key ideas.

• Results from empirical testing.

• Related work and future perspectives.

If you’ve missed out on the previous articles, here they are:

Background: The promise of AI in audio processing
Criticism: What’s wrong with CNNs and spectrograms for audio processing?
Part 1: Human-Like Machine Hearing With AI (1/3)
Part 2: Human-Like Machine Hearing With AI (2/3)

Summary.

Understanding and processing information at an abstract level is not an easy task. Artificial neural networks have moved mountains in this area. Especially for computer vision: Deep 2D CNNs have been shown to capture a hierarchy of visual features, increasing in complexity with each layer of the network. The convolutional neural network was inspired by the Neocognitron which in turn was inspired by the human visual system.

Attempts have been made at reapplying techniques such as style transfer in the audio domain, but the results are rarely convincing. Visual methods don’t seem to reapply well on sounds.

I have argued that sound is a different beast altogether. It’s something to keep in mind when doing feature extraction and designing deep learning architectures. Sounds behave differently. Just like computer vision benefited from modeling the visual system, we can benefit from considering human hearing when working with sound in neural networks.

Sound representation.

To start exploring a modeling approach, we can establish a human baseline:

To the brain, sounds are spectrally represented. Pressure waves are processed by the cochlea and divided into ~3500 logarithmically spaced frequency bands in the range ~20-20,000 Hz.

Sounds are heard at a temporal resolution of 2–5 ms. Sounds (or gaps in sounds) shorter than this are something near imperceptible to humans.

Based on this information, I recommend using a Gammatone filterbank instead of the Fourier Transform. Gammatone filterbanks are a common tool in auditory modeling and generally, filterbanks allow for the decoupling of spectral and temporal resolutions. This way, you can have a many spectral bands and a short window of analysis.

Memory and buffers.

Humans are thought to have memory for storing sensory impressions in the short-term so that they can be compared and integrated. Although experimental results differer slightly, they have shown that humans have ~0.25–4 seconds of echoic memory (sensory memory dedicated to sound).

Any sound that can be understood by a human being can be represented within these limits!

A couple of seconds worth of 2–5 ms windows, each with ~3500 logarithmically spaced frequency bands. But that does add up to a lot of data.

To reduce the dimensionality a bit, I proposed the idea of dilated buffers, where the temporal resolution of a time series is reduced by an increasing factor for older timesteps. This way, a larger time context can be covered.

Listener-processor architecture.

We can conceptualize the inner ear as a spectral feature extractor and the auditory cortex as an analytical processor, deriving “cognitive properties” from memories of auditory impressions.

In between, there is a set of steps often forgotten about. They’re referred to as the cochlear nuclei. There’s a lot we don’t know about these, but they do a sort of initial neural coding of sounds: Encoding basic features essential to localization and sound identification.

This led me to explore a listener-processor model, where sound buffers are embedded to a low-dimensional space by a general-purpose LSTM autoencoder (a “listener”) before being passed to a task-specific neural network (a “processor”). That essentially makes the autoencoder a reusable preprocessing step for doing some task-specific analytical work on a sound.

Results.

I like to think this article series has presented a couple of fresh ideas for working with sound in neural networks. Using these principles, I built a model to do environmental sound classification using the UrbanSound8K dataset.

Due to limited computing resources, I settled for humble representation:

• 100 Gammatone filters.

• 10 ms analysis windows.

• 8-step dilated buffers covering ~1.25 seconds of sound.

To train the listener, I fed thousands of dilated buffers to an autoencoder with 2 LSTM layers on each side, encoding the 800-dimensional sequential input into a latent space with 250 “static” dimensions.

After training for ~50 epochs, the autoencoder was able to capture the coarse structures for most of the input buffers. Being able to produce an embedding that captures the complex sequential movements of frequencies in sounds is very interesting!

But is it useful?

This is the question to ask. To test this, I trained a 5-layer self-normalizing neural network to predict the sound class (UrbanSound8K defines 10 possible classes) using embeddings from the LSTM encoder.

After 50 epochs of training, this network predicted sound class with an accuracy of ~70%.

In 2018, Z. Zhang et. al. achieved state of the art on the UrbanSound8K dataset with a 77.4% prediction accuracy. They achieved this by doing 1D convolutions on Gammatone spectrograms. With data augmentation, that accuracy can be pushed higher yet. I did not have resources to explore data augmentation, so I am going to compare on non-augmented versions of the system.

In comparison, my approach was 7.4% less accurate. However, my technique works on 10 ms time frames (usually with some memory attached), meaning that the 70% accuracy covers a 10 ms moment at any given time of a sound in the dataset. This reduces the latency of the system by a factor of 300, making it well suited for realtime processing. Put simply, this approach was less accurate but introduced significantly less delay when processing.

Perspectives.

Working through this project has been very interesting indeed. I hope to have supplied you with some ideas for how you can work with sound in neural networks. Though I am happy with my initial results, I believe they can be significantly improved given a finer spectral resolution and computational resources for training the neural networks.

I hope that someone will pick this up and experiment with a listener-processor approach on new problems. In particular, I am curious to try a variational autoencoder approach to see what happens to reconstructed sounds when their latent spaces are adjusted — Maybe this can reveal some intuitions about what makes the basic statistical features of sound itself.

Related work.

If you’re interested in sound representation using autoencoders, here are some projects that have inspired me and that I recommend looking into:

Audio Word2Vec

These folks worked on a similar approach for encoding speech using MFCC’s and Seq2Seq autoencoders. They found that the phonetic structures of speech can be adequately represented this way.

A Universal Musical Translation Network

This is very impressive and the closest I’ve seen to style transfer in sound yet. It came out from Facebook AI Research last year. By using a shared WaveNet encoder, they compressed a number of raw-sample musical sequences to a latent space, then decoded with separate WaveNet decoders for each desired output “style”.

Modeling Non-Linear Audio Effects with Neural Networks

Marco Martinez and Joshua Reiss successfully modeled non-linear audio effects (like distortion) with neural networks. They achieved this by using 1D convolutions to encode raw-sample sequencies, transform these encodings (!) with a deep neural network, and then resynthesize these encodings back to raw samples with deconvolution.

Dear reader, thanks so much for coming on this journey with me. I feel privileged by the amount of positive, critical and informative response I’ve had to this article series.

Having been swamped with work, this final article has been a long time coming. Now that it’s wrapped up, I am looking forward to next chapters, new projects and ideas to explore.

The importance of memory

Hi, and welcome back! This article series details a framework for real-time audio signal processing with AI which I have worked on in cooperation with Aarhus University and intelligent loudspeaker manufacturer Dynaudio.

If you’ve missed out on the previous articles, click below to get up to speed:

Background: The promise of AI in audio processing
Criticism: What’s wrong with CNNs and spectrograms for audio processing?
Part 1: Human-Like Machine Hearing With AI (1/3)

In the previous part, we mapped the fundamentals of how humans experience sound as spectral impressions formed in the cochlea which are then “coded” by a sequence of brainstem nuclei. This article will explore how we can integrate memory when producing spectral sounds embeddings with an artificial neural network for sound understanding.

Echoic memory

The meaning of a sound event arises, in large part, from the temporal interplay between spectral features.

One demonstration of this can be seen in the fact that the human auditory system encodes identical phonemes of speech in different ways depending on their temporal context [1]. That means that a phoneme /e/ can mean different things to us neurologically, depending on what came before it.

Memory is crucial to performing sound analysis since it is only possible to compare impressions of “the moment” to previous impressions if they are actually stored somewhere.

Human short term memory is made up of an collection of components which integrate both sensory and working memory [2]. In examination of sound perception, auditory sensory memory (sometimes referred to as echoic memory) has been found in humans. C. Alain et. al. describe auditory sensory memory as “a critical first stage in auditory perception that permits listeners to integrate incoming acoustic information with stored representations of preceding auditory events” [2].

Computationally, we can think of echoic memory as a short buffer of immediate auditory impressions.

There has been disagreements about the duration of echoic memory. On the basis of pure-tone and spoken-vowel masking studies, D. Massaro has argued for ~250 ms while A. Treisman argued for ~4 seconds based on a dichotic listening experiment [3]. For the purpose of integrating the idea of echoic memory with neural networks, we might not need to settle on a fixed duration of sensory store but we can experiment with memory within a range of a few seconds.

Going in circles

Implementing sensory memory in digital spectral representations could be quite straightforward. We can simply allocate a circular buffer to store a predetermined number of spectra at previous timesteps.

A circular buffer is a data structure consisting of an array which is treated as circular, where its indices loop back to 0 after the array’s length is reached [4].

In our case, this could be a multidimensional array with the length of the desired amount of memory, each indice of the circular buffer holding a full frequency spectrum for a specific timestep. As new spectra are calculated, they are written to the buffer, overwriting the oldest timesteps if the buffer is already full.

As the buffer fills, two pointers are updated: A tail pointer marking the newest element added and a head pointer marking the oldest element and, as such, the beginning of the buffer [4].

Here is an example of a circular buffer in Python adapted from Eric Wieser:

import numpy as np

class CircularBuffer():
    # Initializes NumPy array and head/tail pointers
    def __init__(self, capacity, dtype=float):
        self._buffer = np.zeros(capacity, dtype)
        self._head_index = 0
        self._tail_index = 0
        self._capacity = capacity

    # Makes sure that head and tail pointers cycle back around
    def fix_indices(self):
        if self._head_index >= self._capacity:
            self._head_index -= self._capacity
            self._tail_index -= self._capacity
        elif self._head_index < 0:
            self._head_index += self._capacity
            self._tail_index += self._capacity

    # Inserts a new value in buffer, overwriting old value if full
    def insert(self, value):
        if self.is_full():
            self._head_index += 1

        self._buffer[self._tail_index % self._capacity] = value
        self._tail_index += 1
        self.fix_indices()
        
    # Returns the circular buffer as an array starting at head index
    def unwrap(self):
        return np.concatenate((
            self._buffer[self._head_index:min(self._tail_index, self._capacity)],
            self._buffer[:max(self._tail_index - self._capacity, 0)]
        ))

    # Indicates whether the buffer has been filled yet
    def is_full(self):
        return self.count() == self._capacity

    # Returns the amount of values currently in buffer
    def count(self):
        return self._tail_index - self._head_index

Reducing input size

In order to store a full second of frequency spectra at a resolution of 5 ms per timestep, a buffer with a capacity of 200 elements are needed. Each of these elements would contain an array of frequency magnitudes. If human-like spectral resolution is desired, those arrays would contain 3500 values. For a total of 200 timesteps, that is 700,000 values to be processed.

If passed to an artificial neural network, an input with a length of 700,000 values runs the risk of being computationally expensive. This risk can be mitigated by reducing the spectral and temporal resolution or keeping a shorter duration of spectral information in memory.

We can also draw inspiration from the Wavenet architecture, which utilizes dilated causal convolutions in order to optimize analysis of the large amount of sequential data in raw sample audio. As explained by A. Van Den Oord et al., “A dilated convolution (also called á trous, or convolution with holes) is a convolution where the filter is applied over an area larger than its length by skipping input values with a certain step” [5].

Under the assumption that the most recent incoming frequency data is the largest determining factor in momentary sound analysis, a dilated spectral buffer could be a useful tool for reducing the size of computational memory.

By dilating each timestep in a new buffer by some ratio (such as an exponential incrementation of 2^t, for example) in proportion to the original buffer, the dimensionality can be drastically reduced while retaining a high resolution of spectral developments at the most recent timesteps. The values of the dilated buffer can either be obtained from the original buffer by simply looking up single values further and further back, but it is also possible to combine the number of timesteps to be collapsed by extracting the average or median spectrum across the duration.

The driving concept behind a dilated spectral buffer is to keep the most recent spectral impressions in memory while also retaining some information about the “big picture” context in an efficient way.

Below is a simplified code snippet for making dilated spectral frames using a Gammatone filterbank. Note that this example uses offline processing, but the filterbank can be applied real-time as well, inserting spectral frames to a circular buffer.

from gammatone import gtgram
import numpy as np

class GammatoneFilterbank:
    # Initialize Gammatone filterbank
    def __init__(self, 
                 sample_rate, 
                 window_time, 
                 hop_time, 
                 num_filters, 
                 cutoff_low):
        self.sample_rate = sample_rate
        self.window_time = window_time
        self.hop_time = hop_time
        self.num_filters = num_filters
        self.cutoff_low = cutoff_low

    # Make a spectrogram from a number of audio samples
    def make_spectrogram(self, audio_samples):
        return gtgram.gtgram(audio_samples,
                             self.sample_rate,
                             self.window_time,
                             self.hop_time,
                             self.num_filters,
                             self.cutoff_low)

    # Divide audio samples into dilated spectral buffers
    def make_dilated_spectral_frames(self, 
                                     audio_samples, 
                                     num_frames, 
                                     dilation_factor):

        spectrogram = self.make_spectrogram(audio_samples)
        spectrogram = np.swapaxes(spectrogram, 0, 1)
        dilated_frames = np.zeros((len(spectrogram), 
                                  num_frames, 
                                  len(spectrogram[0])))

        for i in range(len(spectrogram)):
            for j in range(num_frames):
                dilation = np.power(dilation_factor, j)

                if i - dilation < 0:
                    dilated_frames[i][j] = spectrogram[0]
                else:
                    dilated_frames[i][j] = spectrogram[i - dilation]

        return dilated_frames

Embedding the buffer

In many models of human memory, selective attention is applied after sensory memory as a filter to prevent overload of information in short-term memory [3]. Since humans have limited cognitive resources, it is advantageous to allocate attention to certain auditory sensations to optimize consumption of mental energy.

This process can be approached with an expansion of the autoencoder neural network architecture. Using this architecture, it’s possible to combine sensory memory of sounds with the bottleneck of selective attention by feeding it dilated frequency spectrum buffers for producing embeddings rather than feeding only the momentary frequency information. To cope with sequential information, a special type of architecture called a sequence-to-sequence autoencoder can be used [6].

Sequence-to-sequence (Seq2Seq) models typically use LSTM units to encode a sequence of data (an English sentence, for example) to an internal representation containing a compressed “meaning” about the sequence as a whole. This internal representation can then be decoded back into a sequence (the same sentence, but in Spanish, for example) [7].

A feature of embedding sounds in this way, is that it makes it possible to analyze and process them using simple feed-forward neural networks which are cheaper to run.

After training a network such as the one drawn below, the right half (the decoding part) can be “cut off”, thus generating a network for encoding temporal frequency information into a compressed space. Good results have been achieved in this area of research by Y. Chung et. al. with Audio Word2Vec which succeeded in generating embeddings describing the sequential phonetic structures of vocal recordings by applying a Seq2Seq autoencoder architecture [6]. With greater diversity of input data, it could also be possible to produce embeddings that describe sounds in a more general way.

Listening with Keras

Using the approach described above, we can implement a Seq2Seq autoencoder with Keras to produce audio embeddings. I call this a Listener Network, since its purpose is to “listen” to incoming sequences of sounds and reduce them to a more compact and meaningful representation which we can then analyze and process.

To train this network, ~3 hours of audio from the UrbanSound8K dataset were used. This dataset contains a selection of environmental sound clips divided into categories. The sounds were processed using a Gammatone filterbank and was segmented into dilated spectral buffers of 8 timesteps with 100 spectral filters each.

from keras.models import Model
from keras.layers import Input, LSTM, RepeatVector

def prepare_listener(timesteps,
                     input_dim,
                     latent_dim,
                     optimizer_type,
                     loss_type):
    """Prepares Seq2Seq autoencoder model

        Args:
            :param timesteps: The number of timesteps in sequence
            :param input_dim: The dimensions of the input
            :param latent_dim: The latent dimensionality of LSTM
            :param optimizer_type: The type of optimizer to use
            :param loss_type: The type of loss to use

        Returns:
            Autoencoder model, Encoder model
    """

    inputs = Input(shape=(timesteps, input_dim))

    encoded = LSTM(int(input_dim / 2), 
                   activation="relu", 
                   return_sequences=True)(inputs)

    encoded = LSTM(latent_dim, 
                   activation="relu", 
                   return_sequences=False)(encoded)

    decoded = RepeatVector(timesteps)(encoded)

    decoded = LSTM(int(input_dim / 2), 
                   activation="relu", 
                   return_sequences=True)(decoded)

    decoded = LSTM(input_dim, 
                   return_sequences=True)(decoded)

    autoencoder = Model(inputs, decoded)
    encoder = Model(inputs, encoded)

    autoencoder.compile(optimizer=optimizer_type, 
                        loss=loss_type,  
                        metrics=['acc'])

    return autoencoder, encoder

For my data, this code produces the network architecture below:

This Listener Network was trained using mean squared error and Adagrad optimization for 50 epochs on a NVIDIA GTX 1070 GPU arriving at a reconstruction accuracy of 42%. Training took a while so I stopped early, though progress did not seem to have stagnated yet. I’m very interested to see the performance of such a model with a larger dataset and more computational capacity for training.

There is certainly room for improvement here, but the images below show that the coarse structure of the sequences were captured after compressing the input by a factor of 3.2.

This was the second part of my article series on audio processing with neural networks. In the final article we will put the concepts to use in creating a network for analyzing audio embeddings.

Follow to stay updated and feel free to leave claps if you enjoyed the article!

References

[1] J. J. Eggermont, “Between sound and perception: reviewing the search for a neural code.,” Hear. Res., vol. 157, no. 1–2, pp. 1–42, Jul. 2001.

[2] C. Alain, D. L. Woods, and R. T. Knight, “A distributed cortical network for auditory sensory memory in humans,” Brain Res., vol. 812, no. 1–2, pp. 23–37, Nov. 1998.

[3] A. Wingfield, “Evolution of Models of Working Memory and Cognitive Resources,” Ear Hear., vol. 37, p. 35S–43S, 2016.

[4] “Implementing Circular/Ring Buffer in Embedded C”, Embedjournal.com, 2014. [Online]. Available: https://embedjournal.com/implementing-circular-buffer-embedded-c/.

[5] A. Van Den Oord et al., “Wavenet: A Generative Model for Raw Audio.”

[6] Y.-A. Chung, C.-C. Wu, C.-H. Shen, H.-Y. Lee, and L.-S. Lee, “Audio Word2Vec: Unsupervised Learning of Audio Segment Representations Using Sequence-to-Sequence Autoencoder,” in Proceedings of the Annual Conference of the International Speech Communication Association, 2016, pp. 765–769.

[7] F. Chollet, “A ten-minute introduction to sequence-to-sequence learning in Keras”, Blog.keras.io, 2018. [Online]. Available: https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html.

Applying neural networks in real-time audio signal processing

Significant breakthroughs in AI technology have been achieved through modeling human systems. While artificial neural networks (NNs) are mathematical models which are only loosely coupled with the way actual human neurons function, their application in solving complex and ambiguous real-world problems has been profound. Additionally, modeling the architectural depth of the brain in NNs has opened up broad possibilities in learning more meaningful representations of data.

If you’ve missed out on the other articles, click below to get up to speed:

Background: The promise of AI in audio processing
Criticism: What’s wrong with CNNs and spectrograms for audio processing?
Part 2: Human-Like Machine Hearing With AI (2/3)

In image recognition and processing, the inspiration from the complex and more spatially invariant cells of the visual system in CNNs has also produced great improvements to our technologies. If you’re interested in applying image recognition technologies on audio spectrograms, check out my article “What’s wrong with CNNs and spectrograms for audio processing?”.

As long as human perceptual capacity exceeds that of machines, we stand to gain by understanding the principles of human systems. Humans are very skillful when it comes to perceptual tasks and the contrast between human understanding and the status quo of AI becomes particularly apparent in the area of machine hearing. Considering the benefits reaped from getting inspired by human systems in visual processing, I propose that we stand to gain from a similar process in machine hearing with neural networks.

In this article series, I will detail a framework for real-time audio signal processing with AI which was developed in cooperation with Aarhus University and intelligent loudspeaker manufacturer Dynaudio A/S. Its inspiration is primarily drawn from cognitive science which attempts to combine perspectives of biology, neuroscience, psychology and philosophy to gain greater understanding of our cognitive faculties.

Cognitive Sound Properties

Perhaps the most abstract domain of sound is how we, as humans, perceive it. While a solution for a signal processing problem has to operate within the parameters of intensity, spectral and temporal properties on a low level, the end goal is most often a cognitive one: Transforming a signal in such a way that our perceptions of the sounds it contains are altered.

If one wishes to programatically change the gender of a recorded spoken voice for example, it is necessary to describe this problem in more meaningful terms before defining its lower level characteristics. The gender of a speaker can be conceived as a cognitive property which is constructed from many factors: General pitch and timbre of a voice, differences in pronunciation, differences in choice of words and language and a common understanding of how these properties relate to gender.

These parameters can be described in lower level features like intensity, spectral and temporal properties but only in more complex combinations do they form high-level representations. This forms a hierarchy of audio features from which the “meaning” of a sound can be derived. The cognitive property representing a human voice can be thought of as a combinatory pattern of temporal developments in a sound’s intensity, spectral and statistical properties.

NNs are great at extracting abstracted representations of data and are therefore well suited for the task of detecting cognitive properties in sound. In order to build a system for this purpose, let’s examine how sound is represented in human auditory organs that we can use to inspire representation of sound for processing with NNs.

Cochlear Representation

Hearing in humans starts at the outer ear which firstly consists of the pinna. The pinna acts as a form of spectral preprocessing in which the incoming sound is modified depending on its direction in relation to the listener. Sound then travels through the opening in the pinna into the ear canal which further acts to modify spectral properties of incoming sound by resonating in a way that amplifies frequencies in the range ~1–6 kHz [1].

As sound waves reach the end of the ear canal, they excite the eardrum onto which the ossicles (the smallest bones in the body) are attached. These bones transmit the pressure from the ear canal to the fluid-filled cochlea in the inner ear [1]. The cochlea is of great interest in guiding sound representation for NNs because this is the organ responsible for transducing acoustic vibrations into neural activity in humans .

It is a coiled tube which is separated along its length by two membranes being the Reissner’s membrane and the basilar membrane. Along the length of the cochlea, there is a row of around 3,500 inner hair cells [1]. As pressures enter the cochlea, its two membranes are pushed down. The basilar membrane is narrow and stiff at its base but loose and wide at its apex so that each place along its length responds more intensely at a particular frequency.

To simplify, the basilar membrane can be thought of as a continuous array of bandpass filters which, along the length of the membrane, acts to separate sounds into their spectral components.

This is the primary mechanism by which humans convert sound pressures into neural activity. Therefore, it is reasonable to assume that spectral representations of audio would be beneficial in modeling sound perception with AI. Since frequency responses along the basilar membrane vary exponentially [2], logarithmic frequency representations might prove most efficient. One such representation could be derived using a gammatone filterbank. These filters are commonly applied in modeling spectral filtering in the auditory system since they approximate the impulse response of human auditory filters derived from the measured auditory nerve fiber response to white noise stimuli called the “revcor” function [3].

Since the cochlea has ~3500 inner hair cells and humans can detect gaps in sounds down to ~2–5 ms in length [1], a spectral resolution of 3500 gammatone filters separated into 2 ms windows seem optimal parameters for achieving human-like spectral representation in machines. In practical scenarios however, I assume that lesser resolutions could still achieve desirable effects in most analysis and processing tasks while being more viable from a computational standpoint.

A number of software libraries for auditory analysis are available online. A notable example is the Gammatone Filterbank Toolkit by Jason Heeris. It provides adjustable filters as well as tools for spectrogram-like analysis of audio signals with gammatone filters.

Neural Coding

As neural activity moves from the cochlea onto the auditory nerve and the ascending auditory pathways, a number of processes are applied in brainstem nuclei before it reaches the auditory cortex.

These processes form a neural code which represents an interface between stimulus and perception [4]. Much knowledge about the specific inner workings of these nuclei is still speculative or unknown, so I will detail these nuclei only at their higher levels of functioning.

Humans have a set of these nuclei for each ear that are interconnected, but for simplicity, I’ve illustrated the flow for only one ear. The cochlear nucleus is the first coding step for neural signals coming from the auditory nerve. It consists of a variety of neurons with different properties which serve to perform initial processing of sound features, some of which are directed to the superior olive which is associated with sound localization while others are directed to the lateral lemniscus and inferior colliculus, commonly associated with more advanced features [1].

J. J. Eggermont details this flow of information from the cochlear nucleus in “Between sound and perception: reviewing the search for a neural code” as follows: “The ventral [cochlear nucleus] (VCN) extracts and enhances the frequency and timing information that is multiplexed in the firing patterns of the [auditory nerve] fibers, and distributes the results via two main pathways: the sound localization path and the sound identification path. The anterior part of the VCN (AVCN) mainly serves the sound localization aspects and its two types of bushy cells provide input to the superior olivary complex (SOC), where interaural time differences (ITDs) and level differences (ILDs) are mapped for each frequency separately” [4].

The information carried by the sound identification pathway is a representation of complex spectra such as vowels. This representation is mainly created in the ventral cochlear nucleus by special types of units dubbed “chopper” (stellate) neurons [4]. The details of these auditory encodings are difficult to specify but they indicate to us that a form of “coding” of incoming frequency spectra could improve understanding of low level sound features as well as making sound impressions less expensive to process in NNs.

Spectral Sound Embeddings

We can apply the unsupervised autoencoder NN architecture as an attempt to learn common properties associated with complex spectra. Like word embeddings, its possible to find commonalities in frequency spectra that represent select features (or a more tightly condensed meaning) of sounds.

An autoencoder is trained to encode an input into a compressed representation that can be reconstructed back into a representation with a high similarity to the input. This means that the autoencoder’s target output is the input itself [5]. If an input can be reconstructed without great loss, the network has learnt to encode it in such a way that the compressed internal representation contains enough meaningful information. This internal representation is then what we refer to as the embedding. The encoding part of the autoencoder can be decoupled from the decoder to generate embeddings for other applications.

Embeddings also have the benefit that they are often of lower dimensionality than the original data. For instance, an autoencoder could compress a frequency spectrum with a total of 3500 values into a vector with a length of 500 values. Put simply, each value of such a vector could describe higher level factors of a spectrum such as vowel, harshness or harmonicity - These are only examples, as the meaning of statistically common factors derived by an autoencoder might often be difficult to label in plain language.

In the next article, we will expand upon this idea with added memory to produce embeddings for temporal developments of audio frequency spectra.

This wraps up the first part of my article series on audio processing with artificial intelligence. Next, we will discuss the essential concepts of sensory memory and temporal dependencies in sound.

Follow to stay updated and feel free to leave claps if you enjoyed the article!

References

[1] C. J. Plack, The Sense of Hearing, 2nd ed. Psychology Press, 2014.

[2] S. J. Elliott and C. A. Shera, “The cochlea as a smart structure,” Smart Mater. Struct., vol. 21, no. 6, p. 64001, Jun. 2012.

[3] A.M. Darling, “Properties and implementation of the gammatone filter: A tutorial”, Speech hearing and language, University College London, 1991.

[4] J. J. Eggermont, “Between sound and perception: reviewing the search for a neural code.,” Hear. Res., vol. 157, no. 1–2, pp. 1–42, Jul. 2001.

[5] T. P. Lillicrap et al., Learning Deep Architectures for AI, vol. 2, no. 1. 2015.

In recent years, great results have been achieved in generating and processing images with neural networks. This can partly be attributed to the great performance of deep CNNs to capture and transform high-level information in images. A notable example of this is the process of image style transfer using CNNs proposed by L. Gatys et. al. which can render semantic content of an image in a different style [1].

The process of neural style transfer is well explained by Y. Li et. al: “this method used Gram matrices of the neural activations from different layers of a CNN to represent the artistic style of a image. Then it used an iterative optimization method to generate a new image from white noise by matching the neural activations with the content image and the Gram matrices with the style image” [2].

In simpler terms, these results can be thought of as being achieved by generating images according to combinations of features from source content and style images at different levels of abstraction. As an example, this could be maintaining the high level structures and contours of the content image while incorporating colors and lower level texture features of the style image.

The performance of style transfer in the realm of visual processing has been quite impressive and lends itself to optimism for “smarter” audio processing algorithms if similar results can be achieved. Since spectrograms are two-dimensional representations of audio frequency spectra over time, attempts have been made in analyzing and processing them with CNNs. It has been shown, that it is possible to process spectrograms as images and perform neural style transfer with CNNs [3] but, so far, the results have not been nearly as compelling as they have been for visual images [4].

To overcome this challenge and to produce better results in neural audio processing we might then need to consider why style transfer with CNNs do not perform as well on spectrograms. Essentially, these techniques apply machine vision in order to do machine hearing. I believe this poses an essential problem which could be hindering the progress of AI-assisted technologies in audio processing. While the problem can undoubtedly be approached from many angles, it might be worthwhile exploring the differences between images and spectrograms and, as a consequence, some of the differences between seeing and hearing.

Sounds are “transparent”

One challenge posed in the comparison between visual images and spectrograms is the fact that visual objects and sound events do not accumulate in the same manner. To use a visual analogy, one could say that sounds are always “transparent” [4] whereas most visual objects are opaque.

When encountering a pixel of a certain color in an image, it can most often be assumed to belong to a single object. Discrete sound events do not separate into layers on a spectrogram: Instead, they all sum together into a distinct whole. That means that a particular observed frequency in a spectrogram cannot be assumed to belong to a single sound as the magnitude of that frequency could have been produced by any number of accumulated sounds or even by the complex interactions between sound waves such as phase cancellation. This makes it difficult to separate simultaneous sounds in spectrogram representations.

The axes of spectrograms do not carry the same meaning

CNNs for images use two-dimensional filters that share weights across the x and y dimensions [4]. As earlier described, this builds on the assumption that features of an image carry the same meaning regardless of their location. For this to be true, you should also assume that the x and y axes of the data have the same implications to the meaning of the content. For example, a face is still a face regardless of whether it is moved horizontally or vertically in an image.

In spectrograms, the two dimensions represent fundamentally different units, one being strength of frequency and the other being time. Moving a sound event horizontally offsets its position in time and it can be argued that a sound event means the same thing regardless of when it happens. However, moving a sound vertically might influence its meaning: Moving the frequencies of a male voice upwards could change its meaning from man to child or goblin, for example. Performing a frequency shifts of a sound event can also changes its spatial extent [4]. Therefore, the spatial invariance that 2D CNNs provide might not perform as well for this form of data.

The spectral properties of sounds are non-local

In images, similar neighboring pixels can often be assumed to belong to the same visual object but in sound, frequencies are most often non-locally distributed on the spectrogram [4]. Periodic sounds are typically comprised of a fundamental frequency and a number of harmonics which are spaced apart by relationships dictated by the source of the sound. It is the mixture of these harmonics that determines the timbre of the sound.

In the instance of a female vocal, the fundamental frequency at a moment in time might be 200Hz while the first harmonic is 400Hz, the next 600Hz and so on. These frequencies are not locally grouped but they move together according to a common relationship. This further complicates the task of finding local features in spectrograms using 2D convolutions as they are often unevenly spaced apart even though they move according to the same factors.

Sound is inherently serial

When assessing a visual environment, we can “scan” our surroundings multiple times to locate each visual object in a scene. Since most objects are non-moving, light will reflect from them in a predictable manner and one can make a mental map of their placement in a physical scene. From a perceptual point of view then, the visual objects are assumed to continue to exist at their observed location, even when you look elsewhere.

This is not true for sounds. Sound takes the physical form of pressure waves and, from the point of view of a listener, such waves exist only in their current state at one moment in time. Once the moment has passed, the wave has passed by, traveling away from the observer. This is why it makes sense to refer to these phenomena as sound events rather than objects. From a physical perspective, this means that listeners experience sound only a moment at a time. Where images can be regarded to contain larger amounts of static parallel information, sound, then, is highly serial.

A more fitting comparison is one between audio and video. Both these media can be conceptualized as depicting movements over time where dependencies across time are essential to the experienced meaning of the content. Since video is constructed from collections of images (frames), it contains much more parallel information.

One way to illustrate this is to “freeze” a moment of time in both media. Looking at a single frame of a video (often depicting ~1/25 seconds of light exposure) it is still often possible to gather a significant amount of meaning about the context, actions and scene of the video: Individual objects can be identified and, sometimes, actions and movements can be assessed. When “freezing” a single moment of audio (such as a corresponding aggregate of ~1/25 seconds) with spectral analysis however, assessments cannot be nearly as comprehensive. Some context about the overall tonal balance and characteristics of the signal can be gathered, but not nearly to the same degree as for video.

For example, it is not possible to identify separate sound events outside the context of time to see which spectral developments happen according to the same temporal patterns. The only thing that can be established for certain is the tonal balance of the heard sound(s) at that specific moment in time. An explanation for this comes back to the previously discussed physical form of sound as waves: Sounds do not exist as static objects which can be observed in parallel, they arrive as sequences of air pressure and meaning about these pressures must be established over time.

These reasons suggest that audio as a medium for conveying meaning is fundamentally serial and more temporally dependent than video which presents another reason why visual spectrogram representations of sounds fed into image processing networks without temporal awareness might not work optimally.

A case for modeling the human experience

Significant breakthroughs in AI technology have been achieved through modeling human systems. While artificial neural networks are mathematical models which are only loosely coupled with the way actual human neurons function, their application in solving complex and ambiguous real-world problems has been profound. Modeling the architectural depth of the brain in these neural networks has opened up broad possibilities in learning more meaningful representations of data. In image recognition and processing, the inspiration from the complex and more spatially invariant cells of the visual system in CNNs have also produced great improvements to the state of our technologies.

As argued by J. B. Allen in “How Do Humans Process and Recognize Speech?”, as long as human perceptual capacity exceeds that of machines, we stand to gain by understanding the principles of human systems [5]. Humans are generally very skillful when it comes to perceptual tasks and the contrast between human understanding and the status quo of AI becomes particularly apparent in the area of machine hearing. Considering the benefits reaped from getting inspired by human systems in visual processing (and the arguments presented that visual models do not perform as well for sound), I propose that we stand to gain from a similar process in machine hearing with neural networks.

This is part of a bigger machine hearing project. If you’ve missed out on the other articles, click below to get up to speed:

Background: The promise of AI in audio processing
Part 1: Human-Like Machine Hearing With AI (1/3)
Part 2: Human-Like Machine Hearing With AI (2/3)
Part 3: Human-Like Machine Hearing With AI (3/3)

References:

[1] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image Style Transfer Using Convolutional Neural Networks,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 2414–2423.

[2] Y. Li, N. Wang, J. Liu, and X. Hou, “Demystifying Neural Style Transfer,” Jan. 2017.

[3] P. Verma and J. O. Smith, “Neural Style Transfer for Audio Spectrograms,” Jan. 2018.

[4] L. Wyse. 2017. Audio Spectrogram Representations for Processing with Convolutional Neural Networks. Proceedings of the First International Workshop on Deep Learning and Music joint with IJCNN. Anchorage, US. May, 2017. 1(1). pp 37–41. DOI: 10.13140/RG.2.2.22227.99364/1

[5] J. B. Allen, “How Do Humans Process and Recognize Speech?,” IEEE Trans. Speech Audio Process., vol. 2, no. 4, pp. 567–577, 1994.

2017 has been a good year for AI, deep learning in particular. We have seen a rise of AI technologies for image and video processing. Even though things tend to take a little while longer making it to the world of audio, here we have also seen impressive technological advances.

In this article, I will summarize some of these advances, outline further potentials of AI in audio processing as well as describe some of the possible pitfalls and challenges we might encounter in pursuing this cause.

Towards smarter audio

The kicker for my interest in AI use cases for audio processing was the publication of Google Deepmind’s “WaveNet” — A deep learning model for generating audio recordings [1] which was released during the end of 2016. Using an adapted network architecture, a dilated convolutional neural network, Deepmind researchers succeeded in generating very convincing text-to-speech and some interesting music-like recordings trained from classical piano recordings.

In the commercial world, we have also seen more applications of machine learning in products — Take for example LANDR, an automated audio mastering service which relies on AI to set parameters for digital audio processing and refinement.

This year, pro audio software mogul iZotope released Neutron 2, an audio mixing tool that features a “track assistant” which utilizes AI to detect instruments and suggest fitting presets to the user. In more direct processing of audio with AI, iZotope also featured a utility for isolating dialogue in their audio restoration suite RX 6.

Potentials of AI in digital signal processing

We are still in the early days of AI application in audio processing. Deep learning methods allow us to approach signal processing problems from a new perspective which is still largely ignored in the audio industry. Up to this point, we have had our focus on formulaic treatment: Coming to a deep understanding of a problem and manually devising functions to solve it. However, the understanding of sound is a very complex task and problems that we, as humans, intuitively find quite easy often turn out to be very difficult to describe formulaically.

Take, for instance, source separation: You find yourself in a scenario where two people are speaking over each other. After the fact, in your mind, you can imagine either person speaking in isolation without much effort. But how do we describe a formula for separating these two voices? Well, it depends:

Is there a unified way to describe how human voices sound? If yes, how are parameters of this description affected by sex, age, energy, personality? How does physical proximity to the listener and room acoustics impact this understanding? What about non-human noise that can occur during the recording? On which parameters can we discriminate one voice over another?

As you can see, devising a formula for the full extent of this problem would require attention to a lot of parameters. Here, AI can provide a more pragmatic approach — By setting up the proper conditions for learning, we can statistically estimate the complexities of this function automatically. In fact, researchers at Eriksholm (a research center for the hearing aid manufacturer Oticon) recently proposed a method for achieving improved source separation in real-time applications using a convolutional recurrent neural network architecture [2].

As methods for processing audio with deep neural networks are improving, we can only begin to imagine the difficult problems we could solve — Here are a few of my imaginations for deep learning in real-time audio processing:

• Selective noise canceling, removing only certain elements like car traffic

• Hi-fi audio reconstruction, from small and low-quality microphones

• Analog audio emulation, estimating the complex interactions between non-linear analog audio components

• Speech processing, changing speaker, dialect or language in recordings

• Improved spatial simulations, for reverb and binaural processing

Challenges in representation and architecture

WaveNet was among the first successful attempts to generate audio on a raw sample level. The one big problem here is that CD quality audio is usually stored with 44100 samples per second and thus, generating seconds of sound with WaveNet takes hours. This excludes the method from having use in real-time applications. It’s just a lot of data to make sense of.

On the other hand, many current solutions for audio processing with neural networks utilize spectrogram representations and convolutional networks. In these solutions, the audio frequency spectrum is essentially represented visually as magnitudes over time on a 2D image and the convolutional network is used to scan and process that image [3]. Usually, results from these methods are not nearly as compelling as those we see in the visual field such as CycleGAN that can do some impressive style transfer for movies [4].

Movies and audio clips have something in common in the sense that they both depict movements over time. Considering innovations in image processing networks like CycleGAN, one would assume that such style transfer could be possible for audio as well.

But movies and audio clips are not the same things — If you freeze a frame of a movie, quite a lot of information can still be gathered about the actions in the frame. If you freeze a “frame” of audio, however, very little meaning can be gathered about what is actually being heard. This suggests that audio is fundamentally more temporally dependent than movies. In spectrograms, you can never assume that a pixel belongs to a single object either: Audio is always “transparent”, and therefore spectrograms display all audible sounds overlapping each other in the same frame [3].

We are doing machine vision to do machine hearing.

Convolutional neural networks are designed with inspiration from the human visual system, loosely based on how information flows into the visual cortex [5]. I believe this presents a problem worth paying consideration to. Essentially, we are taking audio, translating it to images and performing visual processing on that image before translating it back to audio. So, we are doing machine vision to do machine hearing. But, as we are intuitively aware, these two senses don’t function in the same way. Looking at the spectrogram below, how much meaning can you (with your smart human brain) actually gather about the contents of the audio? If you could listen to it, you would quickly get an intuitive understanding of what is happening. Maybe this is a problem that is hindering our progress in AI-assisted technologies for audio.

Therefore I propose that to achieve better results with neural networks for audio processing, we should allocate energy to figuring out better representations and neural network architectures for audio specifically. One such representation could be the autocorrelogram, a three-dimensional representation of sound where time, frequency and periodicity is included [6]. It turns out that humans can separate sound sources by intuitively comparing the periodicity of sounds to find patterns of similarity. Pitch and rhythm are also results of temporal factors. Therefore more temporally focused representations, such as the autocorrelogram, might be useful.

Furthermore, we could start thinking about architecturally modeling the neural pathways of the auditory system. When sound excites our eardrums and travels into the cochlea, it is transformed into magnitudes for different frequencies. Then it travels to the central auditory system for temporal pattern processing. Which modes of analysis do we utilize to gather meaning from audio in our central auditory system that can be modeled in an artificial neural network? Periodicity, maybe [6]. Statistical groupings of sound events, maybe [7]. Dilated time frames of analysis, maybe [1].

Conclusion

Developments in AI presents a big potential for smarter audio signal processing. But to better understand sound in neural networks, we might need to part ways with perspectives that are inherently visual and consider new techniques based instead on the auditory system.

In this article, I have posed more questions than I have provided answers, hoping to kick-start your thought about sound in this context.

This is part of a bigger machine hearing project. If you’ve missed out on the other articles, click below to get up to speed:

Criticism: What’s wrong with CNNs and spectrograms for audio processing?
Part 1: Human-Like Machine Hearing With AI (1/3)
Part 2: Human-Like Machine Hearing With AI (2/3)
Part 3: Human-Like Machine Hearing With AI (3/3)

Sources

[1] A. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, K. Kavukcuoglu: WaveNet: A Generative Model for Raw Audio, 2016

[2] G. Naithani, T. Barker, G. Parascandolo, L. Bramsløw, N. Pontoppidan, T- Virtanan: Low Latency Sound Source Separation Using Convolutional Recurrent Neural Networks, 2017

[3] L. Wyse: Audio spectrogram representations for processing with Convolutional Neural Networks, 2017

[4] J. Zhu, T. Park, P. Isola, A. Efros: Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, 2017

[5] Y. Bengio: Learning Deep Architectures for AI (p. 44), 2009

[6] M. Slaney, R. Lyon: On the importance of time — A temporal representation of sound, 1993

[7] E. Piazza, T. Sweeny, D. Wessel, M. Silver, D. Whitney: Humans Use Summary Statistics to Perceive Auditory Sequences, 2013