Earlier, Faster, Better, Stronger: Catch Breaking Changes by Diffing API Behavior

If you work on web apps, you may sometimes feel that your tools have forsaken you. And you’re not wrong. Because of the way modern tech stacks are set up, it’s getting harder and harder to make sure your code is doing what it’s supposed to.

In this blog post, we’ll show you a bug that slips through the cracks today, explain why it’s nobody’s fault, and show you how we diff API behaviors to catch bugs (while generating API specs along the way!!).

😱 The Bug That (Almost) Ruined A Friendship

You work on a web app. Your co-worker and #bff, Aki, opened a pull request to remove a property from a commonly used API. You two are tight, so you jump in to review the PR. Aki’s PR includes a link showing that this property’s usage has been removed from the website, once again proving why Aki is your #bff. You +1 the PR enthusiastically, with both of you believing that Aki has taken into account all of the dependencies.

The change gets merged. Then: BETRAYAL! There are now outages in the mobile clients, and it’s late nights for you and Aki. (Good thing you started out liking Aki so much.) It turns out that Aki’s API had been so good that the mobile APIs had started depending on the removed property, unbeknownst to both you and Aki.

If this has happened to you, you are not to blame! These bugs are nasty little things: they have to do with dependencies on the code you wrote rather than the code itself, making it hard to catch them through source diffs. In fact, as soon as bugs cross API boundaries, they become extremely hard to catch because they elude common testing and software analysis (think static and dynamic analysis) solutions. Developers are left to rely on documentation, word of mouth, and crossing their fingers really hard.

⚡️ Find Bugs by Diffing API Behavior

To catch these kinds of bugs, you can simply install Akita to analyze API-impacting changes whenever you make a pull request. Here is an example of a comment that Akita leaves on your pull request:

Akita enables you to compare API behavior across pull requests, multiple environments (ex production vs. test), and even pre-existing specs. If Aki had used Akita to compare the API behavior between their production and test environments, they surely would have caught the bug and saved you some late-night debugging!

By diffing on behaviors rather than code, Akita summarizes how a pull request changes your API, including:

• Endpoints added that expose new functionality
• Removed/changed parameters that may break existing clients
• Simple type changes (ex string to int) that can pollute data pipelines
• Complex type changes (ex phone number to datetime) that can break dependencies
• 👷🏻‍♀️ Coming soon: Impacted Clients, telling you exactly which mobile and web clients are affected

Just this week, we released our GitHub integration to deliver insight on every pull request. You can now use Akita without changing any code or config files, without having to proxy—and as part of your normal developer workflow.

Akita does this by showing you semantic diffs between implicit API contracts. In other words, this means Akita does some magic to figure out what your API normally does and then diffs on that, instead of diffing syntactically on the source code. Diffing on observed behaviors, rather than the code itself or a static graph of what APIs could call each other, allows Akita’s reports to pinpoint potential issues much more precisely. Akita is flexible enough to run in both CI, as shown here, and in production environments!

⚒ Under the Hood: Advanced API Analysis

You may be wondering how we do this, since no existing linter, static analysis, or dynamic analysis gets anywhere close. Under the hood, Akita analyzes API traffic to build a model of your API from scratch. As a foundation, Akita builds an API spec for your API. Here we show an OpenAPI spec that Akita automatically generated:

The next step is where the magic comes from. Akita uses advanced programming languages technology to detect not just the basic spec properties, but also implicit API contracts, for instance, specific types like datetime, email, phone number, and more:

All you have to do is set up Akita to watch your API traffic. No code changes and no proxying necessary:

👀 What’s Next?

We’ve just released the spec viewer and GitHub integration in our private beta and would love to have you try out our spec generation and/or semantic API behavior diffs. Here’s how you can help:

• Sign up for our private beta if you’re interested in trying things out!
• You may also be interested in this talk and demo we gave at the API Specs Conference last week.
• Spread the word about us! We’d love all the feedback we can get!