API Testing and API Monitoring: The Complete Guide
API testing and API monitoring are two essential processes in any company that runs a public API program or maintains internal APIs at scale. Both are part of the software quality assurance (QA) stage. They help API providers ensure that their APIs are doing what they’re supposed to do and fulfilling API consumers' expectations. In this article, I would like to explain the difference between testing and monitoring when it comes to APIs and make the case for why you should have a strategy for both as part of your development cycle.
What is API Testing?
In quality assurance, verification is the assertion that a product matches its specification. For APIs, especially for functional testing, verification means that the API fulfills the contract specified by its OpenAPI definition. If the API provider doesn't use OpenAPI (which they really should!), the contract can be a dedicated test specification. In other words, API testing is making sure the APIs function like they are supposed to and can be relied on. API performance testing ensures that the API still works even if there’s a certain amount of load.
Why You Should Run API Tests
The goal of API Testing is to provide quick and actionable feedback to developers. With API testing, you can make sure that you don’t ship failing software to customers or break things for your development team. API tests have clearly defined "pass" and "fail" criteria, and without passing, software delivery should not continue. For example, a developer should not commit failing code to the repository. A failing build should not go into the staging environment. If the API doesn't work correctly, i.e., pass all tests, in staging, it should not enter the production environment.
By running performance tests in addition to functional tests, you can get feedback on how code changes affect the overall API's performance.
How To Run API tests
API tests can run in any environment, for example, on the developer's computer or in a staging environment in the cloud. API testing should be a part of your continuous delivery or integration pipeline. Any change to the implementation or any deployment triggers a test, so the tests run on demand.
Functional API tests: You can run many functional tests locally or in the build pipeline. To test the real-world conditions involved with network connectivity, you can run tests from cloud-based tools like BlazeMeter against your remotely accessible staging environment, too.
Performance API tests: Performance tests can be one-off tests that you run manually as part of delivery milestones. They can also be triggered on-demand as part of a build pipeline or run on a schedule. Whether you run them on every commit or on a regular schedule depends on how often you commit and deploy. Of course, you want performance tests to reflect the behavior in the production environment. Still, most performance tests are better running in a staging environment that you modeled on the production environment.
The reason is that many variants of performance tests, such as stress testing, soak testing, peak testing, and spike testing, involve putting a strain on the API that may cause it to break. Your hundreds of real users will not be happy if you degrade their performance with VUs for the sake of finding out whether the API would support thousands of users.
Now that we’ve covered API testing, let’s look at API monitoring.
What is API Monitoring?
API monitoring is a way to ensure that the API fulfills the performance criteria and the business requirements, and that it does so consistently. API monitoring is not about "pass" and "fail”, but rather about various metrics and KPIs that you want to observe. It can also generate insights over time as you see specific metrics change.
Why You Should Monitor APIs
By implementing API monitoring, you can keep an eye on your production environment, catch issues before your customers do, and make sure you can deliver on the performance you promised. If you specify requirements in a service level agreement (SLA) for customers, API monitoring can warn you when you’re about to break that. For example, you might guarantee an uptime and an error rate of less than 0.1%, acknowledging that things can go wrong, but they go right 99.9% of the time. By looking at the error rate as a metric in your API monitoring, you can ensure that you fulfill the requirement.
How to Monitor APIs
For monitoring your deployed API, you should regularly run tests so that you can catch problems quickly and early. Tests can run as often as once every minute. You can think of the API monitor behaving like a regular user that consistently accesses different functionalities in your API, not a user who's trying to test the limits of your API or your infrastructure (like the VUs in a performance test).
The basic version of API monitoring is uptime monitoring, where you just ensure that the server responds, and the HTTP status code is 200 ("OK"). You can also monitor various endpoints and compare the responses against expected schemas or even set up workflows with multiple API requests, but you don't need to run the most sophisticated tests every time.
When you deploy an API or other software, you can look at metrics like the CPU, memory, disk, and network utilization of your servers. You are also likely to have some logging mechanism that writes errors or other noteworthy events into log files or sends them to a logging service so you can get insights on your production performance. Application Performance Monitoring (APM) systems unify these different metrics. However, they only provide a view of your API from the inside.
API monitoring with cloud-based tools centered on tests like BlazeMeter adds the outside view because they generate traffic from different locations outside your server. Through several pre built integrations and webhook capabilities, you can send results from BlazeMeter to your APM system to combine the inside and outside perspectives and derive a holistic view of your API's performance.
The relevant area to set up monitoring is your production environment. Of course, you can set up API monitoring on the staging environment as a courtesy for your development team, but monitoring is primarily about live usage.
Benefits of Testing and Monitoring APIs
API testing and API monitoring both play a role in the API lifecycle. Following my attempt to explain their differences in this article, one conclusion is that they relate, at least broadly, to different parts of the API lifecycle. API testing is central to building an API, while API monitoring is central to running an API. Or, if you want to put it that way, the Dev and Ops of DevOps.
A successful API program requires that you consider every step of the API lifecycle. You need API testing because a bug-ridden API provides little value even if it’s always available and runs fast. Also, your development team appreciates if things don’t break and they need to debug what their colleagues did when an automated test would have caught it. On the other hand, a perfectly tested API, even tested for performance, provides no business value if it breaks in production regularly and is unreliable. Therefore, both API Testing and API Monitoring is essential.
Both, however, require planning an individual approach suitable for your company and the resources you can dedicate to it. With BlazeMeter, you have an integrated solution where you can access tools for both through a single account.
BlazeMeter API Testing and Monitoring
BlazeMeter provides solutions for API testing and monitoring through a single account and separates its functionality into four dedicated areas: functional testing, performance testing, mock services, and monitoring. Sign up now to start testing and monitoring your APIs.