How to choose a load testing tool: self-hosted or SaaS, code or no-code?
Load testing tools fall into four broad groups: code-based open-source tools, desktop test plan tools, licensed enterprise suites and SaaS cloud services. Which one fits depends on who writes the tests, where the load comes from and where the test data has to stay. This page compares the groups without naming products, says where Spitfire fits, and where another approach is the better choice.
Six questions to answer before choosing
- Who will write the tests? Only developers, or testers, operations and product people too? A tool that needs a scripting language stays with the people who write code.
- Where are the systems under test? Internal services, databases and message queues are closed to load from the cloud; the load has to come from inside that network.
- May test data leave the network? In banking, government and healthcare, request bodies, tokens and results often may not leave the organization (data protection rules, internal audit).
- Which protocols do you need? Only HTTP, or gRPC, WebSocket, Kafka, SQL, MQTT, AMQP, Redis or MongoDB too? A protocol that comes as a plugin brings setup and upkeep with it.
- How much load, from how many places? Load beyond one machine, or traffic from other cities, needs distributed runs whose results merge in one place.
- What is the total cost? Next to the license, count the servers, setup, training and the team's time keeping the tool running (cost calculation).
The kinds of tools side by side
The table shows each group's typical shape; tools within a group differ. Check the details of the tool you pick in its own documentation.
| Kind of tool | Who writes the test | Where the load comes from | Where the data stays | Distributed runs | Reports and sharing | License model |
|---|---|---|---|---|---|---|
| Code-based open-source tools | Developers, in a scripting language | Your machine or your servers | With you | Usually by hand or with an add-on | Usually a separate dashboard is set up | Free; some features in a paid cloud service |
| Desktop / GUI test plan tools | Testers, in a desktop application | Your machine or your servers | With you | Possible; setup and merging results by hand | File-based reports | Usually free |
| Licensed enterprise suites | Testers, by recording and scripting | Your infrastructure | With you | Yes | Extensive | Often per virtual user and protocol; training and setup services often extra |
| SaaS cloud load testing | Developers or testers, in the browser | The provider's cloud regions; internal systems are out of reach without an extra agent | With the provider: test definitions and results leave your network | Yes | Ready dashboards | Usually a subscription, by virtual user hours or test count |
| Spitfire | A no-code web editor; anyone on the team | Your runners: Docker, Kubernetes, Windows, remote locations | With you: the controller, the database and the results on your servers | Built in: runners dial out, results merge into one run | Live charts in the web UI, PDF reports, run comparison | Per plan; the license is verified offline |
Self-hosted or SaaS?
SaaS needs no setup and is a quick start for testing a public website from several regions. The price is that data and access sit with the provider: test definitions, tokens and results leave your network, and internal systems still need an agent you install. With a self-hosted tool the load generators sit next to the system under test, internal services stay off the internet and the data stays in the organization. The price is running a server and keeping it up to date. Spitfire brings that down to a one-command install, signed updates and runners that dial out (self-hosted load testing).
Code or no-code?
Code-based tools suit developer teams who want tests in the same repository as the application, under the same review process. The catch is that a test stays with the people who know the scripting language: testers, operations or product people wait for a developer to change it. In a no-code editor steps, extraction, checks and thresholds are set up with forms, and anyone can read and change a test. In Spitfire the test is still stored as JSON, can be versioned and runs from the command line in CI (performance testing in CI/CD). Generating a test from an OpenAPI document, a Postman collection or a HAR file also gets the first test done in minutes (load testing with Postman and OpenAPI).
When another approach is better
- You want tests to be code. If everyone on the team writes code and you want tests written and reviewed with the application, a code-based tool feels more natural.
- You need a protocol Spitfire lacks. For mainframe applications or custom binary protocols, pick a tool that speaks that protocol. Spitfire's protocols are listed on the how it works page.
- You only test a public site now and then. If you do not want a server, do not mind data leaving the network and do not test internal systems, a SaaS service is less work.
If you already have tests
Scripts and test plans written in other tools become Spitfire tests with Tests → Open file; the migration guide says what comes over by itself and what to finish by hand. You can try it on your own system with the free edition before deciding; plans and prices are on the pricing page.
Frequently asked questions
What is the best load testing tool?Getting started
What is the difference between self-hosted and SaaS load testing?Getting started
Can you load test without writing code?Tests
How should you compare the cost of load testing tools?Licensing and pricing
Spitfire installs on Docker or Kubernetes with one command; every testing feature and protocol is open in the free edition.