Last updated: 2026-09-23
Testing Fundamentals
Testing shows the presence, not the absence of bugs." — Edsger W. Dijkstra 1
Testing Pyramid (Revisited)
| Layer | Purpose | Tools | Target Coverage |
|---|---|---|---|
| Property-based | Discover edge cases automatically | Hypothesis (Py), fast-check (JS), jqwik (Java) | Core algorithms |
| Mutation | Assess test suite quality | Pitest, mutmut, stryker | Critical paths |
| Contract | Consumer-driven API verification | Pact, Spring Cloud Contract | Service boundaries |
| Unit | Fast feedback on logic | pytest, JUnit, Vitest | >90% on domain logic |
| Integration | Verify component wiring | Testcontainers, WireMock | Happy paths + failures |
| E2E | Critical user journeys | Playwright, Cypress | <10 critical flows |
Test Categories Deep Dive
1. Unit Tests — Fast, Isolated, Deterministic
# Good: Single behavior, no I/O, descriptive name
def test_calculate_discount_applies_tiered_rates():
customer = Customer(tier="gold", years_active=3)
order = Order(total=1000, items=5)
discount = PricingEngine().calculate(customer, order)
assert discount == 150 # 10% base + 5% loyalty
# Bad: Multiple assertions, external dependency, flaky
def test_user_registration():
response = requests.post("/api/users", json={...})
assert response.status_code == 201
assert User.exists(email="...")
assert sent_welcome_email()
// Good: Single behavior, no I/O, descriptive name
TEST(PricingEngineTest, AppliesTieredRatesForGoldCustomer) {
Customer customer("gold", 3);
Order order(1000, 5);
Money discount = PricingEngine().calculate(customer, order);
ASSERT_EQ(Money(150), discount);
}
// Bad: Multiple assertions, external dependency, flaky
TEST(UserRegistrationTest, BadExample) {
auto response = Post("/api/users", json{...});
ASSERT_EQ(201, response.status_code());
ASSERT_TRUE(User::Exists("alice@example.com"));
ASSERT_TRUE(EmailSent("welcome"));
}
// Good: Single behavior, no I/O, descriptive name
@Test
void testCalculateDiscountAppliesTieredRates() {
Customer customer = new Customer("gold", 3);
Order order = new Order(1000, 5);
Money discount = pricingEngine.calculate(customer, order);
assertEquals(150, discount.getCents());
}
// Bad: Multiple assertions, external dependency, flaky
@Test
void testUserRegistration() {
var response = post("/api/users", json);
assertEquals(201, response.getStatusCode());
assertTrue(User.exists("alice@example.com"));
assertTrue(Email.wasSent("welcome"));
}
// Good: Single behavior, no I/O, descriptive name
[Fact]
public void CalculateDiscount_AppliesTieredRates_ForGoldCustomer() {
var customer = new Customer("gold", 3);
var order = new Order(1000, 5);
var discount = _pricingEngine.Calculate(customer, order);
Assert.Equal(new Money(150), discount);
}
// Bad: Multiple assertions, external dependency, flaky
[Fact]
public void UserRegistration_BadExample() {
var response = _client.Post("/api/users", json);
Assert.Equal(201, response.StatusCode);
Assert.True(User.Exists("alice@example.com"));
Assert.True(Email.WasSent("welcome"));
}
# Good: Single behaviour, no I/O, descriptive name
RSpec.describe PricingEngine do
it "applies tiered rates when calculating discounts" do
customer = Customer.new(tier: "gold", years_active: 3)
order = Order.new(total: 1000, items: 5)
discount = described_class.new.calculate(customer, order)
expect(discount).to eq(150) # 10% base + 5% loyalty
end
end
# Bad: Multiple assertions, external dependency, flaky
RSpec.describe "user registration" do
it "registers a user" do
response = HTTP.post("/api/users", json: { name: "..." })
expect(response.status).to eq(201)
expect(User.exists?(email: "...")).to be(true)
expect(sent_welcome_email?).to be(true)
end
end
| Principle | Practice |
|---|---|
| F.I.R.S.T. | Fast, Independent, Repeatable, Self-validating, Timely |
| AAA Pattern | Arrange → Act → Assert |
| One logical assert | Multiple physical assertions OK if single concept |
| No I/O | Mock filesystem, network, time, randomness |
Mocking the clock and the RNG buys determinism, but it isn't the whole story once randomness, unpredictable timing, or an LLM call is the thing under test rather than an incidental dependency to route around — see Testing Non-Deterministic Systems for that fuller treatment, and Test Doubles for the difference between a mock, a stub, and a fake this section's "mock" glosses over.
2. Property-Based Testing — Discover the Unknown
# Instead of hand-picked examples, generate thousands
from hypothesis import given, strategies as st
@given(st.lists(st.integers(min_value=0, max_value=100)))
def test_sort_is_idempotent(xs):
assert sorted(xs) == sorted(sorted(xs))
@given(st.text(alphabet=st.characters(blacklist_categories=("Cc", "Cs"))))
def test_escaped_string_roundtrip(s):
assert unescape(escape(s)) == s
#include <rapidcheck/rapidcheck.h>
void test_sort_is_idempotent() {
rc::check("sort is idempotent", [](const std::vector<int>& xs) {
auto sorted_once = xs;
std::sort(sorted_once.begin(), sorted_once.end());
auto sorted_twice = sorted_once;
std::sort(sorted_twice.begin(), sorted_twice.end());
RC_ASSERT(sorted_once == sorted_twice);
});
}
#include <rapidcheck/rapidcheck.h>
void test_escaped_string_roundtrip() {
rc::check("escape/unescape roundtrip", [](const std::string& s) {
RC_ASSERT(unescape(escape(s)) == s);
});
}
import net.jqwik.api.*;
import java.util.*;
class PropertyTests {
@Property
boolean sortIsIdempotent(@ForAll List<Integer> xs) {
var sortedOnce = new ArrayList<>(xs);
Collections.sort(sortedOnce);
var sortedTwice = new ArrayList<>(sortedOnce);
Collections.sort(sortedTwice);
return sortedOnce.equals(sortedTwice);
}
@Property
boolean escapeUnescapeRoundtrip(@ForAll String s) {
return unescape(escape(s)).equals(s);
}
}
// FsCheck for C#
using FsCheck;
using FsCheck.Xunit;
public class PropertyTests {
[Property]
public bool SortIsIdempotent(List<int> xs) {
var sortedOnce = xs.OrderBy(x => x).ToList();
var sortedTwice = sortedOnce.OrderBy(x => x).ToList();
return sortedOnce.SequenceEqual(sortedTwice);
}
[Property]
public bool EscapeUnescapeRoundtrip(string s) {
return Unescape(Escape(s)) == s;
}
}
# Instead of hand-picked examples, generate thousands
require "prop_check"
G = PropCheck::Generators
PropCheck.forall(G.array(G.choose(0..100))) do |xs|
raise "sort is not idempotent" unless xs.sort == xs.sort.sort
end
PropCheck.forall(G.printable_ascii_string) do |s|
raise "roundtrip failed" unless unescape(escape(s)) == s
end
| Benefit | Example |
|---|---|
| Finds edge cases | Empty list, single element, duplicates, overflow |
| Documents invariants | sorted(xs) == sorted(sorted(xs)) = idempotency |
| Shrinks failures | Minimal counterexample for debugging |
3. Mutation Testing — Test Your Tests
┌─────────────────────────────────────────────────────┐
│ Original Code │ Mutant │
│ ────────────────────── ────────────────────────── │
│ if (x > 0) │ if (x >= 0) ← Killed? │
│ return x │ return x │
│ else │ else │
│ return -x │ return -x │
└─────────────────────────────────────────────────────┘
| Metric | Target | Action if Low |
|---|---|---|
| Mutation Score | ≥80% (core) | Add tests for surviving mutants |
| Kill Rate | ≥90% | Improve assertion specificity |
| Coverage ≠ Quality | — | Mutation score > line coverage |
4. Contract Testing — Consumer-Driven
# Consumer (frontend) defines expected contract
from pact import Consumer, Provider, Given, UponReceiving, WithState
pact = Consumer('OrderService').has_pact_with(Provider('PaymentService'))
pact.given('a valid payment request')
.upon_receiving('a charge request')
.with_request('POST', '/payment/charge', body={
'amount_cents': 1999,
'currency': 'USD',
'source': 'tok_visa'
})
.will_respond_with(200, body={
'transaction_id': 'txn_123',
'status': 'succeeded'
})
pact.verify()
// Pact C++ is limited
// Use pact-verifier CLI for verification
# pact-verifier --provider-base-url=http://localhost:8080 \
# --pact-url=./pacts/order_service-payment_service.json
// Pact JVM
@ExtendWith(PactConsumerTestExt.class)
@PactTestFor(providerName = "PaymentService")
class PaymentContractTest {
@Test
@Pact(consumer = "OrderService", provider = "PaymentService")
V2RequestResponsePact createPact(PactDslWithProvider builder) {
return builder
.given("a valid payment request")
.uponReceiving("a charge request")
.path("/payment/charge")
.method("POST")
.body(newJsonBody(o -> o
.numberType("amount_cents", 1999)
.stringType("currency", "USD")
.stringType("source", "tok_visa")
))
.willRespondWith()
.status(200)
.body(newJsonBody(o -> o
.stringType("transaction_id", "txn_123")
.stringType("status", "succeeded")
))
.toPact(V2Pact.class);
}
}
// PactNet
using PactNet;
using Xunit;
public class PaymentContractTests {
[Fact]
public void PaymentContract() {
var config = new PactConfig { PactDir = "./pacts" };
var pact = new PactVerifier(new PactConfig { PactDir = "./pacts" });
var builder = new V2PactBuilder("OrderService", "PaymentService");
builder
.Given("a valid payment request")
.UponReceiving("a charge request")
.With(new ProviderServiceRequest {
Method = HttpVerb.Post,
Path = "/payment/charge",
Body = new { amount_cents = 1999, currency = "USD", source = "tok_visa" }
})
.WillRespondWith(200, new {
transaction_id = "txn_123",
status = "succeeded"
});
}
}
# Consumer (frontend) defines expected contract — pact gem
require "pact/consumer/rspec"
Pact.service_consumer "OrderService" do
has_pact_with "PaymentService" do
mock_service :payment_service do
port 1234
end
end
end
payment_service
.given("a valid payment request")
.upon_receiving("a charge request")
.with(method: :post, path: "/payment/charge",
body: { amount_cents: 1999, currency: "USD", source: "tok_visa" })
.will_respond_with(status: 200,
body: { transaction_id: "txn_123", status: "succeeded" })
| Approach | Pros | Cons |
|---|---|---|
| Consumer-driven (Pact) | Prevents breaking changes | Requires broker |
| Provider-driven (OpenAPI) | Single source of truth | May miss consumer needs |
| Bi-directional | Both sides validated | More complex setup |
5. Integration Testing — Real Dependencies
# Testcontainers: Real DB in container, auto-cleanup
@pytest.fixture
def postgres():
with PostgresContainer("postgres:16") as pg:
yield pg.get_connection_url()
def test_user_repository_saves_and_loads(postgres):
repo = UserRepository(postgres)
user = User(name="Alice", email="alice@example.com")
repo.save(user)
loaded = repo.find_by_email("alice@example.com")
assert loaded.name == "Alice"
assert loaded.id is not None
# Testcontainers C++ (using testcontainers-cpp or similar)
#include <testcontainers/postgres.hpp>
using namespace testcontainers;
auto postgres = PostgresContainer("postgres:16").start();
auto conn = postgres.get_connection_url();
// Or use testcontainers-cpp library
auto container = PostgresContainer()
.withImageName("postgres:16")
.withStartupTimeout(std::chrono::seconds(60))
.start();
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
@Testcontainers
class UserRepositoryTest {
@Container
static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>("postgres:16");
@Test
void testUserRepositorySavesAndLoads() {
var repo = new UserRepository(postgres.getJdbcUrl());
var user = new User("Alice", "alice@example.com");
repo.save(user);
var loaded = repo.findByEmail("alice@example.com");
assertEquals("Alice", loaded.getName());
assertNotNull(loaded.getId());
}
}
using Testcontainers.PostgreSql;
public class UserRepositoryTests : IAsyncLifetime {
private readonly PostgreSqlContainer _postgres = new PostgreSqlBuilder()
.WithImage("postgres:16")
.Build();
public async Task InitializeAsync() => await _postgres.StartAsync();
[Fact]
public async Task TestUserRepositorySavesAndLoads() {
var repo = new UserRepository(_postgres.GetConnectionString());
var user = new User("Alice", "alice@example.com");
await repo.SaveAsync(user);
var loaded = await repo.FindByEmailAsync("alice@example.com");
Assert.Equal("Alice", loaded.Name);
Assert.NotNull(loaded.Id);
}
public async Task DisposeAsync() => await _postgres.DisposeAsync();
}
# Testcontainers: Real DB in container, auto-cleanup
require "testcontainers/postgres"
RSpec.describe UserRepository do
around do |example|
container = Testcontainers::PostgresContainer.new("postgres:16").start
@database_url = container.database_url
example.run
ensure
container&.stop
end
it "saves and loads users" do
repo = UserRepository.new(@database_url)
user = User.new(name: "Alice", email: "alice@example.com")
repo.save(user)
loaded = repo.find_by_email("alice@example.com")
expect(loaded.name).to eq("Alice")
expect(loaded.id).not_to be_nil
end
end
| Anti-pattern | Better Alternative |
|---|---|
| Shared test DB | Testcontainers per test/class |
| Mocking repository | Real DB + transaction rollback |
| Sleep/wait for async | Polling with timeout + condition |
Test Quality Metrics
| Metric | Good | Warning | Action |
|---|---|---|---|
| Execution time (unit) | <100ms | >1s | Refactor slow tests |
| Flakiness rate | 0% | >1% | Quarantine & fix |
| Mutation score | >80% | <60% | Add missing assertions |
| Branch coverage | >85% | <70% | Target uncovered branches |
| Test:code ratio | 1:1 to 2:1 | >3:1 | Possible over-testing |
Testing in CI/CD
| Stage | Tests | Time Budget |
|---|---|---|
| Pre-commit | Unit (changed files), lint | <30s |
| PR Pipeline | Full unit, contract, mutation (diff) | <5min |
| Merge/Main | All unit, integration, contract | <15min |
| Nightly | Full mutation, E2E, load, chaos | Unlimited |
Anti-Patterns to Avoid
| Anti-pattern | Symptom | Fix |
|---|---|---|
| Test Pyramid Inversion | Many E2E, few unit | Push logic down; contract test boundaries |
| Testing Implementation | Brittle tests on refactor | Test behaviour, not private methods |
| Over-mocking | Tests pass, prod fails | Use real dependencies for integration |
| Test-only Code Paths | Coverage ≠ confidence | Mutation testing reveals gaps |
| Ignoring Flaky Tests | "It passes sometimes" | Quarantine → root cause → fix/delete |
Related Topics
- Types of Testing — the same testing-pyramid layers reorganised around the specific question each layer is actually answering.
- Mutation Testing — measuring whether the tests this page's pyramid describes would actually notice a real bug, not just execute the code.
- The Anatomy of a Debugging Strategy — what to do with a failing test once it actually fails.
- Test Doubles: Mocks, Stubs, Fakes — the mock/stub/fake vocabulary this page's speed-and-isolation advice assumes.
Further Reading
| Resource | Focus |
|---|---|
| Effective Software Testing (Aniche, 2022) 2 | Modern testing toolbox |
| xUnit Test Patterns (Meszaros, 2007) 3 | Pattern catalogue for test code |
| Growing Object-Oriented Software, Guided by Tests (Freeman & Pryce, 2009) 4 | TDD at scale |
| The Little Book of Mutation Testing (Pizana, 2023) 5 | Practical mutation testing |
| ISO/IEC/IEEE 29119 6 | International testing standards |
Going Further
This page covers the fundamentals; four companion pages go deeper:
- Types of Testing — untangling scope, knowledge and intent, and how the types cooperate.
- Code Path Analysis — control-flow graphs, cyclomatic complexity and coverage criteria.
- Mutation Testing — validating that your tests would actually notice a bug.
- The V Model — with a Third V for Values — pairing every decision (including ethical ones) with its check.
References
-
Dijkstra, E.W. (1972). "The Humble Programmer." ACM Turing Award Lecture. https://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/EWD340.html ↩
-
Aniche, M. (2022). Effective Software Testing: A Developer's Guide. Manning Publications. https://www.manning.com/books/effective-software-testing ↩
-
Meszaros, G. (2007). xUnit Test Patterns: Refactoring Test Code. Addison-Wesley Professional. https://www.oreilly.com/library/view/xunit-test-patterns/9780131495050/ ↩
-
Freeman, S. & Pryce, N. (2009). Growing Object-Oriented Software, Guided by Tests. Addison-Wesley Professional. https://www.growing-object-oriented-software.com/ ↩
-
Pizana, N.J. (2023). The Little Book of Mutation Testing. Leanpub. https://leanpub.com/mutation-testing ↩
-
ISO/IEC/IEEE. (2013). ISO/IEC/IEEE 29119:2013 — Software and systems engineering — Software testing. https://www.iso.org/standard/44889.html ↩