Last updated: 2026-09-23

U
Undergraduate level

Testing Fundamentals

Testing shows the presence, not the absence of bugs." — Edsger W. Dijkstra 1

Testing Pyramid (Revisited)

graph TD A[E2E Tests] --> B[Integration Tests] B --> C[Unit Tests]
Layer Purpose Tools Target Coverage
Property-based Discover edge cases automatically Hypothesis (Py), fast-check (JS), jqwik (Java) Core algorithms
Mutation Assess test suite quality Pitest, mutmut, stryker Critical paths
Contract Consumer-driven API verification Pact, Spring Cloud Contract Service boundaries
Unit Fast feedback on logic pytest, JUnit, Vitest >90% on domain logic
Integration Verify component wiring Testcontainers, WireMock Happy paths + failures
E2E Critical user journeys Playwright, Cypress <10 critical flows

Test Categories Deep Dive

1. Unit Tests — Fast, Isolated, Deterministic

# Good: Single behavior, no I/O, descriptive name
def test_calculate_discount_applies_tiered_rates():
    customer = Customer(tier="gold", years_active=3)
    order = Order(total=1000, items=5)

    discount = PricingEngine().calculate(customer, order)

    assert discount == 150  # 10% base + 5% loyalty

# Bad: Multiple assertions, external dependency, flaky
def test_user_registration():
    response = requests.post("/api/users", json={...})
    assert response.status_code == 201
    assert User.exists(email="...")
    assert sent_welcome_email()
// Good: Single behavior, no I/O, descriptive name
TEST(PricingEngineTest, AppliesTieredRatesForGoldCustomer) {
    Customer customer("gold", 3);
    Order order(1000, 5);

    Money discount = PricingEngine().calculate(customer, order);

    ASSERT_EQ(Money(150), discount);
}

// Bad: Multiple assertions, external dependency, flaky
TEST(UserRegistrationTest, BadExample) {
    auto response = Post("/api/users", json{...});
    ASSERT_EQ(201, response.status_code());
    ASSERT_TRUE(User::Exists("alice@example.com"));
    ASSERT_TRUE(EmailSent("welcome"));
}
// Good: Single behavior, no I/O, descriptive name
@Test
void testCalculateDiscountAppliesTieredRates() {
    Customer customer = new Customer("gold", 3);
    Order order = new Order(1000, 5);

    Money discount = pricingEngine.calculate(customer, order);

    assertEquals(150, discount.getCents());
}

// Bad: Multiple assertions, external dependency, flaky
@Test
void testUserRegistration() {
    var response = post("/api/users", json);
    assertEquals(201, response.getStatusCode());
    assertTrue(User.exists("alice@example.com"));
    assertTrue(Email.wasSent("welcome"));
}
// Good: Single behavior, no I/O, descriptive name
[Fact]
public void CalculateDiscount_AppliesTieredRates_ForGoldCustomer() {
    var customer = new Customer("gold", 3);
    var order = new Order(1000, 5);

    var discount = _pricingEngine.Calculate(customer, order);

    Assert.Equal(new Money(150), discount);
}

// Bad: Multiple assertions, external dependency, flaky
[Fact]
public void UserRegistration_BadExample() {
    var response = _client.Post("/api/users", json);
    Assert.Equal(201, response.StatusCode);
    Assert.True(User.Exists("alice@example.com"));
    Assert.True(Email.WasSent("welcome"));
}
# Good: Single behaviour, no I/O, descriptive name
RSpec.describe PricingEngine do
  it "applies tiered rates when calculating discounts" do
    customer = Customer.new(tier: "gold", years_active: 3)
    order = Order.new(total: 1000, items: 5)

    discount = described_class.new.calculate(customer, order)

    expect(discount).to eq(150) # 10% base + 5% loyalty
  end
end

# Bad: Multiple assertions, external dependency, flaky
RSpec.describe "user registration" do
  it "registers a user" do
    response = HTTP.post("/api/users", json: { name: "..." })
    expect(response.status).to eq(201)
    expect(User.exists?(email: "...")).to be(true)
    expect(sent_welcome_email?).to be(true)
  end
end
Principle Practice
F.I.R.S.T. Fast, Independent, Repeatable, Self-validating, Timely
AAA Pattern Arrange → Act → Assert
One logical assert Multiple physical assertions OK if single concept
No I/O Mock filesystem, network, time, randomness

Mocking the clock and the RNG buys determinism, but it isn't the whole story once randomness, unpredictable timing, or an LLM call is the thing under test rather than an incidental dependency to route around — see Testing Non-Deterministic Systems for that fuller treatment, and Test Doubles for the difference between a mock, a stub, and a fake this section's "mock" glosses over.

Note well. Mocking the clock and the RNG buys determinism, but it isn't the whole story once randomness, unpredictable timing, or an LLM call is the thing under test rather than an incidental dependency to route around.

2. Property-Based Testing — Discover the Unknown

# Instead of hand-picked examples, generate thousands
from hypothesis import given, strategies as st

@given(st.lists(st.integers(min_value=0, max_value=100)))
def test_sort_is_idempotent(xs):
    assert sorted(xs) == sorted(sorted(xs))

@given(st.text(alphabet=st.characters(blacklist_categories=("Cc", "Cs"))))
def test_escaped_string_roundtrip(s):
    assert unescape(escape(s)) == s
#include <rapidcheck/rapidcheck.h>

void test_sort_is_idempotent() {
    rc::check("sort is idempotent", [](const std::vector<int>& xs) {
        auto sorted_once = xs;
        std::sort(sorted_once.begin(), sorted_once.end());
        auto sorted_twice = sorted_once;
        std::sort(sorted_twice.begin(), sorted_twice.end());
        RC_ASSERT(sorted_once == sorted_twice);
    });
}

#include <rapidcheck/rapidcheck.h>

void test_escaped_string_roundtrip() {
    rc::check("escape/unescape roundtrip", [](const std::string& s) {
        RC_ASSERT(unescape(escape(s)) == s);
    });
}
import net.jqwik.api.*;
import java.util.*;

class PropertyTests {
    @Property
    boolean sortIsIdempotent(@ForAll List<Integer> xs) {
        var sortedOnce = new ArrayList<>(xs);
        Collections.sort(sortedOnce);
        var sortedTwice = new ArrayList<>(sortedOnce);
        Collections.sort(sortedTwice);
        return sortedOnce.equals(sortedTwice);
    }

    @Property
    boolean escapeUnescapeRoundtrip(@ForAll String s) {
        return unescape(escape(s)).equals(s);
    }
}
// FsCheck for C#
using FsCheck;
using FsCheck.Xunit;

public class PropertyTests {
    [Property]
    public bool SortIsIdempotent(List<int> xs) {
        var sortedOnce = xs.OrderBy(x => x).ToList();
        var sortedTwice = sortedOnce.OrderBy(x => x).ToList();
        return sortedOnce.SequenceEqual(sortedTwice);
    }

    [Property]
    public bool EscapeUnescapeRoundtrip(string s) {
        return Unescape(Escape(s)) == s;
    }
}
# Instead of hand-picked examples, generate thousands
require "prop_check"
G = PropCheck::Generators

PropCheck.forall(G.array(G.choose(0..100))) do |xs|
  raise "sort is not idempotent" unless xs.sort == xs.sort.sort
end

PropCheck.forall(G.printable_ascii_string) do |s|
  raise "roundtrip failed" unless unescape(escape(s)) == s
end
Benefit Example
Finds edge cases Empty list, single element, duplicates, overflow
Documents invariants sorted(xs) == sorted(sorted(xs)) = idempotency
Shrinks failures Minimal counterexample for debugging

3. Mutation Testing — Test Your Tests

┌─────────────────────────────────────────────────────┐
│  Original Code           Mutant                    │
│  ────────────────────── ────────────────────────── │
│  if (x > 0)              if (x >= 0)    Killed?   │
│      return x                return x              │
│  else                    else                      │
│      return -x               return -x             │
└─────────────────────────────────────────────────────┘
Metric Target Action if Low
Mutation Score ≥80% (core) Add tests for surviving mutants
Kill Rate ≥90% Improve assertion specificity
Coverage ≠ Quality Mutation score > line coverage

4. Contract Testing — Consumer-Driven

# Consumer (frontend) defines expected contract
from pact import Consumer, Provider, Given, UponReceiving, WithState

pact = Consumer('OrderService').has_pact_with(Provider('PaymentService'))

pact.given('a valid payment request')
    .upon_receiving('a charge request')
    .with_request('POST', '/payment/charge', body={
        'amount_cents': 1999,
        'currency': 'USD',
        'source': 'tok_visa'
    })
    .will_respond_with(200, body={
        'transaction_id': 'txn_123',
        'status': 'succeeded'
    })

pact.verify()
// Pact C++ is limited
// Use pact-verifier CLI for verification
# pact-verifier --provider-base-url=http://localhost:8080 \
#   --pact-url=./pacts/order_service-payment_service.json
// Pact JVM
@ExtendWith(PactConsumerTestExt.class)
@PactTestFor(providerName = "PaymentService")
class PaymentContractTest {
    @Test
    @Pact(consumer = "OrderService", provider = "PaymentService")
    V2RequestResponsePact createPact(PactDslWithProvider builder) {
        return builder
            .given("a valid payment request")
            .uponReceiving("a charge request")
            .path("/payment/charge")
            .method("POST")
            .body(newJsonBody(o -> o
                .numberType("amount_cents", 1999)
                .stringType("currency", "USD")
                .stringType("source", "tok_visa")
            ))
            .willRespondWith()
            .status(200)
            .body(newJsonBody(o -> o
                .stringType("transaction_id", "txn_123")
                .stringType("status", "succeeded")
            ))
            .toPact(V2Pact.class);
    }
}
// PactNet
using PactNet;
using Xunit;

public class PaymentContractTests {
    [Fact]
    public void PaymentContract() {
        var config = new PactConfig { PactDir = "./pacts" };
        var pact = new PactVerifier(new PactConfig { PactDir = "./pacts" });

        var builder = new V2PactBuilder("OrderService", "PaymentService");
        builder
            .Given("a valid payment request")
            .UponReceiving("a charge request")
            .With(new ProviderServiceRequest {
                Method = HttpVerb.Post,
                Path = "/payment/charge",
                Body = new { amount_cents = 1999, currency = "USD", source = "tok_visa" }
            })
            .WillRespondWith(200, new {
                transaction_id = "txn_123",
                status = "succeeded"
            });
    }
}
# Consumer (frontend) defines expected contract — pact gem
require "pact/consumer/rspec"

Pact.service_consumer "OrderService" do
  has_pact_with "PaymentService" do
    mock_service :payment_service do
      port 1234
    end
  end
end

payment_service
  .given("a valid payment request")
  .upon_receiving("a charge request")
  .with(method: :post, path: "/payment/charge",
        body: { amount_cents: 1999, currency: "USD", source: "tok_visa" })
  .will_respond_with(status: 200,
                     body: { transaction_id: "txn_123", status: "succeeded" })
Approach Pros Cons
Consumer-driven (Pact) Prevents breaking changes Requires broker
Provider-driven (OpenAPI) Single source of truth May miss consumer needs
Bi-directional Both sides validated More complex setup

5. Integration Testing — Real Dependencies

# Testcontainers: Real DB in container, auto-cleanup
@pytest.fixture
def postgres():
    with PostgresContainer("postgres:16") as pg:
        yield pg.get_connection_url()

def test_user_repository_saves_and_loads(postgres):
    repo = UserRepository(postgres)
    user = User(name="Alice", email="alice@example.com")

    repo.save(user)
    loaded = repo.find_by_email("alice@example.com")

    assert loaded.name == "Alice"
    assert loaded.id is not None
# Testcontainers C++ (using testcontainers-cpp or similar)
#include <testcontainers/postgres.hpp>

using namespace testcontainers;

auto postgres = PostgresContainer("postgres:16").start();
auto conn = postgres.get_connection_url();

// Or use testcontainers-cpp library
auto container = PostgresContainer()
    .withImageName("postgres:16")
    .withStartupTimeout(std::chrono::seconds(60))
    .start();
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;

@Testcontainers
class UserRepositoryTest {
    @Container
    static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>("postgres:16");

    @Test
    void testUserRepositorySavesAndLoads() {
        var repo = new UserRepository(postgres.getJdbcUrl());
        var user = new User("Alice", "alice@example.com");

        repo.save(user);
        var loaded = repo.findByEmail("alice@example.com");

        assertEquals("Alice", loaded.getName());
        assertNotNull(loaded.getId());
    }
}
using Testcontainers.PostgreSql;

public class UserRepositoryTests : IAsyncLifetime {
    private readonly PostgreSqlContainer _postgres = new PostgreSqlBuilder()
        .WithImage("postgres:16")
        .Build();

    public async Task InitializeAsync() => await _postgres.StartAsync();

    [Fact]
    public async Task TestUserRepositorySavesAndLoads() {
        var repo = new UserRepository(_postgres.GetConnectionString());
        var user = new User("Alice", "alice@example.com");

        await repo.SaveAsync(user);
        var loaded = await repo.FindByEmailAsync("alice@example.com");

        Assert.Equal("Alice", loaded.Name);
        Assert.NotNull(loaded.Id);
    }

    public async Task DisposeAsync() => await _postgres.DisposeAsync();
}
# Testcontainers: Real DB in container, auto-cleanup
require "testcontainers/postgres"

RSpec.describe UserRepository do
  around do |example|
    container = Testcontainers::PostgresContainer.new("postgres:16").start
    @database_url = container.database_url
    example.run
  ensure
    container&.stop
  end

  it "saves and loads users" do
    repo = UserRepository.new(@database_url)
    user = User.new(name: "Alice", email: "alice@example.com")

    repo.save(user)
    loaded = repo.find_by_email("alice@example.com")

    expect(loaded.name).to eq("Alice")
    expect(loaded.id).not_to be_nil
  end
end
Anti-pattern Better Alternative
Shared test DB Testcontainers per test/class
Mocking repository Real DB + transaction rollback
Sleep/wait for async Polling with timeout + condition

Test Quality Metrics

Metric Good Warning Action
Execution time (unit) <100ms >1s Refactor slow tests
Flakiness rate 0% >1% Quarantine & fix
Mutation score >80% <60% Add missing assertions
Branch coverage >85% <70% Target uncovered branches
Test:code ratio 1:1 to 2:1 >3:1 Possible over-testing

Testing in CI/CD

Stage Tests Time Budget
Pre-commit Unit (changed files), lint <30s
PR Pipeline Full unit, contract, mutation (diff) <5min
Merge/Main All unit, integration, contract <15min
Nightly Full mutation, E2E, load, chaos Unlimited

Anti-Patterns to Avoid

Anti-pattern Symptom Fix
Test Pyramid Inversion Many E2E, few unit Push logic down; contract test boundaries
Testing Implementation Brittle tests on refactor Test behaviour, not private methods
Over-mocking Tests pass, prod fails Use real dependencies for integration
Test-only Code Paths Coverage ≠ confidence Mutation testing reveals gaps
Ignoring Flaky Tests "It passes sometimes" Quarantine → root cause → fix/delete

Further Reading

Resource Focus
Effective Software Testing (Aniche, 2022) 2 Modern testing toolbox
xUnit Test Patterns (Meszaros, 2007) 3 Pattern catalogue for test code
Growing Object-Oriented Software, Guided by Tests (Freeman & Pryce, 2009) 4 TDD at scale
The Little Book of Mutation Testing (Pizana, 2023) 5 Practical mutation testing
ISO/IEC/IEEE 29119 6 International testing standards

Going Further

This page covers the fundamentals; four companion pages go deeper:

References


  1. Dijkstra, E.W. (1972). "The Humble Programmer." ACM Turing Award Lecture. https://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/EWD340.html 

  2. Aniche, M. (2022). Effective Software Testing: A Developer's Guide. Manning Publications. https://www.manning.com/books/effective-software-testing 

  3. Meszaros, G. (2007). xUnit Test Patterns: Refactoring Test Code. Addison-Wesley Professional. https://www.oreilly.com/library/view/xunit-test-patterns/9780131495050/ 

  4. Freeman, S. & Pryce, N. (2009). Growing Object-Oriented Software, Guided by Tests. Addison-Wesley Professional. https://www.growing-object-oriented-software.com/ 

  5. Pizana, N.J. (2023). The Little Book of Mutation Testing. Leanpub. https://leanpub.com/mutation-testing 

  6. ISO/IEC/IEEE. (2013). ISO/IEC/IEEE 29119:2013 — Software and systems engineering — Software testing. https://www.iso.org/standard/44889.html