Testing strategy

Short version: TDD is mandatory for core, but TDD alone would leave this project badly under-tested. jdx reads files written by other people's compilers; its worst bugs are not logic errors in code we wrote, they are "the real world contains a jar shaped like that" errors. Tests written from imagination cannot find those. So we pair TDD with five families of test that generate their own inputs or borrow their expected answers from an external oracle.

1. Why not just TDD?#

TDD's loop is: write a failing test for the behaviour you want → make it pass → refactor. That works beautifully when you define correct behaviour.

Roughly 40 % of this project is like that — the symbol-ref grammar, member resolution, generic substitution, truncation, rendering. For all of it, TDD is required (§3).

The other 60 % is different in kind:

The strategy below is the answer to "more tests guarantee fewer bugs" — not more hand-written examples, which plateau fast, but test families that keep generating new cases without anyone writing them.

2. The ten families at a glance#

Tiers (so the fast loop stays fast):

A tier-1 suite that creeps past 30 s kills TDD, and then it kills testing. Guard it: anything touching disk, network, or a real jar belongs in tier 2 or above.

Tags are the mechanism (T-053). JUnit @Tags decide which tier runs a test class:

The wiring lives in the root build.gradle.kts: per-module integrationTest/tier2Test/soakTest/ benchTest tasks with includeTags, test excluding all four tags (integration, tier2, soak, bench), check depending on integrationTest, and root soak/bench/mutationTest/testReport commands. SoakExclusionProofTest (core/.../tiers/) is the deliberate proof that a soak-tagged test cannot run in tier 1 or 2: it asserts -Djdx.tier=soak, a property only the soakTest task sets, so broken exclusion fails the build instead of silently slowing the loop. Soak tests read the corpus directory from -Djdx.corpusDir (default ~/.gradle/caches, override with -Pcorpus=<dir>) and must skip — never fail — when it is absent, so check stays green on machines with no corpus.

The per-test budget is enforced, not aspirational. Every test and integrationTest task is finalised by verifyPerTestBudget, which inspects JUnit XML times, prints total test time, the 10 slowest tests, and any budget violations at the end of the run. Unit tests (test) must finish in <= 2.5 s (override with -Punit.test.budget=<seconds>), while integration tests (integrationTest) must finish in <= 120.0 s (override with -Pintegration.test.budget=<seconds>). When a unit test exceeds its budget, the fix is to optimize it or move it to integration with @Tag("integration").

CI mode (-Pci, issues #57/#61). CI runners are slow, shared, and variable — the same suite passes locally and reds on CI with 0 test failures (issue #2). CI workflows pass -Pci (also auto-detected via CI/GITHUB_ACTIONS env), which relaxes only wall-clock test gates: verifyPerTestBudget becomes report-only (a slow runner with 0 failures stays green) and PIT timeoutConstInMillis lifts from 10 s to 60 s in every gated module. No per-task Test timeout is set in either mode (fail-open by policy). Real product timeouts (Vineflower/javap 30 s, Maven connect/read) are degradation behavior and stay intact — never weaken product behavior to green CI. A local verifyPerTestBudget red (issue #2) is a machine-variance signal to move a suite to tier 2; a CI-infra red is a runner problem — do not confuse the two.

3. TDD — where it is mandatory#

core is developed test-first. No exceptions. core has no dependencies and no IO (D-001, CONTRIBUTING.md), which means there is never an excuse for "hard to test".

The loop, concretely, for T-003 (symbol-ref parser):

// RED — write this first, watch it fail for the right reason
@Test fun `nested type accepts dollar and dot forms identically`() {
    assertEquals(parseRef("java.util.Map\$Entry"), parseRef("java.util.Map.Entry"))
}

// GREEN — the smallest change that passes. Resist generalising here.

// REFACTOR — now make it clean, with the test holding the behaviour still.

Rules that make this work rather than become ceremony:

• One behaviour per test. If the name needs "and", it is two tests.
• Test names are sentences, in backticks: fun `inherited members from a generic supertype are substituted`(). The test report should read like a specification of what jdx does.
• Watch the test fail first, and check the failure message is the one you expect. A test that was green before you wrote the code is testing nothing.
• Assert on values, not on calls. No mocking framework in core. If you need a mock, the design is wrong — push the IO outward.
• Every bug fix starts with a failing regression test that reproduces the report. Name it after the issue: fun `regression #42 - bridge method shadows its own override`().

Outside core, test-first is encouraged but not required, because you often cannot know the expected value until you have looked at what ASM or Vineflower actually returns. There the honest workflow is: spike → look → write the differential or golden test that pins it → delete the spike. What is not acceptable is landing untested code and promising tests later.

4. Property-based testing (§family 2)#

Hand-written examples stop finding bugs once they cover what you already thought of. Properties keep generating inputs forever.

Library: kotest-property. Minimum 1,000 cases per property in tier 1 (JDX_PROPERTY_ITERATIONS in core/src/test/kotlin/dev/jdx/core/gen/PropertySupport.kt). When a property fails, kotest prints the failing seed (Repeat this test by using seed …); pin it as a regression test with one line — checkAll(1_000, pinnedConfig(<seed>L), arbFoo()) { foo -> … } — and remove the pin once the bug is fixed and the property passes unpinned again.

Properties we owe the project:

// Symbol refs (T-003)
"parse(print(ref)) == ref"                              // round-trip is a fixed point
"print(parse(s)) is stable"                             // idempotent normalisation
"a malformed ref never throws"                          // returns a positioned error

// Descriptors and signatures (T-002, generators + properties T-055)
"descriptor -> parsed -> printed == descriptor"         // for generated nested generics,
                                                        // wildcards, arrays, type vars
"binaryName -> fromBinaryName -> binaryName == identity" // TypeNamePropertyTest
"parseClass(classSignature).signature == signature"     // the T-009 class-file entry point
// Member resolution (T-009)
"resolve(type) is deterministic across runs"
"resolve(type) contains every member of declared(type)"
"resolve(type) never contains a private member of a strict supertype"
"resolve of a cyclic hierarchy terminates"              // synthesised cyclic ClassInfo

// Truncation (T-047)
"truncated output never splits an entity"
"shown <= total, and hint is present iff shown < total"

Generators live in core/src/test/kotlin/.../gen/ and are shared. Writing a good Arb<TypeName> once pays for itself across every later property.

5. Differential / oracle testing (§family 3)#

This is the single most valuable family in the project, because it gives us correct answers we did not have to write down.

5.1 javap as the oracle for bytecode reading#

For every class in the fixture corpus and every class in a sampled slice of the local jar corpus:

javap -p -s -c <class>   ──►  parse  ──►  { member names + descriptors + flags }
jdx  members --declared --access all --include-synthetic --json  ──►  same set
assert the two sets are EQUAL

Any disagreement is a bug in jdx (or, rarely, a javap quirk worth documenting). This one test family covers a huge surface — access flags, synthetics, bridges, varargs, nested classes, enum internals, records — without anyone enumerating those cases.

5.2 javac/kotlinc as the oracle for source-derived facts#

The fixture corpus is compiled from source we control, so for every fixture we independently know the truth. Assert jdx's answer against a machine-readable expectation file generated from the fixture's own annotations:

@ExpectedMembers({"public int add(int,int)", "private static final long serialVersionUID"})
public class Arithmetic { … }

The expectation travels with the fixture, so adding a fixture adds a test automatically.

5.3 Compile-back testing for decompilation#

Do not assert decompiler output text — it changes with every Vineflower release and the test becomes noise. Assert the property that actually matters:

decompile(class) ──► javac ──► class'          (must compile)
members(class) == members(class')              (same API surface)

A decompilation that recompiles to the same API is a decompilation an agent can trust. When it fails, that is real information (⚠ reconstructed already warns the user; this test tells us how often it matters).

5.4 The JDK as a free oracle#

java.base ships ~6,000 classes plus src.zip. It is a large, stable, sources-paired corpus that every contributor has. Use it: jdx members java.util.HashMap --inherited compared against the JDK's own source declarations is a strong end-to-end check that needs no downloads.

6. Metamorphic testing (§family 5)#

When you cannot state the right answer, you can often state a relation between answers that must hold. These catch deep index and resolver bugs that example tests never reach.

members(T, --inherited) ⊇ members(T, --declared)
members(T, --inherited) ⊇ members(super(T), --inherited) minus private/overridden
hierarchy(T, --down) contains S  ⟺  hierarchy(S, --up) contains T
usages(X) — every hit, when re-read, genuinely references X
callers(M) contains C  ⟺  calls(C) contains M
search(exact-fqn-of(T)) returns exactly T
show(T) member counts == |members(T, --declared, --access all)|
body(M) parsed back has signature == signature(M)
index(jar) twice ⟹ byte-identical index rows      (determinism)
run(cmd) twice ⟹ byte-identical stdout            (D-007 determinism promise)

Run these over the fixture corpus in tier 2, and over a random sample of the local jar corpus in tier 3. They are cheap to write and they are the tests most likely to find a real bug six months from now.

7. Fault injection (§family 6)#

jdx's promise is degrade, don't fail (AGENTS.md). That promise needs adversarial tests, generated programmatically rather than collected by hand:

• Truncated class file (cut at every 10 % boundary)
• Class-file major version from the future
• Zip entry with ../ traversal (zip-slip) — must be rejected, D-017
• Zip bomb: huge declared uncompressed size — must hit the cap, not OOM
• Empty jar; jar with only resources; jar with a directory named X.class
• Sources jar that does not match its binary (member added, removed, renamed) — must emit SOURCES_VERSION_MISMATCH, not silently disagree
• Sources jar for a different artifact entirely
• Class compiled -g:none (no debug info ⟹ no parameter names) — must fall back, and say the names are synthesised
• Multi-release jar with conflicting versions
• Duplicate FQN across two jars (shading) — must warn, D-016
• Unreadable file / permission denied / file deleted mid-read
• Decompiler timeout and decompiler crash — must fall back to javap, not propagate

Every one of these asserts two things: the process exits with a documented code (D-015), and the output names the problem. A stack trace reaching an agent is a test failure.

8. Corpus soak testing (§family 7)#

The highest-value-per-line test in the project, and the one most teams skip.

The owner's machine has ~2,183 real jars in ~/.gradle/caches, including sources jars and a large obfuscated minecraft-client.jar. That is a free adversarial corpus written by dozens of different compilers and build tools.

for each jar in the corpus:
    index it
    for a random sample of classes in it:
        run show / outline / members / signature / doc / body / source
        assert:
          - exit code ∈ {0,1,2}                       (never 5 or 6)
          - no exception reached stdout/stderr
          - --json output parses and validates against the envelope schema
          - text and JSON carry the same entity set   (D-007)
          - running it twice produces identical bytes (determinism)
    record every warning code emitted, with a count

Notes:

• Tagged, off by default (./gradlew soak) — it depends on artifacts not every contributor has, and it takes minutes. ./gradlew check must never require it.
• It asserts invariants, not values. It cannot know the right answer for 2,183 jars; it can know that jdx must never crash, never emit invalid JSON, and never be non-deterministic.
• The warning histogram is a deliverable. "247 jars emitted MULTI_RELEASE_VARIANT" is how we learn which real-world shapes matter. Record it in the progress log.
• Point it at a directory via -Pcorpus=<dir> so contributors without the owner's cache can run it against their own ~/.m2 or ~/.gradle.

9. Parity / contract testing (§family 8)#

Four front-ends (D-004) will drift unless a test forbids it.

• Adapter parity: the same query through CLI --json, HTTP, and MCP must produce byte-identical result payloads. One test, parameterised over every command.
• Text↔JSON completeness: every entity and warning in the text rendering must appear in the JSON. Enforced structurally, not by eyeball (D-007).
• Exit-code contract: a table-driven test asserting the D-015 code for each documented situation. Changing a code must break a test — that is the point.
• --help coverage: every flag documented in docs/PROPOSAL.md Appendix B exists, and every implemented flag is documented. Drift in either direction fails.
• Envelope schema: the JSON envelope is validated against a checked-in JSON Schema, so "jdx": 1 genuinely means something.

10. Mutation testing (§family 10)#

The direct answer to "don't stand with a few test cases." Line coverage measures which code ran; mutation score measures whether the tests would notice if that code were wrong. Pitest mutates the bytecode (flips conditionals, removes calls, changes return values) and reports how many mutants your tests killed.

Targets:

Run in tier 4. A surviving mutant in core is a genuine gap: either write the test that kills it, or delete the unreachable code.

Runbook: ./gradlew mutationTest -PconfirmTier4=true (or -PallowHeavy=true) runs PIT over core, index, sources and decompile (only core's ≥ 80 % gate fails the build; the rest are measured and reported per D-021). Reports land in <module>/build/reports/pitest (XML + HTML, un-timestamped); JaCoCo line reports sit beside them under build/reports/jacoco/test/. The full run takes ~30 min (the index suite dominates) and never runs in check. Explicit permission is required via -PconfirmTier4=true or -PallowHeavy=true. For fast iteration on one surviving mutant, ./gradlew :core:pitest -PpitestScope='dev.jdx.core.render.Truncation*' narrows both the class and test filters to that glob — never quote a scoped run as the module's score (L-056). Three PIT/Kotlin wrinkles are handled in the build files, not worked around per-case: test discovery goes through pitest-junit5-plugin (JUnit-5-only upstream, verified working on this build's JUnit 6); kotlin.jvm.internal.Intrinsics calls are excluded via avoidCallsTo (equivalent mutants by construction); and build-wiring proof tests (core.tiers, the index soak suites) are excluded from PIT targets — they assert runner invariants, not product behaviour.

10.1 A mutation gate that cannot run is not a gate#

PIT runs its suite once unmutated first, to compute line coverage, and refuses to start if that run is red: "N tests did not pass without mutation … Mutation testing requires a green suite." Nothing is mutated, no report is written, and the module's score is simply never produced. On a contributor's machine that reads as "a slow run"; in CI it reads as a red nightly, which is easy to mistake for a mutation problem.

Two environment facts make that pre-scan fail on a clean checkout only (issue #66, the reason it survived for weeks):

1. A PIT minion is a fresh JVM. It inherits no test/tier2Test system property, so the corpus property jdx.fixturesDir never reaches it and the resolver falls back to walking up for testfixtures/build/libs — which a developer's earlier build has left behind and a runner has not.
2. dependsOn(":testfixtures:jar", …) was on test/tier2Test only, and ./gradlew mutationTest reaches neither.

So the corpus wiring is shared, in the root build, for every module that applies the PIT plugin — one place, not a private copy in index:

• the four :testfixtures jar tasks are dependencies of the pitest task;
• jdx.fixturesDir and jdx.diffFixturesDir go on pitest { jvmArgs } (the minion arguments — PitestTask's own JavaExec args are the launcher's and would not help).

./gradlew verifyPitestWiring asserts both halves for every PIT module and runs in every check, so a module added tomorrow inherits the wiring and a regression is red on a push rather than in a 20-minute weekly run. It also asserts the PIT modules are exactly the ones mutationTest runs, and that it checked something at all.

A missing corpus is a machine fact, not a defect. The shared resolvers expose binaryCorpusAvailable() / sourcesCorpusAvailable() (and DiffFixtureJars.available()), and the suites that need the corpus assumeTrue on them. Resolving optimistically and letting the resolver throw is what turned one missing environment detail into a red build whose stack trace pointed at product code. FixtureResolutionOwnershipTest in :testfixtures pins both halves: exactly one copy of the walk-up resolver may exist (plus core's documented second copy), and the suites rewritten in #66 must stay gated.

11. The fixture corpus (T-006)#

Every family above leans on this, so it is built early and deliberately. Small, nasty Java and Kotlin classes compiled by Gradle at test time into both a binary jar and a sources jar.

Must include: bounded/wildcard/recursive generics · bridge methods from covariant overrides · inner, static-nested, anonymous and local classes · records · sealed hierarchies · enums with constant bodies · repeatable and array-valued annotations · varargs · synchronized/native /strictfp · @Deprecated(forRemoval=true) · package-private and private members · a class compiled -g:none · a class with a static initialiser that writes a marker file (to prove D-017: it must never appear) · Kotlin: suspend, extension functions, default arguments, data/sealed/value/object/companion, @JvmName, @JvmStatic, nullable types, properties with custom accessors, typealiases, inline/reified.

Requirements:

• Deterministic builds — same bytes on repeat runs, or golden tests become flaky.
• Self-describing — fixtures carry @ExpectedMembers-style annotations (§5.2) so adding a fixture adds test coverage automatically.
• A Fixtures helper resolves jar paths with no hard-coded absolute paths.

11.2 The v1/v2 diff pair (issue #23)#

jdx diff needs a corpus that is two artifacts rather than one, so testfixtures carries a second, independent pair: src/diffV1/ and src/diffV2/, the same package dev.jdx.diffapi compiled into build/diff-fixtures/diff-fixtures-v1.jar and -v2.jar and resolved by dev.jdx.testsupport.fixtures.DiffFixtureJars (property jdx.diffFixturesDir). Same package on both sides is the whole point: version the package and every type reads as a removal plus an addition, and the intra-type rules never fire.

Three rules make it work without disturbing the main corpus:

• A different directory, and a base name that does not start with testfixtures-. FixtureJars, core's Fixtures and ArtifactTestJars all require(size == 1) over build/libs, so a second jar there turns three unrelated assertions into exceptions.
• A third source set, src/diffAbsent/, that compiles but is never packaged. javac refuses a class whose supertype is missing, so the fixture for "the supertype is in neither artifact" has to exist in order to compile and then be kept out of both jars — which is exactly the real-world case (a superclass in another dependency) the differ's inheritance not checked detail exists for.
• A deliberately unchanged type (Stable.java, byte-for-byte in both trees), so the pair can prove the differ is not noisy as well as not blind.

Rule coverage, not @ExpectedMembers. The pair is self-describing through DiffFixtureRuleCoverageTest in :index rather than through per-type annotations: the property worth pinning is a property of the pair — that every rule in core/diff/CompatRule.kt fires against real javac output. A rule can pass every hand-built example in core and still be unreachable, because the reader never produces the shape it expects; that is the one class of bug this corpus exists to catch. The test lists every rule with the fixture that triggers it, so a new rule with no fixture fails the build — and a rule that fires with no fixture fails it too, so a fixture cannot grow silently.

It also carries an explicit unreachableFromJavac list, with reasons, for rules that are correct but that no compiler will produce input for — VARARGS_CHANGED is the example, because f(String) and f(String...) have different descriptors, so that change is a PARAMETER_TYPE_CHANGED. Those are pinned by hand-built snapshots instead. Adding one to the required list "to make it pass" would claim coverage the corpus does not have.

11.1 Adding a fixture#

1. Add the source under testfixtures/src/main/java (or src/main/kotlin). The odd one out is src/nodebug/java: classes there compile with -g:none (see testfixtures/build.gradle.kts), so only a fixture that must lack debug info goes there.
2. Annotate every source-declared type with @ExpectedMembers — one entry per declared field, method and constructor, exactly as javap -p prints the member line (modifiers plus fully-qualified signature, no trailing ;). List synthetic members (this$0, bridges, $default stubs) like any other; never list static {}. A type declaring nothing carries an explicit empty list. Anonymous/local classes, enum constant bodies and the Kotlin file facade cannot carry the annotation — they live in FixtureCorpusTest's pinned exempt set instead, which you must extend consciously.
3. Run ./gradlew :testfixtures:jar and copy the member lines from javap -p -classpath testfixtures/build/libs/testfixtures-<version>.jar <YourClass> into the annotation — never hand-write them from memory. The corpus test diffs annotation against javap member-for-member and fails on any deviation, including order.
4. If the fixture must never execute (a static-initialiser probe like StaticInitMarker), say so in its KDoc, and never reference it from test code — read bytes via Fixtures.classBytes(...), never reflection (D-017).
5. Rebuild twice and compare sha256 of both jars; identical bytes are the acceptance bar.

11.3 Resolving the corpus from a test#

dev.jdx.testsupport.fixtures.FixtureJars is the single copy of the resolution, and DiffFixtureJars the single copy for the v1/v2 pair. Both expose a non-throwing *CorpusAvailable() / available() probe next to the throwing resolvers, and a suite that needs the corpus gates on it:

private fun fixtureSourcesJar(): File {
    assumeTrue(
        FixtureJars.sourcesCorpusAvailable(),
        "no testfixtures corpus: build :testfixtures first (docs/TESTING.md §11)",
    )
    return FixtureJars.sourcesJar()
}

Ask, then resolve — never resolve and catch. A missing corpus is a property of the machine, so it deserves a visible SKIP; a stack trace through singleJar reads like a product defect and, in a mutation pre-scan, is indistinguishable from a real failure (§10.1, issue #66).

Do not write your own resolver. FixtureResolutionOwnershipTest in :testfixtures scans every module's test sources for the walk-up marker and fails on any copy outside FixtureJars and core's Fixtures (core's test source set must not depend on a project — that is the one documented exception). Before adding a copy, ask whether the module can take testImplementation(testFixtures(project(":testfixtures"))); every module that reads the corpus already does.

12. What we deliberately do not test#

Testing time is finite; spend it where bugs live.

• Third-party libraries. We do not test that ASM parses class files or that SQLite commits transactions. We test our use of them.
• Exact decompiler output text. §5.3 explains the alternative.
• Rendering cosmetics beyond what golden files already pin. Do not write assertions on individual spaces.
• Getters, data class members, trivial delegation. Mutation testing will flag these as uncovered; that is acceptable and expected.
• Gradle's own behaviour.

13. Running the tests#

./gradlew test                       # tier 1 — fast, the TDD loop (<30s)
./gradlew check                      # tier 2 — + property, golden, fault injection (<3min)
./gradlew check -Pci                 # CI mode: budgets relaxed (report-only), same gates otherwise
./gradlew test -Pgolden.update=true  # rewrite golden files — THEN READ THE DIFF
./gradlew soak                       # tier 3 — corpus, needs local jars
./gradlew soak -Pcorpus=~/.m2        # ...against your own corpus
./gradlew bench -PconfirmTier4=true        # tier 4 — performance budgets (PROPOSAL §15)
./gradlew mutationTest -PconfirmTier4=true # tier 4 — mutation score
./gradlew testReport                 # aggregated HTML report
./gradlew verifyPitestWiring         # the PIT corpus wiring gate (also runs in every `check`)

Definition of done for any task: tier 1 and tier 2 green; tier 3 green if you touched artifact reading, indexing, or rendering; new behaviour covered by at least one family from §2 that generates cases (2, 3, 5, 6, 7) rather than only hand-written examples.

14. A note on updating golden files#

-Pgolden.update=true is the most dangerous command in this repo. It makes every failing output test pass, instantly, whether or not the new output is correct.

Always read the resulting diff before committing it. A golden file you updated without reading is a test you deleted. If the diff is large, that is a signal to check whether you changed behaviour you did not intend to change — not a signal to skim faster.

All golden suites share one helper — GoldenFiles.verifyAll with the UnifiedDiff renderer (testfixtures/src/testFixtures/.../testsupport/golden/, T-054). Do not hand-roll comparison, update-mode, or orphan logic in a new suite; consume the helper via testFixtures(project(":testfixtures")).