systematic-debugging

Systematic Debugging

Overview

Random fixes waste time and create new bugs. Quick patches mask underlying issues.

Core principle: ALWAYS find root cause before attempting fixes. Symptom fixes are failure.

Violating the letter of this process is violating the spirit of debugging.

The Iron Law

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

If you haven't completed Phase 1, you cannot propose fixes.

When to Use

Use for ANY technical issue:

Use this ESPECIALLY when:

Don't skip when:

The Four Phases

You MUST complete each phase before proceeding to the next.

Phase 1: Root Cause Investigation

BEFORE attempting ANY fix:

WHEN system has multiple components (CI → build → signing, API → service → database):

BEFORE proposing fixes, add diagnostic instrumentation:

```

For EACH component boundary:

Run once to gather evidence showing WHERE it breaks

THEN analyze evidence to identify failing component

THEN investigate that specific component

```

Example (multi-layer system):

```bash

# Layer 1: Workflow

echo "=== Secrets available in workflow: ==="

echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"

# Layer 2: Build script

echo "=== Env vars in build script: ==="

env | grep IDENTITY || echo "IDENTITY not in environment"

# Layer 3: Signing script

echo "=== Keychain state: ==="

security list-keychains

security find-identity -v

# Layer 4: Actual signing

codesign --sign "$IDENTITY" --verbose=4 "$APP"

```

This reveals: Which layer fails (secrets → workflow ✓, workflow → build ✗)

WHEN error is deep in call stack:

See root-cause-tracing.md in this directory for the complete backward tracing technique.

Quick version:

Phase 2: Pattern Analysis

Find the pattern before fixing:

Phase 3: Hypothesis and Testing

Scientific method:

Phase 4: Implementation

Fix the root cause, not the symptom:

Pattern indicating architectural problem:

STOP and question fundamentals:

Discuss with your human partner before attempting more fixes

This is NOT a failed hypothesis - this is a wrong architecture.

Red Flags - STOP and Follow Process

If you catch yourself thinking:

ALL of these mean: STOP. Return to Phase 1.

If 3+ fixes failed: Question the architecture (see Phase 4.5)

your human partner's Signals You're Doing It Wrong

Watch for these redirections:

When you see these: STOP. Return to Phase 1.

Common Rationalizations

| Excuse | Reality |

|--------|---------|

| "Issue is simple, don't need process" | Simple issues have root causes too. Process is fast for simple bugs. |

| "Emergency, no time for process" | Systematic debugging is FASTER than guess-and-check thrashing. |

| "Just try this first, then investigate" | First fix sets the pattern. Do it right from the start. |

| "I'll write test after confirming fix works" | Untested fixes don't stick. Test first proves it. |

| "Multiple fixes at once saves time" | Can't isolate what worked. Causes new bugs. |

| "Reference too long, I'll adapt the pattern" | Partial understanding guarantees bugs. Read it completely. |

| "I see the problem, let me fix it" | Seeing symptoms ≠ understanding root cause. |

| "One more fix attempt" (after 2+ failures) | 3+ failures = architectural problem. Question pattern, don't fix again. |

Quick Reference

| Phase | Key Activities | Success Criteria |

|-------|---------------|------------------|

| 1. Root Cause | Read errors, reproduce, check changes, gather evidence | Understand WHAT and WHY |

| 2. Pattern | Find working examples, compare | Identify differences |

| 3. Hypothesis | Form theory, test minimally | Confirmed or new hypothesis |

| 4. Implementation | Create test, fix, verify | Bug resolved, tests pass |

When Process Reveals "No Root Cause"

If systematic investigation reveals issue is truly environmental, timing-dependent, or external:

But: 95% of "no root cause" cases are incomplete investigation.

Supporting Techniques

These techniques are part of systematic debugging and available in this directory:

Related skills:

Real-World Impact

From debugging sessions: