A source-code extractor that reads your Go or Python code, walks each file with a tree-sitter grammar, and writes per-package .md files into your spec tree. Every function, type, and method becomes a reqmd requirement. The source is annotated as provenance, the symbol kind is preserved, and the doc comment becomes the requirement body. The result is a spec tree where the code and the spec share an ID space.

This is what closes the traceability gap from code to spec. Without it, you have two artifacts that reference each other in different vocabularies. With it, the agent (and the human) can follow a trace from a stakeholder requirement to the system requirement to the software requirement to the function that implements it.

What it does#

reqmd-import extract ./internal ./spec/03-software/imported

The extractor:

Walks ./internal for files matching any registered language plugin.
Parses each file with the language’s tree-sitter grammar.
Extracts every symbol: package, function, method, type, constant, variable.
Renders one .md file per package into ./spec/03-software/imported/, with one ## IMP-... heading per symbol.
Writes a schema.yaml for the target dir (id-prefix IMP-, marks everything as status: approved and x-reqmd.imported: true).
The reqmd CLI now validates the target with the rest of the spec — same trace checks, same CI gate.

The output is a normal reqmd document. It can be checked, listed, exported, and diff’d like any other. The x-reqmd.imported: true flag and the  HTML-comment marker tell readers and tooling that the file is auto-generated; the writer’s idempotent write-if-changed path won’t touch hand-authored content in the same target tree.

Sample output#

Given internal/cli/check_test.go, the extractor emits a file like spec/03-software/imported/cli/package.md:

# Package: cli



This file is auto-generated by reqmd-import. Do not edit by hand.

## IMP-cli-BenchmarkRunValidationPipeline-80e6286: BenchmarkRunValidationPipeline
```attr
status: approved
x-reqmd.imported: true
x-reqmd.source-file: internal/cli/check_test.go
x-reqmd.source-line: 19
x-reqmd.symbol-kind: function
trace: []

BenchmarkRunValidationPipeline measures the cost of the full validation pipeline (Pass 1 schema compile + per-req validation, plus Pass 2 graph build + trace checks) against the spec/ fixture (6 documents, 249 requirements).


The ID format is `IMP---<7charhash>`. The hash is computed over `(package, symbol name)` so IDs survive line-number shifts — a refactor that moves code around does not invalidate the trace.

The attr block carries:

- `status: approved` — generated items count as upstream coverage providers by default.
- `x-reqmd.imported: true` — flag that tells the schema validator these came from the importer.
- `x-reqmd.source-file` and `x-reqmd.source-line` — provenance back to the source.
- `x-reqmd.symbol-kind` — `function`, `method`, `type`, `const`, `var`, etc.
- `trace: []` — empty by default. Populated by `--heuristic-traces` (see below) or by hand.

## The trace gap

Plain Markdown traceability has a directional problem. A software requirement can `trace: [system-requirements/SYS-001]` and reqmd validates that SYS-001 exists in the upstream document. But the *implementation* — the function in `internal/foo/bar.go` that actually implements the software requirement — has no native concept of a "reqmd ID". Without help, you have to either:

- Manually maintain a mapping table somewhere (a spreadsheet, a comment, a separate `.md` file with hand-rolled `IMP-foo-FooBar-...` IDs that you have to keep in sync with the code).
- Use a separate IDE plugin to jump from code to spec.
- Give up and accept that the spec only goes down to the test layer.

`reqmd-import` does the first option automatically. After a one-shot `reqmd-import extract`, the IDs exist in the spec tree. The agent and the human can grep for them, link to them, and the CI check enforces that they still resolve. When the function is renamed or moved, the next `reqmd-import extract` updates the file; if the ID prefix is unchanged (because it's a hash, not a position), the spec tree still references the same requirement.

## Usage

```sh
# Build
go build -o reqmd-import ./cmd/reqmd-import

# One-shot extraction
reqmd-import extract ./internal ./spec/03-software/imported

# Restrict to one language
reqmd-import extract --lang go ./internal ./spec/03-software/imported-go
reqmd-import extract --lang python ./src ./spec/03-software/imported-py

# Also extract bare requirement IDs from prose (not just explicit markers)
reqmd-import extract --heuristic-traces ./internal ./spec/03-software/imported

# Custom id-prefix (default: IMP-)
reqmd-import extract --id-prefix MY- ./src ./spec/mine

--heuristic-traces looks for bare IDs in the source code’s doc comments and prose (e.g. // implements SYS-001) and populates the trace: field automatically. This is how you wire the spec tree to the source without writing each trace by hand.

After the extraction, validate like any other spec dir:

reqmd check ./spec/

The generated files participate in every check just like hand-authored ones. If you delete a function and re-run the extractor, the corresponding requirement disappears from the spec, and any upstream trace: [IMP-...] line becomes a broken-ref WARNING.

Languages#

Language	Status	Tree-sitter grammar	Notes
Go	Stable	`tree-sitter-go`	Functions, methods, types, constants, variables. Method receivers are part of the ID (`Foo.Bar`).
Python	Stable	`tree-sitter-python`	Functions, classes, methods, constants. Async functions are tagged in the symbol-kind.
Rust	Not implemented	—	The registry slot is reserved; contributions welcome.
Zig	Not implemented	—	Same.

To add a new language: implement the lang.Language interface in a new package under internal/lang/<lang>/, and self-register it via lang.Register(&impl{}) in an init() block. The blank import in cmd/reqmd-import/main.go pulls all plugins into the binary.

Where it fits in the workflow#

A typical end-to-end loop:

# 1. Author the spec by hand for stakeholder / system / software
reqmd init my-feature/
$EDITOR my-feature/stakeholder/goals.md
$EDITOR my-feature/system/features.md
$EDITOR my-feature/software/components.md
reqmd check my-feature/                      # pass

# 2. Write the code, referencing the spec
$EDITOR src/foo/bar.go
# ...function that implements [system-features/SYS-001]...

# 3. Extract the spec from the code
reqmd-import extract ./src ./my-feature/03-software/imported

# 4. Re-validate
reqmd check my-feature/                      # still pass

# 5. CI runs the same check on every PR
# .github/workflows/reqmd-check.yml:
#   - run: reqmd check spec/

# 6. The agent can now follow a trace end-to-end
# "Find the function that implements SYS-001"
$ grep -r "IMP-foo-Bar" spec/ src/
spec/03-software/imported/foo/package.md:## IMP-foo-Bar-...
src/foo/bar.go:func Bar() error {  // implements SYS-001

The agent — and the human — can now answer questions like:

What functions implement SYS-001? — grep trace spec/03-software/imported/ | grep SYS-001
What spec IDs does foo/bar.go correspond to? — reqmd ls spec/03-software/imported/foo/ | grep -i "source-file:.*bar"
What code is unstested? — load CTRF test results with reqmd check spec/ --results ctrf.json; the existing missing-verdict check fires on measures with no test.

What it doesn’t do#

Three things, deliberately:

It doesn’t guess traceability. The default is trace: [] for every generated item. You either annotate the source with // reqmd:trace: [SYS-001] markers, or run with --heuristic-traces, or hand-fill the trace: field in the generated .md. The tool surfaces the source; the trace is yours to declare.
It doesn’t update the spec in place on every commit. You re-run the extractor when you want to refresh. This is a deliberate trade-off — incremental generation adds complexity (and the possibility of subtle drift) for a workflow that’s typically a once-per-PR or once-per-release concern.
It doesn’t reach into binary artefacts. It reads source. It doesn’t introspect compiled binaries or runtime behaviour. If you need that, you’d add a separate runtime-instrumentation step.

Architecture#

The extractor is structured as a small pipeline:

  walkdir        (per-file)
    → language plugin (tree-sitter parse)
      → symbol list (functions, types, constants)
        → trace inference (explicit markers + heuristic)
          → writer (renders per-package .md, idempotent)

Each stage is independently testable; the writer’s idempotency guarantee (byte-identical output for unchanged inputs) is what makes the import step safe to re-run in CI. The tree-sitter grammars are loaded once at startup; the per-file parse is dispatched to a worker pool sized to runtime.NumCPU().