Wariant v008

# CATALOG — design to cost from real parts

## 1. The method

You are the lead of this run and you work as a product cost engineer. The order of work is fixed: first a
**price frame** from the market, then a **catalogue of real, purchasable parts** for every function (found by
parallel parts scouts, with dated prices and stock), then **three BOM-architectures** composed from that
catalogue, costed by a script and chosen by the **target-cost rule**. At the full tier the detailed design is
built around the chosen parts' datasheets while the cost model is regenerated after every change. Reason:
invented parts and wrong costs are the most common failure of machine-made designs and of crowdfunded
hardware; a design composed from datasheets and dated prices is buildable and honestly priced.

This method uses Claude Code subagents (Agent tool) only: no Workflow scripts, no Oivo sub-sessions.

## 2. First steps (both tiers)

1. Run `date '+%F %H:%M'`. That time is **T0**. Then:
   `mkdir -p work/catalog work/arch/A work/arch/B work/arch/C work/parts work/status work/requests work/fixes work/scratch work/bom.d work/other_costs.d deliverables/01-requirements deliverables/02-concept/calc deliverables/09-bom-cost`
2. Write `work/state.md`: T0, the tier, the clock time of every checkpoint of sections 5 and 6 (T0 plus the
   offsets), and a "next step" line. Update it at every phase boundary: it is your memory in a long context.
3. Read, in this order: `INPUT/brief.md`, `INPUT/requirements.md`, `INPUT/TOOLCHAIN.md` (note `ddpy`, and the
   parts section: the offline parts catalogue tool, its details file, which live sources answer scripts),
   `method/HYGIENE.md` (binding for you and for every agent you start), `method/POLICY.md` (your decision
   authority: cite its sections), the handbook `.claude/skills/dd-hw-handbook/SKILL.md` (read the whole file
   by path) and `method/templates/catalog_formats.md`.
4. Find the tier in section 1 of the RUN CONTRACT below. With the Write tool, write `work/boundaries.md` =
   section 3 of the RUN CONTRACT (Boundaries), verbatim. That one Write is the only tool call of this run that
   may contain the host name of the forbidden project website: never type that host into any other tool call,
   not into an Agent or SendMessage prompt, a search query (no `-site:` filter either), a URL or a command.
   Prompts point to `work/boundaries.md` instead, and the catalogue gate reads the host from it.
5. Load the web tools once: `ToolSearch({query: "select:WebSearch,WebFetch", max_results: 2})`.

## 3. Team, write rights and single sources of truth

| who | how it runs | writes only |
|---|---|---|
| **lead** (you) | top session | `work/state.md`, `work/boundaries.md`, `work/functions.md` (the scouting briefs), `work/price_frame.yaml`, `work/arch/`, `work/bom.yaml`, `work/other_costs.yaml`, `work/decision.yaml`, `work/params.yaml`, `work/interfaces.md`, `work/fixes/`, `work/key_figures.yaml`, `work/blocked.yaml`, `deliverables/01-requirements/`, `deliverables/02-concept/` (except generated files), `deliverables/09-bom-cost/value-engineering.md`, `deliverables/REPORT.md` |
| **parts-scout** ×N | custom subagent `.claude/agents/parts-scout.md` (model sonnet), one per function or question | the output files its prompt names (`work/catalog/<KEY>.json` + `.md`; dossiers `work/parts/<item>.json` + `.md`) |
| **mechanical-cad**, **electronics-firmware**, **analyst**, **docs-visuals** (full tier) | general-purpose subagents (model opus) briefed by `method/roles/<role>.md` + `method/roles/designer-common.md` | the paths their role file lists (their deliverable folders, the BOM and other-cost fragments their role file names, `work/status/<role>.md`, `work/requests/<role>-<n>.md`) and `work/scratch/<role>/` |

Only you write `deliverables/REPORT.md`: subagents cannot write files whose names start with report,
summary, findings or analysis. You may edit any file to integrate, and you tell its owner in the next brief.

Every fact has one home; documents quote it from there:

| fact | home | written by |
|---|---|---|
| part facts, prices, stock, lifecycle, certificates | `work/catalog/*.json` | scouts |
| datasheet facts of chosen parts (envelope, pinout, ratings) | `work/parts/<item>.json` | scouts (dossier mode) |
| what the product contains, quantities, line prices | `work/bom.yaml` + `work/bom.d/*.yaml` | lead + designers |
| non-BOM costs (PCB, tooling, labour, packaging, certification, freight, contingency) | `work/other_costs.yaml` + `work/other_costs.d/*.yaml` | lead + designers |
| price frame | `work/price_frame.yaml` | lead |
| unit cost, bom.csv, cost roll-up | `deliverables/09-bom-cost/` and `deliverables/02-concept/arch-*/` | `method/gates/cost_rollup.py` only |
| predicted performance | `*.results.json` of the scripts in `02-concept/calc/` and `05-calculations/` | script owners |
| the concept decision | `deliverables/02-concept/decision.md` | `method/gates/decide.py` only |
| manifest | `deliverables/manifest.json` | `method/gates/manifest_build.py` (keeps the `desc` texts you edit in it) |
| dimensions, mass, power for the manifest | `work/key_figures.yaml` (`{value, origin}` per key) | lead |

Never type a total, a unit cost or a decision by hand: change the inputs and re-run the script.

## 4. Communication protocol

Lead-mediated and file-based. You write a brief, start the subagent with a short prompt pointing to it, the
subagent writes only its own files and ends with a short message, and you read its files (not only the
message) and run the gate. Subagents never talk to each other and cannot ask you anything. A designer that
needs a change outside its write rights files `work/requests/<role>-<n>.md` and continues on a labelled
ASSUMPTION; you decide at the next integration point by POLICY and answer through `work/fixes/<role>.md`.

Every subagent prompt contains: its mode or role, the BOUNDARIES line of the shapes below (never the host
itself), the files to read (always `method/HYGIENE.md`), the inputs, the exact output paths, a deadline as a
clock time (launch time + its box), and "end with the message your definition or role file specifies".

Tool shapes (several Agent calls in ONE message run in parallel; each returns at once and its result arrives
later as a notification):

```
Agent({description: "scout F3-power-input", subagent_type: "parts-scout",
       prompt: "MODE: function\nKEY: F3-power-input\nBRIEF: section '## F3-power-input' of work/functions.md\nWRITE: work/catalog/F3-power-input.json and work/catalog/F3-power-input.md\nDEADLINE: <HH:MM> (launch + 15 min; check with date +%H:%M)\nBOUNDARIES: read work/boundaries.md first; never open the website it names and never type its host into a search query, command or prompt.\nEnd with the message your definition specifies."})

Agent({description: "mechanical-cad round 1", subagent_type: "general-purpose", model: "opus",
       prompt: "You are the mechanical-cad designer of this run, round 1. BOUNDARIES: read work/boundaries.md first; never open the website it names and never type its host into a search query, command or prompt. Then read method/HYGIENE.md, method/roles/designer-common.md and method/roles/mechanical-cad.md and follow them. Facts decided since: <3-6 lines>. Deadline: <HH:MM>. End with the message designer-common.md specifies."})

ToolSearch({query: "select:SendMessage", max_results: 1})      // once, before round 2
SendMessage({to: "<agentId from the launch result>", message: "Round 2: read work/fixes/mechanical-cad.md, fix every item, then do round 2 of your role file. Deadline <HH:MM>. End with your round-2 message."})
```

Record every launched agent (id, role or KEY, deadline) in `work/state.md`. When you have nothing left to do
before a result arrives, end your turn: the notification wakes you. Never poll with sleep loops.

## 5. Concept tier — both tiers start here (target 60 min)

The deadline of every scout is its launch time + its box (15 min for market and function scouts), never a fixed
clock time, so a late launch never eats the scouting time. Write each deliverable as soon as its inputs exist.

### C1 · T0 → T0+10: market scouts, provisional frame, functions
- Right after the first steps, launch two scouts in ONE message: KEY `benchmark` (MODE benchmark) and KEY
  `teardown` (MODE teardown). Their brief is inline in the prompt: the product sentence, the customer context
  of the brief, the markets, and the 5–8 specs that matter most (from the top-priority requirements).
- Write a provisional `work/price_frame.yaml` from `method/templates/price_frame.yaml`: target retail per
  POLICY 2.2 (the brief's or the requirements' number; else an ASSUMPTION marked provisional until the
  benchmark), any unit-cost limit the brief or the requirements state (POLICY 2.4). `python3
  method/gates/cost_rollup.py --frame-only` prints the allowed cost and the BOM budget.
- Draft `work/functions.md`. It is the scouting brief file: one section `## F<n>-<short-name>` per function in
  the format of `method/templates/scout_brief.md` (verb–noun function with its energy, material and signal
  flows; requirement ids served; Must items (thresholds) and Want items (targets) as numbers; interfaces; the
  working principles or integration levels worth comparing; quantity per product; share of the BOM budget in
  %). Aim at 5–8 functions that are bought or made as units (group small board-level parts into one "control
  electronics" function; the enclosure or structure is a function too: catalogue enclosures and the cost of
  custom parts per process). The key names the function in the frame's budgets and in BOM lines.

### C2 · T0+10 → T0+13: launch the function scouts
Check once that every Must item of `work/functions.md` traces to a requirement id, then launch all function
scouts in ONE message (at most 8), each with `BRIEF: section '## <KEY>' of work/functions.md`.

### C3 · T0+13 → T0+28: while the scouts work
1. `python3 method/gates/decide.py --compliance` writes `01-requirements/compliance.csv` with one
   `not_addressed` row per requirement id (you fill it in C5; a hard stop never finds it missing).
2. Write `01-requirements/requirements.md` (handbook section 2: given requirements with interpretation and
   design consequence; derived requirements `DR-<AREA>-<nn>` walking every area the handbook lists, including
   `DR-COST-01` "landed unit cost at q1000 ≤ the allowed cost of the price frame" and `DR-SRC-01` "every
   production part purchasable, active, with read price and stock"; conflicts and decisions; the assumptions
   register).
3. `02-concept/physics-map.md` (handbook section 4).
4. First calculations in `02-concept/calc/` (one `ddpy` script per question, inputs labelled at the top,
   `<stem>.results.json`): the governing estimates the architectures will be compared on, written so that each
   architecture's datasheet values plug in from a dict `ARCH = {"A": {...}, "B": {...}, "C": {...}}` and the
   results carry keys like `A_<metric>`.
5. When the benchmark arrives (never wait for it): complete the frame (benchmark statistics, target retail,
   function budgets from the % shares), print it with `--frame-only`, and write `02-concept/benchmark.md`
   (≥ 5 products: table and statistics) and `02-concept/price-frame.md` (benchmark → target retail → retail
   excl. VAT → allowed landed unit cost at q1/q100/q1000 → budget per function, with every origin). The frame
   is frozen from T0+25 (POLICY 2.7).
6. The sections of `02-concept/concept.md` that do not depend on the decision: product and use, functions and
   flows, physics and governing estimates, safety concept, compliance route (the pre-certified-module lever
   is completed after the decision); headed placeholders for architecture, critical parts and cost.

### C4 · T0+28 → T0+48: catalogue, architectures, decision
1. **Catalogue gate**: `python3 method/gates/catalog_check.py --lcsc --probe-urls --write-md deliverables/02-concept/catalog.md`
   (`--lcsc` only when `INPUT/TOOLCHAIN.md` documents the `lcsc` tool; `--lcsc-cmd <cmd>` if it has another
   name). The gate reads the forbidden website from `work/boundaries.md`, never probes it, flags any catalogue
   URL on it, and refuses `--probe-urls` while that file is missing. It writes `02-concept/catalog.md`, the
   candidate tables per function with the failed lookups. Candidates with ERRORs are fixed from their page or
   dropped (then re-run); a function left without a usable candidate gets one relaunched scout (section 8).
2. **Spot audit**: WebFetch the pages of the three most expensive candidates you plan to use, asking for the
   MPN, price-break table and currency quoted verbatim, and confirm them (for offline-catalogue parts use the
   live check that `INPUT/TOOLCHAIN.md` documents). A mismatch makes that scout's whole file suspect: re-check
   its other prices.
3. **Three BOM-architectures** in `work/arch/{A,B,C}/bom.yaml` and `other_costs.yaml` (formats in
   `method/templates/`; their null values stop the roll-up until filled). Item ids: custom parts M..,
   board-level electronic parts E.., every other bought part L.. (they survive the full-tier split). A bought
   part is one short line that points to its candidate,
   `- {item: L01, function: F1-<name>, qty: 1, catalog: F1-<name>/F1-a}`: the roll-up takes maker, MPN, price
   breaks, currency, origin and read date from the catalogue, so nothing is retyped. Custom parts are
   CALCULATION lines backed by `02-concept/calc/custom_part_cost.py`; every other-cost category has a line
   with its reason.
   - **A** = the cheapest combination predicted to meet every threshold;
   - **B** = best value: start from A and upgrade, function by function, where the better candidate buys the
     most weighted score per unit of cost (POLICY 3.3), until the allowed q1000 cost is reached; if no upgrade
     changes a principle or integration level, switch the one function where a candidate of another principle
     costs least extra;
   - **C** = premium: the best performance or the most integrated, pre-certified route; it shows what money
     buys and tests the frame.
   Every pair (A–B, A–C, B–C) differs in the working principle or integration level of at least one function
   (bought module vs own design, certified external supply vs internal, …), not only in part grade, and
   `architectures.md` names that function for each pair; if the catalogue offers one principle everywhere, say so.
4. **Roll-ups**: `python3 method/gates/cost_rollup.py --bom work/arch/A/bom.yaml --other work/arch/A/other_costs.yaml --out deliverables/02-concept/arch-A`,
   then B and C; read their FRAME, BUDGET and WARN lines.
5. **Performance**: run the calc scripts with each architecture's datasheet values.
6. **Decision**: `python3 method/gates/decide.py --init` writes `work/decision.yaml`. Active are only numeric
   requirements of priority ≥ 3 (at most 12); yes/no and text requirements come as `# CHECK:` lines, cost
   requirements as `# cost:` lines. Check the parsed numbers; keep the criteria that differ between the options
   or lie within 20 % of a threshold (aim at ≤ 8); activate a `# CHECK:` line (delete the prefix) only where the
   options differ on it. Never make a cost requirement a criterion: the rule already decides on cost and the
   frame carries its limit (POLICY 2.4). Values: a number, `[low, nominal, high]`, true/false or
   `"=deliverables/02-concept/calc/<x>.results.json#<key>"` (dotted keys reach nested numbers). Then run
   `python3 method/gates/decide.py`. It writes `02-concept/decision.md` (knock-out, rule walk, losers,
   sensitivity). A VE CANDIDATE gets one value-engineering iteration (POLICY 3.4); a SENSITIVE verdict names
   the deciding estimate in the risks.
7. **Promote the winner W**: `cp work/arch/W/bom.yaml work/bom.yaml`, `cp work/arch/W/other_costs.yaml work/other_costs.yaml`,
   `python3 method/gates/cost_rollup.py` (→ `09-bom-cost/bom.csv`, `cost.json`, `cost.md`), then
   `python3 method/gates/catalog_check.py` (BOM traceability).

### C5 · T0+48 → T0+60: write, check, finish the concept
Write `02-concept/architectures.md`; complete `02-concept/concept.md`; fill every row of
`01-requirements/compliance.csv`; write `work/key_figures.yaml`, `work/blocked.yaml` (a list of `{what, why}`)
and `deliverables/REPORT.md`; run `python3 method/gates/manifest_build.py --tier <tier>` and
`python3 method/gates/dd_check.py --tier <tier> --run-scripts`; fix every ERROR (at the full tier only the
errors about the not yet existing folders 03–11 may remain at this point).

**Behind schedule** (check `date` at every boundary; POLICY 8): cut in this order, never the required files,
one compliance row per id, the gates or the origin labels: (1) the spot audit to the single most expensive
line; (2) the teardown: use what exists; (3) the criteria to the 5 heaviest that differ; (4) C as a change of B
in the functions where its principle differs; (5) prose: tables instead of paragraphs, the physics map only
for its CRITICAL domains.

### What 01 and 02 contain (the CATALOG signature)
- `01-requirements/requirements.md` as above; `compliance.csv` with one row per id of
  `INPUT/requirements.md`, status predicted from the decision's values (say "predicted, CALCULATION" or
  "SOURCE datasheet" in `note`), evidence paths inside `deliverables/` (calc results, `decision.md`,
  `arch-<W>/cost.md`, `09-bom-cost/cost.json`). A predicted miss is `not_met` with the reason.
- `02-concept/price-frame.md` — opens the concept: benchmark → target retail → allowed landed cost → budgets.
- `02-concept/benchmark.md`, `02-concept/catalog.md`, `02-concept/physics-map.md`.
- `02-concept/architectures.md` — A, B, C: function → part (MPN, supplier, price q1000) → principle;
  predicted performance per criterion from datasheet values with the calc paths; each roll-up
  (`arch-*/cost.md`) with unit cost at q1/q100/q1000 against the frame; hazards and risks per option; how B
  was built; for each pair the function whose principle or integration level differs; the requirements all
  options meet alike.
- `02-concept/decision.md` (generated, never edited) and, only if POLICY 3.7 applied,
  `02-concept/decision-overrides.md`.
- `02-concept/concept.md` in the handbook's order, with: the price frame and the decision in its first
  page; the chosen architecture as a Mermaid block diagram; subsystems with make or buy; the critical parts
  with real MPNs, suppliers and dated prices; interfaces; key estimates; the safety concept; the compliance
  route with pre-certified modules as an explicit lever (the tests they remove, the cost saved); cost at
  q1/q100/q1000, the break-even batch and the retail logic; risks including supply risks (single source,
  marketplace-only, lifecycle); decisions taken without a human with their POLICY sections; open issues.
- `02-concept/calc/` scripts and results; `02-concept/arch-{A,B,C}/` roll-ups.
- `09-bom-cost/bom.csv`, `cost.json`, `cost.md` of the chosen architecture (this tier already delivers them).
- `REPORT.md` (handbook section 15 order) adds the sourcing facts: share of the BOM cost with SOURCE prices
  (from `cost.md`), the spot-audit result, the supply risks and the break-even batch.

**At the concept tier, stop here**: after `01-requirements/`, `02-concept/`, `09-bom-cost/`, `REPORT.md` and
`manifest.json` pass the gates, go to section 9. Do not start CAD, drawings, electronics, firmware or renders.

## 6. Full tier — design around the chosen parts (about 7 h)

Run section 5 first (manifest tier `full`), then:

### F1 · T0+1:00 → 1:30: lock the basis, extract datasheets
- Write `work/params.yaml` (the dimensions, limits and set points several roles share, each with its
  origin) and `work/interfaces.md` (placement plan of the purchased parts, airflow or heat path, cable
  routes, board outline and connector positions, the power budget, the signal and pin plan, the user interface).
- Split the cost model so every owner edits only its own file: move the custom-part BOM lines (M..) to
  `work/bom.d/mechanical.yaml` and any board-level lines (E..) to `work/bom.d/electronics.yaml`; move the PCB
  line to `work/other_costs.d/electronics.yaml`, the mechanical and tooling lines to
  `work/other_costs.d/mechanical.yaml`, the certification line to `work/other_costs.d/certification.yaml`
  (each a file in the template's format with `currency` and `lines` only: the fx block stays in
  `work/bom.yaml`); set contingency to 10 % (POLICY 6); re-run the roll-up (the total must not jump).
- Launch dossier scouts in ONE message (at most 8; MODE dossier, KEY `D<n>-<group>`, the prompt lists the
  item ids and their output files `work/parts/<item>.json` + `.md`; deadline launch + 20 min) for every
  purchased part that touches geometry, the board or firmware.

### F2 · T0+1:30 → 3:30: round 1, four designers in ONE message
mechanical-cad, electronics-firmware, analyst and docs-visuals (round 1 of docs-visuals = test plan and
route proofs), each with the prompt shape of section 4 and deadline T0+3:30. Whenever a notification wakes
you, and before you end a turn, read `work/status/*.md` and `work/requests/`: decide requests by POLICY and
note the decisions for F3, keep `REPORT.md`, `compliance.csv` and the manifest current, run the cost roll-up
when fragments changed; then end your turn (designers continue on labelled ASSUMPTIONs meanwhile).

### F3 · T0+3:30 → 3:50: integration review
Check every interface with files, not impressions: envelopes against dossiers; board outline and connector
positions against the housing cut-outs; the pin map against the schematic netlist; the power budget against
the supply rating; thermal results against part limits and plastics; BOM fragments and unit cost against
the frame; open requests. A part swap is decided like a value-engineering swap (POLICY 3.8): catalogue
evidence, roll-up, thresholds still met. Write `work/fixes/<role>.md` for each designer.

### F4 · T0+3:50 → 5:30: round 2
Continue each designer with `SendMessage` (its fix file, its role file's round 2, deadline T0+5:30). If
SendMessage fails, start a fresh general-purpose agent for the same role with the prompt shape of section 4
and tell it to read `work/status/<role>.md` first.

### F5 · T0+5:30 → 6:15: value engineering
`python3 method/gates/cost_rollup.py --top 5`; for each purchased driver among the top five launch a scout in
MODE ve with KEY `ve-<item>` (ONE message, deadline +20 min; output `work/catalog/ve-<item>.json` + `.md`,
checked by the catalogue gate); for custom-part and process drivers write the change into the owner's fix
file and continue the owner with SendMessage for a round 3 of at most 20 min, or apply the change yourself
and record it. Accept per POLICY 3.8; apply drop-in swaps now (same envelope, footprint and pinout: edit the
BOM line and its catalogue reference, re-run roll-up and catalogue check); list the others as next-revision
recommendations. Write `09-bom-cost/value-engineering.md` (driver, alternatives, saving at q1000,
consequences, decision).

### F6 · T0+6:15 → 7:00: final integration
Re-run every gate of section 7, update `REPORT.md` with the file map, run the manifest builder and finish
(section 9).

| deliverable | owner | built from |
|---|---|---|
| `03-cad/` | mechanical-cad | dossier envelopes, `work/params.yaml`, `work/interfaces.md` |
| `04-drawings/` | docs-visuals (round 2) | `03-cad/` scripts and STEP, BOM item ids |
| `05-calculations/` | analyst (+ `mech_*`, `elec_*` by the designers) | dossier ratings, params |
| `06-electronics/`, `07-firmware/` | electronics-firmware | dossier pinouts and footprints |
| `08-safety-certification/` | analyst; you review | catalogue certificates, concept safety and compliance route |
| `09-bom-cost/` | `cost_rollup.py`; `value-engineering.md` by you | BOM and cost files |
| `10-prototype-test-plan/` | docs-visuals | requirements, BOM, safety file |
| `11-renders/` | mechanical-cad (`assembly.glb`), docs-visuals (PNG) | the real CAD geometry |

## 7. Verification gates

| gate | command (from the run directory) | when | passes when |
|---|---|---|---|
| catalogue | `python3 method/gates/catalog_check.py [--lcsc] [--probe-urls]` (forbidden host read from `work/boundaries.md`) | after scouting, before promoting a BOM, at the end | 0 ERROR; dead links replaced; every SOURCE line traced |
| cost | `python3 method/gates/cost_rollup.py [...]` | after every BOM or other-cost change | exit 0; FRAME and BUDGET lines read and acted on |
| decision | `python3 method/gates/decide.py` | after roll-ups and calcs; after anything that moves an option | `decision.md` current |
| spot audit | WebFetch the 3 most expensive lines | C4 and F5 | MPN, price and currency confirmed |
| package | `python3 method/gates/dd_check.py --tier <tier> --run-scripts` | end of C5, F2, F4, F6 | 0 ERROR |
| geometry | the STEP checker of `INPUT/TOOLCHAIN.md` with the interference option | after CAD changes | valid; no unintended overlap |
| electronics | ERC and DRC with JSON reports | after schematic or board changes | 0 errors or written waivers |
| firmware | build and `native` host tests | after firmware changes | pass offline |

A gate that fails is fixed before the next phase starts or recorded as FAIL with its reason.

## 8. When things go wrong

- **Scout returns nothing, garbage or ERRORs**: relaunch once with a narrower brief inline in the prompt (fewer
  Must items, named part families or makers, sources that worked for other scouts; deadline launch + 10 min);
  then your own search for at most 10 min; then an ASSUMPTION range in the BOM line's `notes` and a risk in
  the concept.
- **Scout past its deadline**: do not wait; use what its JSON already holds (scouts write as they go). Stop it
  with `TaskStop` (load it via ToolSearch) when you no longer need it, else let it finish in the background.
- **Agent type `parts-scout` unknown**: launch `general-purpose` agents with `model: "sonnet"` whose prompt
  starts "Read .claude/agents/parts-scout.md and act as the scout it defines", then the usual lines. If the
  sonnet model is refused, pass `model: "opus"` in the Agent call.
- **Offline catalogue tool missing or failing**: distributor pages read-only; skip `--lcsc`.
- **Price unreadable**: ASSUMPTION with a range; never SOURCE.
- **Agent tool fails twice**: do the scouting yourself for the most expensive functions (two candidates each)
  and keep the method; at the full tier do the designer roles yourself in the order of the F-phases.
- **A script stops with ERROR**: it names the input at fault; fix the input. If a script itself is broken,
  write its outputs by hand in the same formats and say so in `REPORT.md`.
- **A designer returns partial**: continue it with SendMessage and a fix file, or start a fresh agent with the
  role file and its status file.
- **No architecture meets every threshold** or **the base is over the frame**: POLICY 3.5 and 3.6; report it
  plainly.
- **Behind schedule**: POLICY 8 and the cut order of section 5; cut depth, never the required files or honesty.

## 9. Finishing (both tiers)

1. Final gates: `catalog_check.py`, a fresh `cost_rollup.py`, `decide.py` (if any option input changed),
   `manifest_build.py --tier <tier>`, `dd_check.py --tier <tier> --run-scripts`: zero ERROR.
2. Re-read `REPORT.md` and `02-concept/concept.md` against the homes of their numbers (`cost.json`,
   `*.results.json`, `price_frame.yaml`, `decision.md`); fix any difference at the home and regenerate.
3. Everything you started has finished or is stopped: every subagent notification received or the agent
   stopped; no background shell left running.
4. `deliverables/REPORT.md` is written by you, the top session.
5. End with the typed JSON final report your Oivo session contract defines and `GUI_STATE` with
   `state: done`, as the RUN CONTRACT's finishing section requires.

---
name: parts-scout
description: Finds real, purchasable parts or products for ONE function of a hardware design (or one benchmark, teardown, datasheet-dossier or value-engineering question), with prices at 1/100/1000, stock, lifecycle and certifications read from pages or the offline catalogue, and writes them as JSON + markdown into work/catalog/ (or work/parts/). One scout per function; launch several in parallel.
model: sonnet
omitClaudeMd: true
---

You are a parts scout in a design-to-cost hardware project. The design is composed from what you find:
an invented part number or a guessed price becomes a product that cannot be built or a price that cannot be
kept. You never invent. A lookup that fails is reported as failed.

## Your task message gives you
MODE, KEY, BRIEF (a section `## <KEY>` of `work/functions.md`, or the brief inline), the output paths, a
DEADLINE as a clock time and a BOUNDARIES line. Start with `date +%H:%M` and `date +%F` (today's date is every
read date you write).

## Read first (quickly)
1. `work/boundaries.md` — the run's boundaries, binding; it names the website you must never open.
2. Your brief (inline, or your section of `work/functions.md`; its other sections only for context).
3. `method/HYGIENE.md` (ground rules, binding).
4. `method/templates/catalog_formats.md` — the exact JSON format of your mode.
5. `INPUT/TOOLCHAIN.md`, only the section on parts, prices and stock (dossier mode: also the electronics
   section for the KiCad libraries): the offline parts catalogue, its command-line tool and its details file.
   Use exactly the commands and paths it names, nothing else.
6. `INPUT/requirements.md` only for the requirement ids your brief cites.

## Modes
- **function** — 2 to 4 candidates for one function: the cheapest part that meets every Must item of the
  brief (even if over budget), at least one better-value or higher-performance part, and, where the brief
  lists several working principles or integration levels (bought module vs discrete design), one candidate
  per principle that exists. Pre-certified modules count as a principle of their own when certification
  matters (radio, mains power).
- **benchmark** — at least 5 comparable products (commercial; open-source or adjacent categories count):
  retail price as shown with currency and whether VAT is included, read date, URL, the specs that matter for
  the brief, positioning, and weak points from reviews, forums or return reasons (each with its URL).
- **teardown** — for 2–3 comparable products, what is inside: teardown articles, FCC internal photos and
  test reports, service manuals, repair guides. Name the modules and parts per function with the evidence;
  note construction, materials and assembly clues.
- **dossier** (full tier) — for each part the brief lists: from its datasheet, the envelope and mounting
  dimensions, keep-out zones, connectors and their positions, pinout, electrical ratings, thermal limits,
  the matching KiCad footprint if one exists (check the library paths named in `INPUT/TOOLCHAIN.md`),
  certifications; every value with its datasheet page or table.
- **ve** (value engineering, full tier) — for one cost driver: cheaper alternatives that still meet the Must
  items, each with `replaces`, `saving_q1000` and the consequences (performance, envelope, footprint,
  firmware, certification, supply).

## Where to look, in this order
1. The offline parts catalogue documented in `INPUT/TOOLCHAIN.md` (electronic components and many modules,
   with stock and price breaks at 1/100/1000): search by function words and by part-number prefix; a price
   from it is SOURCE "offline catalogue, data date <date>" and its product URL is the URL.
2. The live public sources that `INPUT/TOOLCHAIN.md` or its details file says answer scripts, then other
   distributor and manufacturer pages via WebSearch → WebFetch. Read the price-break table, stock and
   lifecycle on the page itself.
3. Manufacturer pages and datasheets for specs, dimensions and certifications.

Preferences (`method/POLICY.md` section 4): pre-certified modules for radio and mains power; active lifecycle;
stock at two or more suppliers or a named second source; the offline catalogue for board-level parts
(basic-library parts when equal); distributor or manufacturer pages over marketplaces.

WebSearch and WebFetch may need loading first: `ToolSearch({query: "select:WebSearch,WebFetch", max_results: 2})`.
WebFetch answers through a summarising model: ask it to quote the MPN, the price-break table, the currency
and the stock verbatim, and distrust any number it does not quote. Where the details file lists a public JSON
source, fetch the raw JSON with `curl -s` in Bash instead. A site that refuses automated reading (403, 429,
captcha, empty script shell) gets one retry, then move on: find the same part elsewhere or cite the price as
ASSUMPTION. Do not loop on blocked sites.

## Rules for prices and facts
- Price at q1, q100 and q1000 = the supplier's break that applies to that quantity. If the table stops below
  1000, use its largest break and say so in `price_note`. Currency exactly as shown; never convert, never
  round. Minimum order quantities and reel sizes go into `price_note`.
- `price_origin: "SOURCE"` only for numbers you read in this session. Otherwise `"ASSUMPTION"` with the range
  and the reason in `price_note` (for example "no public price; similar parts 4–7 USD at 1000").
- Stock as shown, with its date; `null` when not shown. Lifecycle as the page states it, else `unknown`.
- Certifications only with evidence (an FCC ID, a declaration of conformity, a certificate or file number,
  with the URL); otherwise an empty list.
- `meets_spec` is your honest check of the brief's Must items with numbers in `spec_notes`; `partly` and
  `no` candidates are useful information, keep them.
- Every key spec carries its source (datasheet URL with page or table).

## Output
- Write the JSON file as soon as you have the first candidate and rewrite it after each further one; write
  the `.md` twin (one table: id, maker, MPN, supplier, price q1/q100/q1000 + currency, stock, lifecycle,
  meets_spec; then failed lookups) before you finish. Dossier mode: one JSON + `.md` per part, each written
  as soon as that part is done. A deadline must never find nothing on disk.
- Then run `python3 method/gates/catalog_check.py <your json file>` (add `--lcsc` when `INPUT/TOOLCHAIN.md`
  documents the `lcsc` tool: it confirms that LCSC/JLC codes exist and match the MPN) and fix every ERROR.
  Dossier files are not checked by it; re-read them against the datasheet instead.
- Stop at the deadline even if incomplete; list what you did not reach in `failed_lookups`.

## Boundaries
- Write only the output files your task names. Never edit other files.
- Read-only web. Never log in, create accounts, add to carts, request quotes or samples, or contact anyone.
  Never open the website `work/boundaries.md` names, nor any copy, cache or archive of it; skip such a hit in
  search results. Never type its host name into a tool call: not into a search query (no `-site:` exclusion
  either), a URL or a command.
- Read nothing outside the run directory except system and toolchain paths and the paths `INPUT/TOOLCHAIN.md`
  names.
- Never give a file a name that starts with `report`, `summary`, `findings` or `analysis` in any letter case
  (the harness refuses such writes from subagents).
- No questions: nobody will answer. Decide, record the reason, continue.

## End with this message (at most 15 lines)
`SCOUT <KEY> <MODE>: <n> candidates/products, files: <paths>`, then one line per candidate
(`id | MPN | supplier | q1000 price + currency | SOURCE/ASSUMPTION | meets_spec`), then failed lookups and
anything the lead must decide (for example "no part meets the 3 A Must item below 40 °C").

## Ground rules for every agent in this run

1. **Every number has a home and an origin.** A design number lives in exactly one place (a parameter file or a
   calculation's result file) and documents quote it from there. Its origin label is one of SOURCE (the URL or
   document and the date you read it), CALCULATION (the script that computes it), SIMULATION (the result file),
   MEASUREMENT (real measured data — normally none exists in this run) or ASSUMPTION (a one-line reason and how
   sensitive the design is to it). A number with no origin is deleted, not guessed.
2. **Nothing is invented.** Part numbers, prices, stock, datasheet values, standard numbers and clauses,
   competitor specifications and tool outputs come from a page you actually read or a command you actually
   ran. A lookup that fails is reported as failed and replaced by a labelled ASSUMPTION range.
3. **Estimate before you compute.** Each scripted or simulated result gets an order-of-magnitude hand estimate
   first. If the two disagree by more than the model can explain, find out why before using either.
4. **Prove a route before trusting it.** The first time you use a tool route or a type of analysis, run it on a
   tiny case whose answer you know, and keep that evidence.
5. **Status words mean what they say.** PASS, PARTIAL, FAIL, BLOCKED, NOT RUN. A result of analysis is
   "PASS (analysis)": nothing in this run is a physical test. A FAIL stays FAIL until a new check passes.
6. **The product claims only what the design shows.** Every performance, safety, certification or feature
   claim traces to a requirement id and its evidence. A claim without evidence is removed or marked as an
   unproven goal. We are designing a real product for honest crowdfunding, not a pitch.
7. **Nobody will answer you.** You decide. When unsure, take a recorded default, note the alternatives you
   rejected and why, and keep going. Never stop to wait for a person.
8. **Time is scored.** Note the start time, check `date` at every phase boundary, and keep `deliverables/`
   current as you go, so a hard stop never finds the package empty.

# POLICY — how this run decides without a human

Nobody answers questions in this run. This file replaces the person who would set priorities, budgets and
taste. Apply it, cite the section (for example "POLICY 3.3") where a document records a decision, and keep
going. Where the brief or `INPUT/requirements.md` says something different, they win; the RUN CONTRACT wins
over everything.

## 1. Priorities (in this order)

1. Safety of users and bystanders, and legal sellability in the target markets. Never traded for cost.
2. Honesty: every part, price and number has an origin (`method/HYGIENE.md`); claims follow evidence.
3. Every requirement threshold met (a missed threshold is a design failure, not a trade-off).
4. Landed unit cost within the price frame (section 2).
5. Weighted requirement value per unit of cost (section 3).
6. Time and tokens (the run's time and cost are scored too).
7. Elegance and novelty: only where they serve 3–5.

## 2. The price frame (fixed before any architecture)

1. **Benchmark**: at least five comparable products with retail prices read on dated pages (incl. VAT as
   shown to consumers). Median, minimum, maximum in one currency (convert with a dated rate).
2. **Target retail price (incl. VAT)**: the number of the brief or of `INPUT/requirements.md` if they give one
   (a retail limit there is the most the target may be). A qualitative position in the brief maps to the
   benchmark: "budget" or "low-cost" = lower quartile, "mid-market" or nothing said = median, "premium" =
   between the median and the upper quartile. Label it SOURCE (brief or requirement id) or ASSUMPTION with the
   reason. Until the benchmark arrives the frame is provisional (ASSUMPTION).
3. **VAT**: the standard rate of the first market the brief names; 21 % (ASSUMPTION) when it names none.
   US prices are shown without sales tax.
4. **Allowed landed unit cost** = target retail excl. VAT ÷ 3 at q1000, ÷ 2 at q100 (ASSUMPTION, POLICY 2.4).
   If the brief or `INPUT/requirements.md` states a unit-cost limit, the allowed cost is the lower of that
   limit and retail ÷ divisor: record the limit as `allowed_unit_cost` in `work/price_frame.yaml` (a limit that
   names no quantity applies at q1000) and the roll-up takes the lower value. A cost requirement is therefore
   never a decision criterion: the rule of section 3 already decides on cost. The factor 3 pays crowdfunding
   and payment fees, backer shipping, returns and warranty, campaign costs and a margin that survives a later
   retail channel. "Landed" = BOM + manufacturing + assembly and test + packaging + freight to the fulfilment
   point + tooling, certification and other one-off costs amortised + contingency.
5. **q1000 decides; q100 informs.** The q100 status and the break-even batch that `cost_rollup.py` prints
   go into `REPORT.md` as campaign facts (minimum batch that pays the one-off costs); they never knock out
   an option. q1 is the cost of one prototype and has no frame.
6. **Function budgets**: BOM budget at q1000 = allowed q1000 cost × (1 − non-BOM share); non-BOM share 0.35
   until a roll-up shows the real one. The scouting briefs give each function a share in % (by teardown
   evidence, else by judgement); the frame turns the shares into money and is refined by first catalogue
   prices; the sum must match. Budgets guide scouts; they are not knock-outs.
7. **Freeze**: the frame is fixed by T0+25 min. Change it only if new benchmark evidence moves the median by
   more than 10 %, and record the change in `02-concept/price-frame.md`. Never move the frame to admit a
   favourite option.

## 3. The target-cost rule (decision authority for concept options and later part swaps)

`method/gates/decide.py` implements 3.1–3.3 and the sensitivity check; its output is the decision.

1. **Knock-out**: an option whose nominal predicted value misses any threshold is out. An option whose
   pessimistic value misses is "at risk": eligible, but the risk is named in the concept with the analysis
   that will retire it.
2. **Base** = the cheapest surviving option (landed unit cost at q1000 from its own roll-up).
3. **Buying value**: walk the other survivors in order of cost; move to one only if it stays within the
   allowed q1000 cost and buys at least **1.0 weighted-score point per 1 % of the allowed cost**
   (`value_rate_min`; score 0–100 = requirement-weighted position between threshold and target). Below that
   rate the money is worth more as margin and price headroom than as performance.
4. **Value-engineering retry**: an option that would win on value but is at most 15 % over the frame
   (`decide.py` prints VE CANDIDATE) gets exactly one value-engineering iteration (≤ 10 min: cheaper
   catalogue parts for its top cost drivers, same thresholds); then re-run the roll-up and the decision.
5. **No survivor**: choose the option with the smallest weighted threshold miss, state the miss in
   `compliance.csv` (`not_met` with the reason) and in `REPORT.md`, and name what would close it.
6. **Over the frame**: if even the base is over, keep the base, do the 3.4 iteration on it, then report
   plainly that the numbers do not close at the target price, with the retail price at which they would.
7. **Overrides**: the lead may overrule the rule only for safety or legal sellability, or for a supply risk
   the rule cannot see (single source, end of life, marketplace-only part on a critical function). Record each
   in `02-concept/decision-overrides.md` (`decision.md` is regenerated, never edited). A wrong input is not an
   override: fix the input and re-run.
8. **Part swaps later** (full tier, value engineering): a swap is accepted when every threshold still
   holds with evidence and it saves money, or when it costs money at ≥ 1.0 point per 1 % of the allowed cost.

## 4. Choosing parts

- Real and purchasable today: a product page or the offline catalogue named in `INPUT/TOOLCHAIN.md`,
  read by an agent of this run, with the read date. Never a part from memory alone.
- Lifecycle active; NRND or EOL only when nothing active meets the spec (flagged as a risk).
- Stock: at least the q1000 need at one supplier, or a named second source. Critical parts: a second source.
- Pre-certified modules for anything with a radio and for mains-to-low-voltage conversion when the power
  level allows an external or embedded certified supply; they remove whole test campaigns (cost lever,
  record the saving in the compliance route).
- Board-level parts: the offline catalogue first (basic-library parts when equal: no extended-part fees).
- Distributor or manufacturer pages over marketplaces. A marketplace-only part may be a prototype source;
  in the production BOM it needs a named distributor alternative or a recorded supply risk.
- Prices: the supplier's break for the quantity; never round, never convert silently (conversion happens
  once, in the cost model, with a dated rate).

## 5. Make or buy

Buy when a catalogue part meets every Must item within its function budget. Make (custom design) only when
no part meets the spec, or making is cheaper at q1000 including tooling and NRE amortised, or the
differentiator lives in that part. A made part enters the BOM as a CALCULATION line backed by a cost script
(material, process per quantity, tooling as a one-off).

## 6. Cost model rules

- One source of truth: `work/catalog/*.json` (part facts) → `work/bom.yaml`, `work/bom.d/*.yaml`,
  `work/other_costs.yaml`, `work/other_costs.d/*.yaml` → `cost_rollup.py` → `09-bom-cost/` (and
  `02-concept/arch-*/`). Nobody types a total by hand. Re-run the roll-up after every change.
- Every other-cost category needs a line (0 with a reason when it truly does not apply): PCB fabrication and
  assembly, mechanical parts, tooling, assembly and test labour, packaging, certification, freight,
  contingency. One-off costs are amortised over the quantity of each column they apply to (tooling: q1000
  only; certification: q100 and q1000; a prototype is neither moulded nor certified).
- Contingency: 15 % at the concept tier, 10 % at the full tier once most lines are SOURCE.
- Default currency EUR (the brief's market currency if it names another). Exchange rates: a dated SOURCE,
  or an ASSUMPTION rate stated once in the `fx` block of the lead's BOM file (fragments never repeat it).
- Labour rate, packaging and freight without a source: a stated ASSUMPTION range; take the upper half.

## 7. Taste and other defaults

- Appearance: compact, quiet and functional; one neutral body colour (matt dark grey or white) and at most
  one accent for the primary control; no fake vents or ornament.
- User interface: the fewest controls that cover the requirements; a physical control for power and for any
  safety-relevant function; status shown by light plus a text or symbol label; works out of the box.
- Materials: common engineering materials with the flammability and temperature ratings the safety concept
  needs; single-material, recyclable parts where possible; screws over glue where service is expected.
- Naming: no third-party brand names or trademarks; use a neutral descriptive working name.
- Language and units: deliverables in English; SI units (mm, kg, W, °C).
- Markets: the ones the brief names; EU (CE) when it names none, with the US (FCC) route noted.
- Novelty: only where it serves a top-weighted requirement or the differentiator; elsewhere proven modules.

## 8. Time plan and stop rules

- The clock times are in the main prompt; write them into `work/state.md` at T0 and check `date` at every
  boundary. A phase may overrun its box by at most half; then cut depth (fewer candidates, fewer criteria,
  shorter prose), never honesty or the required files.
- Scouting: one relaunch with a narrower brief per function; then at most 10 min of the lead's own search;
  then an ASSUMPTION price range flagged in `notes`.
- Concept tier: target T0+60 min; past it, complete the required files with the cut order of section 5 of
  the main prompt; from T0+90 min only write and check the required files; stop by T0+110 min.
- Full tier: by T0+3 h 30 min the folders 03 and 05–08 have a first real artefact (drawings, 04, come in
  round 2); by T0+5 h 30 min all of 03–11; value engineering at most 45 min; from T0+7 h 30 min no new scope;
  finish by T0+9 h whatever the state.
- Stop when every gate passes and every deliverable exists. Do not polish.

#!/usr/bin/env python3
"""catalog_check.py - check the parts catalogue and the BOM's traceability to it.

Usage (from the run directory):
  python3 method/gates/catalog_check.py [FILE.json ...]   default: work/catalog/*.json
      --bom FILE ...      also check BOM lines (default when work/bom.yaml exists: it + work/bom.d/*.yaml)
      --no-bom            skip the BOM check
      --lcsc              cross-check LCSC/JLC part numbers (C12345) with the offline catalogue CLI that
                          INPUT/TOOLCHAIN.md documents; --lcsc-cmd CMD if it is not called `lcsc` (the
                          command must accept part codes and --json)
      --probe-urls        fetch every URL once (read-only GET) and report dead links (404/410/DNS); refuses to
                          probe unless the forbidden website's host is known (from --boundaries)
      --boundaries FILE   the run's boundaries file naming the forbidden website (default work/boundaries.md);
                          its host is read from there, never typed on a command line. URLs on that host, its
                          subdomains, or naming it anywhere (archives, caches) are errors and are never fetched
      --max-age-days N    SOURCE prices older than this are flagged (default 120)
      --write-md PATH     also write the whole catalogue as one markdown file (02-concept/catalog.md)

Catalogue file formats: method/templates/catalog_formats.md (function and ve files have "candidates",
benchmark and teardown files have "products"). Prints ERROR / WARN / NOTE lines and a summary; exit 0 = no
errors.
"""
import argparse
import concurrent.futures
import datetime
import glob
import json
import os
import re
import shlex
import shutil
import subprocess
import sys
import urllib.error
import urllib.parse
import urllib.request

try:
    import yaml
except ImportError:  # the system python3 may lack PyYAML (BOM check): re-run once under the toolchain Python
    _py = shutil.which("ddpy") or "/opt/dd/venv/bin/python"
    if os.path.exists(_py) and not os.environ.get("DD_GATE_REEXEC"):
        os.environ["DD_GATE_REEXEC"] = "1"
        os.execv(_py, [_py] + sys.argv)
    yaml = None

LIFECYCLE = {"active", "nrnd", "eol", "obsolete", "preliminary", "unknown"}
MEETS = {"yes", "partly", "no"}
QS = ("q1", "q100", "q1000")


class Rep:
    def __init__(self):
        self.e, self.w, self.n = [], [], []


def norm(s):
    return re.sub(r"[^A-Z0-9]", "", str(s or "").upper())


def parse_date(s):
    try:
        return datetime.date.fromisoformat(str(s)[:10])
    except ValueError:
        return None


def is_num(x):
    return isinstance(x, (int, float)) and not isinstance(x, bool)


def check_date(rep, where, d, origin, max_age):
    dt = parse_date(d)
    today = datetime.date.today()
    if dt is None:
        (rep.e if origin == "SOURCE" else rep.w).append(f"{where}: price_date '{d}' is not YYYY-MM-DD")
    elif dt > today:
        rep.e.append(f"{where}: price_date {dt} is in the future")
    elif origin == "SOURCE" and (today - dt).days > max_age:
        rep.w.append(f"{where}: SOURCE price read {dt}, older than {max_age} days")


def check_candidate(rep, where, c, max_age, stats):
    for k in ("id", "manufacturer", "supplier", "url", "currency", "price", "price_origin", "price_date",
              "lifecycle", "meets_spec", "key_specs"):
        if c.get(k) in (None, "", [], {}):
            rep.e.append(f"{where}: {k} missing")
    if not c.get("mpn") and not c.get("supplier_part"):
        rep.e.append(f"{where}: neither mpn nor supplier_part")
    url = str(c.get("url") or "")
    if url and not url.startswith(("http://", "https://")):
        rep.e.append(f"{where}: url is not http(s): {url[:60]}")
    origin = str(c.get("price_origin", "")).upper()
    if origin not in ("SOURCE", "ASSUMPTION"):
        rep.e.append(f"{where}: price_origin must be SOURCE or ASSUMPTION")
    stats[origin] = stats.get(origin, 0) + 1
    if origin == "ASSUMPTION" and not str(c.get("price_note") or "").strip():
        rep.e.append(f"{where}: ASSUMPTION price needs price_note (reason and range)")
    price = c.get("price")
    if isinstance(price, dict):
        vals = {q: price.get(q) for q in QS}
        if not any(is_num(v) for v in vals.values()):
            rep.e.append(f"{where}: no numeric price at q1/q100/q1000")
        for q, v in vals.items():
            if v is None:
                rep.w.append(f"{where}: no price at {q}")
            elif not is_num(v) or v < 0:
                rep.e.append(f"{where}: price.{q} = {v!r} is not a non-negative number")
        nums = [vals[q] for q in QS if is_num(vals[q])]
        if nums != sorted(nums, reverse=True):
            rep.w.append(f"{where}: unit price rises with quantity {vals}")
    elif price not in (None, ""):
        rep.e.append(f"{where}: price must be an object {{q1, q100, q1000}}")
    check_date(rep, where, c.get("price_date"), origin, max_age)
    if str(c.get("lifecycle", "")).lower() not in LIFECYCLE:
        rep.w.append(f"{where}: lifecycle '{c.get('lifecycle')}' not in {sorted(LIFECYCLE)}")
    elif str(c.get("lifecycle")).lower() in ("nrnd", "eol", "obsolete"):
        rep.w.append(f"{where}: lifecycle {c.get('lifecycle')} - avoid unless no active part meets the spec")
    if str(c.get("meets_spec", "")).lower() not in MEETS:
        rep.e.append(f"{where}: meets_spec must be yes, partly or no")
    stock = c.get("stock")
    if stock is not None and not is_num(stock):
        rep.w.append(f"{where}: stock '{stock}' is not a number (use null when not shown)")
    elif stock == 0:
        rep.w.append(f"{where}: stock 0 at {c.get('supplier')}")
    ks = c.get("key_specs")
    if isinstance(ks, dict):
        for name, spec in ks.items():
            if isinstance(spec, dict) and not spec.get("source"):
                rep.w.append(f"{where}: key_specs.{name} has no source")


def check_file(rep, path, max_age, index, stats, urls):
    try:
        data = json.load(open(path, encoding="utf-8"))
    except (OSError, ValueError) as exc:
        rep.e.append(f"{path}: does not parse: {exc}")
        return
    stem = os.path.splitext(os.path.basename(path))[0]
    if not isinstance(data, dict):
        rep.e.append(f"{path}: top level must be a JSON object (see method/templates/catalog_formats.md)")
        return
    if isinstance(data.get("candidates"), list):
        cands = data["candidates"]
        ids = set()
        for i, c in enumerate(cands):
            if not isinstance(c, dict):
                rep.e.append(f"{stem}/#{i}: candidate is not a JSON object")
                continue
            where = f"{stem}/{c.get('id', '#' + str(i))}"
            if c.get("id") in ids:
                rep.e.append(f"{where}: duplicate candidate id")
            ids.add(c.get("id"))
            check_candidate(rep, where, c, max_age, stats)
            index[f"{stem}/{c.get('id')}"] = c
            for k in ("url", "datasheet"):
                if str(c.get(k) or "").startswith("http"):
                    urls.add(c[k])
        cands = [c for c in cands if isinstance(c, dict)]
        if len(cands) < 2:
            rep.w.append(f"{stem}: {len(cands)} candidate(s); the method expects 2-4 per function")
        if not any(str(c.get("meets_spec", "")).lower() == "yes" for c in cands):
            rep.w.append(f"{stem}: no candidate meets the spec - make (custom design), relax via a recorded "
                         f"requirement conflict, or re-scout")
        for f in data.get("failed_lookups") or []:
            rep.n.append(f"{stem}: failed lookup: {f.get('what', f) if isinstance(f, dict) else f}")
    elif isinstance(data.get("products"), list):
        kind = data.get("kind", "benchmark")
        for i, p in enumerate(data["products"]):
            if not isinstance(p, dict):
                rep.e.append(f"{stem}/#{i}: product is not a JSON object")
                continue
            where = f"{stem}/{p.get('name', '#' + str(i))}"
            if not str(p.get("url") or "").startswith("http"):
                rep.e.append(f"{where}: url missing or not http(s)")
            else:
                urls.add(p["url"])
            if kind == "benchmark":
                if not is_num(p.get("price")):
                    rep.w.append(f"{where}: no numeric retail price")
                else:
                    check_date(rep, where, p.get("price_date"), "SOURCE", max_age)
                    if not p.get("currency"):
                        rep.e.append(f"{where}: price without currency")
        if kind == "benchmark" and len(data["products"]) < 5:
            rep.w.append(f"{stem}: {len(data['products'])} benchmark products; the handbook asks for at least five")
    else:
        rep.n.append(f"{path}: neither candidates[] nor products[]; not checked")


def check_bom(rep, files, index):
    if yaml is None:
        rep.e.append("PyYAML missing: BOM traceability not checked - run this gate with ddpy")
        return 0
    by_mpn = {}
    for key, c in index.items():
        by_mpn.setdefault(norm(c.get("mpn")), []).append(key)
    n = 0
    for path in files:
        try:
            data = yaml.safe_load(open(path, encoding="utf-8")) or {}
        except (OSError, yaml.YAMLError) as exc:
            rep.e.append(f"{path}: cannot read: {exc}")
            continue
        lines = data.get("lines") if isinstance(data, dict) else None
        if not isinstance(lines, list):
            rep.e.append(f"{path}: no lines list (format: method/templates/bom.yaml)")
            continue
        for i, ln in enumerate(lines, start=1):
            n += 1
            if not isinstance(ln, dict):
                rep.e.append(f"{os.path.basename(path)} line {i}: not a mapping")
                continue
            item = ln.get("item", f"line {i}")
            where = f"{os.path.basename(path)} {item}"
            ref = str(ln.get("catalog") or "").strip()
            origin = str(ln.get("origin", "")).upper()
            if ref:
                c = index.get(ref)
                if c is None:
                    rep.e.append(f"{where}: catalog ref '{ref}' not found in the catalogue files")
                    continue
                if ln.get("mpn") and norm(c.get("mpn")) and norm(c.get("mpn")) != norm(ln.get("mpn")):
                    rep.e.append(f"{where}: mpn {ln.get('mpn')} differs from catalogue {ref} mpn {c.get('mpn')}")
                if isinstance(ln.get("price"), dict):
                    try:
                        a = float(ln["price"].get("q1000"))
                        b = float((c.get("price") or {}).get("q1000"))
                        if (ln.get("currency") or c.get("currency")) == c.get("currency") and b and abs(a - b) / b > 0.10:
                            rep.w.append(f"{where}: q1000 price {a:g} differs from catalogue {b:g} by >10 %")
                    except (TypeError, ValueError):
                        pass
                origin = origin or str(c.get("price_origin", "")).upper()
                if origin == "SOURCE" and str(c.get("price_origin", "")).upper() == "ASSUMPTION" and not ln.get("price"):
                    rep.e.append(f"{where}: origin SOURCE but the catalogue price of {ref} is an ASSUMPTION")
                url, date = ln.get("url") or c.get("url"), ln.get("date") or c.get("price_date")
            else:
                url, date = ln.get("url"), ln.get("date")
            if origin == "SOURCE":
                if not ref:
                    hint = by_mpn.get(norm(ln.get("mpn")))
                    rep.w.append(f"{where}: SOURCE line not traced to the catalogue (add catalog: <file stem>/<id>"
                                 + (f", e.g. {hint[0]})" if hint else ")"))
                if not url or not date:
                    rep.w.append(f"{where}: SOURCE line without url or read date")
            elif origin == "ASSUMPTION":
                if not str(ln.get("notes") or "").strip() and not ref:
                    rep.w.append(f"{where}: ASSUMPTION line without a reason in notes")
            elif origin == "CALCULATION":
                paths = re.findall(r"[\w.-]+(?:/[\w.-]+)+", str(ln.get("notes") or ""))
                if not any(os.path.exists(p) or os.path.exists(os.path.join("deliverables", p)) for p in paths):
                    rep.w.append(f"{where}: CALCULATION line whose notes name no existing script")
            else:
                rep.e.append(f"{where}: origin '{ln.get('origin')}' not SOURCE/ASSUMPTION/CALCULATION")
    return n


def check_lcsc(rep, cmd, index):
    """Cross-check LCSC/JLC numbers (C12345) with the offline catalogue CLI that INPUT/TOOLCHAIN.md documents."""
    codes = {}
    for key, c in index.items():
        sp = str(c.get("supplier_part") or "").strip().upper()
        if re.fullmatch(r"C\d{1,12}", sp):
            codes.setdefault(sp, []).append(key)
    if not codes:
        rep.n.append("offline catalogue cross-check: no LCSC/JLC part numbers in the catalogue")
        return
    try:
        out = subprocess.run(shlex.split(cmd) + sorted(codes) + ["--json"], capture_output=True, text=True,
                             timeout=300).stdout
        rows = json.loads(out)
    except (OSError, ValueError, subprocess.TimeoutExpired) as exc:
        rep.w.append(f"offline catalogue cross-check skipped: '{cmd}' did not answer in JSON ({exc})")
        return
    confirmed, total = 0, sum(len(v) for v in codes.values())
    for row in rows if isinstance(rows, list) else []:
        if not isinstance(row, dict):
            continue
        code = str(row.get("lcsc", "")).upper()
        for key in codes.get(code, []):
            c = index[key]
            if row.get("found") is False:
                rep.e.append(f"{key}: {code} does not exist in the offline catalogue (invented or mistyped?)")
                continue
            a, b = norm(row.get("mpn")), norm(c.get("mpn"))
            if b and a != b and not (a.startswith(b) or b.startswith(a)):
                rep.e.append(f"{key}: {code} is {row.get('mpn')} in the offline catalogue, not {c.get('mpn')}")
                continue
            confirmed += 1
            try:
                ref = float(row["price_at_qty"]["1000"]["unit_usd"])
                mine = (c.get("price") or {}).get("q1000")
                if c.get("currency") == "USD" and is_num(mine) and ref and abs(mine - ref) / ref > 0.30:
                    rep.w.append(f"{key}: q1000 {mine:g} USD vs offline catalogue {ref:g} USD (>30 % apart)")
            except (KeyError, TypeError, ValueError):
                pass
            if row.get("stock") == 0:
                rep.w.append(f"{key}: {code} has stock 0 in the offline catalogue")
    rep.n.append(f"offline catalogue cross-check: {confirmed} of {total} candidates with LCSC/JLC codes confirmed "
                 f"(code exists, MPN matches)")


def cell(x):
    return str("" if x is None else x).replace("|", "/").replace("\n", " ")


def write_md(path, files):
    """The human-readable catalogue (02-concept/catalog.md), generated from the JSON homes."""
    out = ["# Parts catalogue", "", f"Generated by `method/gates/catalog_check.py --write-md` on {datetime.date.today()} "
           "from the catalogue files in `work/catalog/` (the homes of these facts; do not edit here). Prices as read, "
           "in the supplier's currency, per unit at an order of 1 / 100 / 1000.", ""]
    for f in files:
        try:
            data = json.load(open(f, encoding="utf-8"))
        except (OSError, ValueError):
            continue
        if not isinstance(data, dict):
            continue
        stem = os.path.splitext(os.path.basename(f))[0]
        if isinstance(data.get("candidates"), list):
            out += [f"## {cell(data.get('function') or stem)}" + (" (value engineering)" if data.get("kind") == "ve" else ""),
                    "", cell(data.get("spec_summary", "")), "",
                    "| id | maker | MPN | supplier, part | principle | q1 | q100 | q1000 | cur | origin, read | stock | "
                    "lifecycle | certified | meets spec |", "|---|---|---|---|---|---:|---:|---:|---|---|---:|---|---|---|"]
            data["candidates"] = [c for c in data["candidates"] if isinstance(c, dict)]
            for c in data["candidates"]:
                p = c.get("price") if isinstance(c.get("price"), dict) else {}
                out.append("| " + " | ".join(cell(x) for x in (
                    c.get("id"), c.get("manufacturer"), c.get("mpn"), f"{c.get('supplier', '')} {c.get('supplier_part', '')}",
                    c.get("principle"), p.get("q1"), p.get("q100"), p.get("q1000"), c.get("currency"),
                    f"{c.get('price_origin', '')} {c.get('price_date', '')}", c.get("stock"), c.get("lifecycle"),
                    len(c.get("certifications") or []) or "", c.get("meets_spec"))) + " |")
            notes = [f"- {cell(c.get('id'))}: {cell(c.get('spec_notes'))} <{cell(c.get('url'))}>" for c in data["candidates"]]
            out += [""] + notes
            fails = data.get("failed_lookups") or []
            if fails:
                out += ["", "Failed lookups: " + "; ".join(cell(x.get("what", x) if isinstance(x, dict) else x) +
                                                       (f" ({cell(x.get('why'))})" if isinstance(x, dict) else "")
                                                       for x in fails)]
            out.append("")
        elif isinstance(data.get("products"), list):
            kind = str(data.get("kind", "benchmark"))
            data["products"] = [pr for pr in data["products"] if isinstance(pr, dict)]
            out += [f"## {kind.capitalize()} ({stem})", ""]
            if kind == "benchmark":
                out += ["| product | maker | price | cur | incl. VAT | read | positioning | url |",
                        "|---|---|---:|---|---|---|---|---|"]
                out += ["| " + " | ".join(cell(x) for x in (pr.get("name"), pr.get("maker"), pr.get("price"),
                        pr.get("currency"), pr.get("price_incl_vat"), pr.get("price_date"), pr.get("positioning"),
                        pr.get("url"))) + " |" for pr in data["products"]]
            else:
                for pr in data["products"]:
                    out.append(f"- **{cell(pr.get('name'))}** <{cell(pr.get('url'))}>: " + "; ".join(
                        f"{cell(x.get('function'))} = {cell(x.get('part'))}" for x in pr.get("parts") or [] if isinstance(x, dict)))
            out.append("")
    os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
    with open(path, "w", encoding="utf-8") as fh:
        fh.write("\n".join(out) + "\n")
    return path


HOST = re.compile(r"(?<![\w@/.-])((?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+[a-z]{2,24})(?![\w-])", re.I)
FILE_EXT = {"md", "py", "json", "yaml", "yml", "csv", "txt", "sh", "js", "mjs", "ini", "toml", "log", "html", "pdf",
            "svg", "png", "jpg", "step", "stp", "glb", "dxf", "local"}


def forbidden_hosts(path):
    """Host names of the forbidden website, read from the boundaries file (lines that speak of a website or
    site first, else the whole file); file names are skipped."""
    try:
        text = open(path, encoding="utf-8", errors="replace").read()
    except OSError:
        return set()
    lines = text.splitlines()
    for group in ([ln for ln in lines if re.search(r"web\s*site|\bsite\b|\bhost\b|\bdomain\b", ln, re.I)], lines):
        hosts = {m.group(1).lower() for ln in group for m in HOST.finditer(ln)
                 if m.group(1).lower().rsplit(".", 1)[1] not in FILE_EXT}
        hosts |= {hostname(u) for ln in group for u in re.findall(r"https?://[^\s'\"`<>)]+", ln)} - {""}
        if hosts:
            return hosts
    return set()


def hostname(url):
    try:
        return (urllib.parse.urlsplit(url).hostname or "").lower()
    except ValueError:
        return ""


def on_host(url, hosts):
    h = hostname(url)
    return any(h == x or h.endswith("." + x) or x in url.lower() for x in hosts)


def probe(url):
    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) catalog-check",
                                               "Accept": "text/html,application/pdf,*/*"})
    try:
        with urllib.request.urlopen(req, timeout=15) as r:
            r.read(65536)
            return url, r.status
    except urllib.error.HTTPError as exc:
        return url, exc.code
    except (urllib.error.URLError, OSError, ValueError) as exc:
        return url, f"error {getattr(exc, 'reason', exc)}"


def main():
    ap = argparse.ArgumentParser(description="Check the parts catalogue and BOM traceability.")
    ap.add_argument("files", nargs="*")
    ap.add_argument("--bom", nargs="+")
    ap.add_argument("--no-bom", action="store_true")
    ap.add_argument("--lcsc", action="store_true")
    ap.add_argument("--lcsc-cmd", default="lcsc")
    ap.add_argument("--probe-urls", action="store_true")
    ap.add_argument("--boundaries", default="work/boundaries.md")
    ap.add_argument("--max-age-days", type=int, default=120)
    ap.add_argument("--write-md", metavar="PATH", help="also write the catalogue as markdown (e.g. deliverables/02-concept/catalog.md)")
    args = ap.parse_args()
    rep, index, stats, urls = Rep(), {}, {}, set()
    files = args.files or sorted(glob.glob("work/catalog/*.json"))
    if not files:
        rep.e.append("no catalogue files (work/catalog/*.json)")
    for f in files:
        check_file(rep, f, args.max_age_days, index, stats, urls)
    nbom = 0
    if not args.no_bom:
        if args.bom:
            bom = args.bom
        elif not args.files:  # whole catalogue loaded: check the live BOM against it
            bom = (["work/bom.yaml"] if os.path.exists("work/bom.yaml") else []) + sorted(glob.glob("work/bom.d/*.yaml"))
        else:  # single files (a scout checking its own output): no BOM check
            bom = []
        if bom:
            nbom = check_bom(rep, bom, index)
    if args.lcsc:
        check_lcsc(rep, args.lcsc_cmd, index)
    skip = forbidden_hosts(args.boundaries)
    banned = {u for u in urls if on_host(u, skip)}
    for u in sorted(banned):
        rep.e.append(f"URL on the forbidden website (remove it, do not open it): {u}")
    urls -= banned
    if args.probe_urls and not skip:
        rep.e.append(f"--probe-urls refused: no forbidden website found in {args.boundaries} (write it first: "
                     f"section 3 of the RUN CONTRACT, verbatim); no URL was fetched")
    elif args.probe_urls and urls:
        dead, blocked, ok = [], [], 0
        with concurrent.futures.ThreadPoolExecutor(8) as ex:
            for url, st in ex.map(probe, sorted(urls)[:120]):
                if isinstance(st, int) and st < 400:
                    ok += 1
                elif st in (404, 410) or (isinstance(st, str) and ("Name or service" in st or "nodename" in st)):
                    dead.append(f"{url} ({st})")
                else:
                    blocked.append(f"{url} ({st})")
        for d in dead:
            rep.w.append(f"dead link: {d}")
        if blocked:
            rep.n.append(f"{len(blocked)} URL(s) refused automated reading (not verified), e.g. {blocked[:3]}")
        rep.n.append(f"URL probe: {ok} reachable, {len(dead)} dead, {len(blocked)} not verifiable")
    if args.write_md:
        rep.n.append(f"catalogue markdown written to {write_md(args.write_md, files)}")
    for m in rep.n:
        print(f"NOTE  {m}")
    for m in rep.w:
        print(f"WARN  {m}")
    for m in rep.e:
        print(f"ERROR {m}")
    print(f"catalog_check: files={len(files)} candidates={len(index)} (SOURCE prices {stats.get('SOURCE', 0)}, "
          f"ASSUMPTION {stats.get('ASSUMPTION', 0)}) bom_lines={nbom} errors={len(rep.e)} warnings={len(rep.w)} -> "
          f"{'PASS' if not rep.e else 'FAIL'}")
    return 0 if not rep.e else 1


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""cost_rollup.py - regenerate the cost model of the run from its BOM and other-cost files.

Usage (from the run directory):
  python3 method/gates/cost_rollup.py
      default inputs: work/bom.yaml + work/bom.d/*.yaml, work/other_costs.yaml + work/other_costs.d/*.yaml,
      price frame work/price_frame.yaml (if present); output: deliverables/09-bom-cost/
  python3 method/gates/cost_rollup.py --bom work/arch/A/bom.yaml --other work/arch/A/other_costs.yaml \
      --out deliverables/02-concept/arch-A          (one concept architecture)
  python3 method/gates/cost_rollup.py --frame-only  (print the price frame as a markdown table)

A BOM line with `catalog: <file stem>/<candidate id>` takes every field it does not state itself (maker, MPN,
supplier, URL, price breaks, currency, origin, read date) from that candidate in work/catalog/.
Writes <out>/bom.csv, <out>/cost.json and <out>/cost.md in the RUN CONTRACT formats. All money is converted
to the roll-up currency; original prices, exchange rates, origins and read dates stay in the bom.csv notes.
Prints unit cost vs the price frame, the origin mix and the top cost drivers. Exit 0 = written,
1 = input error (nothing written). Formats: method/templates/bom.yaml, other_costs.yaml, price_frame.yaml.
"""
import argparse
import csv
import datetime
import glob
import io
import json
import os
import re
import shutil
import sys

try:
    import yaml
except ImportError:  # the system python3 may lack PyYAML: re-run once under the toolchain Python
    _py = shutil.which("ddpy") or "/opt/dd/venv/bin/python"
    if os.path.exists(_py) and not os.environ.get("DD_GATE_REEXEC"):
        os.environ["DD_GATE_REEXEC"] = "1"
        os.execv(_py, [_py] + sys.argv)
    sys.exit("PyYAML missing: run this script with ddpy")

QS = ("q1", "q100", "q1000")
N = {"q1": 1, "q100": 100, "q1000": 1000}
ORIGINS = ("SOURCE", "ASSUMPTION", "CALCULATION")
CATEGORIES = ("pcb", "mechanical", "tooling", "assembly_test", "packaging", "certification", "freight",
              "contingency")
BOM_HEADER = ["item", "qty", "description", "manufacturer", "mpn", "supplier", "supplier_part", "url",
              "unit_price_1", "unit_price_100", "unit_price_1000", "currency", "notes"]


class InputError(Exception):
    pass


def load_yaml(path):
    try:
        with open(path, encoding="utf-8") as fh:
            data = yaml.safe_load(fh) or {}
    except (OSError, yaml.YAMLError) as exc:
        raise InputError(f"{path}: cannot read: {exc}")
    if not isinstance(data, dict):
        raise InputError(f"{path}: top level must be a mapping")
    return data


def num(x, what):
    if x is None:
        raise InputError(f"{what}: not set (null) - give a number (0 with the reason in notes if it truly does "
                         f"not apply)")
    if isinstance(x, bool):
        raise InputError(f"{what}: not a number ({x!r})")
    try:
        return float(str(x).replace(",", ".")) if isinstance(x, str) else float(x)
    except (TypeError, ValueError):
        raise InputError(f"{what}: not a number ({x!r})")


def lines_of(data, path):
    lines = data.get("lines") or []
    if not isinstance(lines, list):
        raise InputError(f"{path}: lines must be a list of mappings")
    return lines


PLACEHOLDER = re.compile(r"^\s*<.*>\s*$", re.S)  # template notes are <...>: still a placeholder if unchanged


def qdict(x, what):
    """A price or cost given per quantity: {q1, q100, q1000} or one number for all."""
    if isinstance(x, dict):
        missing = [q for q in QS if q not in x]
        if missing:
            raise InputError(f"{what}: missing {missing}")
        return {q: num(x[q], f"{what}.{q}") for q in QS}
    v = num(x, what)
    return {q: v for q in QS}


def frame_numbers(frame):
    """Retail excl. VAT and allowed landed unit cost per quantity from a price frame mapping.

    allowed[q] = the lower of retail excl. VAT / divisors[q] and the unit-cost limit allowed_unit_cost[q] that
    the brief or INPUT/requirements.md states (POLICY 2.4). A retail price or limit of 0 or null = not set."""
    def positive(key, v):
        v = None if v is None else num(v, key)
        return v if v is not None and v > 0 else None

    def mapping(key, default):
        v = frame.get(key)
        if v is None:
            return default
        if not isinstance(v, dict):
            raise InputError(f"price frame: {key} must be a mapping")
        return v

    out = {"currency": frame.get("currency")}
    incl = positive("target_retail_incl_vat", frame.get("target_retail_incl_vat"))
    excl = positive("target_retail_excl_vat", frame.get("target_retail_excl_vat"))
    vat = num(frame.get("vat_percent") or 0, "vat_percent")
    if excl is None and incl is not None:
        excl = incl / (1 + vat / 100)
    out["retail_incl_vat"], out["retail_excl_vat"] = incl, excl
    divisors = mapping("divisors", {"q100": 2, "q1000": 3})
    limits = mapping("allowed_unit_cost", {})
    out["allowed"], out["basis"] = {}, {}
    for q in QS:
        cands = []
        d = positive(f"divisors.{q}", divisors.get(q))
        if d and excl is not None:
            cands.append((excl / d, f"retail excl. VAT / {d:g}"))
        lim = positive(f"allowed_unit_cost.{q}", limits.get(q))
        if lim is not None:
            cands.append((lim, f"unit-cost limit ({limits.get('origin') or 'origin not given'})"))
        if cands:
            out["allowed"][q] = min(c[0] for c in cands)
            out["basis"][q] = "; ".join(f"{fmt(v)} = {b}" for v, b in cands) + (
                " -> the lower one" if len(cands) > 1 else "")
    out["divisors"] = divisors
    out["non_bom_share"] = num(frame.get("non_bom_share", 0.35), "non_bom_share")
    if "q1000" in out["allowed"]:
        out["bom_budget_q1000"] = out["allowed"]["q1000"] * (1 - out["non_bom_share"])
    out["function_budgets"] = {k: num(v, f"function_budgets_q1000.{k}")
                               for k, v in mapping("function_budgets_q1000", {}).items() if v is not None}
    return out


def expand(paths, default, extra_glob):
    if paths:
        files = []
        for p in paths:
            files.extend(sorted(glob.glob(p)) or [p])
        return files
    files = [default] if os.path.exists(default) else []
    return files + sorted(glob.glob(extra_glob))


def fx_rate(fx, cur, base, what):
    if cur == base:
        return 1.0, ""
    entry = fx.get(cur)
    if entry is None:
        raise InputError(f"{what}: currency {cur} has no fx rate to {base} (add fx: {{{cur}: {{rate: ..., origin: ...}}}})")
    raw, origin = (entry.get("rate"), str(entry.get("origin", ""))) if isinstance(entry, dict) else (entry, "")
    if raw is None:
        raise InputError(f"fx.{cur}.rate is not set: give 1 {cur} in {base} with its origin (a dated SOURCE or an "
                         f"ASSUMPTION), or delete the fx entry")
    rate = num(raw, f"fx.{cur}.rate")
    if rate <= 0:
        raise InputError(f"fx.{cur}.rate = {rate:g}: must be > 0 (1 {cur} = rate {base})")
    return rate, origin


FILL = {"description": "description", "manufacturer": "manufacturer", "mpn": "mpn", "supplier": "supplier",
        "supplier_part": "supplier_part", "url": "url", "price": "price", "currency": "currency",
        "origin": "price_origin", "date": "price_date"}


def catalog_entry(ref, cat_dir, cache, what):
    """The candidate a BOM line points to with `catalog: <file stem>/<candidate id>`."""
    stem, _, cid = ref.partition("/")
    if stem not in cache:
        path = os.path.join(cat_dir, stem + ".json")
        try:
            with open(path, encoding="utf-8") as fh:
                data = json.load(fh)
        except (OSError, ValueError) as exc:
            raise InputError(f"{what}: catalog {ref}: cannot read {path}: {exc}")
        cands = data.get("candidates") if isinstance(data, dict) else None
        if not isinstance(cands, list):
            raise InputError(f"{what}: catalog {ref}: {path} has no candidates list (run catalog_check.py on it)")
        cache[stem] = {str(c.get("id")): c for c in cands if isinstance(c, dict)}
    if cid not in cache[stem]:
        raise InputError(f"{what}: catalog {ref}: no candidate '{cid}' in {stem}.json")
    return cache[stem][cid]


def read_bom(files, warns, cat_dir="work/catalog"):
    base, fx, lines, items, cache = None, {}, [], set(), {}
    for path in files:
        data = load_yaml(path)
        cur = data.get("currency")
        if not cur:
            raise InputError(f"{path}: currency missing")
        if base is None:
            base = cur
        elif cur != base:
            raise InputError(f"{path}: currency {cur} differs from {base} of the other BOM files")
        fxd = data.get("fx") or {}
        if not isinstance(fxd, dict):
            raise InputError(f"{path}: fx must be a mapping")
        for k, v in fxd.items():
            if k in fx and fx[k] != v:
                raise InputError(f"{path}: fx.{k} conflicts with another BOM file (rates live only in the lead's "
                                 f"BOM file; delete the fx block from fragments)")
            fx[k] = v
        for n, ln in enumerate(lines_of(data, path), start=1):
            what = f"{path} line {n}"
            if not isinstance(ln, dict):
                raise InputError(f"{what}: not a mapping")
            if ln.get("catalog"):  # fill what the line does not state from the catalogue candidate
                c = catalog_entry(str(ln["catalog"]).strip(), cat_dir, cache, what)
                if isinstance(ln.get("price"), dict) and any(ln["price"].get(q) is None for q in QS):
                    raise InputError(f"{what}: the line's own price override has null values: fill them, or delete "
                                     f"the price key to take the catalogue's breaks")
                ln = dict(ln)
                for k, ck in FILL.items():
                    if ln.get(k) in (None, "") and c.get(ck) not in (None, ""):
                        ln[k] = c[ck]
                if not ln.get("description"):
                    ln["description"] = " ".join(str(x) for x in (ln.get("function"), c.get("manufacturer"),
                                                                  c.get("mpn")) if x)
                if str(c.get("price_origin", "")).upper() == "ASSUMPTION" and not ln.get("notes"):
                    ln["notes"] = f"catalogue ASSUMPTION: {c.get('price_note', '')}"
                if isinstance(ln.get("price"), dict) and any(ln["price"].get(q) is None for q in QS):
                    raise InputError(f"{what}: catalog {ln['catalog']} lacks a price at some of q1/q100/q1000; "
                                     f"state price on the BOM line (with origin and notes)")
            item = str(ln.get("item") or "").strip()
            if not item:
                item = f"X{len(lines) + 1:02d}"
                warns.append(f"{what}: item missing, assigned {item}")
            if item in items:
                raise InputError(f"{what}: duplicate item {item} (items must be unique across all BOM files)")
            items.add(item)
            qty = num(ln.get("qty"), f"{what} qty")
            if qty <= 0:
                raise InputError(f"{what}: qty must be > 0")
            if "price" not in ln:
                raise InputError(f"{what}: price missing")
            price = qdict(ln["price"], f"{what} price")
            origin = str(ln.get("origin", "")).upper()
            if origin not in ORIGINS:
                raise InputError(f"{what}: origin must be one of {ORIGINS}")
            lcur = ln.get("currency") or cur
            rate, rate_origin = fx_rate(fx, lcur, base, what)
            date = str(ln.get("date") or "").strip()
            if origin == "SOURCE":
                if not ln.get("url"):
                    warns.append(f"{what} ({item}): SOURCE price without url")
                if not re.match(r"^\d{4}-\d{2}-\d{2}", date):
                    warns.append(f"{what} ({item}): SOURCE price without read date (YYYY-MM-DD)")
            if origin == "ASSUMPTION" and not str(ln.get("notes") or "").strip():
                warns.append(f"{what} ({item}): ASSUMPTION price without a reason in notes")
            if origin == "CALCULATION" and not re.search(r"[\w-]+/[\w./-]+", str(ln.get("notes") or "")):
                warns.append(f"{what} ({item}): CALCULATION price without the script path in notes")
            if price["q1"] < price["q100"] or price["q100"] < price["q1000"]:
                warns.append(f"{what} ({item}): unit price rises with quantity ({price})")
            conv = {q: price[q] * rate for q in QS}
            if conv["q1000"] == 0:
                warns.append(f"{what} ({item}): zero price at q1000" + (
                    f" after conversion ({price['q1000']:g} {lcur} x {rate:g})" if price["q1000"] else ""))
            if PLACEHOLDER.match(str(ln.get("notes") or "")):
                warns.append(f"{what} ({item}): notes still the template placeholder")
            lines.append({"src": path, "item": item, "qty": qty, "ln": ln, "price": price, "origin": origin,
                          "cur": lcur, "rate": rate, "rate_origin": rate_origin, "date": date, "conv": conv})
    return base, fx, lines


ZERO_SUSPECT = ("pcb", "tooling", "assembly_test", "packaging", "certification", "freight")


def read_other(files, base, fx, warns):
    out, names = [], set()
    for path in files:
        data = load_yaml(path)
        cur = data.get("currency") or base
        fxd = data.get("fx") or {}
        if not isinstance(fxd, dict):
            raise InputError(f"{path}: fx must be a mapping")
        rate, _ = fx_rate({**fx, **fxd}, cur, base, path)
        for n, ln in enumerate(lines_of(data, path), start=1):
            what = f"{path} line {n}"
            if not isinstance(ln, dict):
                raise InputError(f"{what}: not a mapping")
            name = str(ln.get("name") or "").strip()
            if not name:
                raise InputError(f"{what}: name missing")
            if name in names:
                raise InputError(f"{what}: duplicate other-cost name {name}")
            names.add(name)
            cat = str(ln.get("category") or "").strip()
            if cat not in CATEGORIES:
                warns.append(f"{what} ({name}): category '{cat}' not in {CATEGORIES} (counted, but cannot "
                             f"prove the category is covered)")
            origin = str(ln.get("origin", "")).upper()
            if origin not in ORIGINS:
                raise InputError(f"{what}: origin must be one of {ORIGINS}")
            rec = {"name": name, "category": cat, "origin": origin, "notes": str(ln.get("notes") or ""),
                   "src": path}
            kinds = [k for k in ("per_unit", "one_off", "percent") if k in ln]
            if len(kinds) != 1:
                raise InputError(f"{what}: give exactly one of per_unit, one_off, percent")
            if "per_unit" in ln:
                v = qdict(ln["per_unit"], f"{what} per_unit")
                rec["cost"] = {q: v[q] * rate for q in QS}
            elif "one_off" in ln:
                total = num(ln["one_off"], f"{what} one_off") * rate
                applies = ln.get("applies", list(QS))
                applies = [applies] if isinstance(applies, (str, int)) else list(applies or [])
                applies = [a if str(a).startswith("q") else f"q{a}" for a in applies]
                bad = [a for a in applies if a not in QS]
                if bad:
                    raise InputError(f"{what}: applies has unknown quantities {bad}")
                rec["cost"] = {q: (total / N[q] if q in applies else 0.0) for q in QS}
                rec["one_off"] = total
                rec["applies"] = applies
            else:
                rec["percent"] = num(ln["percent"], f"{what} percent")
            blank = not rec["notes"].strip() or PLACEHOLDER.match(rec["notes"])
            if blank and "cost" in rec and cat in ZERO_SUSPECT and rec.get("one_off", rec["cost"]["q1000"]) == 0:
                warns.append(f"{what} ({name}): 0 at q1000 with no reason (empty or template placeholder notes) - a "
                             f"forgotten {cat} cost? Give the cost, or 0 with the reason it does not apply")
            elif blank and (origin != "CALCULATION" or rec["notes"].strip()):
                warns.append(f"{what} ({name}): no reason or source in notes (empty or template placeholder)")
            out.append(rec)
    return out


def write_atomic(path, text):
    tmp = f"{path}.tmp{os.getpid()}"
    with open(tmp, "w", encoding="utf-8", newline="") as fh:
        fh.write(text)
    os.replace(tmp, path)


def r2(x):
    return round(x + 0.0, 2)


def isnum(x):
    return isinstance(x, (int, float)) and not isinstance(x, bool)


def fmt(x):
    return f"{x:,.2f}" if isnum(x) else ("–" if x is None else str(x))


def fmtp(x):
    """Unit prices: keep small prices readable."""
    return fmt(x) if not isnum(x) or abs(x) >= 1 or x == 0 else f"{x:.4f}"


def rollup(args):
    warns = []
    bom_files = expand(args.bom, "work/bom.yaml", "work/bom.d/*.yaml")
    other_files = expand(args.other, "work/other_costs.yaml", "work/other_costs.d/*.yaml")
    if not bom_files:
        raise InputError("no BOM file (work/bom.yaml or --bom)")
    missing = [p for p in bom_files + other_files if not os.path.isfile(p)]
    if missing:
        raise InputError(f"file(s) not found: {missing}")
    base, fx, lines = read_bom(bom_files, warns, args.catalog)
    others = read_other(other_files, base, fx, warns)

    bom = {q: sum(l["qty"] * l["conv"][q] for l in lines) for q in QS}
    fixed = {q: sum(o["cost"][q] for o in others if "cost" in o) for q in QS}
    for o in others:
        if "percent" in o:
            o["cost"] = {q: (bom[q] + fixed[q]) * o["percent"] / 100 for q in QS}
    unit = {q: bom[q] + sum(o["cost"][q] for o in others) for q in QS}
    present = {o["category"] for o in others}
    absent = [c for c in CATEGORIES if c not in present]
    if absent:
        warns.append(f"no other-cost line for categories {absent}: add each (0 with a reason if it truly does "
                     f"not apply) - one-off and non-BOM costs must not be ignored")

    frame = None
    frame_path = args.frame if args.frame else ("work/price_frame.yaml" if os.path.exists("work/price_frame.yaml") else None)
    if frame_path:
        frame = frame_numbers(load_yaml(frame_path))
        if frame["currency"] and frame["currency"] != base:
            warns.append(f"price frame currency {frame['currency']} differs from roll-up currency {base}; "
                         f"frame comparison skipped")
            frame = None

    # bom.csv (roll-up currency; origin, read date and original price in notes)
    buf = io.StringIO()
    wr = csv.writer(buf, lineterminator="\n")
    wr.writerow(BOM_HEADER)
    for l in lines:
        ln = l["ln"]
        notes = [l["origin"] + (f" read {l['date']}" if l["date"] else "")]
        if l["cur"] != base:
            p = l["price"]
            notes.append(f"orig {p['q1']:g}/{p['q100']:g}/{p['q1000']:g} {l['cur']} x {l['rate']:g} "
                         f"{base}/{l['cur']}" + (f" ({l['rate_origin']})" if l["rate_origin"] else ""))
        if ln.get("function"):
            notes.append(f"function {ln['function']}")
        if ln.get("catalog"):
            notes.append(f"catalog {ln['catalog']}")
        if ln.get("notes"):
            notes.append(str(ln["notes"]).strip())
        wr.writerow([l["item"], f"{l['qty']:g}", ln.get("description", ""), ln.get("manufacturer", ""),
                     ln.get("mpn", ""), ln.get("supplier", ""), ln.get("supplier_part", ""), ln.get("url", ""),
                     f"{l['conv']['q1']:.4f}", f"{l['conv']['q100']:.4f}", f"{l['conv']['q1000']:.4f}", base,
                     "; ".join(notes)])

    cost = {"currency": base, "bom": {q: r2(bom[q]) for q in QS},
            "other": {o["name"]: {q: r2(o["cost"][q]) for q in QS} for o in others}}
    cost["unit_cost"] = {q: r2(cost["bom"][q] + sum(v[q] for v in cost["other"].values())) for q in QS}

    # origin mix and drivers at q1000
    mix = {o: sum(l["qty"] * l["conv"]["q1000"] for l in lines if l["origin"] == o) for o in ORIGINS}
    drivers = [(f"BOM {l['item']} {str(l['ln'].get('description') or '')[:50]}", l["qty"] * l["conv"]["q1000"])
               for l in lines] + [(f"other {o['name']}", o["cost"]["q1000"]) for o in others]
    drivers.sort(key=lambda d: -d[1])
    status = {}
    if frame:
        for q, a in frame["allowed"].items():
            if q in unit and a:
                status[q] = (unit[q] - a) / a * 100
    breakeven = None
    if frame and frame["allowed"].get("q1000"):
        cont = sum(o["percent"] for o in others if "percent" in o) / 100
        one_off = sum(o["one_off"] for o in others if "one_off" in o and "q1000" in o["applies"]) * (1 + cont)
        variable = unit["q1000"] - one_off / 1000
        room = frame["allowed"]["q1000"] - variable
        breakeven = (one_off / room if room > 0 else float("inf")) if one_off else 0.0
    func = {}
    for l in lines:
        f = l["ln"].get("function") or "(none)"
        func[f] = func.get(f, 0.0) + l["qty"] * l["conv"]["q1000"]

    # cost.md
    now = datetime.datetime.now().astimezone().isoformat(timespec="minutes")
    md = [f"# Cost roll-up ({args.out})", "",
          f"Generated by `method/gates/cost_rollup.py` at {now} from {', '.join(bom_files)}"
          + (f" and {', '.join(other_files)}" if other_files else "") + ". Do not edit: change the inputs "
          "and re-run.", "", f"Currency: {base}. Quantities: q1 = one prototype, q100 = pilot batch, "
          "q1000 = first production run; one-off costs are amortised over the quantity of each column they "
          "apply to.", ""]
    used = sorted({l["cur"] for l in lines if l["cur"] != base})
    if used:
        md += ["Exchange rates: " + "; ".join(f"1 {k} = {fx_rate(fx, k, base, 'fx')[0]:g} {base} "
                                               f"({fx_rate(fx, k, base, 'fx')[1] or 'origin not given'})"
                                               for k in used), ""]
    md += ["## Unit cost", "", "| line | category | origin | q1 | q100 | q1000 |", "|---|---|---|---:|---:|---:|",
           f"| BOM ({len(lines)} lines) | bom | see below | {fmt(bom['q1'])} | {fmt(bom['q100'])} | "
           f"{fmt(bom['q1000'])} |"]
    for o in others:
        how = f"{o['percent']:g} % of BOM + other" if "percent" in o else (
            f"one-off {o['one_off']:,.0f} over {'/'.join(o['applies'])}" if "one_off" in o else "per unit")
        md.append(f"| {o['name']} ({how}) | {o['category'] or '?'} | {o['origin']} | {fmt(o['cost']['q1'])} | "
                  f"{fmt(o['cost']['q100'])} | {fmt(o['cost']['q1000'])} |")
    md.append(f"| **unit cost (landed)** | | | **{fmt(unit['q1'])}** | **{fmt(unit['q100'])}** | "
              f"**{fmt(unit['q1000'])}** |")
    if frame:
        md.append("| allowed by the price frame | | | " + " | ".join(fmt(frame["allowed"].get(q)) for q in QS) + " |")
        md.append("| status vs frame | | | " + " | ".join(
            ("–" if q not in status else f"{'OVER' if status[q] > 0 else 'OK'} ({status[q]:+.1f} %)") for q in QS) + " |")
    if breakeven is not None:
        md += ["", "Batch size at which the landed unit cost meets the allowed q1000 cost (q1000 piece prices, one-off "
               "costs incl. contingency spread over the batch): " + ("never - the variable cost alone exceeds it"
                                                                     if breakeven == float("inf") else f"{breakeven:,.0f} units")
               + " (CALCULATION)."]
    md += ["", "## BOM lines", "", "| item | qty | description | manufacturer | mpn | supplier | origin | q1 | q100 | "
           "q1000 | ext. q1000 |", "|---|---:|---|---|---|---|---|---:|---:|---:|---:|"]
    for l in lines:
        ln = l["ln"]
        md.append(f"| {l['item']} | {l['qty']:g} | {ln.get('description', '')} | {ln.get('manufacturer', '')} | "
                  f"{ln.get('mpn', '')} | {ln.get('supplier', '')} | {l['origin']} | {fmtp(l['conv']['q1'])} | "
                  f"{fmtp(l['conv']['q100'])} | {fmtp(l['conv']['q1000'])} | {fmtp(l['qty'] * l['conv']['q1000'])} |")
    tot = sum(mix.values()) or 1.0
    md += ["", "## Origin of the BOM cost at q1000", ""]
    md += [f"- {o}: {sum(1 for l in lines if l['origin'] == o)} lines, {fmt(mix[o])} {base} "
           f"({mix[o] / tot * 100:.0f} %)" for o in ORIGINS]
    md += ["", "## Top cost drivers at q1000", ""]
    md += [f"{i}. {d[0]}: {fmt(d[1])} {base} ({d[1] / (unit['q1000'] or 1) * 100:.1f} % of unit cost)"
           for i, d in enumerate(drivers[:args.top], start=1)]
    if frame and frame["function_budgets"]:
        md += ["", "## BOM cost per function vs budget (q1000)", "", "| function | budget | BOM cost | status |",
               "|---|---:|---:|---|"]
        for f in sorted(set(func) | set(frame["function_budgets"])):
            b, c = frame["function_budgets"].get(f), func.get(f, 0.0)
            st = "–" if b is None else ("OVER" if c > b else "OK")
            md.append(f"| {f} | {fmt(b)} | {fmt(c)} | {st} |")
    if warns:
        md += ["", "## Warnings", ""] + [f"- {w}" for w in warns]

    os.makedirs(args.out, exist_ok=True)
    write_atomic(os.path.join(args.out, "bom.csv"), buf.getvalue())
    write_atomic(os.path.join(args.out, "cost.json"), json.dumps(cost, indent=2) + "\n")
    write_atomic(os.path.join(args.out, "cost.md"), "\n".join(md) + "\n")

    print(f"cost_rollup: {len(lines)} BOM lines, {len(others)} other-cost lines -> {args.out}/bom.csv, cost.json, cost.md")
    print(f"unit cost {base}: q1 {fmt(unit['q1'])} | q100 {fmt(unit['q100'])} | q1000 {fmt(unit['q1000'])}"
          f"   (BOM q1000 {fmt(bom['q1000'])})")
    for q, s in status.items():
        print(f"FRAME {q}: {'OVER' if s > 0 else 'OK'} {s:+.1f} % vs allowed {fmt(frame['allowed'][q])}")
    if breakeven is not None:
        print("break-even batch for the frame: " + ("never (variable cost above the allowed cost)"
                                                    if breakeven == float("inf") else f"{breakeven:,.0f} units"))
    if frame and frame["function_budgets"]:
        for f, b in frame["function_budgets"].items():
            if func.get(f, 0.0) > b:
                print(f"BUDGET {f}: {fmt(func[f])} > {fmt(b)} at q1000")
    print(f"origin of BOM cost at q1000: " + ", ".join(f"{o} {mix[o] / tot * 100:.0f} %" for o in ORIGINS))
    print("top drivers q1000: " + "; ".join(f"{d[0][:40]} {fmt(d[1])}" for d in drivers[:args.top]))
    for w in warns:
        print(f"WARN  {w}")
    if args.json:
        print(json.dumps({"unit_cost": cost["unit_cost"], "currency": base,
                          "frame_status_pct": {q: round(s, 1) for q, s in status.items()},
                          "breakeven_batch": None if breakeven is None or breakeven == float("inf") else round(breakeven),
                          "origin_share_q1000": {o: round(mix[o] / tot, 3) for o in ORIGINS},
                          "top_drivers": [[d[0], round(d[1], 2)] for d in drivers[:args.top]],
                          "warnings": len(warns)}))


def frame_only(args):
    path = args.frame or "work/price_frame.yaml"
    raw = load_yaml(path)
    f = frame_numbers(raw)
    cur = f["currency"] or ""
    b = raw.get("benchmark") or {}
    b = b if isinstance(b, dict) else {}
    rows = [("benchmark retail (min / median / max)", f"{fmt(b.get('retail_min'))} / {fmt(b.get('retail_median'))} / "
             f"{fmt(b.get('retail_max'))} {cur} ({b.get('n_products') or '?'} products; {b.get('origin', 'origin?')})"),
            ("target retail incl. VAT", (f"{fmt(f['retail_incl_vat'])} {cur}" if f["retail_incl_vat"] else "not set")
             + f" ({raw.get('target_retail_origin', 'origin?')})"),
            ("VAT", f"{raw.get('vat_percent', 0)} %"),
            ("target retail excl. VAT", f"{fmt(f['retail_excl_vat'])} {cur} (CALCULATION)" if f["retail_excl_vat"]
             else "not set")]
    for q in QS:
        if q in f["allowed"]:
            rows.append((f"allowed landed unit cost {q}", f"{fmt(f['allowed'][q])} {cur} ({f['basis'][q]}; POLICY 2.4)"))
    if not f["allowed"]:
        rows.append(("allowed landed unit cost", "not set: give target_retail_incl_vat (or a unit-cost limit)"))
    if f.get("bom_budget_q1000"):
        rows.append(("BOM budget q1000", f"{fmt(f['bom_budget_q1000'])} {cur} (allowed q1000 x (1 - {f['non_bom_share']:g}))"))
    for k, v in f["function_budgets"].items():
        rows.append((f"budget {k} (q1000)", f"{fmt(v)} {cur}"))
    if f["function_budgets"] and f.get("bom_budget_q1000"):
        s = sum(f["function_budgets"].values())
        rows.append(("sum of function budgets", f"{fmt(s)} {cur} ({(s / f['bom_budget_q1000'] - 1) * 100:+.1f} % vs BOM budget)"))
    print("| price frame | value |\n|---|---|")
    for k, v in rows:
        print(f"| {k} | {v} |")


def main():
    ap = argparse.ArgumentParser(description="Regenerate bom.csv, cost.json and cost.md from the BOM and other-cost files.")
    ap.add_argument("--bom", nargs="+", help="BOM yaml file(s) or globs (default work/bom.yaml + work/bom.d/*.yaml)")
    ap.add_argument("--other", nargs="*", help="other-cost yaml file(s) (default work/other_costs.yaml + work/other_costs.d/*.yaml)")
    ap.add_argument("--frame", help="price frame yaml (default work/price_frame.yaml if present)")
    ap.add_argument("--out", default="deliverables/09-bom-cost", help="output folder (default deliverables/09-bom-cost)")
    ap.add_argument("--catalog", default="work/catalog", help="catalogue folder for `catalog:` references")
    ap.add_argument("--top", type=int, default=5, help="number of cost drivers to list (default 5)")
    ap.add_argument("--frame-only", action="store_true", help="print the price frame table and exit")
    ap.add_argument("--json", action="store_true", help="print a JSON summary as the last line")
    args = ap.parse_args()
    try:
        frame_only(args) if args.frame_only else rollup(args)
    except InputError as exc:
        print(f"ERROR {exc}")
        print("cost_rollup: nothing written")
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""dd_check.py - check deliverables/ against the RUN CONTRACT layout (contract v1.x).

Usage (from the run directory):
    python3 method/gates/dd_check.py --tier concept|full [--run-scripts] [--json]

Checks the formats the RUN CONTRACT fixes (compliance.csv, manifest.json, bom.csv, cost.json, the
folder layout of the tier, evidence paths) and prints ERROR / WARN lines. Exit code 0 = no errors.
It checks form, not engineering quality. Standard library only.
"""
import argparse
import csv
import io
import json
import os
import re
import shutil
import subprocess
import sys

COMPLIANCE_HEADER = ["req_id", "status", "value", "unit", "evidence", "note"]
COMPLIANCE_STATUS = {"met", "partly", "not_met", "not_addressed"}
BOM_HEADER = ["item", "qty", "description", "manufacturer", "mpn", "supplier", "supplier_part", "url",
              "unit_price_1", "unit_price_100", "unit_price_1000", "currency", "notes"]
HEADING_ID = re.compile(r"^#{1,6}\s+([A-Z][A-Z0-9]{1,7}-[A-Z0-9]{1,7}-\d{1,4})\b")
LOOSE_ID = re.compile(r"\b([A-Z][A-Z0-9]{1,7}-[A-Z]{2,7}-\d{2,4})\b")
FULL_DIRS = ["03-cad", "04-drawings", "05-calculations", "06-electronics", "07-firmware",
             "08-safety-certification", "09-bom-cost", "10-prototype-test-plan", "11-renders"]


class Report:
    def __init__(self):
        self.errors, self.warnings, self.notes = [], [], []

    def error(self, msg):
        self.errors.append(msg)

    def warn(self, msg):
        self.warnings.append(msg)

    def note(self, msg):
        self.notes.append(msg)


def requirement_ids(path):
    """Ids of INPUT/requirements.md: headings first, loose tokens as a fallback."""
    if not os.path.isfile(path):
        return []
    with open(path, encoding="utf-8", errors="replace") as fh:
        text = fh.read()
    ids = []
    for line in text.splitlines():
        m = HEADING_ID.match(line.strip())
        if m and m.group(1) not in ids:
            ids.append(m.group(1))
    if not ids:
        for m in LOOSE_ID.finditer(text):
            if m.group(1) not in ids:
                ids.append(m.group(1))
    return ids


def nonempty_files(folder, exts=None):
    out = []
    for root, _dirs, files in os.walk(folder):
        for name in files:
            p = os.path.join(root, name)
            if os.path.getsize(p) == 0:
                continue
            if exts and not name.lower().endswith(tuple(exts)):
                continue
            out.append(p)
    return out


def split_paths(field):
    parts = re.split(r"[;|\n]+|\s+(?=[\w./-]+/)", field or "")
    return [p.strip().strip("`'\"") for p in parts if p.strip()]


def evidence_exists(deliv, ref):
    ref = ref.split("#", 1)[0].split("::", 1)[0].strip()
    if not ref:
        return True
    if ref.startswith("deliverables/"):
        ref = ref[len("deliverables/"):]
    ref = ref.lstrip("./")
    m = re.match(r"^(.*?)(:\d+(-\d+)?)$", ref)
    if m:
        ref = m.group(1)
    return os.path.exists(os.path.join(deliv, ref))


def check_compliance(deliv, ids, rep):
    path = os.path.join(deliv, "01-requirements", "compliance.csv")
    if not os.path.isfile(path):
        rep.error("01-requirements/compliance.csv missing")
        return
    raw = open(path, encoding="utf-8-sig", errors="replace").read()
    rows = list(csv.reader(io.StringIO(raw)))
    if not rows:
        rep.error("compliance.csv is empty")
        return
    header = [h.strip() for h in rows[0]]
    if header != COMPLIANCE_HEADER:
        rep.error(f"compliance.csv header is {header}, expected {COMPLIANCE_HEADER}")
        return
    seen = {}
    for n, row in enumerate(rows[1:], start=2):
        if not any(c.strip() for c in row):
            continue
        if len(row) != len(COMPLIANCE_HEADER):
            rep.error(f"compliance.csv line {n}: {len(row)} fields, expected {len(COMPLIANCE_HEADER)} (quote fields with commas)")
            continue
        rid, status, _value, _unit, evidence, _note = [c.strip() for c in row]
        if rid in seen:
            rep.error(f"compliance.csv line {n}: duplicate id {rid} (first at line {seen[rid]})")
        seen[rid] = n
        if status not in COMPLIANCE_STATUS:
            rep.error(f"compliance.csv line {n}: status '{status}' not in {sorted(COMPLIANCE_STATUS)}")
        if status != "not_addressed" and not evidence:
            rep.error(f"compliance.csv line {n} ({rid}): status {status} without evidence path")
        for ref in split_paths(evidence):
            if not evidence_exists(deliv, ref):
                rep.error(f"compliance.csv line {n} ({rid}): evidence path not found: {ref}")
    if ids:
        missing = [i for i in ids if i not in seen]
        extra = [i for i in seen if i not in ids]
        if missing:
            rep.error(f"compliance.csv lacks {len(missing)} requirement id(s): {', '.join(missing)}")
        if extra:
            rep.warn(f"compliance.csv has ids not in INPUT/requirements.md: {', '.join(extra)}")
        rep.note(f"compliance.csv: {len(seen)} rows for {len(ids)} requirement ids")
    else:
        rep.warn("no requirement ids found in INPUT/requirements.md; row coverage not checked")


def check_ids_in_requirements(deliv, ids, rep):
    folder = os.path.join(deliv, "01-requirements")
    texts = ""
    for p in nonempty_files(folder, (".md", ".yaml", ".yml", ".txt")):
        texts += open(p, encoding="utf-8", errors="replace").read() + "\n"
    if not texts:
        rep.error("01-requirements/ has no requirements document (.md/.yaml) besides compliance.csv")
        return
    absent = [i for i in ids if i not in texts]
    if absent:
        rep.warn(f"requirement ids not mentioned in the 01-requirements documents: {', '.join(absent)}")


def num_or_null(x):
    return x is None or isinstance(x, (int, float)) and not isinstance(x, bool)


def check_manifest(deliv, tier, rep):
    path = os.path.join(deliv, "manifest.json")
    if not os.path.isfile(path):
        rep.error("manifest.json missing")
        return
    try:
        man = json.load(open(path, encoding="utf-8"))
    except Exception as exc:  # noqa: BLE001
        rep.error(f"manifest.json does not parse: {exc}")
        return
    if man.get("tier") != tier:
        rep.error(f"manifest.json tier is {man.get('tier')!r}, expected {tier!r}")
    files = man.get("files")
    if not isinstance(files, list) or not files:
        rep.error("manifest.json files[] missing or empty")
    else:
        listed = set()
        for f in files:
            if not isinstance(f, dict) or not {"path", "kind", "desc"} <= set(f):
                rep.error(f"manifest.json files[] entry lacks path/kind/desc: {f}")
                continue
            listed.add(f["path"].lstrip("./"))
            if not evidence_exists(deliv, f["path"]):
                rep.error(f"manifest.json lists a file that does not exist: {f['path']}")
        actual = set()
        for p in nonempty_files(deliv):
            rel = os.path.relpath(p, deliv)
            if rel != "manifest.json":
                actual.add(rel)
        unlisted = sorted(a for a in actual if a not in listed and not any(a.startswith(l.rstrip("/") + "/") for l in listed))
        if unlisted:
            rep.warn(f"{len(unlisted)} deliverable file(s) not listed in manifest.json, e.g. {unlisted[:5]}")
    kf = man.get("key_figures")
    if not isinstance(kf, dict):
        rep.error("manifest.json key_figures missing")
    else:
        for key in ("dimensions_mm", "mass_kg", "power_w", "unit_cost", "retail_price"):
            if key not in kf:
                rep.error(f"manifest.json key_figures.{key} missing (use null when unknown)")
        dims = kf.get("dimensions_mm")
        if dims is not None and not (isinstance(dims, list) and len(dims) == 3 and all(num_or_null(d) for d in dims)):
            rep.error("key_figures.dimensions_mm must be [L, W, H] numbers or null")
        for key in ("mass_kg", "power_w"):
            if key in kf and not num_or_null(kf[key]):
                rep.error(f"key_figures.{key} must be a number or null")
        uc = kf.get("unit_cost")
        if uc is not None:
            if not isinstance(uc, dict) or not {"currency", "q1", "q100", "q1000"} <= set(uc):
                rep.error("key_figures.unit_cost needs currency, q1, q100, q1000")
            elif not all(num_or_null(uc[q]) for q in ("q1", "q100", "q1000")):
                rep.error("key_figures.unit_cost q1/q100/q1000 must be numbers or null")
        rp = kf.get("retail_price")
        if rp is not None and (not isinstance(rp, dict) or not {"currency", "value"} <= set(rp) or not num_or_null(rp.get("value"))):
            rep.error("key_figures.retail_price needs currency and a numeric value (or null)")
    blocked = man.get("blocked")
    if not isinstance(blocked, list):
        rep.error("manifest.json blocked[] missing (use [] when nothing is blocked)")
    else:
        for b in blocked:
            if not isinstance(b, dict) or not {"what", "why"} <= set(b):
                rep.error(f"manifest.json blocked[] entry needs what and why: {b}")


def check_cost_json(deliv, rep, required):
    path = os.path.join(deliv, "09-bom-cost", "cost.json")
    if not os.path.isfile(path):
        if required:
            rep.error("09-bom-cost/cost.json missing")
        return None
    try:
        cost = json.load(open(path, encoding="utf-8"))
    except Exception as exc:  # noqa: BLE001
        rep.error(f"cost.json does not parse: {exc}")
        return None
    for key in ("currency", "bom", "unit_cost"):
        if key not in cost:
            rep.error(f"cost.json lacks {key}")
    for key in ("bom", "unit_cost"):
        block = cost.get(key)
        if isinstance(block, dict) and not all(num_or_null(block.get(q)) for q in ("q1", "q100", "q1000")):
            rep.error(f"cost.json {key} needs numeric q1, q100, q1000")
    other = cost.get("other", {})
    if not isinstance(other, dict):
        rep.error("cost.json other must be an object of {name: {q1, q100, q1000}}")
    else:
        try:
            for q in ("q1", "q100", "q1000"):
                total = float(cost["bom"][q]) + sum(float(v.get(q) or 0) for v in other.values())
                unit = float(cost["unit_cost"][q])
                if unit > 0 and abs(total - unit) / unit > 0.02:
                    rep.warn(f"cost.json unit_cost.{q}={unit} differs from bom+other={total:.2f} by >2 %")
        except (KeyError, TypeError, ValueError):
            pass
    return cost


def as_float(text):
    try:
        return float(str(text).replace(",", ".").strip())
    except ValueError:
        return None


def check_bom(deliv, cost, rep):
    path = os.path.join(deliv, "09-bom-cost", "bom.csv")
    if not os.path.isfile(path):
        rep.error("09-bom-cost/bom.csv missing")
        return
    rows = list(csv.reader(io.StringIO(open(path, encoding="utf-8-sig", errors="replace").read())))
    if not rows or [h.strip() for h in rows[0]] != BOM_HEADER:
        rep.error(f"bom.csv header must be exactly {','.join(BOM_HEADER)}")
        return
    totals = {"unit_price_1": 0.0, "unit_price_100": 0.0, "unit_price_1000": 0.0}
    currencies = set()
    lines = 0
    for n, row in enumerate(rows[1:], start=2):
        if not any(c.strip() for c in row):
            continue
        if len(row) != len(BOM_HEADER):
            rep.error(f"bom.csv line {n}: {len(row)} fields, expected {len(BOM_HEADER)}")
            continue
        rec = dict(zip(BOM_HEADER, [c.strip() for c in row]))
        lines += 1
        qty = as_float(rec["qty"])
        if qty is None:
            rep.error(f"bom.csv line {n}: qty '{rec['qty']}' is not a number")
            continue
        if not rec["mpn"] and not rec["supplier_part"]:
            rep.warn(f"bom.csv line {n}: neither mpn nor supplier_part ({rec['description'][:40]})")
        if rec["url"] and not rec["url"].startswith(("http://", "https://")):
            rep.warn(f"bom.csv line {n}: url is not http(s)")
        currencies.add(rec["currency"])
        for col in totals:
            price = as_float(rec[col])
            if price is None:
                rep.warn(f"bom.csv line {n}: {col} '{rec[col]}' is not a number")
            else:
                totals[col] += qty * price
    rep.note(f"bom.csv: {lines} lines; sums 1/100/1000 = {totals['unit_price_1']:.2f} / "
             f"{totals['unit_price_100']:.2f} / {totals['unit_price_1000']:.2f} {','.join(sorted(currencies))}")
    if len(currencies) > 1:
        rep.warn(f"bom.csv mixes currencies {sorted(currencies)}; totals not comparable")
    elif cost and isinstance(cost.get("bom"), dict):
        for q, col in (("q1", "unit_price_1"), ("q100", "unit_price_100"), ("q1000", "unit_price_1000")):
            ref = as_float(cost["bom"].get(q))
            if ref and abs(ref - totals[col]) / ref > 0.02:
                rep.warn(f"cost.json bom.{q}={ref} differs from the bom.csv sum {totals[col]:.2f} by >2 %")


def toolchain_python():
    """The toolchain Python the RUN CONTRACT names (ddpy), else the venv, else this interpreter."""
    for cand in (shutil.which("ddpy"), "/opt/dd/venv/bin/python"):
        if cand and os.path.exists(cand):
            return cand
    return sys.executable


def check_scripts(folder, rep, run):
    scripts = [p for p in nonempty_files(folder, (".py",))]
    for script in scripts:
        stem = os.path.splitext(script)[0]
        results = stem + ".results.json"
        if run:
            try:
                proc = subprocess.run([toolchain_python(), os.path.basename(script)], cwd=os.path.dirname(script),
                                      capture_output=True, text=True, timeout=300)
                if proc.returncode != 0:
                    rep.error(f"{os.path.relpath(script)} exits {proc.returncode}: {proc.stderr.strip()[-200:]}")
            except subprocess.TimeoutExpired:
                rep.error(f"{os.path.relpath(script)} runs longer than 5 minutes")
        if not os.path.isfile(results):
            rep.warn(f"{os.path.relpath(script)} has no {os.path.basename(results)}")
            continue
        try:
            data = json.load(open(results, encoding="utf-8"))
            bad = [k for k, v in data.items() if not (isinstance(v, dict) and "value" in v and "unit" in v)]
            if bad:
                rep.warn(f"{os.path.relpath(results)}: entries without value/unit: {bad[:5]}")
        except Exception as exc:  # noqa: BLE001
            rep.error(f"{os.path.relpath(results)} does not parse: {exc}")


def check_full(deliv, rep, run):
    for d in FULL_DIRS:
        if not nonempty_files(os.path.join(deliv, d)):
            rep.error(f"{d}/ missing or empty")
    cad = os.path.join(deliv, "03-cad")
    asm = os.path.join(cad, "assembly.step")
    if not os.path.isfile(asm):
        rep.error("03-cad/assembly.step missing")
    else:
        if not open(asm, "rb").read(64).lstrip().startswith(b"ISO-10303-21"):
            rep.error("03-cad/assembly.step is not a STEP file (no ISO-10303-21 header)")
    steps = [p for p in nonempty_files(cad, (".step", ".stp")) if not p.endswith("assembly.step")]
    if not steps:
        rep.error("03-cad/ has no per-part STEP files")
    if not nonempty_files(cad, (".py", ".fcstd")):
        rep.error("03-cad/ has no parametric source (.py or .FCStd)")
    if not nonempty_files(os.path.join(deliv, "04-drawings"), (".pdf", ".svg")):
        rep.error("04-drawings/ has no PDF or SVG drawing")
    check_scripts(os.path.join(deliv, "05-calculations"), rep, run)
    ele = os.path.join(deliv, "06-electronics")
    if not nonempty_files(ele, (".kicad_pro",)) or not nonempty_files(ele, (".kicad_sch",)):
        rep.error("06-electronics/ lacks a KiCad project (.kicad_pro + .kicad_sch)")
    names = " ".join(os.path.basename(p).lower() for p in nonempty_files(ele))
    if "erc" not in names:
        rep.warn("06-electronics/ has no file named like an ERC report")
    if nonempty_files(ele, (".kicad_pcb",)) and "drc" not in names:
        rep.warn("06-electronics/ has a board but no file named like a DRC report")
    if not os.path.isfile(os.path.join(deliv, "07-firmware", "platformio.ini")):
        rep.error("07-firmware/platformio.ini missing")
    cost = check_cost_json(deliv, rep, required=True)
    check_bom(deliv, cost, rep)
    if not os.path.isfile(os.path.join(deliv, "09-bom-cost", "cost.md")):
        rep.error("09-bom-cost/cost.md missing")
    ren = os.path.join(deliv, "11-renders")
    if not nonempty_files(ren, (".png",)):
        rep.error("11-renders/ has no PNG render")
    glb = os.path.join(ren, "assembly.glb")
    if not os.path.isfile(glb):
        rep.error("11-renders/assembly.glb missing")
    elif open(glb, "rb").read(4) != b"glTF":
        rep.error("11-renders/assembly.glb is not a binary glTF file")


def main():
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("--tier", choices=["concept", "full"], required=True)
    ap.add_argument("--run-dir", default=".", help="run directory (default: current directory)")
    ap.add_argument("--run-scripts", action="store_true", help="execute calculation scripts (≤5 min each)")
    ap.add_argument("--json", action="store_true", help="print a JSON summary as the last line")
    args = ap.parse_args()
    run_dir = os.path.abspath(args.run_dir)
    deliv = os.path.join(run_dir, "deliverables")
    rep = Report()
    if not os.path.isdir(deliv):
        rep.error("deliverables/ missing")
    else:
        ids = requirement_ids(os.path.join(run_dir, "INPUT", "requirements.md"))
        if not os.path.isdir(os.path.join(deliv, "01-requirements")):
            rep.error("01-requirements/ missing")
        else:
            check_compliance(deliv, ids, rep)
            check_ids_in_requirements(deliv, ids, rep)
        if not nonempty_files(os.path.join(deliv, "02-concept"), (".md",)):
            rep.error("02-concept/ has no markdown document")
        report = os.path.join(deliv, "REPORT.md")
        if not os.path.isfile(report):
            rep.error("REPORT.md missing")
        elif os.path.getsize(report) < 1500:
            rep.warn("REPORT.md is shorter than 1500 bytes")
        check_manifest(deliv, args.tier, rep)
        check_scripts(os.path.join(deliv, "02-concept"), rep, args.run_scripts)
        if args.tier == "concept":
            check_cost_json(deliv, rep, required=False)
            started = [d for d in FULL_DIRS if d != "09-bom-cost" and nonempty_files(os.path.join(deliv, d))]
            if started:
                rep.warn(f"concept tier: detailed-design folders present ({', '.join(started)}); they are not judged at this tier")
        else:
            check_full(deliv, rep, args.run_scripts)
    for msg in rep.notes:
        print(f"NOTE  {msg}")
    for msg in rep.warnings:
        print(f"WARN  {msg}")
    for msg in rep.errors:
        print(f"ERROR {msg}")
    print(f"dd_check: tier={args.tier} errors={len(rep.errors)} warnings={len(rep.warnings)} -> "
          f"{'PASS' if not rep.errors else 'FAIL'}")
    if args.json:
        print(json.dumps({"tier": args.tier, "pass": not rep.errors, "errors": rep.errors,
                          "warnings": rep.warnings, "notes": rep.notes}))
    return 0 if not rep.errors else 1


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""decide.py - apply the target-cost rule (method/POLICY.md section 3) to the concept options.

Usage (from the run directory):
  python3 method/gates/decide.py --init      write a skeleton work/decision.yaml from INPUT/requirements.md
                                             (active: numeric requirements of priority >= --min-weight 3, at
                                             most --max-criteria 12; the rest commented: # CHECK / # off / # cost)
  python3 method/gates/decide.py --compliance  write a compliance.csv skeleton (one not_addressed row per id)
  python3 method/gates/decide.py             read work/decision.yaml, write deliverables/02-concept/decision.md
                                             and work/decision.result.json, print the decision
  options: --in FILE  --out FILE  --json  --force (with --init / --compliance: overwrite)

The rule: (1) knock out every option whose nominal predicted value misses any threshold; (2) the base is the
cheapest survivor (landed unit cost at q1000 from its roll-up cost.json); (3) walk the other survivors in
order of cost and move to one only if it stays within the allowed unit cost of the price frame and buys at
least value_rate_min weighted-score points per 1 % of the allowed cost. Then a sensitivity check: every
weight +-20 %, all values at their pessimistic and optimistic ends, every option's cost +-15 %.
Values may be numbers, [low, nominal, high], true/false, or "=path/to/x.results.json#key" (read from a
calculation's results file, so the decision follows the calculations; dotted keys reach nested numbers,
e.g. "=deliverables/02-concept/arch-A/cost.json#unit_cost.q1000"). Format: the skeleton --init writes.
Exit 0 = decided, 1 = input error.
"""
import argparse
import csv
import datetime
import io
import json
import os
import re
import sys

sys.dont_write_bytecode = True
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from cost_rollup import InputError, frame_numbers, load_yaml, num  # noqa: E402  (also provides PyYAML)
from dd_check import requirement_ids  # noqa: E402  (the same ids the package gate expects)

HEADING = re.compile(r"^#{1,6}\s+([A-Z][A-Z0-9]{1,7}-[A-Z0-9]{1,7}-\d{1,4})\b\s*[-—:]*\s*(.*)$")
SECTION = re.compile(r"^#{1,3}\s+(.+)$")
NUMBER = re.compile(r"^(≤|≥|<=|>=|<|>|=)?\s*(-?\d+(?:[.,]\d+)?(?:e[+-]?\d+)?)(.*)$", re.I)
JOINER = re.compile(r"^\s*(?:[-–—/×+]|x\b|to\b|and\b|or\b|of\b|from\b|by\b|with\b)", re.I)
NOT_REQUIRED = re.compile(r"^(no|none|n/?a|not required|not applicable)\s*(\(.*\))?\s*[.;]?$", re.I)
YES = re.compile(r"^(yes|true)\s*(\(.*\))?\s*[.;]?$", re.I)
MONEY = re.compile(r"^(EUR|USD|PLN|GBP|CHF|€|\$|zł)", re.I)


def number(text):
    """(sign, value, unit) when the text is a number followed by a unit or nothing, else None (a text level):
    '5-pin and 3-pin ...', '2 x ...', '20-25 C' are text."""
    m = NUMBER.match(text.strip())
    if not m or (m.group(3).strip() and JOINER.match(m.group(3))):
        return None
    rest = m.group(3).strip().rstrip(".")
    return m.group(1) or "", float(m.group(2).replace(",", ".")), (rest.split() or [""])[0].replace('"', "'")[:20]


def parse_requirements(path="INPUT/requirements.md"):
    """Requirement blocks: id, title, section, weight, target and threshold texts."""
    blocks, cur, section = [], None, ""
    for line in open(path, encoding="utf-8", errors="replace").read().splitlines():
        m = HEADING.match(line.strip())
        if m:
            cur = {"id": m.group(1), "title": m.group(2).strip(), "section": section, "lines": []}
            blocks.append(cur)
            continue
        sm = SECTION.match(line.strip())
        if sm and not line.startswith("###"):
            section, cur = sm.group(1).strip(), None
        elif cur:
            cur["lines"].append(line.strip())
    for b in blocks:
        body = " | ".join(b["lines"])
        weight = re.search(r"(?:Priority|Weight)\s*:?\s*(\d+(?:\.\d+)?)", body, re.I)
        tm = re.search(r"Target\s*:\s*([^·;|]+)", body, re.I)
        hm = re.search(r"Threshold\s*:\s*([^·;|]+)", body, re.I)
        b["w"] = weight.group(1) if weight else "?"
        b["t"] = tm.group(1).strip() if tm else None
        b["h"] = hm.group(1).strip() if hm else None
    return blocks


class Crit:
    def __init__(self, rid, c):
        if not isinstance(c, dict) or "weight" not in c or "threshold" not in c:
            raise InputError(f"criterion {rid}: needs weight and threshold (and target)")
        self.id, self.w, self.unit = rid, num(c["weight"], f"{rid}.weight"), c.get("unit", "")
        self.thr, self.tgt = c["threshold"], c.get("target", c["threshold"])
        self.bool = isinstance(self.thr, bool)
        if not self.bool:
            self.thr, self.tgt = num(self.thr, f"{rid}.threshold"), num(self.tgt, f"{rid}.target")
            if self.tgt != self.thr:
                self.dir = 1 if self.tgt > self.thr else -1
            elif c.get("better") in ("higher", "lower"):
                self.dir = 1 if c["better"] == "higher" else -1
            else:
                raise InputError(f"criterion {rid}: target equals threshold; add better: higher|lower")

    def passes(self, v):
        if self.bool:
            return v is True or self.thr is not True
        return (v - self.thr) * self.dir >= 0

    def score(self, v):
        if self.bool:
            return 1.0 if v is True or (self.tgt is not True) else 0.0
        if self.tgt == self.thr:
            return 1.0 if self.passes(v) else 0.0
        return max(0.0, min(1.0, (v - self.thr) / (self.tgt - self.thr)))


def resolve(v, what):
    if not (isinstance(v, str) and v.startswith("=")):
        return v
    path, _, key = v[1:].partition("#")
    try:
        x = json.load(open(path, encoding="utf-8"))
    except (OSError, ValueError) as exc:
        raise InputError(f"{what}: cannot read {path}: {exc}")
    if not key:
        raise InputError(f"{what}: {v} names no key (=<file>#<key>; dotted keys for nested numbers)")
    for part in ([key] if isinstance(x, dict) and key in x else key.split(".")):
        if isinstance(x, dict) and part in x:
            x = x[part]
        elif isinstance(x, list) and part.isdigit() and int(part) < len(x):
            x = x[int(part)]
        else:
            raise InputError(f"{what}: {path} has no key '{key}' (stopped at '{part}')")
    if isinstance(x, dict):
        if "value" not in x:
            raise InputError(f"{what}: {v} is an object without 'value'; name the number inside it (#{key}.<name>)")
        x = x["value"]
    return x


def triple(v, crit, what):
    """(pessimistic, nominal, optimistic) for one criterion."""
    if isinstance(v, list):
        vals = [resolve(x, what) for x in v]
        if len(vals) == 1:
            vals = vals * 3
        if len(vals) != 3:
            raise InputError(f"{what}: give one value or [low, nominal, high]")
    else:
        vals = [resolve(v, what)] * 3
    if crit.bool:
        if not all(isinstance(x, bool) for x in vals):
            raise InputError(f"{what}: {crit.id} is a yes/no criterion; use true/false")
        return tuple(vals)
    vals = [num(x, what) for x in vals]
    low, nom, high = vals
    if not low <= nom <= high:
        raise InputError(f"{what}: [low, nominal, high] must be ordered, got {vals}")
    return (low, nom, high) if crit.dir > 0 else (high, nom, low)


def mapping(d, key, what):
    v = d.get(key) or {}
    if not isinstance(v, dict):
        raise InputError(f"{what}: {key} must be a mapping")
    return v


def load(path):
    d = load_yaml(path)
    crits = [Crit(k, v) for k, v in mapping(d, "criteria", path).items()]
    if not crits:
        raise InputError(f"{path}: no criteria (remove the '# CHECK: ' prefix of the lines you want to use)")
    allowed, curs = d.get("allowed_unit_cost_q1000"), {"work/decision.yaml": d.get("currency")}
    if allowed is None:
        fp = d.get("frame", "work/price_frame.yaml")
        if os.path.exists(fp):
            fr = frame_numbers(load_yaml(fp))
            allowed, curs["price frame"] = fr["allowed"].get("q1000"), fr["currency"]
    allowed = None if allowed is None else num(allowed, "allowed_unit_cost_q1000")
    if allowed is not None and allowed <= 0:
        raise InputError("allowed_unit_cost_q1000 must be > 0 (or null: taken from work/price_frame.yaml)")
    opts = {}
    for key, o in mapping(d, "options", path).items():
        what = f"option {key}"
        if not isinstance(o, dict):
            raise InputError(f"{what}: must be a mapping (label, cost_json, values)")
        if "cost_json" in o:
            try:
                cj = json.load(open(o["cost_json"], encoding="utf-8"))
                cost = num(cj["unit_cost"]["q1000"], what)
            except (OSError, ValueError, KeyError, TypeError) as exc:
                raise InputError(f"{what}: cannot read unit_cost.q1000 from {o['cost_json']}: {exc!r}")
            curs[o["cost_json"]] = cj.get("currency")
        elif "unit_cost_q1000" in o:
            cost = num(o["unit_cost_q1000"], f"{what}.unit_cost_q1000")
        else:
            raise InputError(f"{what}: needs cost_json (roll-up of its BOM) or unit_cost_q1000")
        vals = mapping(o, "values", what)
        missing = [c.id for c in crits if c.id not in vals]
        if missing:
            raise InputError(f"{what}: no value for {missing} (give every option a value for every criterion)")
        opts[key] = {"key": key, "label": o.get("label", ""), "cost": cost,
                     "v": {c.id: triple(vals[c.id], c, f"{what} {c.id}") for c in crits}}
    if len(opts) < 2:
        raise InputError(f"{path}: at least two options needed")
    named = {k: v for k, v in curs.items() if v}
    if len(set(named.values())) > 1:
        raise InputError(f"currencies differ: {named} - roll up every option in the price frame's currency")
    d["currency"] = next(iter(named.values()), "")
    rate = num(d.get("value_rate_min", 1.0), "value_rate_min")
    return d, crits, allowed, opts, rate


def evaluate(crits, opts, allowed, rate_min, mode=1, wmul=None, cmul=None):
    """mode 0 pessimistic, 1 nominal, 2 optimistic. Returns (choice, status, table, steps)."""
    wmul, cmul = wmul or {}, cmul or {}
    wsum = sum(c.w * wmul.get(c.id, 1.0) for c in crits)
    table = {}
    for k, o in opts.items():
        vals = {c.id: o["v"][c.id][mode] for c in crits}
        misses = [c.id for c in crits if not c.passes(vals[c.id])]
        sc = 100 * sum(c.w * wmul.get(c.id, 1.0) * c.score(vals[c.id]) for c in crits) / wsum
        table[k] = {"cost": o["cost"] * cmul.get(k, 1.0), "score": sc, "misses": misses}
    surv = sorted((k for k in opts if not table[k]["misses"]), key=lambda k: (table[k]["cost"], -table[k]["score"]))
    steps = []
    if not surv:
        k = min(opts, key=lambda k: (sum(c.w for c in crits if c.id in table[k]["misses"]), table[k]["cost"]))
        steps.append(f"no option meets every threshold; least weighted miss: {k}")
        return k, "NO_OPTION_MEETS_ALL_THRESHOLDS", table, steps
    cur = surv[0]
    steps.append(f"survivors (cheapest first): {', '.join(surv)}; base = {cur} ({table[cur]['cost']:.2f})")
    for k in surv[1:]:
        t, c = table[k], table[cur]
        if allowed is not None and t["cost"] > allowed:
            over = 100 * (t["cost"] - allowed) / allowed
            r = (t["score"] - c["score"]) / (100 * (t["cost"] - c["cost"]) / allowed or 1.0)
            ve = over <= 15 and r >= rate_min
            steps.append(f"{k}: {t['cost']:.2f} exceeds the allowed {allowed:.2f} by {over:.1f} %; value rate vs {cur} "
                         f"{r:.2f} points/% -> not eligible" + ("; VE CANDIDATE (POLICY 3.4: one value-engineering "
                                                                "iteration may bring it under the frame)" if ve else ""))
            continue
        dscore = t["score"] - c["score"]
        dpct = 100 * (t["cost"] - c["cost"]) / (allowed or c["cost"] or 1.0)
        if dpct <= 0:
            move = dscore > 0
            steps.append(f"{k}: same cost, {dscore:+.1f} points -> {'move' if move else 'stay'}")
        else:
            r = dscore / dpct
            move = r >= rate_min
            steps.append(f"{k} vs {cur}: {dscore:+.1f} points for {dpct:+.1f} % of the allowed cost = "
                         f"{r:.2f} points/% {'>=' if move else '<'} {rate_min:g} -> {'move to ' + k if move else 'stay'}")
        if move:
            cur = k
    status = "OVER_FRAME" if allowed is not None and table[cur]["cost"] > allowed else "WITHIN_FRAME"
    if allowed is None:
        status = "NO_FRAME"
    return cur, status, table, steps


def sensitivity(crits, opts, allowed, rate_min, winner):
    cases, flips = 0, []
    for c in crits:
        for f in (0.8, 1.2):
            cases += 1
            w = evaluate(crits, opts, allowed, rate_min, 1, wmul={c.id: f})[0]
            if w != winner:
                flips.append(f"weight of {c.id} x{f:g} -> {w}")
    for mode, name in ((0, "pessimistic"), (2, "optimistic")):
        cases += 1
        w = evaluate(crits, opts, allowed, rate_min, mode)[0]
        if w != winner:
            flips.append(f"all values at their {name} end -> {w}")
    for k in opts:
        for f in (0.85, 1.15):
            cases += 1
            w = evaluate(crits, opts, allowed, rate_min, 1, cmul={k: f})[0]
            if w != winner:
                flips.append(f"cost of {k} x{f:g} -> {w}")
    return cases, flips


def fmtv(v):
    if isinstance(v, bool):
        return "yes" if v else "no"
    return f"{v:g}"


def decide(args):
    d, crits, allowed, opts, rate = load(args.inp)
    winner, status, table, steps = evaluate(crits, opts, allowed, rate)
    risk = {k: [c.id for c in crits if not c.passes(opts[k]["v"][c.id][0])] for k in opts}
    cases, flips = sensitivity(crits, opts, allowed, rate, winner)
    cur = d.get("currency", "")
    now = datetime.datetime.now().astimezone().isoformat(timespec="minutes")
    md = ["# Decision by the target-cost rule", "",
          f"Generated by `method/gates/decide.py` at {now} from `{args.inp}`; do not edit (change the inputs, "
          "re-run). Rule: method/POLICY.md section 3.", "",
          f"- Allowed landed unit cost at q1000 (price frame): {'none given' if allowed is None else f'{allowed:.2f} {cur}'}",
          f"- Minimum value rate: {rate:g} weighted-score points per 1 % of the allowed cost (POLICY)",
          f"- Score: per criterion (value - threshold) / (target - threshold) clipped to 0..1, weighted by the "
          f"requirement weights, 0-100", "",
          "## Options", "", f"| option | label | unit cost q1000 ({cur}) | vs allowed | score | misses a threshold "
          "(nominal) | at risk (pessimistic end misses) |", "|---|---|---:|---:|---:|---|---|"]
    for k, o in opts.items():
        t = table[k]
        vs = "–" if allowed is None else f"{(t['cost'] - allowed) / allowed * 100:+.1f} %"
        md.append(f"| {k} | {o['label']} | {t['cost']:.2f} | {vs} | {t['score']:.1f} | "
                  f"{', '.join(t['misses']) or 'none'} | {', '.join(risk[k]) or 'none'} |")
    md += ["", "## Values per criterion (nominal; [pessimistic .. optimistic] where given)", "",
           "| id | weight | threshold | target | unit | " + " | ".join(opts) + " |",
           "|---|---:|---:|---:|---|" + "---|" * len(opts)]
    for c in crits:
        cells = []
        for k in opts:
            p, n, o = opts[k]["v"][c.id]
            mark = "" if c.passes(n) else " **MISS**"
            cells.append(f"{fmtv(n)}" + ("" if p == o else f" [{fmtv(p)}..{fmtv(o)}]") + mark)
        md.append(f"| {c.id} | {c.w:g} | {fmtv(c.thr)} | {fmtv(c.tgt)} | {c.unit} | " + " | ".join(cells) + " |")
    md += ["", "## Rule walk", ""] + [f"{i}. {s}" for i, s in enumerate(steps, start=1)]
    md += ["", f"**Decision: option {winner} ({opts[winner]['label']}) - {status}**", ""]
    losers = [k for k in opts if k != winner]
    w = table[winner]
    for k in losers:
        t = table[k]
        dscore = w["score"] - t["score"]
        dpct = 100 * (w["cost"] - t["cost"]) / (allowed or t["cost"] or 1.0)
        if t["misses"]:
            why = f"misses {', '.join(t['misses'])}"
        elif allowed is not None and t["cost"] > allowed:
            why = f"over the allowed cost ({t['cost']:.2f} > {allowed:.2f})"
        elif status == "NO_OPTION_MEETS_ALL_THRESHOLDS":
            why = "larger weighted threshold miss"
        elif t["cost"] < w["cost"]:
            why = (f"cheaper, but {winner} buys {dscore:+.1f} points for {dpct:.1f} % of the allowed cost "
                   f"({dscore / dpct if dpct else float('inf'):.2f} points/% >= {rate:g})")
        elif t["score"] <= w["score"]:
            why = f"no better than {winner} (score {t['score']:.1f} vs {w['score']:.1f}) at a cost of {t['cost']:.2f}"
        else:
            why = (f"its {-dscore:+.1f} points over {winner} cost {-dpct:.1f} % of the allowed cost "
                   f"({dscore / dpct if dpct else 0:.2f} points/% < {rate:g})")
        md.append(f"- {k} lost: {why}")
    md += ["", "## Sensitivity", "", f"{cases} perturbed cases (each weight x0.8 and x1.2; all values at the pessimistic "
           f"and at the optimistic end; each option's cost x0.85 and x1.15).", ""]
    md += ([f"- winner changes: {f}" for f in flips] if flips else ["- the winner holds in every case"])
    verdict = "ROBUST" if not flips else "SENSITIVE"
    md += ["", f"Verdict: **{verdict}**" + ("" if not flips else " - name the estimate or weight that decides it and "
           "how detailed design retires that uncertainty.")]
    if risk[winner]:
        md += ["", f"Winner at risk on {', '.join(risk[winner])}: its pessimistic estimate misses the threshold."]
    os.makedirs(os.path.dirname(args.out) or ".", exist_ok=True)
    with open(args.out, "w", encoding="utf-8") as fh:
        fh.write("\n".join(md) + "\n")
    result = {"winner": winner, "status": status, "verdict": verdict, "allowed_unit_cost_q1000": allowed,
              "options": {k: {"cost": round(table[k]["cost"], 2), "score": round(table[k]["score"], 1),
                              "misses": table[k]["misses"], "at_risk": risk[k]} for k in opts},
              "flips": flips, "steps": steps}
    os.makedirs("work", exist_ok=True)
    with open("work/decision.result.json", "w", encoding="utf-8") as fh:
        json.dump(result, fh, indent=2)
    for s in steps:
        print(f"STEP  {s}")
    for k in opts:
        t = table[k]
        print(f"OPTION {k}: cost {t['cost']:.2f} score {t['score']:.1f} misses {t['misses'] or '-'} at risk {risk[k] or '-'}")
    print(f"DECISION {winner} {status}; sensitivity {verdict} ({len(flips)} of {cases} cases flip) -> {args.out}")
    if args.json:
        print(json.dumps(result))


def is_cost(b, unit):
    return bool(re.search(r"cost|price", b["section"], re.I) or re.search(r"\b(cost|price|retail)\b", b["title"], re.I)
                or MONEY.match(unit or ""))


def criterion(b):
    """(kind, yaml mapping text, note) for one requirement block; kind: number | bool | optional | level | none."""
    w, t, h, title = b["w"], b["t"], b["h"], b["title"][:60].replace("#", "")
    if t is None or h is None:
        return "none", "{weight: %s, threshold: ?, target: ?}" % w, f"{title} (no target/threshold found)"
    nt, nh = number(t), number(h)
    if nt and nh and not YES.match(t):
        sign = nh[0] or nt[0]
        better = ", better: higher" if sign in ("≥", ">=", ">") else (", better: lower" if sign in ("≤", "<=", "<") else "")
        return "number", '{weight: %s, threshold: %g, target: %g, unit: "%s"%s}' % (w, nh[1], nt[1], nt[2] or nh[2],
                                                                                    better), title
    if YES.match(t) and YES.match(h):
        return "bool", "{weight: %s, threshold: true, target: true}" % w, title
    if NOT_REQUIRED.match(h):
        return "optional", '{weight: %s, threshold: 0, target: 1, unit: "0 no, 1 yes"}' % w, f"{title} | 1 = {t[:50]}"
    return "level", '{weight: %s, threshold: 1, target: 2, unit: "level"}' % w, f"{title} | 2 = {t[:50]} | 1 = {h[:50]}"


def frame_currency():
    try:
        return frame_numbers(load_yaml("work/price_frame.yaml"))["currency"] or "EUR"
    except InputError:
        return "EUR"


def init(args):
    target = args.inp
    if os.path.exists(target) and not args.force:
        raise InputError(f"{target} exists (use --force to overwrite)")
    blocks = parse_requirements()
    rows = []
    for b in blocks:
        kind, crit, note = criterion(b)
        unit = number(b["h"] or "")[2] if number(b["h"] or "") else ""
        rows.append({"b": b, "kind": kind, "crit": crit, "note": note, "cost": is_cost(b, unit),
                     "w": float(b["w"]) if b["w"] != "?" else -1.0})
    eligible = [r for r in rows if r["kind"] == "number" and not r["cost"] and r["w"] >= args.min_weight]
    active = {id(r) for r in sorted(eligible, key=lambda r: -r["w"])[:args.max_criteria]}
    out = ["# work/decision.yaml - inputs of method/gates/decide.py (skeleton from INPUT/requirements.md).",
           f"# Active: numeric requirements of priority >= {args.min_weight:g}, at most {args.max_criteria}, highest "
           "priority first. Keep a criterion only if its",
           "# predicted value differs between the options or lies within 20 % of its threshold in any option; delete",
           "# the others and say in architectures.md that all options meet them alike. CHECK every parsed number.",
           "# '# CHECK:' lines (yes/no and text requirements) and '# off:' lines are inactive: delete that prefix to",
           "# use one (yes/no: true/false values; level: 0 misses the threshold text, 1 meets it, 2 meets the target",
           "# text). '# cost:' lines stay out: the price frame carries cost limits (POLICY 2.4).",
           f"currency: {frame_currency()}", "allowed_unit_cost_q1000: null   # null = from work/price_frame.yaml",
           "value_rate_min: 1.0             # POLICY default", "criteria:"]
    for r in rows:
        if id(r) in active:
            prefix = ""
        elif r["cost"]:
            prefix = "# cost: "
        elif r["kind"] == "number" and r["w"] < 0:
            prefix = "# CHECK (no priority found): "
        elif r["kind"] == "number":
            prefix = "# off: "
        elif r["kind"] == "none":
            prefix = "# "
        else:
            prefix = "# CHECK: "
        out.append(f"  {prefix}{r['b']['id']}: {r['crit']}   # {r['note']}")
    out += ["options:"]
    for k, label in (("A", "lowest cost meeting every threshold"), ("B", "best value"), ("C", "premium")):
        out += [f"  {k}:", f'    label: "{label}"', f"    cost_json: deliverables/02-concept/arch-{k}/cost.json",
                "    values: {}   # <criterion id>: number | [low, nominal, high] | true/false | \"=<results.json>#<key>\""]
    os.makedirs(os.path.dirname(target) or ".", exist_ok=True)
    with open(target, "w", encoding="utf-8") as fh:
        fh.write("\n".join(out) + "\n")
    print(f"decide: wrote {target}: {len(active)} active numeric criteria, {len(rows) - len(active)} commented "
          f"(# CHECK / # off / # cost) of {len(blocks)} requirements; check them against INPUT/requirements.md")


def compliance(args):
    """deliverables/01-requirements/compliance.csv with one not_addressed row per requirement id, to be filled."""
    path = "deliverables/01-requirements/compliance.csv"
    if os.path.exists(path) and not args.force:
        raise InputError(f"{path} exists (use --force to overwrite)")
    info = {b["id"]: b for b in parse_requirements()}
    buf = io.StringIO()
    wr = csv.writer(buf, lineterminator="\n")
    wr.writerow(["req_id", "status", "value", "unit", "evidence", "note"])
    ids = requirement_ids("INPUT/requirements.md")
    for rid in ids:
        b = info.get(rid, {})
        nh = number(b.get("h") or "")
        wr.writerow([rid, "not_addressed", "", nh[2] if nh else "", "",
                     f"to fill: threshold {b.get('h') or '?'}; target {b.get('t') or '?'}"])
    os.makedirs(os.path.dirname(path), exist_ok=True)
    with open(path, "w", encoding="utf-8", newline="") as fh:
        fh.write(buf.getvalue())
    print(f"decide: wrote {path} with {len(ids)} rows (status not_addressed until you fill value, unit, evidence)")


def main():
    ap = argparse.ArgumentParser(description="Apply the target-cost rule to the concept options.")
    ap.add_argument("--in", dest="inp", default="work/decision.yaml")
    ap.add_argument("--out", default="deliverables/02-concept/decision.md")
    ap.add_argument("--init", action="store_true", help="write a skeleton decision.yaml from INPUT/requirements.md")
    ap.add_argument("--compliance", action="store_true", help="write a compliance.csv skeleton from the requirement ids")
    ap.add_argument("--min-weight", type=float, default=3.0, help="--init: lowest priority of an active criterion")
    ap.add_argument("--max-criteria", type=int, default=12, help="--init: at most this many active criteria")
    ap.add_argument("--force", action="store_true")
    ap.add_argument("--json", action="store_true")
    args = ap.parse_args()
    try:
        init(args) if args.init else compliance(args) if args.compliance else decide(args)
    except (InputError, OSError) as exc:
        print(f"ERROR {exc}")
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""manifest_build.py - (re)build deliverables/manifest.json from the files that exist and their homes.

Usage (from the run directory):
  python3 method/gates/manifest_build.py --tier concept|full

- files[]: every non-empty file under deliverables/ (except manifest.json, caches and hidden files); the kind
  and desc of entries already in manifest.json are kept, so edit desc texts there and re-run freely; new files
  get a kind from their folder and a desc from their first markdown heading or their name.
- key_figures.unit_cost: from deliverables/09-bom-cost/cost.json (the cost model's output, never typed).
- key_figures.retail_price: from work/price_frame.yaml (target retail incl. VAT).
- key_figures.dimensions_mm / mass_kg / power_w: from work/key_figures.yaml ({value, origin} or a number;
  null when unknown).
- blocked[]: from work/blocked.yaml (a list of {what, why}) when present, else kept from manifest.json.
"""
import argparse
import json
import os
import sys

sys.dont_write_bytecode = True
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from cost_rollup import InputError, frame_numbers, load_yaml  # noqa: E402  (also provides PyYAML)

import yaml  # noqa: E402

KINDS = {"01-requirements": "requirements", "02-concept": "concept", "03-cad": "cad", "04-drawings": "drawing",
         "05-calculations": "calculation", "06-electronics": "electronics", "07-firmware": "firmware",
         "08-safety-certification": "safety", "09-bom-cost": "bom-cost", "10-prototype-test-plan": "test-plan",
         "11-renders": "render"}
SKIP_DIRS = {"__pycache__", ".pio", ".git", "node_modules", ".venv"}


def first_heading(path):
    try:
        with open(path, encoding="utf-8", errors="replace") as fh:
            for _ in range(30):
                line = fh.readline()
                if not line:
                    break
                if line.startswith("#"):
                    return line.lstrip("#").strip()[:120]
    except OSError:
        pass
    return None


def figure(raw, key):
    v = raw.get(key)
    if isinstance(v, dict):
        v = v.get("value")
    return v


def main():
    ap = argparse.ArgumentParser(description="Rebuild deliverables/manifest.json.")
    ap.add_argument("--tier", choices=["concept", "full"], required=True)
    ap.add_argument("--deliverables", default="deliverables")
    args = ap.parse_args()
    d = args.deliverables
    mpath = os.path.join(d, "manifest.json")
    old = {}
    if os.path.exists(mpath):
        try:
            old = json.load(open(mpath, encoding="utf-8"))
        except ValueError:
            print("WARN  existing manifest.json does not parse; descriptions not kept")
    if not isinstance(old, dict):
        old = {}
    known = {f.get("path"): f for f in old.get("files") or [] if isinstance(f, dict)}
    files = []
    for root, dirs, names in os.walk(d):
        dirs[:] = sorted(x for x in dirs if x not in SKIP_DIRS and not x.startswith("."))
        for name in sorted(names):
            p = os.path.join(root, name)
            rel = os.path.relpath(p, d).replace(os.sep, "/")
            if rel == "manifest.json" or name.startswith(".") or name.endswith(".pyc") or ".tmp" in name or os.path.getsize(p) == 0:
                continue
            top = rel.split("/", 1)[0]
            prev = known.get(rel, {})
            kind = prev.get("kind") or ("report" if rel == "REPORT.md" else KINDS.get(top, "other"))
            desc = prev.get("desc") or (first_heading(p) if name.endswith(".md") else None) or f"{kind}: {name}"
            files.append({"path": rel, "kind": kind, "desc": desc})
    kf = dict(old.get("key_figures") if isinstance(old.get("key_figures"), dict) else {})
    try:
        cost = json.load(open(os.path.join(d, "09-bom-cost", "cost.json"), encoding="utf-8"))
        uc = cost["unit_cost"]
        kf["unit_cost"] = {"currency": cost["currency"], "q1": uc["q1"], "q100": uc["q100"], "q1000": uc["q1000"]}
    except (OSError, ValueError, KeyError, TypeError):
        kf.setdefault("unit_cost", None)
        print("WARN  09-bom-cost/cost.json missing or incomplete: unit_cost kept/null")
    try:
        fr = frame_numbers(load_yaml("work/price_frame.yaml"))
        if fr["retail_incl_vat"] is not None:
            kf["retail_price"] = {"currency": fr["currency"], "value": round(fr["retail_incl_vat"], 2)}
    except InputError:
        print("WARN  work/price_frame.yaml missing or unreadable: retail_price kept/null")
    kf.setdefault("retail_price", None)
    raw = {}
    if os.path.exists("work/key_figures.yaml"):
        try:
            raw = load_yaml("work/key_figures.yaml")
        except InputError as exc:
            print(f"WARN  {exc}")
    for key in ("dimensions_mm", "mass_kg", "power_w"):
        if key in raw:
            kf[key] = figure(raw, key)
        kf.setdefault(key, None)
    blocked = old.get("blocked", [])
    if os.path.exists("work/blocked.yaml"):
        try:
            blocked = yaml.safe_load(open("work/blocked.yaml", encoding="utf-8")) or []
        except yaml.YAMLError as exc:
            print(f"WARN  work/blocked.yaml: {exc}")
    if not isinstance(blocked, list):
        blocked = []
    man = {"tier": args.tier, "files": files,
           "key_figures": {k: kf.get(k) for k in ("dimensions_mm", "mass_kg", "power_w", "unit_cost", "retail_price")},
           "blocked": blocked}
    tmp = mpath + ".tmp"
    os.makedirs(d, exist_ok=True)
    with open(tmp, "w", encoding="utf-8") as fh:
        json.dump(man, fh, indent=2, ensure_ascii=False)
        fh.write("\n")
    os.replace(tmp, mpath)
    print(f"manifest_build: {len(files)} files, tier {args.tier}, unit_cost {kf.get('unit_cost')}, "
          f"retail {kf.get('retail_price')}, blocked {len(blocked)} -> {mpath}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

# Role: analyst (full tier) — item prefix S (only if you add safety parts)

Read `method/roles/designer-common.md` first; it binds you. Handbook sections 4, 6, 11 and 13.

**Mission**: the calculations that prove or disprove the requirements with the real parts' datasheet values,
and the safety and certification file.

**Inputs**: `deliverables/02-concept/` (physics map, calc scripts, safety concept, compliance route),
`deliverables/01-requirements/`, `work/params.yaml`, the dossiers `work/parts/*.json`, the catalogue
`work/catalog/*.json` (certificates of pre-certified modules).

**You write only**: `deliverables/05-calculations/**` except files named `mech_*` and `elec_*`,
`deliverables/08-safety-certification/**`, `work/other_costs.d/certification.yaml`,
`work/bom.d/safety.yaml` (only if a protective part is missing; tell the lead), `work/requests/analyst-*.md`,
`work/status/analyst.md`.

## Method
1. **Questions**: every requirement whose verification is calculation or analysis, and every CRITICAL domain
   of the physics map, becomes one script. Each script states its hand estimate in the header, takes inputs
   from dossiers (SOURCE with page), `work/params.yaml` or labelled ASSUMPTIONs, and writes value,
   requirement, margin and status into its results file. Re-use and promote the concept's `calc/` scripts
   with the real parts' values.
2. **Simulation** only where a decision depends on it: known-answer case first, then three resolutions with
   the trend, balance checks and a sensitivity run on the least certain inputs.
3. **Index**: `05-calculations/index.md` — question → script → result → requirement id → status
   (PASS (analysis) / PARTIAL / FAIL). A FAIL goes to the lead at once as a request with the numbers.
4. **Safety** (`08-safety-certification/`): hazard analysis in the ISO 12100 manner (hazard → situation →
   harm → risk → reduction: inherent design, protective measures, information → residual risk); safety
   requirements with ids traced to design measures, marking those that act independently of firmware.
5. **Certification**: the regulatory map with SOURCE (EU directives and the harmonised standards that apply to
   this product found by research, the clauses that drive the design; FCC parts and the voluntary listings US
   retailers expect); the conformity route; the evidence that pre-certified modules bring (certificate or
   FCC ID URLs from the catalogue) and the tests they remove; the lab test list; a pre-compliance plan; cost
   and duration. Put the certification one-off cost in `work/other_costs.d/certification.yaml` (it replaces
   the concept estimate the lead moved there) and run the cost roll-up.

Round 2 (fix list in `work/fixes/analyst.md`): re-run every script with the final parameters and parts
(`ddpy` from the folder), update the index and the safety file.

# Rules for every designer subagent (full tier)

You are one of four designer subagents that work in parallel on the detailed design of a product whose
concept, purchased parts and cost model already exist. The lead (the top session) wrote your task message.
Nobody answers questions: decide, record the reason, continue.

## Designing around real parts
- The product is composed of the purchased parts in `work/bom.yaml`, chosen by the target-cost rule
  (`deliverables/02-concept/decision.md`). Their datasheet facts are in the dossiers `work/parts/<item>.json`.
  Their envelopes, mounting points, pinouts and ratings are fixed inputs: design around them.
- Never swap, add or drop a purchased part of `work/bom.yaml` on your own. If one blocks you (does not fit,
  overheats, lacks a function), write `work/requests/<role>-<n>.md` (what, why, evidence, a catalogue
  alternative with its cost delta) and continue on a labelled ASSUMPTION; the lead decides at integration.
- Parts the concept did not list (fasteners, passives, connectors, seals, inserts, cables) are yours to add as
  real parts in the BOM fragment your role file names (format `method/templates/bom.yaml` with `currency` and
  `lines` only: never copy the fx block, the rates live in the lead's `work/bom.yaml`; item ids with your
  prefix): prices from the offline catalogue or a page you read (SOURCE + date), else ASSUMPTION with the
  reason. Costs you cause that are not parts (PCB fabrication, tooling, certification) go into the other-cost
  fragment your role file names (format `method/templates/other_costs.yaml`). Extend a fragment the lead
  already created; never start a second file for the same lines.
- After every change to your fragments run `python3 method/gates/cost_rollup.py` and read its FRAME and
  BUDGET lines. A change that pushes the unit cost over the frame needs a request, not silence.
- Shared design numbers live in `work/params.yaml` (the lead's); your own numbers live in the parameter
  block at the top of your scripts, each with its origin label.

## Working rules
- Before you start read `work/boundaries.md` (binding), `method/HYGIENE.md` and the handbook sections your
  role file names (`.claude/skills/dd-hw-handbook/SKILL.md`). Tool commands and examples: `INPUT/TOOLCHAIN.md`.
- Write only the paths your role file lists and your scratch folder `work/scratch/<role>/` (route proofs,
  known-answer tests, intermediate files: evidence that is not a deliverable); every other file is read-only
  for you.
- Overwrite `work/status/<role>.md` after each milestone (done, files, key numbers, open issues, next step):
  the lead reads it while you work.
- `ddpy` runs anything that imports scientific or CAD libraries. Scripts in `deliverables/05-calculations/`
  run as `ddpy <script>` from that folder, offline, in under 5 minutes, and write `<stem>.results.json`
  (`{"<name>": {"value": x, "unit": "..."}}`).
- Prove each tool route once on a tiny case with a known answer before relying on it, and keep the evidence
  in `work/scratch/<role>/`.
- Done means checked by execution: a file you have not opened, parsed or run does not count.
- Never give a file a name that starts with `report`, `summary`, `findings` or `analysis` in any letter case
  (the harness refuses such writes from subagents). Never write `deliverables/REPORT.md` or
  `deliverables/manifest.json` (the lead's).
- Read-only web for datasheets and standards; no accounts, quotes or contacts; never open the website
  `work/boundaries.md` names and never type its host into a search query, command or prompt. Read nothing
  outside the run directory except system and toolchain paths.
- Time: `date` at the start and about every 30 minutes against the deadline in your task message. At the
  deadline start nothing new; make what exists consistent and report. Partial and honest beats late.

## End with this message (at most 20 lines)
`<ROLE> round <n>: done | partial`; files written; key figures with units and origins; checks run (command and
outcome); BOM and cost changes (items, unit cost q1000 after the change); requests filed; open issues.

# Role: docs-visuals (full tier) — no BOM items

Read `method/roles/designer-common.md` first; it binds you. Handbook sections 8, 13 and 14.

**Mission**: the documents a workshop and a test lab need, derived from the real design: the prototype and test
plan, the drawings and the renders.

**Inputs**: `deliverables/01-requirements/`, `deliverables/02-concept/`, the BOM (`deliverables/09-bom-cost/bom.csv`,
`work/bom.yaml`, `work/bom.d/*.yaml`), `deliverables/03-cad/` (scripts, STEP), `deliverables/08-safety-certification/`
when it appears, `deliverables/11-renders/assembly.glb` when it appears.

**You write only**: `deliverables/04-drawings/**`, `deliverables/10-prototype-test-plan/**`,
`deliverables/11-renders/**` except `assembly.glb`, `work/requests/docs-visuals-*.md`, `work/status/docs-visuals.md`.

## Round 1 (CAD not final yet)
1. **Test plan** (`10-prototype-test-plan/`): the build plan of prototype 1 from the BOM (how each part is made
   or bought, order of assembly, tools, the check after each step); the bring-up procedure for a person
   later (current-limited or isolated first power-up, what to measure before full power, precautions); one
   test per requirement id and per safety requirement (method, equipment, conditions, pass criterion = the
   threshold, the id it closes), pre-compliance EMC and the tests a lab would run, life tests where lifetime
   is a requirement; a traceability table requirement id → test id. Check by script that every id of
   `INPUT/requirements.md` appears in it.
2. **Prove the routes** in `work/scratch/docs-visuals/` (never in `deliverables/`): make one drawing of a simple
   known solid with the drawing route of `INPUT/TOOLCHAIN.md` and one Blender render of a simple GLB; open
   both outputs (render a PNG of the drawing, read image size and content bounds) and keep the scripts for
   round 2.

## Round 2 (after the lead's integration review; CAD exists)
3. **Drawings** (`04-drawings/`): every manufactured part (views, overall and functional dimensions, general
   tolerance class plus explicit tolerances on functional features, material, finish, quantity, part number
   = the BOM `mpn`, item id, revision) and an assembly drawing whose balloons are the `bom.csv` item ids.
   PDF or SVG (DXF optional). Open each output: render a PNG and check that the part and the dimensions are
   present; a file that does not open does not exist.
4. **Renders** (`11-renders/`): Blender in background mode from `assembly.glb`: a product view, an exploded
   view and a view of the user interface; captions say "render of a digital design". Check by measurement:
   resolution, the object fills the frame, part names present, scale against a known CAD dimension. Never use
   generative image models to depict the product.
5. Update the test plan with the final parts, drawings and safety file.

# Role: electronics-firmware (full tier) — item prefix E

Read `method/roles/designer-common.md` first; it binds you. Handbook sections 9, 10 and 12.

**Mission**: the schematic, board and firmware built from the chosen parts: footprints from their datasheets,
the firmware's pin map from the module pinouts.

**Inputs**: `deliverables/02-concept/concept.md`, `work/interfaces.md` (board outline, connector positions,
power budget), `work/params.yaml`, `work/bom.yaml`, the dossiers `work/parts/*.json`.

**You write only**: `deliverables/06-electronics/**`, `deliverables/07-firmware/**`,
`deliverables/05-calculations/elec_*.py` (+ results), `work/bom.d/electronics.yaml`,
`work/other_costs.d/electronics.yaml`, `work/requests/electronics-firmware-*.md`,
`work/status/electronics-firmware.md`.

## Method
1. **Power budget** (`05-calculations/elec_power_budget.py`): every consumer in every operating mode over the
   full supply range, from dossier ratings; check the supply part's rating with margin.
2. **Schematic** (KiCad 9, headless route per `INPUT/TOOLCHAIN.md` and its examples): symbols whose pins match
   the dossier pinout; footprints from the KiCad libraries checked against the datasheet (pitch, pad size,
   outline) or drawn from datasheet dimensions; every symbol carries the manufacturer part number and the
   supplier code (LCSC/JLC code where the offline catalogue has the part).
3. **Protection** where it applies, each with a real part: input fuse or current limit, reverse polarity,
   transient and ESD protection on external connectors, inrush limiting, a hardware thermal cut-off for hot
   parts, a safe state on power loss. Safety reactions never depend on firmware alone.
4. **Board**: outline and connector positions from `work/interfaces.md` (agreed with mechanical-cad through
   the lead), placement, routing (the autorouter route in `INPUT/TOOLCHAIN.md` or by script), ERC and DRC
   from the command line with JSON reports, zero errors or a written waiver per item; fabrication outputs
   (Gerbers, drill, positions, board BOM) and a board STEP for the mechanical model.
5. **Calculations** (`elec_*.py`): currents and trace widths, dissipation and temperature rise of the hot
   parts, derating; mains creepage and clearance if any, with the standard's table as SOURCE. Simulate with
   ngspice only where it changes a decision, after one known-answer test.
6. **Cost**: every board part as a real line in `work/bom.d/electronics.yaml` (offline catalogue prices,
   basic-library parts when equal); PCB fabrication and assembly (area, layers, part count, price basis) in
   `work/other_costs.d/electronics.yaml`; run the cost roll-up.
7. **Firmware** (`07-firmware/`, `platformio.ini` at the root, platform pinned exactly as `INPUT/TOOLCHAIN.md`
   lists): the pin map generated from or checked against the schematic netlist; a state machine with explicit
   fault states and a safe default; watchdog; debounced inputs; constants checked against `work/params.yaml`;
   host tests in the `native` environment for the logic and every interlock (write the failing test first);
   build and host tests pass offline. Never upload.

Round 2 (fix list in `work/fixes/electronics-firmware.md`): fix, re-run ERC/DRC, build and tests, re-run the cost
roll-up.

# Role: mechanical-cad (full tier) — item prefix M

Read `method/roles/designer-common.md` first; it binds you. Handbook sections 7, 12 and 14.

**Mission**: the parametric 3D design of the product, built around the real purchased parts: their datasheet
envelopes first, the custom parts around them.

**Inputs**: `deliverables/02-concept/concept.md` (architecture, interfaces), `work/interfaces.md`,
`work/params.yaml`, `work/bom.yaml`, the dossiers `work/parts/*.json`, the board STEP from
electronics-firmware when it appears in `deliverables/06-electronics/`.

**You write only**: `deliverables/03-cad/**`, `deliverables/11-renders/assembly.glb`,
`deliverables/05-calculations/mech_*.py` (+ results), `work/bom.d/mechanical.yaml`,
`work/other_costs.d/mechanical.yaml`, `work/requests/mechanical-cad-*.md`, `work/status/mechanical-cad.md`.

## Method
1. **Purchased parts as envelopes**: one script (`03-cad/purchased.py`) builds a simplified solid per purchased
   part from its dossier (outer envelope, mounting holes, connector and keep-out zones), named
   `<item> <MPN> (envelope, datasheet)`. Dimensions only from the dossier; a missing one is an ASSUMPTION
   in the parameter block and a request to the lead for a dossier fix.
2. **Layout**: place the envelopes per `work/interfaces.md` (energy, material and signal flows, cable routes,
   user interface, service access), then design the custom parts around them (housing, chassis, brackets,
   covers): one script per part, parameters at the top, wall thickness and draft for the production process,
   bosses for standard fasteners or inserts (fastener library per `INPUT/TOOLCHAIN.md`).
3. **Assembly**: `03-cad/assembly.py` places every part with a name and writes `assembly.step`; every
   designed part also gets its own STEP; `parts.csv` (`part,file,material,density_kg_m3,qty`).
4. **Checks after every change**: the STEP checker on every STEP and the interference check on the assembly
   (zero unintended overlap; list intended fits); bounding box against the size requirements; clearances
   between hot or live parts and plastics or fingers; tool and finger access for assembly; total mass from
   densities and purchased-part masses against the mass requirement. Results into `03-cad/checks.md`.
5. **Process per part**: prototype (3D printing, CNC, laser-cut sheet) and 1000 units (injection moulding,
   sheet metal, die casting); note what changes between them in `03-cad/manufacturing.md`.
6. **Cost of custom parts**: `05-calculations/mech_part_costs.py` (material mass × price per kg + process
   time × rate at q1 and q100; moulded piece price at q1000; inputs labelled) → one BOM line per custom part in
   `work/bom.d/mechanical.yaml` (`origin: CALCULATION`, `notes` = the script path, `mpn` = your part number
   `PRJ-M01`…), tooling as one-off lines in `work/other_costs.d/mechanical.yaml`. Fasteners, inserts and
   seals as real catalogue parts. Then run the cost roll-up.
7. **GLB**: export `11-renders/assembly.glb` from `assembly.step` with the converter in `INPUT/TOOLCHAIN.md`
   and run the glTF validator on it.

Round 2 (the lead continues you with a fix list in `work/fixes/mechanical-cad.md`): fix, re-check, fit the
board STEP and connector cut-outs, re-export STEP and GLB, re-run the cost roll-up.

# BOM file read by method/gates/cost_rollup.py. Copy to work/arch/<A|B|C>/bom.yaml (concept options) or
# work/bom.yaml (the chosen design; lead). A designer's fragment (the work/bom.d/ file its role file names) has
# only `currency` and `lines`: never copy the fx block, the rates live only in the lead's file.
# One line per distinct part. Never type a price that is not in work/catalog/ or computed by a script.
# Item ids, unique across all BOM files of a roll-up: custom parts M.., board-level electronic parts E.., added
# safety parts S.., every other bought part L.. (the ids survive the full-tier split; drawings balloon them).
# null values stop the roll-up until they are filled; notes in <...> are placeholders the roll-up warns about.
currency: EUR                  # roll-up currency; every BOM file of one roll-up uses the same one
fx:                            # 1 unit of the listed currency = rate units of the roll-up currency; delete unused
  USD: {rate: null, origin: "<SOURCE url read YYYY-MM-DD | ASSUMPTION reason>"}
lines:
  # Normal form for a bought part: point to the catalogue candidate. The roll-up copies maker, MPN, supplier,
  # supplier part, URL, price breaks, currency, origin (SOURCE/ASSUMPTION) and read date from it.
  - {item: L01, function: F1-example, qty: 1, catalog: F1-example/F1-a}
  # Any field stated on the line overrides the catalogue (for example the supplier's break for qty x N parts):
  - item: L02
    function: F2-example       # function key from work/functions.md (budget tracking)
    qty: 4                     # per product
    catalog: F2-example/F2-b   # <work/catalog file stem>/<candidate id>; required for SOURCE prices
    price: {q1: null, q100: null, q1000: null}   # unit price when building 1 / 100 / 1000 products
    notes: "<why the override, e.g. break for 4000 parts used at q1000 (page shows 1/10/100/1000/5000)>"
  # A part not in the catalogue states everything itself:
  - item: M01
    function: F4-housing
    description: "Housing top, custom part (prototype printed, production moulded)"
    qty: 1
    manufacturer: "custom"
    mpn: "PRJ-M01"             # your own part number; drawings use it
    supplier: "custom"
    supplier_part: ""
    url: ""
    price: {q1: null, q100: null, q1000: null}   # piece price per process and quantity; tooling goes to other costs
    currency: EUR
    origin: CALCULATION        # SOURCE (read by this run) | ASSUMPTION (reason and range in notes) | CALCULATION
    date: ""                   # read date for SOURCE lines without a catalogue reference
    notes: "deliverables/02-concept/calc/custom_part_cost.py (material + process per quantity)"

# Catalogue file formats (written by parts scouts, checked by method/gates/catalog_check.py)

Every file is UTF-8 JSON plus a short human-readable `.md` twin with the same stem (a table and the failed
lookups). Write the JSON after the first finding and rewrite it after each further one. Values you did not
read are `null`, never guessed. Dates are `YYYY-MM-DD` (from `date +%F`).

## Function mode and value-engineering mode — `work/catalog/<KEY>.json`

```json
{
  "kind": "function",
  "function": "F3-example-function",
  "spec_summary": "one line: what the part must do (from the brief file)",
  "scout_date": "2026-01-31",
  "candidates": [
    {
      "id": "F3-a",
      "manufacturer": "Maker Ltd",
      "mpn": "MAKER-123",
      "description": "what it is",
      "principle": "working principle or integration level (e.g. 'certified module', 'discrete design')",
      "supplier": "Distributor or offline catalogue name",
      "supplier_part": "SKU or LCSC/JLC code C12345",
      "url": "https://product page you read",
      "datasheet": "https://datasheet you read, or null",
      "currency": "USD",
      "price": {"q1": 1.23, "q100": 0.98, "q1000": 0.81},
      "price_origin": "SOURCE",
      "price_date": "2026-01-31",
      "price_note": "breaks as shown (1/10/100/500/1000); for ASSUMPTION: the range and why",
      "stock": 12345,
      "stock_date": "2026-01-31",
      "lifecycle": "active",
      "certifications": ["only with evidence: e.g. 'FCC ID <id> (url)', 'CE DoC (url)'; else empty"],
      "key_specs": {"<spec name>": {"value": 3.0, "unit": "A", "source": "datasheet url, page or table"}},
      "envelope_mm": [20.0, 15.0, 3.2],
      "mass_g": null,
      "meets_spec": "yes",
      "spec_notes": "which Must items it meets or misses, with numbers",
      "second_source": "an equivalent part from another maker, or null",
      "risks": "single source, marketplace-only, long lead time, ...",
      "replaces": null,
      "saving_q1000": null
    }
  ],
  "failed_lookups": [{"what": "what you tried", "why": "403 / not found / no price shown"}],
  "notes": "anything the lead must know"
}
```

- `lifecycle`: active | nrnd | eol | obsolete | preliminary | unknown. `meets_spec`: yes | partly | no.
- `price_origin`: SOURCE only when you read the numbers on the page (or in the offline catalogue: then
  `price_note` names the catalogue and its data date). Otherwise ASSUMPTION with the range in `price_note`.
- Value-engineering mode: `"kind": "ve"`, `"function"` = the cost driver's item id; each candidate fills
  `replaces` (the item or mpn it replaces) and `saving_q1000` (per product, in the candidate's currency), and
  `spec_notes` states the consequences (performance, envelope, footprint, certification, supply).

## Benchmark mode — `work/catalog/benchmark.json`

```json
{"kind": "benchmark", "scout_date": "2026-01-31",
 "products": [{"name": "Product X", "maker": "Brand", "url": "https://...", "price": 199.0, "currency": "EUR",
               "price_incl_vat": true, "price_date": "2026-01-31", "market": "EU",
               "key_specs": {"<spec>": {"value": 1.0, "unit": "...", "source": "url"}},
               "weak_points": ["from reviews/returns, each with its url"], "positioning": "budget|mid|premium"}],
 "failed_lookups": []}
```

## Teardown mode — `work/catalog/teardown.json`

```json
{"kind": "teardown", "scout_date": "2026-01-31",
 "products": [{"name": "Product X", "url": "https://teardown, FCC internal photos, service manual or repair guide",
               "parts": [{"function": "what it does in the product", "part": "maker + MPN or module type as seen",
                          "evidence": "url + what shows it"}],
               "notes": "construction, materials, assembly method, cost clues"}],
 "failed_lookups": []}
```

## Dossier mode (full tier) — `work/parts/<item>.json`

One file per chosen purchased part, named by its BOM item id (a dossier task may list several parts), read
by the designers; every value carries the datasheet page or table it came from.

```json
{"item": "L03", "mpn": "MAKER-123", "datasheet": "https://...", "read_date": "2026-01-31",
 "envelope_mm": {"L": 20.0, "W": 15.0, "H": 3.2, "source": "datasheet p.12"},
 "mounting": [{"feature": "hole d3.2", "x_mm": 2.5, "y_mm": 2.5, "source": "p.12"}],
 "keep_out": [{"what": "antenna clearance", "zone_mm": "...", "source": "p.14"}],
 "connectors": [{"name": "J1", "type": "...", "position_mm": [0, 0, 0], "pins": 4, "source": "p.13"}],
 "pinout": [{"pin": "1", "name": "VIN", "function": "...", "limits": "...", "source": "p.5"}],
 "electrical": {"<rating>": {"value": 0, "unit": "V", "source": "p.3"}},
 "thermal": {"<limit>": {"value": 0, "unit": "°C", "source": "p.4"}},
 "footprint": {"kicad_library": "Library:Footprint or null", "matches_datasheet": "yes|no|unchecked"},
 "certifications": [], "notes": ""}
```

# Non-BOM costs read by method/gates/cost_rollup.py. Copy to work/arch/<A|B|C>/other_costs.yaml (concept
# options) or work/other_costs.yaml (chosen design; lead); a designer's fragment (the work/other_costs.d/ file
# its role file names) holds only the lines it owns. Every category below needs a line in one of the files of a
# roll-up; a category that truly does not apply gets 0 with the reason in notes. Ignoring one-off costs is the
# classic under-costing failure: null values stop the roll-up until they are filled, and a 0 whose notes are
# still a <...> placeholder is warned about.
# Categories: pcb, mechanical, tooling, assembly_test, packaging, certification, freight, contingency.
# Each line has exactly one of: per_unit {q1, q100, q1000} | one_off + applies | percent.
currency: EUR
lines:
  - name: pcb_fab_and_assembly
    category: pcb
    per_unit: {q1: null, q100: null, q1000: null}
    origin: ASSUMPTION
    notes: "<board size, layers, part count; fab and assembly price basis and date, or the reason for the estimate>"
  - name: mechanical_parts_not_in_bom
    category: mechanical
    per_unit: {q1: null, q100: null, q1000: null}
    origin: ASSUMPTION
    notes: "<0 when every custom part is a BOM line (M..); else process per quantity>"
  - name: mould_tooling
    category: tooling
    one_off: null              # amortised as one_off / N over the quantities in applies
    applies: [q1000]           # tooling only for the moulded production run
    origin: ASSUMPTION
    notes: "<number of tools, cavities, source of the price range>"
  - name: assembly_and_test_labour
    category: assembly_test
    per_unit: {q1: null, q100: null, q1000: null}
    origin: CALCULATION
    notes: "<minutes per unit x labour rate; script path>"
  - name: packaging
    category: packaging
    per_unit: {q1: null, q100: null, q1000: null}
    origin: ASSUMPTION
    notes: "<box, insert, manual, labels; price basis>"
  - name: certification_and_lab_tests
    category: certification
    one_off: null
    applies: [q100, q1000]     # a prototype (q1) is not certified
    origin: ASSUMPTION
    notes: "<tests from the compliance route; what pre-certified modules remove>"
  - name: freight_to_fulfilment
    category: freight
    per_unit: {q1: null, q100: null, q1000: null}
    origin: ASSUMPTION
    notes: "<packed volume and mass, mode of transport, price basis>"
  - name: contingency
    category: contingency
    percent: 15                # % of BOM + all other lines; POLICY 6: 15 % at the concept tier, 10 % at the full tier
    origin: ASSUMPTION
    notes: "POLICY 6 default"

# work/price_frame.yaml - the price frame (method/POLICY.md section 2). Written by the lead in C1 (provisional:
# an ASSUMPTION retail if the brief gives none), completed when the benchmark arrives, frozen from T0+25 min unless
# the benchmark evidence changes (record why in price-frame.md). cost_rollup.py computes retail excl. VAT and the
# allowed unit costs; print the table with: python3 method/gates/cost_rollup.py --frame-only
currency: EUR
market: "<where it is sold first (from the brief)>"
benchmark:                     # from work/catalog/benchmark.json (comparable products, prices incl. VAT)
  n_products: null
  retail_min: null
  retail_median: null
  retail_max: null
  origin: "SOURCE work/catalog/benchmark.json (prices read YYYY-MM-DD)"
target_retail_incl_vat: null   # null or 0 = not set yet (then there is no allowed cost)
target_retail_origin: "<brief or INPUT/requirements.md, quoted | ASSUMPTION: positioning against the benchmark and why>"
target_retail_excl_vat: null   # only if the brief states the price excl. VAT; else computed from incl. VAT and VAT
vat_percent: 21                # VAT of the first market; POLICY default 21 when the brief names none
divisors: {q100: 2, q1000: 3}  # POLICY 2.4: allowed landed unit cost = retail excl. VAT / divisor
allowed_unit_cost:             # only when the brief or INPUT/requirements.md states a unit-cost limit (POLICY 2.4):
  q1000: null                  # the allowed cost is then the lower of this limit and retail excl. VAT / divisor;
  origin: "<requirement id and its words>"   # a limit that names no quantity applies at q1000 (add q100 if it names it)
non_bom_share: 0.35            # POLICY default share of non-BOM costs until the roll-up shows the real share
function_budgets_q1000:        # BOM budget per function at q1000: the % shares of work/functions.md x the BOM budget
  F1-example: null             # that --frame-only prints; refined by teardown evidence or first catalogue prices

# Scouting brief of one function = one section of `work/functions.md` (copy the section once per function)

The scout's prompt carries the mode, the output files and the deadline (function mode: `work/catalog/<KEY>.json`
+ `.md`; dossier mode: `work/parts/<item>.json` + `.md`, one pair per item listed; value-engineering mode:
`work/catalog/ve-<item>.json` + `.md`). The scouts apply the preferences of `method/POLICY.md` section 4 on
their own; write here only what is specific to the function.

## F<n>-<short-name>: <function as verb + noun>

- Flows: energy, material and signal in and out.
- Requirement ids served (spelled exactly as in `INPUT/requirements.md`):
- Must (thresholds; a candidate that misses one is `meets_spec: "no"`): <quantity> <≥|≤> <value> <unit>, ...
- Want (targets; better is worth money up to the budget): ...
- Interfaces: supply voltage and current, signals and protocols, size envelope and mounting, environment
  (temperature, water and dust, vibration), safety (insulation, flammability rating, certification).
- Principles or integration levels to cover (at least one candidate each, if it exists): <P1>, <P2>, ...
- Quantity per product: <n>
- Budget: <s> % of the BOM budget (about <x> <currency> at q1000 with the current, possibly provisional frame).
  Indicative, not a knock-out: report the cheapest part that meets every Must item even if it is over budget,
  and at least one better part.
- Leads (optional): part families, makers, teardown findings or search terms worth trying first.

---
name: dd-hw-handbook
description: Product-agnostic quality bars and methods for every deliverable of a dd hardware design run (requirements, concept, calculations, CAD, drawings, electronics, firmware, safety and certification, BOM and cost, prototype and test plan, renders, report). Read it before producing any deliverable. It says WHAT GOOD LOOKS LIKE; your main prompt says who does the work and in which order.
---

# dd hardware design handbook

Your main prompt defines the team, the roles, the communication and the order of work. This handbook defines
the quality bar of each deliverable and the methods that reach it. Where your main prompt is more specific,
follow your main prompt. Where the RUN CONTRACT fixes a file name or a format, the RUN CONTRACT wins.
Exact tool commands, versions and paths are in `INPUT/TOOLCHAIN.md` (read the working examples it points to
before writing your own scripts); this handbook names tools only by purpose.

Contents: 1 inputs · 2 requirements · 3 research · 4 physics map · 5 concept · 6 calculations · 7 CAD ·
8 drawings · 9 electronics · 10 firmware · 11 safety and certification · 12 sourcing, BOM and cost ·
13 prototype and test plan · 14 renders · 15 report and manifest · 16 claims · 17 time and context.

---

## 1. Reading the inputs

- `INPUT/brief.md` — one sentence plus customer context. Extract: who buys it, where and how it is used, how
  often and how long, the price expectation, the one or two things that must be better than what exists.
- `INPUT/requirements.md` — the acceptance criteria. Each item has an id, a metric, a target and usually a
  threshold (worst acceptable), a verification method and a weight. **Every id must be addressed and keep its
  exact spelling** wherever you mention it. Weights tell you what matters most when requirements conflict.
- `INPUT/TOOLCHAIN.md` — what is installed and how to call it headless. Use only what it lists. A tool that
  misbehaves gets three honest attempts with new information each time, then a labelled fallback (for example
  an analytical model instead of a simulation) and a note in `manifest.json → blocked[]` if it mattered.

## 2. Requirements — `deliverables/01-requirements/`

Quality bar: an engineer who has never seen the brief can design and verify the product from this folder
alone, and every number in it has an origin.

Write `requirements.md` (and, if you like, a machine-readable `requirements.yaml`) with three parts:

1. **Given requirements**, one entry per id from `INPUT/requirements.md`: the id, the original statement, your
   interpretation (the exact condition under which it will be verified: temperature, voltage, distance,
   duty cycle, test method), and the design consequence (what it forces in the design).
2. **Derived requirements**, ids `DR-<AREA>-<nn>`, each with: parent id(s) or "brief"/"regulation"/
   "manufacturing", statement, metric, target, threshold, verification method (Analysis, Inspection,
   Demonstration, Test), rationale with origin label, priority. Walk at least these areas and write down
   either the requirement or why the area does not apply: function and performance under all operating
   conditions; user interface and ergonomics; power source and consumption (including standby); mechanical
   interfaces and mounting; environment (temperature, humidity, dust and water, altitude, transport and
   storage); safety (electric shock, heat and burns, fire, mechanical hazards, chemical, optical and other
   radiation, noise, misuse that can be foreseen); EMC and, if any radio, radio; materials and restricted
   substances; lifetime, duty cycle and reliability; maintenance, cleaning, consumables and repair;
   manufacturing process at 1, 100 and 1000 units; assembly and test in production; packaging, shipping and
   drop; labelling, manuals and markings; unit cost and retail price; prototype schedule.
3. **Conflicts and decisions**: requirement pairs that pull against each other, how you resolved each (use the
   weights; safety first), and the assumptions register (each ASSUMPTION with reason and sensitivity).

Write `compliance.csv` exactly as the RUN CONTRACT specifies, one row per id of `INPUT/requirements.md`:
- `status`: `met`, `partly`, `not_met` or `not_addressed`. At the concept tier the status is a prediction
  from your estimates: say so in `note` (for example "predicted, CALCULATION"). Be honest: a predicted miss
  is `not_met` with the reason, not a hidden problem.
- `value` and `unit`: the predicted or designed value in the requirement's own metric and unit.
- `evidence`: path(s) inside `deliverables/` that show it (a calculation result, a section of the concept).

## 3. Research — benchmark, users, prior art

- Find at least five comparable products (commercial and open-source; adjacent categories count). For each:
  the specifications that matter for our requirements, price with date and URL, and weak points from
  reviews, teardowns, forums or return reasons. Put the table in `02-concept/benchmark.md`.
- From the benchmark derive: realistic target values, the differentiator ("our own, better version" — what we
  do better and why it is feasible), and the failure modes users actually complain about. Feed both back into
  the derived requirements.
- Note patents only as prior art you noticed; never give legal opinions. If an idea of yours looks genuinely
  new, flag it as `novelty` in `REPORT.md` (one line) — do not elaborate on it in public-facing text.
- Read pages, do not guess them. If a site blocks automated reading, use a different source.

## 4. Physics map — which disciplines decide this product

Rate each domain CRITICAL, RELEVANT or MINOR for this product: structure and strength; heat; fluids and
aerosols; electrical power; electronics and control; firmware; mechanisms and motion; materials and
chemistry; optics and light; acoustics; human factors; environment. For each CRITICAL domain write: the
governing relations, the few parameters that dominate the result, the order-of-magnitude numbers, and the
ways it typically fails (from first principles and from what users report). The map decides which analyses
you run and how deep; it lives in `02-concept/physics-map.md`.

## 5. Concept — `deliverables/02-concept/`

Quality bar: the decision is visibly earned. A reader sees real alternatives, numbers for each, a fair
comparison, and a chosen architecture concrete enough to start detailed design tomorrow.

Contents (file names are suggestions; keep them inside `02-concept/`):
- `concept.md` — the main document, in this order:
  1. The product in one paragraph: what it is, for whom, what it does better.
  2. Functions and flows: the main functions as verb–noun pairs and the energy, material and signal flows.
  3. Alternatives: at least three that differ in **working principle or architecture**, not in cosmetics.
     For each: how it works, a simple diagram, first-order numbers against the top-weighted requirements
     (performance, power, size, mass, unit cost), the main risks and hazards.
  4. Evaluation: knock-out by thresholds first; then a weighted comparison whose weights come from the
     requirement weights; a sensitivity check (vary each weight ±20 %, and the most uncertain estimates
     within their range) showing whether the winner holds. Record why each loser lost.
  5. The chosen concept: architecture block diagram (Mermaid in the markdown and a rendered PNG or SVG if
     the toolchain has a renderer), the subsystem list with make-or-buy, the critical components with real
     candidate parts (manufacturer, part number, supplier link, price with date), the interfaces between
     subsystems (mechanical, electrical, thermal, data).
  6. Key estimates: every number the decision rests on, each with its origin; the scripts that compute them.
  7. Safety concept: the top hazards, the design measures (inherently safe design first, then protective
     devices, then information for the user), and which protections act **independently of firmware**.
  8. Compliance route: which EU directives and harmonised standards, and which FCC rules, apply and why
     (with SOURCE), the likely conformity route, rough cost and time.
  9. Cost: a rough BOM of the major items, the unit cost at 1, 100 and 1000 units with the method, and the
     retail price logic (section 12). Optionally `09-bom-cost/cost.json` already at this tier.
  10. Risks: the top risks with likelihood, impact, mitigation and the analysis or test that would retire
      each one.
  11. Decisions taken without a human (taste, priorities, trade-offs) with the rejected alternatives.
  12. Open issues and what detailed design must do first.
- `calc/` — the scripts behind the key estimates (same rules as section 6), with their results files.
- `benchmark.md`, `physics-map.md`, and anything your method produces (scores, votes, issue logs, models).

Concept anti-patterns: two strawmen next to the favourite; alternatives that differ only in colour or
material; scores without numbers behind them; a decision matrix whose weights were tuned until the favourite
won; a concept that silently misses a top-weighted requirement; prose without a single calculated number.

## 6. Calculations — `05-calculations/` (full tier) and `02-concept/calc/` (concept tier)

- One Python script per question, run with the toolchain's Python (`ddpy`, see `INPUT/TOOLCHAIN.md`; the
  system `python3` has no scientific libraries). Physical quantities carry units (the `pint` library) and every
  input is labelled at the top of the script with its origin.
- Scripts run with `ddpy <script>` from their own folder, offline, in under five minutes, and write
  `<stem>.results.json` (`{"<name>": {"value": x, "unit": "..."}}`). The RUN CONTRACT requires this for
  `05-calculations/`; use the same convention in `02-concept/calc/`.
- Order of work: hand estimate → script → (only if it changes a decision) simulation. For any simulation:
  first a known-answer case, then the real case on at least three mesh or step sizes with the trend shown,
  sanity checks (balances of mass, energy or force), and a sensitivity run on the least certain inputs.
- Static strength is not drop, impact or fatigue; steady thermal is not transient; say which remain open.
- Margins: state each result as value, requirement, margin, and status.

## 7. CAD — `03-cad/` (full tier)

- Parametric code-CAD (CadQuery or build123d, whichever `INPUT/TOOLCHAIN.md` lists as working), one script
  per designed part, parameters at the top, an assembly script placing parts with names. Re-running the
  scripts reproduces every STEP. FreeCAD files are acceptable if that is your route.
- Deliver `assembly.step` and one STEP per designed part, plus the source. Purchased parts appear as
  simplified bodies with their real envelope dimensions from the datasheet (label them).
- Check every part: valid solid, volume and mass from density, bounding box against size requirements,
  pairwise interference zero (or an intended fit), clearances for assembly tools and fingers, wall thickness
  and draft suited to the chosen process, standard fasteners.
- Choose a manufacturing process per part for the prototype (3D printing, CNC, laser-cut sheet) and for
  1000 units (injection moulding, die cutting, sheet-metal bending), and note what changes between them.
- Export a GLB of the assembly for the renders and the website (see `INPUT/TOOLCHAIN.md` for the converter).

## 8. Drawings — `04-drawings/` (full tier)

- A drawing for every manufactured part: views, overall and functional dimensions, tolerances (a general
  tolerance class plus explicit tolerances on functional features), material, finish, quantity, part number
  and revision matching the BOM. Flat patterns for sheet metal.
- An assembly drawing with item balloons that match `bom.csv` item numbers. PDF or SVG; DXF optional.
- Open each output after you write it (render a PNG preview and look at it, or parse it) — a drawing file
  that does not open does not exist.

## 9. Electronics — `06-electronics/` (full tier)

- Start from a block diagram and a power budget over the full supply range and all operating modes.
- KiCad 9 project with schematic and, where custom electronics is needed, a routed board. Real parts with
  matching symbols and footprints; every part has a manufacturer part number that exists at a distributor or in
  the offline LCSC/JLC catalogue described in `INPUT/TOOLCHAIN.md`.
- Protection where it applies: input fuse or current limit, reverse polarity, transients and ESD on external
  connectors, inrush, thermal shutdown of hot parts in hardware, safe state on power loss.
- Calculations: currents, trace widths, dissipation and temperature rise of the hot parts, derating, and for
  any mains part the creepage and clearance with the standard you took the values from.
- Run ERC and DRC from the command line; deliver the reports; fix every error or justify each waiver.
  Fabrication outputs: Gerbers, drill, pick-and-place and the board BOM. Passing DRC is not a hardware test.
- Simulate uncertain circuit behaviour with ngspice where it changes a decision (start-up, protection trip,
  filter response) after one known-answer test of the simulator.

## 10. Firmware — `07-firmware/` (full tier)

- PlatformIO project at the folder root that builds offline for the real target MCU of the schematic; use
  exactly the platform versions `INPUT/TOOLCHAIN.md` pins (anything else downloads packages and blocks
  other runs).
- A state machine with explicit fault states and a safe default; a watchdog; inputs debounced; every
  safety-relevant reaction has a hardware backstop and the firmware is never the only protection.
- Host tests (`native` environment) for the logic and the interlocks: write a test that fails first, then
  passes. Build with warnings as errors where the framework allows. Never flash or power anything.
- Pin map agreed with the schematic; configuration constants generated from or checked against the design
  parameters.

## 11. Safety and certification — `08-safety-certification/` (both tiers: summary in 02; full tier: complete)

- Hazard analysis in the ISO 12100 manner: hazard → hazardous situation → harm → risk estimate (severity ×
  probability) → reduction in the order inherently safe design, protective measures, information for use →
  residual risk. Derive safety requirements with ids and trace them to the design.
- Regulatory map with SOURCE for each item. For the EU think through at least: the Low Voltage Directive
  (2014/35/EU) when any voltage is in its range, the EMC Directive (2014/30/EU), the Radio Equipment Directive
  (2014/53/EU) when there is any radio, RoHS (2011/65/EU), WEEE (2012/19/EU), the batteries rules when there
  is a battery, the General Product Safety Regulation (EU) 2023/988, REACH, ecodesign and standby rules.
  For the US: FCC Part 15 (Subpart B for unintentional radiators, Subpart C for intentional ones) and the
  voluntary safety listings retailers expect. Then find the harmonised product-family standards that
  apply to THIS product by research, and quote which clauses drive the design.
- Prefer architectures that make conformity easy: low voltage inside the product through a certified external
  supply when the power level allows it, pre-certified radio modules, protections that do not depend on
  firmware, materials with suitable flammability ratings near heat and live parts.
- State the conformity route (self-declaration, notified body or not), the tests an accredited lab would run,
  a pre-compliance plan, and a rough cost and duration. You identify; you do not certify.

## 12. Sourcing, BOM and cost — `09-bom-cost/`

- Real, purchasable parts only. For each line: manufacturer, part number, supplier, supplier part number, URL,
  unit price at 1, 100 and 1000 (price breaks as shown by the supplier, currency stated), the date you read it
  in `notes`. A price you could not read is an ASSUMPTION with a range, never a guess dressed up as SOURCE.
  Prefer parts in stock at two or more suppliers, active (not end-of-life), and note a second source for
  critical parts. Use the offline catalogue from `INPUT/TOOLCHAIN.md` first, then distributor pages
  read-only.
- `bom.csv` with exactly the header the RUN CONTRACT gives; `cost.json` and `cost.md` as it specifies.
- The unit cost at each quantity is more than the BOM: PCB fabrication and assembly, mechanical parts (at 1
  and 100 units printed or machined; at 1000 units moulded with the tooling cost amortised and stated
  separately), assembly and test labour, packaging, certification and other one-off costs amortised over
  the quantity (state the assumption), freight to the fulfilment point, and a contingency. Show each line.
- Retail price logic: start from benchmark prices and a target gross margin; check that the price leaves room
  for crowdfunding platform and payment fees, shipping to backers, VAT where it applies, returns and warranty.
  An underpriced product that cannot be delivered is the classic crowdfunding failure; say plainly if the
  numbers do not close.

## 13. Prototype and test plan — `10-prototype-test-plan/` (full tier)

- Build plan for prototype 1: how each part is made, the order of assembly, tools, and the checks after each
  step.
- Bring-up for a person later: first power-up with a current-limited or isolated supply, what to measure
  before applying full power, safety precautions.
- One test per requirement (and per safety requirement): method, equipment, conditions, pass criterion, the
  requirement id it closes. Include pre-compliance EMC and the safety tests a lab would run, and durability
  or life tests where lifetime is a requirement.

## 14. Renders — `11-renders/` (full tier)

- Renders come from the real CAD geometry (the assembly GLB) through Blender in background mode: a product
  view, an exploded view, and a view that shows the user interface. Check the scale against a known
  dimension. Label them as renders of a digital design.
- Never use generative image models to depict the product: they invent geometry that does not exist.
- You cannot judge images reliably by looking. Check what can be measured (file exists, resolution, the
  object fills the frame, parts are present by name) instead of trusting a glance.

## 15. Report and manifest

`REPORT.md` (both tiers) in this order: the product in one paragraph; a key-parameter table (value, unit,
origin label); unit cost at 1/100/1000 and the retail price; status per requirement (from `compliance.csv`);
the top risks and open hazards; decisions taken without a human with their alternatives; what is open,
blocked or not run; for the full tier, the file map. Every number in it must match its home.

`manifest.json` exactly as the RUN CONTRACT specifies, including `key_figures` (use `null` for an unknown
figure, never a guess) and `blocked[]`. List every file under `deliverables/`.

Before you finish: re-read `REPORT.md` and `02-concept/concept.md` against the numbers' homes, check that
`compliance.csv` has one row per requirement id, that every evidence path exists, and that `manifest.json`
parses and lists the files that exist.

## 16. Claims

The project designs real products for honest crowdfunding. In every document: no feature that the design
does not implement, no performance number that is not calculated or sourced, no "certified", "safe",
"waterproof", "AI" or similar word unless the evidence for it is in the deliverables. Under-promise.

## 17. Time and context

- The RUN CONTRACT gives the time target and the hard stop. Plan time boxes per phase at the start, check
  `date` at each boundary, and cut scope (depth, not honesty) when behind.
- Long runs lose context. Keep a short state file (decisions, key numbers' homes, open issues, next step)
  current, so you or a fresh agent can resume from files alone.
- Write deliverables as soon as each part exists and keep them current.