nixmac E2E Report - evolve_review_full_mac_journey
1 passed
0 assertion failed
0 infra/not-run
1/1 selected scenarios produced reports
Evolve review real-Mac journey
evolve_review_full_mac_journey
full-mac on macos-e2e (full-mac)
Real-Mac companion - Attached to Auto-evolve basic package, Discard and restore state, Manual evolve existing changes +2 more
- Commit
33f37e696bb5d5f1217b458e722d756ffb3d1c64- Duration
- 184s
- Replay
tests/e2e/run.sh evolve_review_full_mac_journey
Full-Mac lane: real macOS desktop automation with full-screen recording evidence.
What this checks
Runs the shipped app on a real Mac against a local mock vLLM server, records prompt submission, review, follow-up, question-answer, and discard behavior, and verifies generated git diffs.
Coverage
- Prompt suggestions and Send are usable in the released app
- Mock vLLM response reaches evolve review with a JetBrains Mono diff
- Follow-up prompt preserves the first diff and adds Fira Code
- Discard cancel keeps the review state visible
- Discard confirm returns to the prompt state
- Inline question-answer continues to evolve review
- Reads git diff from a temporary nix-darwin repo on the Mac runner
- Publishes screenshots and a full-screen recording as adjacent real-desktop proof
Known gaps / not covered
- Uses a local mock provider; live provider behavior remains covered by the separate live OpenRouter scenario.
- Does not apply or rebuild the generated nix-darwin configuration.
- Release-app localhost mock reachability is verified only when the full-Mac job actually runs on the Mac runner.
View additional proof artifacts (6)
| Phase | Status | Duration | Summary |
|---|---|---|---|
| Nix installed (already present) | passed | 0s | |
| Provider-guarded state seeded | passed | 0s | |
| Prompt/provider guardrail verified | passed | 0s | |
| Secondary prompt surfaces verified | passed | 0s |