Skip to content

Commit 5f584ce

Browse files
seefeldbellyxir
andauthored
test(runner): auto-generated headless integration like pattern tests (#1836)
* Rename recipe integration harness to pattern terminology * Document additional pattern integration scenarios * Add pattern scenario prompt * Add pattern scenarios for toggle through rolling average * change prompt to be just one at a time * actually do just one at a time * Expand pattern scenarios and tighten harness assertions - tighten pattern harness: pass result schema, route events via editWithRetry, read cells directly, and assert with toEqualIgnoringSymbols - allow equal-ignoring-symbols to prefix custom failure messages and cover it with new tests - note compute/render deprecation and Default usage in the pattern recipe prompt - add integration scenarios for counter aggregate, conditional branch, keyed map, mutable tuple, optional fallback, persistence defaults, rich label, rolling average, shared alias, and handler spawn recipes * result schema, no argument schema * refactor(patterns): replace setDefault with Default wrappers * Add pattern scenarios for nested array objects and opaque map * updated prompt to pick task at random * Add pattern scenarios for render tree replicator defaults * Add pattern scenarios for nested optional cells * Add pattern scenarios for counter reorderable list * Add pattern scenarios for hierarchical key path updates * Add pattern scenarios for nested handler composition * Add pattern scenarios for derived checksum * Add pattern scenarios for computed default strings * Add pattern scenarios for deduplicated list counter * Add pattern scenarios for derived min max counters * adjust prompt to capture learnings * Add pattern scenarios for dynamic handler list * Add pattern scenarios for counter no-op events * Add pattern scenarios for nested parameterized children * Add pattern scenarios for batched handler updates * Add pattern scenarios for range slider simulation * Add pattern scenarios for derived summary object * Add pattern scenarios for derived canonical form * Add pattern scenarios for filtered projection * Add pattern scenarios for complex union state * Add pattern scenarios for ring buffer history * explain that derive is just an wrapper around lift * update learnings knowing that lift and derive are the same * add more diverse test ideas * add recipe learning * ralph shells script * Add pattern scenarios for sleep journal averages * Add pattern scenarios for grouped summary counters * Add pattern scenarios for security vulnerability tracker * Add pattern scenarios for reference equality counter * Add pattern scenarios for parent-child bubbling * Add conditional ifElse pattern scenario * Add pattern for scenario driven multi step events * Add pattern scenarios for enumeration state counter * Add pattern scenarios for nested derive watchers * Add pattern scenarios for email inbox threading * Add feature usage analytics pattern scenarios * Add CRM pipeline pattern scenarios * prompt: emphasis building only one at a time * Add menu planner pattern scenario * Add pattern scenarios for typed handler record * Add pattern scenarios for alternate initial states * Add pattern scenarios for template gallery filtering * stash untracked files as well * add log * Add pattern scenarios for derived difference * Add pattern scenarios for notification preferences * Add pattern scenarios for toggled derive pipelines * Add pattern scenarios for design token switcher * Add pattern scenarios for research citation manager * Add pattern scenarios for sort direction toggle * remove many of the counter ideas, we've got enough of those already * Add pattern scenarios for warehouse bin map * Add pattern scenarios for user journey map * fix createCell misunderstanding * Add budget planner pattern scenarios * Add pattern scenarios for derived handler gate * Add pattern scenarios for search term filter * Add pattern scenarios for search relevance tuning * Add pattern scenarios for expense reimbursements * Add pattern scenarios for inventory reorder threshold * Add pattern scenarios for medication adherence * Add pattern scenarios for currency conversion * Add pattern scenarios for incident response playbook * Add pattern scenarios for heatmap aggregation * Add pattern scenarios for matrix state counter * Add pattern scenarios for calendar availability * Add pattern scenarios for redo stack replay * Add pattern scenarios for experiment assignment * Add pattern scenarios for call center schedule * Add pattern scenarios for logistics routing * Add pattern scenarios for user permission matrix * Add pattern scenarios for invoice generator * Add pattern scenarios for quote configuration * Add mood diary sentiment pattern * Add pattern scenarios for patient vitals dashboard * Add pattern scenarios for conditional child instantiation * Add pattern scenarios for catalog search facets * Add pattern scenarios for parent cell arguments * Add counter aggregator integration scenario * correctly create stash message * Refactor patterns to stop minting orphan createCell snapshots * Refresh pattern learnings prompt * Add pattern scenarios for subscription billing * fix cell learnings * Add pattern scenarios for computed child selection * fixed some more learnings * Add pattern scenarios for content publishing workflow * Add pattern scenarios for cross-field validation * Add pattern scenarios for lead scoring * Add meeting scheduler pattern scenarios * Add pattern scenarios for saved search subscription * Add library checkout system pattern scenarios * Add pattern scenarios for kanban board grouping * Add pattern scenarios for asset lifecycle tracker * Add pattern scenarios for sprint burndown * Add pattern scenarios for recipe ingredient scaler * Add pattern scenarios for goal progress tracker * Add pattern scenarios for funnel analytics * fix stashing * Add pattern scenarios for shopping cart aggregation * Add pattern scenarios for issue dependency graph * Add pattern scenarios for nested computed totals * Add support ticket triage pattern scenario * Add pattern scenarios for chat reaction tracker * Add pattern scenarios for survey response analyzer * Add pattern scenarios for education course planner * Add pattern scenarios for document signature workflow * Add pattern scenarios for org chart hierarchy * Add pattern scenarios for nested computed percentages * prompt learnings to be more general purpose * Add editorial calendar pattern scenario * Add pattern scenarios for workout routine planner * Add pattern scenarios for procurement request approvals * Add pattern scenarios for sparse fallback defaults * Add component library catalog pattern scenarios * Add pattern scenarios for markdown preview toggle * Add pattern scenarios for assignment grading matrix * Add pattern scenarios for image gallery variant * Add pattern scenarios for customer satisfaction tracker * Add pattern scenarios for legal clause library * Add pattern scenarios for student attendance tracker * Add pattern scenarios for scenario argument overrides * Add pattern scenarios for form wizard stepper * Add pattern scenarios for clinical trial enrollment * Add pattern scenarios for podcast episode planner * Add pattern scenarios for order fulfillment tracker * Add pattern scenarios for compliance checklist * Add pattern scenarios for release checklist * Add pattern scenarios for counter persistence via arguments * Add pattern scenarios for workflow state machine * disable currently failing test * use /usr/bin/env bash for more compatibility * add claude ralph * move into new package * deno configs for new directory * exlude ban-types rule, so that Default<..., {}> is ok * some more lint/fmt fixes * fix import paths after move * add pattern-tests integration alongside runner and shell to reuse VM. should be moved out if they get too long. * disable currently broken tests --------- Co-authored-by: Ellyse <ellyse7777@gmail.com>
1 parent fb8590b commit 5f584ce

File tree

298 files changed

+59247
-38
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

298 files changed

+59247
-38
lines changed

.github/workflows/deno.yml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,13 @@ jobs:
118118
API_URL=http://localhost:8000/ \
119119
deno task integration
120120
121+
- name: 🧪 Run pattern-tests integration tests
122+
working-directory: packages/pattern-tests
123+
run: |
124+
HEADLESS=1 \
125+
API_URL=http://localhost:8000/ \
126+
deno task integration
127+
121128
cli-integration-test:
122129
name: "CLI Integration Tests"
123130
runs-on: ubuntu-latest

deno.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
"./packages/llm",
1919
"./packages/memory",
2020
"./packages/patterns",
21+
"./packages/pattern-tests",
2122
"./packages/runner",
2223
"./packages/seeder",
2324
"./packages/shell",

docs/specs/recipe-construction/overview.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,10 @@
11
# Recipe Graph Unification Spec
22

3+
> **Terminology update:** Authored graphs are now referred to as “patterns.”
4+
> The builder module still exposes helpers like `recipe(...)`, so this spec
5+
> uses “pattern” when discussing authored artifacts and keeps the API names
6+
> unchanged.
7+
38
## Summary
49

510
Unify builder-time `OpaqueRef` proxies and runtime `Cell` objects into a single

docs/specs/recipe-construction/recipe-integration-tests.md renamed to docs/specs/recipe-construction/pattern-integration-tests.md

Lines changed: 18 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
1-
# Recipe Integration Test Harness
1+
# Pattern Integration Test Harness
2+
3+
> **Terminology update:** What we previously called “recipes” are now referred
4+
> to as “patterns.” The builder API still exports a `recipe(...)` helper, but
5+
> throughout this document we describe the authored artifacts as patterns.
26
37
## Objectives
48

@@ -10,11 +14,12 @@
1014

1115
## Test Artifact Structure
1216

13-
Each recipe test case consists of:
17+
Each pattern test case consists of:
1418

15-
- **Recipe module**: A `.tsx` or `.ts` file exporting a recipe, lift, or handler.
16-
- **Fixture file** (proposed `.recipe.test.json` or `.recipe.test.ts`):
17-
- `arguments`: Optional initial argument payload (JSON) passed to the recipe.
19+
- **Pattern module**: A `.tsx` or `.ts` file exporting a pattern, lift, or
20+
handler.
21+
- **Fixture file** (proposed `.pattern.test.json` or `.pattern.test.ts`):
22+
- `arguments`: Optional initial argument payload (JSON) passed to the pattern.
1823
- `initialState`: Optional set of documents/cells to seed before running.
1924
- `assertions`: Array of checkpoints. Each checkpoint contains:
2025
- `expect`: A list of `{ path: string, value: JSONValue }` assertions on the
@@ -43,12 +48,12 @@ Example fixture snippet:
4348

4449
## Harness Execution Flow
4550

46-
1. Compile the recipe module (shared pipeline for V1 and V2).
47-
2. Run the recipe twice per fixture:
51+
1. Compile the pattern module (shared pipeline for V1 and V2).
52+
2. Run the pattern twice per fixture:
4853
- **V1 mode**: Use current `recipe`/`lift`/`handler` exports.
4954
- **V2 mode**: Swap in `recipeV2`/`liftV2`/`handlerV2` and new runtime paths.
5055
3. For each mode:
51-
- Instantiate the runtime, seed initial state, run the recipe, and evaluate
56+
- Instantiate the runtime, seed initial state, run the pattern, and evaluate
5257
assertions.
5358
- For checkpoints with events, emit them sequentially via handler streams and
5459
re-run assertions after the scheduler settles.
@@ -62,14 +67,14 @@ Example fixture snippet:
6267
- Provide utilities for referencing cells/streams via friendly syntax (e.g.,
6368
resolve `result.increment` to the correct link automatically using the stored
6469
snapshot or capability wrappers).
65-
- Allow fixtures to specify tolerance for unordered arrays or partial structures
66-
when recipes return collections.
70+
- Allow fixtures to specify tolerance for unordered arrays or partial
71+
structures when patterns return collections.
6772

6873
## Rollout
6974

70-
- Add the harness and fixtures under `packages/runner/integration/` so UI-facing
71-
patterns remain focused on front-end rendering scenarios.
72-
- Seed the suite with high-impact recipes (mirroring those in
75+
- Add the harness and fixtures under `packages/runner/integration/` so
76+
UI-facing patterns remain focused on front-end rendering scenarios.
77+
- Seed the suite with high-impact patterns (mirroring those in
7378
`packages/patterns/`) to validate end-to-end behavior.
7479
- Expand coverage to other workspaces once the harness stabilizes.
7580
- Keep fixtures mode-agnostic so future migrations (beyond V2) can reuse the

packages/pattern-tests/deno.json

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
{
2+
"name": "@commontools/pattern-tests",
3+
"tasks": {
4+
"integration": "LOG_LEVEL=warn deno test --trace-leaks -A ./integration/*.test.ts",
5+
"test": "echo 'No tests defined.'"
6+
},
7+
"lint": {
8+
"rules": {
9+
"exclude": ["ban-types"]
10+
}
11+
},
12+
"exports": {
13+
".": "./mod.ts"
14+
}
15+
}

0 commit comments

Comments
 (0)