Updates to `derive` and `lift` #1838

mathpirate · 2025-09-29T21:37:26Z

Remove the Promise return type option
Use Typescript's inferred types to autogenerate schemas even when the programmer doesn't explicitly provide type template values for both lift and derive

Summary by cubic

Removes async support from derive/lift and adds automatic schema generation using TypeScript type inference when generics are omitted. This simplifies the API and cuts boilerplate by inferring and injecting schemas at build time.

New Features
- Auto-generate argument and result JSONSchema for lift and derive from function parameter and return types (when generics are not provided).
- Added TypeRegistry to pass TypeScript Types between transformer stages for reliable schema generation.
- Schema injection now infers types and fills missing sides with unknown when only one can be determined.
Migration
- derive and lift no longer accept Promise-returning functions; callbacks must be synchronous.
- Update any async derive/lift usage to sync functions or an alternative API that supports async.

…eneration on lift and derive

…mas from them for and

…'unknown'

...s-transformers/test/fixtures/ast-transform/schema-generation-derive-no-generics.expected.tsx

…ixture showing we do correct inference and schema generation now

...s-transformers/test/fixtures/ast-transform/schema-generation-handler-event-only.expected.tsx

ubik2 · 2025-09-29T23:03:05Z

...ansformers/test/fixtures/ast-transform/schema-generation-handler-no-annotations.expected.tsx

+/// <cts-enable />
+import { handler, JSONSchema } from "commontools";
+// No type annotations at all - should generate false schemas (unknown type)
+export const genericHandler = handler(false as const satisfies JSONSchema, false as const satisfies JSONSchema, (event, state) => {


I don't think we need the as const satisfies JSONSchema when we use true or false. It shouldn't be a problem, but it's more verbose.

that makes sense, thanks. i took this linear issue to come back to this to try to wrangle this PR in a more manageable form :) https://linear.app/common-tools/issue/CT-954/dont-generate-as-const-satisfies-jsonschema-for-truefalse-schemas

…odes in the type registry

jsantell · 2025-09-29T23:08:12Z

packages/ts-transformers/src/core/type-registry.ts

+ * Uses WeakMap with node identity as key. Node identity is preserved when
+ * transformers are applied in sequence via ts.transform().
+ */
+export class TypeRegistry {


type TypeRegistry = WeakMap<ts.Node, ts.Type> would suffice here

ooooh yes great point thank you, done

jsantell · 2025-09-29T23:12:04Z

packages/ts-transformers/src/transform.ts

  return [
-    createModularOpaqueRefTransformer(program),
-    createSchemaTransformer(program),
+    createModularOpaqueRefTransformer(program, { typeRegistry }),


What are the lifetime of the ts.Nodes in the weak map? I didn't think the Program held a reference to all nodes it's created, and if it doesn't, what persists between these transformation calls that ensures the weak map does not clean its entries?

I honestly don't have a great theoretical answer here - it appears empirically to be the case that Typescript's transformation pipeline passes the updated-in-place SourceFile from one transformer to the next, such that when the later transformers use the SourceFile object (which is the root node of the AST) they receive to walk the AST, they encounter the Nodes that were created/modified by the previous transformers in the pipeline.

Confirmed that even when using a transformed (new) SourceFile, walking produces the same (instance equality) nodes when it can -- probably stored in the Program

Thanks for checking on that

jsantell · 2025-09-29T23:15:37Z

packages/ts-transformers/src/schema/schema-transformer.ts

+        // First check if we have a registered Type for this node
+        // (from schema-injection when synthetic TypeNodes were created)
+        let type: ts.Type;
+        if (typeRegistry && typeRegistry.has(node)) {


Are these the same Nodes that have registered to the type registry, e.g. the same instance (as opposed to a different instance representing the same underlying syntax fragment)?

Ah I see the comment that node identity is preserved during transformations

yeah it's the same Node object with the same identity

packages/ts-transformers/src/opaque-ref/rules/schema-injection.ts

packages/ts-transformers/src/opaque-ref/type-inference.ts

seefeldb · 2025-09-29T23:26:09Z

...ts-transformers/test/fixtures/ast-transform/schema-generation-derive-inside-jsx.expected.tsx

+/// <cts-enable />
+import { derive, h } from "commontools";
+declare const value: number;
+// This derive is INSIDE JSX and should NOT be transformed with schemas


we should transform derive, handler, etc. inside JSX as well + add schemas, even for the ones that are automatically added

thanks for noticing this, i fixed this now

ubik2 · 2025-09-30T19:09:49Z

packages/schema-generator/src/formatters/common-tools-formatter.ts

+      if (typeof valueSchema === "boolean") {
+        // Boolean schemas (true/false) cannot have properties
+        // Wrap in an object schema
+        return {


rather than { allOf: [valueSchema], default: defaultValue} here, can we do the { default: defaultValue } / {not: true, default: defaultValue}?

sure thing, done!

ubik2 · 2025-09-30T20:14:14Z

packages/ts-transformers/test/type-inference-exploration.ts

Not sure if the inclusion of these three type-inference files was intentional.
They're useful, since other devs will probably want to explore the functionality as well, so reasonable to include.

I was thinking of leaving them in through the merge to main, then maybe deleting them off main later - that way they'd be in the repo history but not taking up space in the folder... but I'm open to whatever, not sure what's best!

ubik2

lgtm!

mathpirate added 4 commits September 29, 2025 10:04

NON-WORKING checkpoint towards supporting type inference for schema g…

98e5a48

…eneration on lift and derive

merge work that was accidentally done on main

60a57ad

mostly working checkpoint for inferring types and autogenerating sche…

a4e7ba6

…mas from them for and

new fixtures for new derive and handler type inference

6573144

mathpirate temporarily deployed to ci September 29, 2025 21:41 — with GitHub Actions Inactive

remove unnneeded fallback, add two new fixture pairs

2769583

mathpirate temporarily deployed to ci September 29, 2025 22:07 — with GitHub Actions Inactive

mathpirate added 2 commits September 29, 2025 15:24

refactor type inference to shared utility file

7a09fb9

based on robin's feedback, generate 'true' for 'any' and 'false' for …

19a2500

…'unknown'

mathpirate requested review from jsantell, seefeldb and ubik2 September 29, 2025 22:33

fmt

ff71730

ubik2 reviewed Sep 29, 2025

View reviewed changes

...s-transformers/test/fixtures/ast-transform/schema-generation-derive-no-generics.expected.tsx Outdated Show resolved Hide resolved

mathpirate added 2 commits September 29, 2025 16:00

fix small type inference bug using ben's logCharmsList example; add f…

164d620

…ixture showing we do correct inference and schema generation now

fmt

9cbef04

ubik2 reviewed Sep 29, 2025

View reviewed changes

...s-transformers/test/fixtures/ast-transform/schema-generation-handler-event-only.expected.tsx Outdated Show resolved Hide resolved

ubik2 reviewed Sep 29, 2025

View reviewed changes

fix for previous overly broad change to how we handle synthetic typen…

01b0bcc

…odes in the type registry

mathpirate temporarily deployed to ci September 29, 2025 23:15 — with GitHub Actions Inactive

jsantell reviewed Sep 29, 2025

View reviewed changes

seefeldb reviewed Sep 29, 2025

View reviewed changes

packages/ts-transformers/src/opaque-ref/rules/schema-injection.ts Outdated Show resolved Hide resolved

seefeldb reviewed Sep 29, 2025

View reviewed changes

packages/ts-transformers/src/opaque-ref/type-inference.ts Outdated Show resolved Hide resolved

seefeldb reviewed Sep 29, 2025

View reviewed changes

mathpirate added 2 commits September 30, 2025 11:13

fmt

c879e7d

update fixtures for new schema version

fb2408d

mathpirate temporarily deployed to ci September 30, 2025 18:20 — with GitHub Actions Inactive

use instead of definitions

4a56b01

mathpirate temporarily deployed to ci September 30, 2025 18:34 — with GitHub Actions Inactive

simplify typeregistry to a type alias for weakmap rather than class

d54af9b

mathpirate temporarily deployed to ci September 30, 2025 18:54 — with GitHub Actions Inactive

ubik2 reviewed Sep 30, 2025

View reviewed changes

clean ups

038bf63

mathpirate temporarily deployed to ci September 30, 2025 19:35 — with GitHub Actions Inactive

correctly inject schemas on/transform derive calls inside JSX

12827cb

mathpirate temporarily deployed to ci September 30, 2025 19:46 — with GitHub Actions Inactive

change format for boolean schemas

8691969

mathpirate temporarily deployed to ci September 30, 2025 19:53 — with GitHub Actions Inactive

ubik2 reviewed Sep 30, 2025

View reviewed changes

ubik2 approved these changes Sep 30, 2025

View reviewed changes

mathpirate merged commit a3b4717 into main Sep 30, 2025
7 checks passed

mathpirate deleted the feature-remove-derive-promise branch September 30, 2025 20:31

Updates to derive and lift #1838

Updates to derive and lift #1838

Uh oh!

Conversation

mathpirate commented Sep 29, 2025 • edited by cubic-dev-ai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by cubic

Uh oh!

Uh oh!

Uh oh!

ubik2 Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ubik2 Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ubik2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Updates to `derive` and `lift` #1838

Updates to `derive` and `lift` #1838

mathpirate commented Sep 29, 2025 •

edited by cubic-dev-ai bot

Loading

ubik2 Sep 29, 2025 •

edited

Loading

ubik2 Sep 30, 2025 •

edited

Loading