Add @source not support

[RobinMalfait]

This PR adds a new source detection feature: @source not "…". It can be used to exclude files specifically from your source configuration without having to think about creating a rule that matches all but the requested file:

@import "tailwindcss";
@source not "../src/my-tailwind-js-plugin.js";

While working on this feature, we noticed that there are multiple places with different heuristics we used to scan the file system. These are:

• Auto source detection (so the default configuration or an @source "./my-dir")
• Custom sources ( e.g. @source "./**/*.bin" — these contain file extensions)
• The code to detect updates on the file system

Because of the different heuristics, we were able to construct failing cases (e.g. when you create a new file into my-dir that would be thrown out by auto-source detection, it'd would actually be scanned). We were also leaving a lot of performance on the table as the file system is traversed multiple times for certain problems.

To resolve these issues, we're now unifying all of these systems into one ignore crate walker setup. We also implemented features like auto-source-detection and the not flag as additional gitignore rules only, avoid the need for a lot of custom code needed to make decisions.

High level, this is what happens after the now:

• We collect all non-negative @source rules into a list of roots (that is the source directory for this rule) and optional globs (that is the actual rules for files in this file). For custom sources (i.e with a custom glob), we add an allowlist rule to the gitignore setup, so that we can be sure these files are always included.
• For every negative @source rule, we create respective ignore rules.
• Furthermore we have a custom filter that ensures files are only read if they have been changed since the last time they were read.

So, consider the following setup:

/* packages/web/src/index.css */
@import "tailwindcss";
@source "../../lib/ui/**/*.bin";
@source not "../../lib/ui/expensive.bin";

This creates a git ignore file that (simplified) looks like this:

# Auto-source rules
*.{exe,node,bin,…}
*.{css,scss,sass,…}
{node_modules,git}/

# Custom sources can overwrite auto-source rules
!lib/ui/**/*.bin

# Negative rules
lib/ui/expensive.bin

We then use this information on top of your existing .gitignore setup to resolve files (i.e so if your .gitignore contains rules e.g. dist/ this line is going to be added before any of the rules lined out in the example above. This allows negative rules to allow-list your .gitignore rules.

To implement this, we're rely on the ignore crate but we had to make various changes, very specific, to it so we decided to fork the crate. All changes are prefixed with a // CHANGED: block but here are the most-important ones:

• We added a way to add custom ignore rules that extend (rather than overwrite) your existing .gitignore rules
• We updated the order in which files are resolved and made it so that more-specific files can allow-list more generic ignore rules.
• We resolved various issues related to adding more than one base path to the traversal and ensured it works consistent for Linux, macOS, and Windows.

Behavioral changes

1. Any custom glob defined via @source now wins over your .gitignore file and the auto-content rules.
   • Resolves .gitignore is too powerful and cannot be stopped #16920
2. The node_modules and .git folders as well as the .gitignore file are now ignored by default (but can be overridden by an explicit @source rule).
   • Resolves Performance issue with versions greater than >4.0.9 #17318
   • Resolves [v4] Tailwind running with Next v15 on Docker takes ton of time and resources for first load #15882
3. Source paths into ignored-by-default folders (like node_modules) now also win over your .gitignore configuration and auto-content rules.
   • Resolves @source doesn't work if dist/ is in the gitignore. #16669
4. Introduced @source not "…" to negate any previous rules.
   • Resolves Plugin in source code is picked up as application code in Standalone CLI mode #17058
5. Negative content rules in your legacy JavaScript configuration (e.g. content: ['!./src']) now work with v4.
   • Resolves [v4] Ignore certain paths from scanning no longer supported #15943
6. The order of @source definitions matter now, because you can technically include or negate previous rules. This is similar to your .gitingore file.
7. Rebuilds in watch mode now take the @source configuration into account
   • Resolves [v4] Tailwind CLI ignores .gitignore in watch mode #15684

Combining with other features

Note that the not flag is also already compatible with @source inline(…) added in an earlier commit:

@import "tailwindcss";
@source not inline("container");

Test plan

• We added a bunch of oxide unit tests to ensure that the right files are scanned
• We updated the existing integration tests with new @source not "…" specific examples and updated the existing tests to match the subtle behavior changes
• We also added a new special tag [ci-all] that, when added to the description of a PR, causes the PR to run unit and integration tests on all operating systems.

[ci-all]

[ur5us]

@RobinMalfait Curious, is there a way to inspect the generated ignore list and emit what file change lead to another rebuild? Even with that new @source not "…"; directive and our .gitnore I’m seeing constant rebuilds in our Rails app whenever a page load happens. I assumed it’s happening due to log files being watched (which should be ignored as per .gitignore file) so I tried to add a @source not directive but no dice. Now I wonder whether there’s a potential bug with path resolution or yet another edge case not covered by the current detection mechanism.

Our setup is more complex as we use 3 separate builds for the marketing site, the admin UI and the booking engine. It’s roughly the following structure:

.
├── app
│   └── assets
│       └── stylesheets
│           ├── admin
│           │   ├── application.tailwind.css
│           │   └── tailwind.config.js
│           ├── booking
│           │   ├── application.tailwind.css
│           │   └── tailwind.config.js
│           └── cms
│               ├── application.tailwind.css
│               └── tailwind.config.js
|…
├──log

The .gitignore file lives in the root folder .. The config files are a variation of

import base from "../../../../tailwind.config.js"

/** @type {import('tailwindcss').Config} */
export default {
  presets: [base],
  content: [
    "./app/views/cms/**/*.erb",
    "./app/views/common/**/*.erb",
    "./app/views/layouts/cms*.erb",
    "./app/helpers/cms*.rb",
    "./app/helpers/cms/**/*.rb",
    "./app/assets/stylesheets/cms/application.tailwind.css",
    "./app/javascript/controllers/pages/**/*.{js,ts}",
  ],
};

with the base being

/** @type {import('tailwindcss').Config} */
export default {
  theme: {
    …
  },
  prefix: "tw"
};

We’re migrating from v3 hence why there are so many content: […] configs. On that note, I assume those are ignored now entirely, or‽ In other words, we need to migrate to @source …

[jclusso]

@ur5us did you figure anything out on this? Using @source '../../' or @source not '../../log' doesn't work for me at all. Doing @import 'tailwindcss/utilities.css' layer(utilities) source('../../'); does though. I don't like that though.

[ur5us]

@jclusso No, I have not. Due to time constraints I've had to stay on v3 and work on actual features. Might revisit in 1 – 2 months.