Ensure @source globs with symlinks are preserved (#20203)

This PR fixes an issue when working with `@source` and `@source not`
that involves symlinks.

Internally our sources are mapped to a source entry where we have a
`base` path and a `pattern`. You can think about this where we insert a
`.gitignore` file in the `base` path for the given pattern.

However, we optimize these entries to move as many "static" parts into
the base path. For example:
```css
@source "./some/folder/here/*.html";
```

Is mapped to something like:
```ts
{ base: "/projects/my-project", pattern: "./some/folder/here/*.html" }
```
We then optimize it by turning it into:
```ts
{ base: "/projects/my-project/some/folder/here", pattern: "*.html" }
```

While doing this, we also use `dunce::canonicalize` to resolve the
actual paths on disk. This means that a symlink is resolved to their
real paths. This can cause issues because the "real" path is not what
you wrote in the `@source` directives.

So before, it could be that you have this:
```css
@source "./some/symlinked-folder/here/*.html";
```
Which was mapped to this on the Rust side:
```ts
{ base: "/projects/my-project", pattern: "./some/symlinked-folder/here/*.html" }
```
But was then optimized to:
```ts
{ base: "/projects/my-project/some/actual-folder/here", pattern: "*.html" }
```

...and we lost the `symlinked-folder` information. This causes issues as
seen in #17985.

With this PR, we keep the symlinked information in those globs since
that's what you wrote in those `@source` directives.

While setting up integration tests, I stumbled upon an issue because I
wanted to test that ignoring a symlinked folder, but including a single
particular file of that ignored folder resulted in that file being
ignored as well. Let's look at an example:

```css
@source     '../lib';
@source not '../lib/ignored';
@source     '../lib/ignored/except.html';
```

Earlier I mentioned that we create `.gitignore` files based on these
`@source` directives. In this case, when we're dealing with a folder, we
use `**/*` as the contents.

Looking at the example above, we should essentially have something like
this:
```gitignore
# lib/.gitignore
# @source '../lib'
!**/*

# lib/ignored/.gitignore
# @source '../lib/ignored'
**/*

# @source '../lib/ignored/except.html'
!except.html
```

Since it's a `.gitignore` file, we have to invert the globs. But the bug
I noticed is that in reality the result of those gitignores didn't look
like the above, it looked like:
```gitignore
# lib/.gitignore
# @source '../lib'
!**/*

# lib/ignored/.gitignore
# @source '../lib/ignored/except.html'
!except.html

# @source '../lib/ignored'
**/*
```

Notice how the `!except.html` and `**/*` are flipped. When dealing with
`.gitignore` files, the order is important.

This was caused because internally we kept a `BTreeMap` of `BTreeSet`s
where the map was the base path and a set of patterns. The patterns were
sorted because of the `BTreeSet`... which is not what we want.

Fixes: #17985
Closes: #20091

## Test plan

1. Added new tests in the scanner tests (on the Rust side)
2. Added integration tests with a symlink to another folder, outside of
the current folder
2. Added integration tests with a symlink to another folder, inside of
the current folder
2. Added integration tests to ensure that the order of `@source` files
with a folder + file is sorted correctly.
3. Since we're dealing with symlinks in these tests, let's test all OSes
[ci-all]

Author: RobinMalfait

CHANGELOG.md

@@ -27,6 +27,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Ensure class candidates are extracted from Twig `addClass(…)` and `removeClass(…)` calls ([#20198](https://github.com/tailwindlabs/tailwindcss/pull/20198))
 - Don't crash in the Ruby or Vue preprocessors when scanning files containing invalid UTF-8 bytes ([#19588](https://github.com/tailwindlabs/tailwindcss/pull/19588))
 - Allow `@variant` to be used inside `addBase` ([#19480](https://github.com/tailwindlabs/tailwindcss/pull/19480))
+- Ensure `@source` globs with symlinks are preserved ([#20203](https://github.com/tailwindlabs/tailwindcss/pull/20203))
+- Ensure later `@source` rules can re-include files excluded by earlier `@source not` rules ([#20203](https://github.com/tailwindlabs/tailwindcss/pull/20203))
 
 ### Changed
 

crates/oxide/src/scanner/mod.rs

@@ -17,7 +17,6 @@ use fxhash::{FxHashMap, FxHashSet};
 use ignore::{gitignore::GitignoreBuilder, WalkBuilder};
 use init_tracing::{init_tracing, SHOULD_TRACE};
 use rayon::prelude::*;
-use std::collections::{BTreeMap, BTreeSet};
 use std::path::{Path, PathBuf};
 use std::sync::{Arc, Mutex};
 use std::time::SystemTime;
@@ -637,7 +636,16 @@ fn walk_parallel(walker: &mut WalkBuilder) -> Vec<WalkEntry> {
 fn create_walker(sources: &Sources) -> Option<WalkBuilder> {
     let mut other_roots: FxHashSet<&PathBuf> = FxHashSet::default();
     let mut first_root: Option<&PathBuf> = None;
-    let mut ignores: BTreeMap<&PathBuf, BTreeSet<String>> = Default::default();
+
+    let mut ignores: Vec<(&PathBuf, Vec<String>)> = Default::default();
+    let mut emit = |base, pattern| match ignores.last_mut() {
+        Some((prev_base, patterns)) if *prev_base == base => {
+            patterns.push(pattern);
+        }
+        _ => {
+            ignores.push((base, vec![pattern]));
+        }
+    };
 
     for source in sources.iter() {
         match source {
@@ -649,7 +657,7 @@ fn create_walker(sources: &Sources) -> Option<WalkBuilder> {
                 }
             }
             SourceEntry::Pattern { base, pattern } => {
-                let mut pattern = pattern.to_string();
+                let pattern = pattern.to_owned();
 
                 if first_root.is_none() {
                     first_root = Some(base);
@@ -658,35 +666,19 @@ fn create_walker(sources: &Sources) -> Option<WalkBuilder> {
                 }
 
                 if !pattern.contains("**") {
-                    // Ensure that the pattern is pinned to the base path.
-                    if !pattern.starts_with("/") {
-                        pattern = format!("/{pattern}");
-                    }
-
                     // Specific patterns should take precedence even over git-ignored files:
-                    ignores
-                        .entry(base)
-                        .or_default()
-                        .insert(format!("!{}", pattern));
+                    emit(base, format!("!{}", pattern));
                 } else {
                     // Assumption: the pattern we receive will already be brace expanded. So
                     // `*.{html,jsx}` will result in two separate patterns: `*.html` and `*.jsx`.
                     if let Some(extension) = Path::new(&pattern).extension() {
                         // Extend auto source detection to include the extension
-                        ignores
-                            .entry(base)
-                            .or_default()
-                            .insert(format!("!*.{}", extension.to_string_lossy()));
+                        emit(base, format!("!*.{}", extension.to_string_lossy()));
                     }
                 }
             }
             SourceEntry::Ignored { base, pattern } => {
-                let mut pattern = pattern.to_string();
-                // Ensure that the pattern is pinned to the base path.
-                if !pattern.starts_with("/") {
-                    pattern = format!("/{pattern}");
-                }
-                ignores.entry(base).or_default().insert(pattern);
+                emit(base, pattern.to_owned());
             }
             SourceEntry::External { base } => {
                 if first_root.is_none() {
@@ -696,16 +688,10 @@ fn create_walker(sources: &Sources) -> Option<WalkBuilder> {
                 }
 
                 // External sources should take precedence even over git-ignored files:
-                ignores
-                    .entry(base)
-                    .or_default()
-                    .insert(format!("!{}", "/**/*"));
+                emit(base, format!("!{}", "/**/*"));
 
                 // External sources should still disallow binary extensions:
-                ignores
-                    .entry(base)
-                    .or_default()
-                    .insert(BINARY_EXTENSIONS_GLOB.clone());
+                emit(base, BINARY_EXTENSIONS_GLOB.clone());
             }
         }
     }

crates/oxide/src/scanner/sources.rs

@@ -1,7 +1,6 @@
-use crate::glob::split_pattern;
 use crate::GlobEntry;
 use bexpand::Expression;
-use std::path::PathBuf;
+use std::path::{Component, Path, PathBuf};
 use tracing::{event, Level};
 
 use super::auto_source_detection::IGNORED_CONTENT_DIRS;
@@ -102,72 +101,215 @@ impl PublicSourceEntry {
     /// resolved path.
     pub fn optimize(&mut self) {
         // Resolve base path immediately
-        let Ok(base) = dunce::canonicalize(&self.base) else {
+        let Ok(mut base) = dunce::canonicalize(&self.base) else {
             event!(Level::ERROR, "Failed to resolve base: {:?}", self.base);
             return;
         };
-        self.base = base.to_string_lossy().to_string();
 
-        // No dynamic part, figure out if we are dealing with a file or a directory.
-        if !self.pattern.contains('*') {
-            let combined_path = if self.pattern.starts_with("/") {
-                PathBuf::from(&self.pattern)
-            } else {
-                PathBuf::from(&self.base).join(&self.pattern)
-            };
+        let mut new_pattern = PathBuf::new();
+        enum ComponentStage {
+            Base,
+            Pattern,
+        }
+        let mut stage = ComponentStage::Base;
+
+        let mut components = Path::new(&self.pattern).components().peekable();
+        while let Some(component) = components.next() {
+            match stage {
+                ComponentStage::Base => {
+                    match component {
+                        // Ignore the current dir, e.g. `.`
+                        Component::CurDir => {}
+
+                        // Go up a directory, e.g. `..`
+                        Component::ParentDir => {
+                            base.pop();
+                        }
+
+                        // Once we hit a component that contains a wildcard character, then we
+                        // can't change the base anymore and we must move to the pattern part.
+                        Component::Normal(part) if part.to_string_lossy().contains("*") => {
+                            new_pattern.push(component);
+                            stage = ComponentStage::Pattern;
+                        }
+
+                        // File or folder, but not the last component
+                        Component::Normal(part) if components.peek().is_some() => {
+                            base.push(part);
+                        }
 
-            match dunce::canonicalize(combined_path) {
-                Ok(resolved_path) if resolved_path.is_dir() => {
-                    self.base = resolved_path.to_string_lossy().to_string();
-                    self.pattern = "**/*".to_owned();
+                        // Last file or folder. If it's a folder, we move it to the base,
+                        // otherwise we move it to the pattern.
+                        Component::Normal(part) => {
+                            let full_path = base.join(part);
+                            if full_path.is_dir() {
+                                base.push(part);
+                            } else {
+                                new_pattern.push(part);
+                            }
+                        }
+
+                        // When we're dealing with an absolute path, then we have to bypass the
+                        // `base` entirely.
+                        Component::Prefix(_) => {
+                            base.clear();
+                            base.push(component);
+                        }
+                        Component::RootDir => {
+                            #[cfg(not(windows))]
+                            base.clear();
+                            base.push(component);
+                        }
+                    }
                 }
-                Ok(resolved_path) if resolved_path.is_file() => {
-                    self.base = resolved_path
-                        .parent()
-                        .unwrap()
-                        .to_string_lossy()
-                        .to_string();
-                    // Ensure leading slash, otherwise it will match against all files in all folders/
-                    self.pattern =
-                        format!("/{}", resolved_path.file_name().unwrap().to_string_lossy());
+                ComponentStage::Pattern => {
+                    new_pattern.push(component);
                 }
-                _ => {}
             }
-            return;
         }
 
-        // Contains dynamic part
-        let (static_part, dynamic_part) = split_pattern(&self.pattern);
-
-        let base: PathBuf = self.base.clone().into();
-        let base = match static_part {
-            Some(static_part) => {
-                // TODO: If the base does not exist on disk, try removing the last slash and try
-                // again.
-                match dunce::canonicalize(base.join(static_part)) {
-                    Ok(base) => base,
-                    Err(err) => {
-                        event!(tracing::Level::ERROR, "Failed to resolve glob: {:?}", err);
-                        return;
-                    }
+        self.base = base.to_string_lossy().to_string();
+        self.pattern = path_to_posix_string(&new_pattern);
+
+        // Ensure we have `**/*` when the base is a folder and we don't have a pattern at all
+        if self.pattern == "" {
+            self.pattern = "/**/*".to_owned();
+        }
+        // Ensure that the pattern is pinned to the base path.
+        else if !self.pattern.starts_with("/") {
+            self.pattern = format!("/{}", self.pattern);
+        }
+    }
+}
+
+fn path_to_posix_string(path: &Path) -> String {
+    let mut parts = Vec::new();
+    let mut is_rooted = false;
+
+    for component in path.components() {
+        match component {
+            Component::Prefix(prefix) => {
+                parts.push(prefix.as_os_str().to_string_lossy().to_string());
+            }
+            Component::RootDir => {
+                is_rooted = true;
+                if parts.is_empty() {
+                    parts.push(String::new());
                 }
             }
-            None => base,
+            Component::CurDir => {
+                parts.push(".".to_string());
+            }
+            Component::ParentDir => {
+                parts.push("..".to_string());
+            }
+            Component::Normal(part) => {
+                parts.push(part.to_string_lossy().to_string());
+            }
+        }
+    }
+
+    let result = parts.join("/");
+    if result.is_empty() && is_rooted {
+        "/".to_string()
+    } else {
+        result
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use pretty_assertions::assert_eq;
+    use std::fs;
+    use tempfile::tempdir;
+
+    #[test]
+    fn path_to_posix_string_serializes_relative_paths() {
+        let path = PathBuf::from("src").join("**").join("*.html");
+
+        assert_eq!(path_to_posix_string(&path), "src/**/*.html");
+    }
+
+    #[test]
+    fn path_to_posix_string_serializes_rooted_paths() {
+        let path = PathBuf::from(std::path::MAIN_SEPARATOR.to_string())
+            .join("src")
+            .join("**")
+            .join("*.html");
+
+        assert_eq!(path_to_posix_string(&path), "/src/**/*.html");
+    }
+
+    #[test]
+    fn path_to_posix_string_serializes_empty_paths() {
+        assert_eq!(path_to_posix_string(&PathBuf::new()), "");
+    }
+
+    #[test]
+    fn optimize_hoists_static_directories_and_keeps_files_in_the_pattern() {
+        let dir = tempdir().unwrap();
+        fs::create_dir_all(dir.path().join("src").join("examples")).unwrap();
+
+        let mut source = PublicSourceEntry {
+            base: dir.path().to_string_lossy().to_string(),
+            pattern: "src/examples/index.html".to_string(),
+            negated: false,
         };
 
-        let pattern = match dynamic_part {
-            Some(dynamic_part) => dynamic_part,
-            None => {
-                if base.is_dir() {
-                    "**/*".to_owned()
-                } else {
-                    "".to_owned()
-                }
-            }
+        source.optimize();
+
+        assert_eq!(
+            source.base,
+            dunce::canonicalize(dir.path().join("src").join("examples"))
+                .unwrap()
+                .to_string_lossy()
+        );
+        assert_eq!(source.pattern, "/index.html");
+    }
+
+    #[test]
+    fn optimize_hoists_folder_patterns() {
+        let dir = tempdir().unwrap();
+        fs::create_dir_all(dir.path().join("src").join("examples")).unwrap();
+
+        let mut source = PublicSourceEntry {
+            base: dir.path().to_string_lossy().to_string(),
+            pattern: "src/examples".to_string(),
+            negated: false,
         };
 
-        self.base = base.to_string_lossy().to_string();
-        self.pattern = pattern;
+        source.optimize();
+
+        assert_eq!(
+            source.base,
+            dunce::canonicalize(dir.path().join("src").join("examples"))
+                .unwrap()
+                .to_string_lossy()
+        );
+        assert_eq!(source.pattern, "/**/*");
+    }
+
+    #[test]
+    fn optimize_keeps_wildcards_in_the_pattern() {
+        let dir = tempdir().unwrap();
+        fs::create_dir_all(dir.path().join("src")).unwrap();
+
+        let mut source = PublicSourceEntry {
+            base: dir.path().to_string_lossy().to_string(),
+            pattern: "src/**/*.html".to_string(),
+            negated: false,
+        };
+
+        source.optimize();
+
+        assert_eq!(
+            source.base,
+            dunce::canonicalize(dir.path().join("src"))
+                .unwrap()
+                .to_string_lossy()
+        );
+        assert_eq!(source.pattern, "/**/*.html");
     }
 }