Skip to content

Fix #223: URI encode links - #224

Merged
bezzad merged 2 commits into
bezzad:developfrom
XanderLuciano:claude/fix-download-input-validation-bcG5a
Apr 25, 2026
Merged

Fix #223: URI encode links#224
bezzad merged 2 commits into
bezzad:developfrom
XanderLuciano:claude/fix-download-input-validation-bcG5a

Conversation

@XanderLuciano

@XanderLuciano XanderLuciano commented Apr 24, 2026

Copy link
Copy Markdown

Closes #223. Originally surfaced via the downstream consumer rdt-client#962.

Problem

Request.cs passes the raw address string straight to Uri.TryCreate:

if (Uri.TryCreate(address, UriKind.Absolute, out Uri uri) == false)
{
    uri = new Uri(new Uri("<http://localhost>"), address);
}

Per RFC 3986, square brackets are reserved for IPv6 literals in the host component only; [, ], {, }, and unencoded spaces are illegal in a path segment and must be percent-encoded. .NET's URI parser is platform-sensitive here:

  • Windows silently tolerates and auto-encodes these characters, hiding the bug.
  • Linux (.NET 8 / 9 / 10) is stricter — TryCreate either returns false (falling through to the http://localhost base-URI branch and producing a 404) or yields a Uri that HttpClient transmits incorrectly.

Real-world triggers are Debrid download links (Real-Debrid, TorBox, AllDebrid) that embed the torrent filename, e.g.:

https://real-debrid.com/d/ABCDEF/[SubGroup] Series - 03 [1080p WEB-DL].mkv

Fix

Added UrlHelper.EnsurePathEncoded(string) in src/Downloader/Extensions/UrlHelper.cs. It:

  • Operates only on the path. Scheme, userinfo, host (including IPv6 literal brackets like [::1]), port, query, and fragment are preserved byte-for-byte.
  • Uses the RFC 3986 pchar safe set — alphanumerics plus -._~!$&'()*+,;=:@. Everything else in a segment is percent-encoded as UTF-8. This matches the behavior of Go's net/url.PathEscape, Python's urllib.parse.quote, and Node's WHATWG URL, so downloads now send the same request line curl or a browser would.
  • Is idempotent. Existing valid %XX triplets pass through unchanged (lowercase hex is normalized to uppercase). Safe on already-encoded or partially-encoded input.
  • Decomposes the URL manually rather than via Uri, because Uri is exactly what rejects these URLs on Linux.

Wired in at every ingestion point where a server- or user-supplied URL string reaches new Uri():

File Line Site
Request.cs 49 Primary entry — every download URL flows here via AbstractDownloadService.InitialDownloader
SocketClient.cs 203 HTTP 30x redirect — server-supplied Location header
SocketClient.cs 143 Referer header

No public API changes. UrlHelper is internal.

Why a helper, not an inline fix?

Three call sites, subtle correctness requirements (idempotency, IPv6-host preservation, pchar set). Centralizing gives one well-tested implementation rather than three near-duplicates.

Tests

Added UrlHelperTest.cs with 26 [Theory] cases plus 3 [Fact] checks:

  • Motivating bug: [SubGroup] ... [1080p].mkv → correctly encoded
  • Curly braces, unencoded spaces
  • Pipe, caret, backtick, double-quote, angle brackets (all illegal in path)
  • Query string preserved verbatim even when it contains [ ]
  • Fragment preserved verbatim
  • IPv6 literal host ([::1], [2001:db8::1]) left untouched — only path encoded
  • Userinfo (user:pass@host) preserved
  • RFC 3986 sub-delims and : @ preserved in path (not over-encoded)
  • file:// scheme works the same way
  • Already-encoded URLs pass through unchanged (idempotency, 6 sample URLs × 2 passes)
  • Lowercase hex triplets normalized to uppercase
  • Standalone % (not followed by two hex digits) encoded to %25
  • Authority-only URLs with ? or # but no path — unchanged
  • Unicode filenames (Arabic) — UTF-8 percent-encoded, round-trips correctly via Uri.UnescapeDataString
  • Request constructor integration: bracketed Debrid URL now yields correct Host and AbsoluteUri

Risk / compatibility

  • Windows: Uri previously encoded many of these characters internally anyway; normalizing one step earlier produces identical wire output. No observable regression.
  • Linux: the failing URLs now succeed. That is the fix.
  • Already-encoded input: idempotency guarantees no double-encoding.
  • Serialized DownloadPackage.Urls: these come from req.Address.OriginalString. After this change that's the normalized form. Resuming a package feeds the URL back through Request → idempotent → identical Address. No migration needed.
  • Public API: unchanged.

claude added 2 commits April 24, 2026 05:26
… URI parsing

Downloads with URLs containing RFC 3986-illegal path characters (e.g. square
brackets in release-group tags like "[SubGroup] Show - 01 [1080p].mkv",
curly braces, or unencoded spaces) were failing on Linux. The .NET URI
parser on Linux is stricter than on Windows, so Request.Address either
fell back to a bogus http://localhost base or produced a URI the HTTP
stack transmitted incorrectly.

Add UrlHelper.EnsurePathEncoded — a small, idempotent normalizer that
percent-encodes non-pchar path characters as UTF-8 while preserving
scheme, userinfo, host (including IPv6 literal brackets like [::1]),
port, query, and fragment. Wire it in at every ingestion point where a
server-supplied or user-supplied URL string reaches new Uri():
Request ctor, SocketClient redirect-Location handling, and the Referer
header. Add UrlHelperTest covering brackets, braces, spaces, Unicode,
IPv6 hosts, userinfo, query/fragment preservation, idempotency, and
lone '%' handling.
Follow-up to the initial issue bezzad#223 fix, prompted by a self-review. No
behavior changes to the encoder itself — only hardening around it:

- Expand UrlHelper class docstring with the four security invariants it
  relies on (never decode, always encode control chars, authority
  untouched, output Uri-parseable) so a future maintainer understands
  why each property matters and doesn't relax one by accident.
- Document the treatment of literal '?' and '#' in callers' filenames
  (must be pre-encoded).
- Replace ReferenceEquals coupling between EnsurePathEncoded and
  EncodePath with straightforward string equality; the fast path still
  allocates nothing.
- Add guard comments at both SocketClient call sites (Referer and
  redirect Location) explaining why the UrlHelper wrapper must not be
  removed in future refactors.

Tests added to lock in the security posture:

- Control characters (CR, LF, CRLF, TAB, NUL, SOH, FF, DEL) are all
  percent-encoded as UTF-8.
- A CRLF injection payload ("\r\nHost: evil.com\r\nGET /admin") is
  defused — the attempted header/request-line smuggle becomes
  percent-escaped content inside the path.
- Already-encoded input (%2e%2e%2f) is never unescaped, so server-side
  path-traversal defenses see the attacker's true input.
- Authority is never altered by path encoding.
- Request ctor handling of a CRLF-containing URL yields a clean
  AbsoluteUri with no raw control chars.
- Empty path segments (double slashes) and scheme-only inputs pass
  through unchanged.

Strengthen RequestConstructorAcceptsBracketedUrl to assert explicitly
that we don't fall back to the http://localhost base URI and that
AbsolutePath contains no raw '[' or ' '.
@bezzad bezzad self-assigned this Apr 25, 2026
@bezzad bezzad added the bug Something isn't working label Apr 25, 2026
@bezzad
bezzad changed the base branch from master to develop April 25, 2026 10:48
@bezzad
bezzad merged commit dcb6cce into bezzad:develop Apr 25, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Downloads fail when URL path contains square brackets

3 participants