Renew star.jquery.com cert (expires 25 June 2025)

[timmywil]

Instructions: https://github.com/jquery/infrastructure-puppet/blob/staging/doc/cdn-cert.md
Previous ticket: #50

Timeline

Date	Action
Tue June 10, 2025	Created a ticket with LF IT to issue new certs
Tue June 10, 2025	LF IT confirmed receipt of the request
Tue June 10, 2025	LFIT ticket assigned to Ryan Aslett
Wed June 11, 2025	Timo clicked "Escalate" on the ticket
Thu June 12, 2025	Timo messaged Ryan directly to confirm timeline
Fri June 13, 2025	Certs issued
Fri June 13, 2025	Certs locally verified
Fri June 13, 2025	Verified cert locally
Fri June 13, 2025	Add new cert to Fastly; enable in staging on code2 by activating it only for t.sni DNS (the CDN and other sites use k.sni).
Fri June 13, 2025	Tested the new cert. New connection failures on iOS 6-9 and macOS 10.9-10.11, and good or same service everywhere else. We think these specific failures will work when deployed to Fastly's k TLS configuration.
Fri June 13, 2025	Waiting at least 5 days after issue date Thu, 12 Jun 2025 00:00:00 GMT which will be Tue 17 June.
Tue June 17, 2025	Deployed to production by activating on k.sni, as used by code.jquery.com. Rolled back due to failure to verify trust on macOS 10.9-10.11.
Thu June 19, 2025	Fixed ca-bundle to swap in cross-sign stamp by AAA root. Deleted new cert and new private key in Fastly. Re-created pem from crt+ca-bundle. Upload key and pem to Fastly, and activate on t.sni for code2 staging. Verified on various browsers. Activate on k.sni for production code.jquery.com. Confirmed to work on macOS 10.9-10.11.

[timmywil]

Assigned to Timo as I am currently on paternity leave.

[Krinkle]

Important dates ahead:

• Sun June 15, 2025: Last ideal date to receive new cert, to have 5 days incubation before go-live, given backward clock drift.
• Fri June 20, 2025: Old cert starts failing for some clients, given forward clock drift.
• Tue June 24, 2025: Old cert officially expires.

[Krinkle]

I've received three files via the LFIT ticket:

• STAR_jquery_com.crt
• STAR_jquery_com_bundle.pem (This is the .ca-bundle file that our docs refer to, but with a different name/suffix)
• STAR_jquery_com.pem (This is the file our docs say to create by concatenating .crt and .ca-bundle, but Ryan has done this for us already. I've confirmed that its contents are indeed the first two files combined).

And via 1Password we got the private key, which I've temporarily saved as STAR_jquery_com.key.

$ ls
STAR_jquery_com.key
STAR_jquery_com.pem
STAR_jquery_com_bundle.key
verify_cert.sh # from jquery/infrastructure-puppet.git:bin/

$ ./verify_cert.sh STAR_jquery_com.pem 

Dates the cert is valid (expect today to be within this range):
notBefore=Jun 12 00:00:00 2025 GMT
notAfter=Jun 26 23:59:59 2026 GMT

Verifying validity of the certificate chain (expect "OK"):
STAR_jquery_com.pem: OK

Verify the public keys match (expect "Keys match"):
Keys match

Verify the PEM file is in the right order (expect issuer to match next subject)
subject=CN = *.jquery.com
issuer=C = GB, O = Sectigo Limited, CN = Sectigo Public Server Authentication CA DV E36

subject=C = GB, O = Sectigo Limited, CN = Sectigo Public Server Authentication CA DV E36
issuer=C = GB, O = Sectigo Limited, CN = Sectigo Public Server Authentication Root E46

subject=C = GB, O = Sectigo Limited, CN = Sectigo Public Server Authentication Root E46
issuer=C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust ECC Certification Authority

[Krinkle]

I've added it to Fastly as a new cert and activated it for t.sni. In DNS, only code2 uses t.sni. The CDN and other sites use k.sni.

I've confirmed it is being served as expected without errors in my local Firefox:

Comparing:

code.jquery.com	code2.jquery.com

[Krinkle]

Ryan asked whether there is still a gap in acceptence between our cert and Let's Encrypt, so I included in my test matrix one of our self-hosted services that use Let's Encrypt with a minimal URL.

Tested:

• Status quo, our 2024 cert as served live from Fastly k.sni (extra ciphers), at https://code.jquery.com/MIT-LICENSE.txt
• Staging, our new 2025 cert as served from Fastly t.sni (fewer ciphers), at https://code2.jquery.com/MIT-LICENSE.txt
• Let's Encrypt, using https://typesense.jquery.com/collections

By the way, the reason I test it this way with staging using a different Fastly server (t.sni instead of k.sni) is that Fastly has a shared certificate store on each edge server, so a cert cannot be "active" for one domain (code2) but not others (code). The cert for *.jquery.com it either active, or not. The lever we do have, is their distinct edge clusters (e.g. k.sni, t.sni, and others) so we pick the best for prod (k.sni), the second-best for t.sni, and separate them by DNS.

-	Status quo	Staging	Let's Encrypt
WinXP/IE8	Fail (OK on HTTP)	Fail	Fail
WinXP/Firefox 52	OK	OK	OK
Win7/IE8, IE9, IE10, IE11	OK	OK	OK
Win7/Chrome 69	OK	OK	OK
Win7/Chrome 109	OK	OK	OK
Win8/IE10	OK	OK	OK
Win10/IE11	OK	OK	OK
Safari 6 (macOS 10.7 Lion)	Fail	Fail	Fail
Safari 6.2 (macOS 10.8 Mountain Lion)	Fail	Fail	Fail
Safari 7.1 (macOS 10.9 Mavericks)	OK	Fail to connect	Fail to verify
Safari 8 (macOS 10.10 Yosemite)	OK	Fail to connect	Fail to verify
Safari 9.1 (macOS 10.11 El Capitan)	OK	Fail to verify	Fail to verify
Safari 10.1 (macOS 10.12 Sierra)	OK	OK	OK
Safari 11 (macOS 10.13 High Sierra)	OK	OK	OK
iOS 6 (iPhone 5 simulator)	OK	Fail to connect	Fail to verify
iOS 7 (iPad Mini simulator)	OK	Fail to connect	Fail to verify
iOS 9.3 (iPad Mini 4 simulator)	OK	Fail to verify	Fail to verify
iOS 10.3 (iPad Pro simulator)	OK	OK	OK
iOS 11 (iPad 6th Gen, 2018 real)	OK	OK	OK
iOS 11 (iPhone 8th Gen, 2017 real)	OK	OK	OK
Android 4.4 (Samsung Galaxy S5 Mini simulator)	OK	OK	Fail to verify
Android 7 (Samsung Galaxy S8, 2017 real)	OK	OK	OK
Android 9 (Huawei P30, 2021 real)	OK	OK	OK

Quote from last year:

Chrome 49/WinXP. This supports TLS 1.2 but appears to be RSA-only. Even with Fastly's extended chipher support, this won't work now and Chrome/WinXP joins IE8/WinXP in being HTTP-only. Note that WinXP remains supported via Firefox 52.

The reason we tested Chrome 49, is that that was the last Chrome for WinXP. The Firefox ESR for WinXP is Firefox 52.

Likewise, Chrome 109 is the last Chrome for Win7, Win8 and Win10. Although afaik Chrome (unlike Firefox) does not bring its own networking/cert layer, so this mostly reflects what Window/IE do on the same device.

Summary about staging:

Like last year, I expect macOS 10.9-10.11 and iOS 6-9 will be OK once deployed to Fastly'sk.sni TLS configuration, as that one supports the additional older ciphers.

Summary about Let's Encrypt:

I'm assuming that failure to connect is reflective of supporting the right TLS version and ciphersuite, hence something that can vary with the same cert depending on how it is configured/deployed.

As such, given that old iOS, macOS, and Android failed at verify step with LE, rather than on connect, this would presumably not be helped by Fastly's k.sni configuration. I assume a switch would mean we lose:

• iOS 6-9,
• Safari (macOS 10.9 Mavericks - macOS 10.11 El Capitan),
• Android 4-6

This is inconclusive as I'm testing our self-hosted nginx and certbot, whereas Fastly's service and Varnish/k.sni configuration may perform differently. I suggest that later this year, we can migrate a secondary service to it and test again. It would have to be on a top-level domain other than jquery.com to avoid a conflict. Perhaps something under jquerymobile.com, like https://demos.jquerymobile.com/ could move under the "miscweb" bucket in our Fastly service.

[Krinkle]

To quantify impact of the iOS, macOS and Android versions, one basically has to check a sampling of devices to see what age is affected. I did this recently for typesense-minibar Browser support and Wikipedia.org Browser support, to understand which browser/OS versions are meaningful cut-off points, and which are merely within a larger mostly inconsequential range.

For example, if we assume Let's Encrypt would drop iOS 6-9, macOS 10.9-10.11, and Android 4-6:

• MacBook Pro:
  • MacBook Pro's from 2007 to Mid-2009: Unreachable, macOS 10.9-10.11 marked the end of their range.
  • From Mid-2010 onwards: OK, may run macOS 10.12 (new major release in 2016, minor updates until 2019).
• iMac: Idem.
  • For iMacs from 2007-2009: Unreachable, macOS 10.11 marked the end of their range.
  • Post-2010 iMacs: OK, may run macOS 10.12.
• iPad Mini:
  • iPad Mini 1st Gen: Unreachable, shipped 2012 with iOS 6, may only run upto iOS 9.
  • iPad Mini 2th Gen: OK, shipped 2013 with iOS 7, and may run upto iOS 12 (released 2018, with minor updates until 2023).
• Samsung Galaxy A-series:
  • Samsung Galaxy A5 2015 and earlier: Unreachable, shipped with Android 4, but may run upto Android 6 only.
  • Samsung Galaxy A5 2016: OK, shipped Android 5, may run upto Android 7.
• Samsung Galaxy S-series:
  • Samsung Galary S5: Unreachable, shipped in 2014 with Android 4, but may only run upto Android 6.
  • Samsung Galary S6: OK, shipped in 2015 with Android 5, and may run upto Android 7.
• Motorola Moto
  • Moto G3: Unreachable, shipped 2015 with Android 5, runs upto Android 6 only.
  • Moto G4: OK, shipped 2016 with Android 6, runs upto Android 7.

Desktop and iOS are pretty safe there in terms of age.

The mobile landscape is really showing just how badly Android-based vendors support their devices. The fact that 2016 devices are "fine" is obscuring this somewhat, because all we test for is TLS support. Moto G4 ships in 2016 and is unsupported a mere year later, running only upto Android 7 from 2017. The very next major release doesn't run there. Compare that to an iPad Mini 2, shipping in 2013, and running upto iOS 12, a new major release from 2018, with minor updates until 2023.

[Krinkle]

I briefly deployed the new cer but rolled it back because it did not pass on Safari 7.1 (macOS 10.9 Mavericks) the way we expected.

This seems to be because the new Sectigo cert is now stamping with a significantly newer root cert. Rather than the one rooted in the 2010 USERTRUST cert, it is using a new standalone root from 2021.

Previous certificate from 2024:

• Subject:
  • Common name: *.jquery.com
  • Not Before: Tue, 25 Jun 2024 00:00:00 GMT
  • Not After: Wed, 25 Jun 2025 23:59:59 GMT
  • Algo / Sig: Elliptic Curve 256 / ECDSA with SHA-256
• Issuer: Sectigo Limited
  • Common name: Sectigo ECC Domain Validation Secure Server CA
  • Not Before: Fri, 02 Nov 2018 00:00:00 GMT
  • Not After: Tue, 31 Dec 2030 23:59:59 GMT
• Issuer: The USERTRUST Network
  • Common name: USERTrust ECC Certification Authority
  • Not Before: Mon, 01 Feb 2010 00:00:00 GMT
  • Not After: Mon, 18 Jan 2038 23:59:59 GMT

New certificate:

• Subject:
  • Common name: *.jquery.com
  • Not Before: Thu, 12 Jun 2025 00:00:00 GMT
  • Not After: Fri, 26 Jun 2026 23:59:59 GMT
  • Algo / Sig: Elliptic Curve 256 / ECDSA with SHA-256
• Issuer: Sectigo Limited
  • Common name: Sectigo Public Server Authentication CA DV E36
  • Not Before: Mon, 22 Mar 2021 00:00:00 GMT
  • Not After: Fri, 21 Mar 2036 23:59:59 GMT
• Issuer: Sectigo Limited
  • Common name: Sectigo Public Server Authentication Root E46
  • Not Before: Mon, 22 Mar 2021 00:00:00 GMT
  • Not After: Wed, 21 Mar 2046 23:59:59 GMT

[Krinkle]

Background knowledge

• The private .key and .crt file are specific to our domain and the 1-year validity.
• These are generated and signed by a particular intermediary.
• The ca-bundle file and its stamps are public information, which build a chain of trust from a root your device already trusts, to your short-term domain cert. In theory, if devices could magically come with all current roots and intermediaries, then your pem file could be reduced to just your short-term crt. This isn't the case in practice, and would have various security disadvantages (i.e. in case of a private key breach). So instead there's a buffer of medium-term intermediaries that can come and go, with the root excercised and exposed less often and thus valid for longer.
• These public stamps (PEM) can be retreived from websites like https://crt.sh.

Context

Up until 2024:

• Sectigo issued each new domain cert (the private .crt file) from the "Sectigo ECC …" (2018) intermediary.
• The ca-bundle must thus start with the public key of this 2018 intermediary.
• The 2018 "Sectigo ECC …" intermediary was cross-signed by the 2010 USERTrust/AAA root.
• The ca-bundle included a second stamp by the 2010 USERTrust root that approves this 2018 intermediary.
• Thus with this bundle, our domain cert is trusted by all devices that trust the 2010 AAA root.

As of 2025:

• Sectigo created a new root in 2021 called "Sectigo … Root E46", and a new intermediary called "Sectigo … DV E36". This new intermediary is also cross-signed by the same 2010 AAA root.
• Sectigo now issues new domain certs from the "Sectigo … DV E36" (2021) intermediary.
• The ca-bundle must thus start with the public key of this 2021 intermediary.
• The ca-bundle they give you includes a second stamp by their new 2021 root, approving their own intermediary.
• Thus with this bundle, our domain cert is not trusted by older devices that don't already know about the "Sectigo … Root E46" root.

Their new "Sectigo …Root E46" did make it into all major platforms (Apple, Microsoft, and Debian/Ubuntu, and the Mozilla CA store as popularized by Curl and used in tons of misc contexts). But three years is a fairly timeline to push a new root.

The good news is that they did, on the same day, cross-sign this new "Sectigo …Root E46" intermediate with the AAA root (2010) root as well. I don't understand why they don't include the stamp of that event in the ca-bundle since they obviously made it for this reason, and if you swap it yourself, it works fine.

Credit to Ryan Aslett for figuring this all out, identifying which stamp we need, and where we can download it.

Details at https://www.sectigo.com/knowledge-base/detail/Sectigo-Public-Intermediates-and-Roots/kA0Uj0000003eovKAA
In particular, the Sectigo CA Hierarchy.pdf file at the very end.

The swap

Highlighted in blue are the two sampes that their ca-bundle includes by default. Highlighted in green is the replacement.

The two public stamps they give you in bundle.pem:

• Sectigo … DV E36 > Sectigo … Root E46 | https://crt.sh/?id=4267304693
• Sectigo … Root E46 > USERTrust | https://crt.sh/?id=11405664274

-----BEGIN CERTIFICATE-----
MIIDXz…
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIIDRj…
-----END CERTIFICATE-----

I downloaded the Sectigo … Root E46 > AAA stamp from the "Download Certificate: PEM "link at https://crt.sh/?id=11405664273 and swapped the second stamp with this one.

What we now use instead:

• Sectigo … DV E36 > Sectigo … Root E46 | https://crt.sh/?id=4267304693
• Sectigo … Root E46 > AAA | https://crt.sh/?id=11405664273

-----BEGIN CERTIFICATE-----
MIIDXz…
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIIDyz…
-----END CERTIFICATE-----

After uploading that to Fastly and deploying it to staging (t.sni / code2):

• macOS 10.11 El Capitan (Safari 9.1): Now succeeds instead of failing to verify.
• iOS 9.3 (iPad Mini 4 simulator): Now succeeds instead of failing to verify.
• macOS 10.9 and 10.11: Still failing to connect (due to older connection ciphers). Note that there was no failure to verify trust, it just doesn't connect.

After deploying to production (k.sni / code.jquery.com):

• macOS 10.9 and 10.11: Now also succeed.

Before (2024)	After (2025 with default ca-bundle)	After (2025 with AAA stamp)

[Krinkle]

Lesson for next year: We don't have to wait until k.sni/production to know whether this is going to happen.

There is an important distinction between macOS 10.9-10.10 (Fail to connect) and macOS 10.11 (Fail to verify).

The difference between staging (t.sni/code2) and production (k.sni/code) is only in the connection chiphers. The verification of trust happens after the connection is established. I should have realized that when code2 failed on macOS 10.11 with a trust prompt (rather than a connection error page), that activating it on production would have yielded the same error.

Connection error (i.e due to new TLS version or new connection ciphers):

Trust prompt (i.e. due to cert being from an unknown root authority):