What you’re up to using TLS certs. Next round of certification joy.

Issue: TLS x509 certificate chain verification fails on a particular host for any Let’s Encrypt cert. We’re using certbot for Let’s Encrypt is providing outstanding work for a great goal. This issue is specific to a particular certificate DST Root CA X3. This is, when using gnutls-cli and openssl s_client. Not so on other hosts, though. And all the while the localhost cert itself being perfectly valid.

TL;DR: Update ca-certificates package, deactivate DST Root CA X3 in favor of ISRG Root X1 and rebuild /etc/ssl/certs/ca-certificates.crt. C.f. Solution.

~# gnutls-cli localhost
Processed 121 CA certificate(s).
Resolving 'localhost:443'...
Connecting to '127.0.0.1:443'...
- Certificate type: X.509
- Got a certificate list of 3 certificates.
- Certificate[0] info:
- subject `CN=localhost', issuer `CN=R3,O=Let's Encrypt,C=US', serial 0xdeadbeef, RSA key 2048 bits, signed using RSA-SHA256, activated `2021-08-19 11:56:19 UTC', expires `2021-11-17 11:56:17 UTC', pin-sha256="deadbeef="
[...]
- Certificate[1] info:
- subject `CN=R3,O=Let's Encrypt,C=US', issuer `CN=ISRG Root X1,O=Internet Security Research Group,C=US', serial 0x00912b084acf0c18a753f6d62e25a75f5a, RSA key 2048 bits, signed using RSA-SHA256, activated `2020-09-04 00:00:00 UTC', expires `2025-09-15 16:00:00 UTC', pin-sha256="jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0="
- Certificate[2] info:
- subject `CN=ISRG Root X1,O=Internet Security Research Group,C=US', issuer `CN=DST Root CA X3,O=Digital Signature Trust Co.', serial 0x4001772137d4e942b8ee76aa3c640ab7, RSA key 4096 bits, signed using RSA-SHA256, activated `2021-01-20 19:14:03 UTC', expires `2024-09-30 18:14:03 UTC', pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="
- Status: The certificate is NOT trusted. The certificate chain uses expired certificate. 
*** PKI verification of server certificate failed...
*** Fatal error: Error in the certificate.

Corollary: Even though the ISRG_Root_X1 cert might be present, it is most likely not (yet) included in the compiled /etc/ssl/certs/ca-certificates.crt. This leading to TLS clients trying to verify the certificate chain against DST_Root_CA_X3. Which is bound to fail after september 30 2021.

As always, there’s 2+ ways around this:

Solution:

~# apt update && apt install ca-certificates && \
  sed 's@^!mozilla/ISRG_Root_X1.crt@mozilla/ISRG_Root_X1.crt@g; s@^mozilla/DST_Root_CA_X3.crt@!mozilla/DST_ROOT_CA_X3.crt@g' \
  -i /etc/ca-certificates.conf && \
  update-ca-certificates

You may want to handle this manually:

dpkg-reconfigure ca-certificates -> yes -> activate/select mozilla/ISRG Root X1” -> OK

This updates the system certificates, activates ISRG_Root_X1.crt and deactivates the old mozilla/DST_Root_CA_X3.crt cert.

After these modifcations, testing yields:

~# gnutls-cli localhost
[...]
- Status: The certificate is trusted.

Check.