Sign in Subscribe

Troubleshooting a broken certificate chain

Troubleshooting a broken certificate chain

Goodday my fellow internet dweller. Thanks for joining me. Today we're troubleshooting a rather 'strange' issue. We can open the website of a testing environment in a browser and it's secured by SSL. One of the connecting parties, however, tells us they can't validate our chain and thus fail to connect to us securely.

How odd. This means a browser and a third-party application validate the chain differently. Let's validate his claim: 

➜  ~ openssl s_client -connect stb.acc1.bxxxxxr.nl:443
Connecting to 144.43.244.129
CONNECTED(00000005)
depth=0 C=NL, L=Den Haag, O=Logius, CN=stb.acc1.bxxxxxr.nl
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C=NL, L=Den Haag, O=Logius, CN=stb.acc1.bxxxxxr.nl
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 C=NL, L=Den Haag, O=Logius, CN=stb.acc1.bxxxxxr.nl
verify return:1
---
Certificate chain
 0 s:C=NL, L=Den Haag, O=Logius, CN=stb.acc1.bxxxxxr.nl
   i:C=IE, O=DigiCert Ireland Limited, CN=DigiCert G2 TLS EU RSA4096 SHA384 2022 CA1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar  5 00:00:00 2024 GMT; NotAfter: Mar  4 23:59:59 2025 GMT
 1 s:C=NL, O=QuoVadis Trustlink B.V., CN=QuoVadis Europe SSL CA G2
   i:C=BM, O=QuoVadis Limited, CN=QuoVadis Root CA 2 G3
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jul 22 18:33:11 2020 GMT; NotAfter: Jul 20 18:33:11 2030 GMT
 2 s:C=BM, O=QuoVadis Limited, CN=QuoVadis Root CA 2 G3
   i:C=BM, O=QuoVadis Limited, CN=QuoVadis Root CA 2 G3
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jan 12 18:59:32 2012 GMT; NotAfter: Jan 12 18:59:32 2042 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
(...)

Hold on. The first certificate in the list, our "claim" was issued by DigiCert G2 TLS EU. But the intermediate certificate the server sent, the certificate who is supposed to validate our claim and "glue" it to the CA, is QuoVadis Europe SSL CA G2, which validates against QuoVadis Root CA 2 G3. Yup, that's a broken chain right there. And yet it validates in the browser. What's happening there?

Digging down

The basic mental model of a certificate chain, for crypto-muggles like me, looks a bit like this:

Certificate 1:
Subject: "My fancy website"
Issuer: "Cool provider 1" ---.
                             | Trust 1
Certificate 2:                |
Subject: "Cool provider 1" <-'
Issuer: "Super certificate authority" ---.
                                        |
"Certificate 3:                          | Trust 2
Subject: "Super certificate authority"<--'
Issuer: "Super certificate authority"

It's a linked list, in which the issuer for each certificate should match with the subject of the issueing party. In our case, it doesn't match. This means it's kaput. But the browser still validates the certificate. How?

In firefox, I click the little lock icon, and dig through the SSL settings - and when I arrive at the certificate chain, I'm presented with this:

An entirely different chain! And, as you can see, the intermediate and root CA are actually correct for the certificate the server offered us.

The reason for this is simple. Browsers use their own truststore. It is unclear to me if they fall back to their own truststore once "Trust 1" fails, or if they do it all the time. But there you have it. Broken chain, schmoken chain, according to firefox.

Fixing the broken certificate chain

Seeing as the endpoint we offer up is used by automated processes and not as much by people opening it in their browsers, we get to work.

I know what apache configuration file we need, so I ssh into the server and take a look at where I can find the offending chain.

[therder@stb-a1]$ sudo cat /etc/httpd/conf.d/proxy.conf | grep -i 'SSLCertificate'

SSLCertificateFile /etc/pki/tls/certs/stb.acc1.bxxxxxr.nl.crt
SSLCertificateKeyFile /etc/pki/tls/certs/stb.acc1.bxxxxxr.nl.key
SSLCertificateChainFile /etc/pki/tls/certs/TRAIL.pem

Alright. Let's verify our certificate.

[therder@stb-a1]$ openssl verify /etc/pki/tls/certs/stb.acc1.bxxxxxr.nl.crt

/etc/pki/tls/certs/stb.acc1.bxxxxxr.nl.crt: C = NL, L = Den Haag, O = Logius, CN = stb.acc1.bxxxxxr.nl
error 20 at 0 depth lookup:unable to get local issuer certificate

We succeeded in failing the verification.

Let's take a look at the certificate for this server.

[therder@stb-a1]$ openssl x509 -noout -subject -issuer -in /etc/pki/tls/certs/stb.acc1.bxxxxxr.nl.crt

subject= /C=NL/L=Den Haag/O=Logius/CN=stb.acc1.bxxxxxr.nl
issuer= /C=IE/O=DigiCert Ireland Limited/CN=DigiCert G2 TLS EU RSA4096 SHA384 2022 CA1

This certificate is vouched for by DigiCert G2 TLS EU. We should offer up this intermediate certificate in order to get a valid chain. What are we currently offering up?

[therder@stb-a1] $ openssl crl2pkcs7 -nocrl -certfile /etc/pki/tls/certs/TRAIL.pem | openssl pkcs7 -print_certs -noout

subject=/C=NL/O=QuoVadis Trustlink B.V./CN=QuoVadis Europe SSL CA G2
issuer=/C=BM/O=QuoVadis Limited/CN=QuoVadis Root CA 2 G3

subject=/C=BM/O=QuoVadis Limited/CN=QuoVadis Root CA 2 G3
issuer=/C=BM/O=QuoVadis Limited/CN=QuoVadis Root CA 2 G3

Not the right intermediate for our certificate. Let's fix this.

Creating a certificate chain

When offering up a chain, you only need to go as far as including, in order, the list of intermediates. Sending the root CA is optional.

When you want to manually verify the chain, however, you'll need the root CA as well. Luckily for us we can grab the intermediate and root .pem files straight from firefox when inspecting the certificate!

Having saved these certificates as root.pem and intermediate.pem , we can now check if we have all the components for a validating chain.

[therder@stb-a1]$ openssl verify -CAfile root.pem -untrusted intermediate.pem /etc/pki/tls/certs/stb.acc1.bxxxxxr.nl.crt

/etc/pki/tls/certs/stb.acc1.bxxxxxr.nl.crt: OK

Alright! Let's pour them, in the correct order, into a big happy chain. First we convert the .crt file into .pem format for easy of use.

[therder@stb-a1]$ openssl x509 -in /etc/pki/tls/certs/stb.acc1.bxxxxxr.nl.crt -out mycert.pem -outform PEM

Then we cat them, in order, into our very own TRAIL.pem file (from the apache config above, remember?). Note that the root.pem is optional, and doesn't need to be included. The reason I'm doing it is so I can automate the checking and refreshing of the chain in the future.

[therder@stb-a1]$ cat mycert.pem intermediate.pem root.pem > TRAIL.pem

And verify that this chain works.

[therder@stb-a1]$ openssl crl2pkcs7 -nocrl -certfile TRAIL.pem | openssl pkcs7 -print_certs -noout

subject=/C=NL/L=Den Haag/O=Logius/CN=stb.acc1.bxxxxxr.nl
issuer=/C=IE/O=DigiCert Ireland Limited/CN=DigiCert G2 TLS EU RSA4096 SHA384 2022 CA1

subject=/C=IE/O=DigiCert Ireland Limited/CN=DigiCert G2 TLS EU RSA4096 SHA384 2022 CA1
issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root G2

subject=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root G2
issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root G2

Yay! Now, let's move this thing into position, restart apache and verify that we've fixed the issue.

mv TRAIL.pem /etc/pki/tls/certs/TRAIL_FIXED.pem
sudo ln -sf /etc/pki/tls/certs/TRAIL_FIXED.pem /etc/pki/tls/certs/TRAIL.pem
sudo systemctl restart httpd

And on our local machine, let's grab the SSL chain once more..

openssl s_client -connect stb.acc1.bxxxxxr.nl:443

Connecting to 144.43.244.129
CONNECTED(00000005)
depth=2 C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root G2
verify return:1
depth=1 C=IE, O=DigiCert Ireland Limited, CN=DigiCert G2 TLS EU RSA4096 SHA384 2022 CA1
verify return:1
depth=0 C=NL, L=Den Haag, O=Logius, CN=stb.acc1.stb.acc1.bxxxxxr.nl.nl
verify return:1
---
Certificate chain
 0 s:C=NL, L=Den Haag, O=Logius, CN=stb.acc1.stb.acc1.bxxxxxr.nl.nl
   i:C=IE, O=DigiCert Ireland Limited, CN=DigiCert G2 TLS EU RSA4096 SHA384 2022 CA1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar  5 00:00:00 2024 GMT; NotAfter: Mar  4 23:59:59 2025 GMT
 1 s:C=NL, L=Den Haag, O=Logius, CN=stb.acc1.bsn-koppelregister.nl
   i:C=IE, O=DigiCert Ireland Limited, CN=DigiCert G2 TLS EU RSA4096 SHA384 2022 CA1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar  5 00:00:00 2024 GMT; NotAfter: Mar  4 23:59:59 2025 GMT
 2 s:C=IE, O=DigiCert Ireland Limited, CN=DigiCert G2 TLS EU RSA4096 SHA384 2022 CA1
   i:C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root G2
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA384
   v:NotBefore: Sep 19 00:00:00 2022 GMT; NotAfter: Sep 18 23:59:59 2032 GMT
 3 s:C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root G2
   i:C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root G2
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug  1 12:00:00 2013 GMT; NotAfter: Jan 15 12:00:00 2038 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----

(...omitted for brevity...)

    Start Time: 1710154381
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---
closed

Voila! A working chain!

Hope this braindump was useful to you. Writing this certainly cleared up things for me. For one, I didn't know browser truststores did the fallback thing. I also didn't know they contained intermediaries. You live, you learn. Until next time!