Enhance SslInfo to support multiple certificate stores.

[JoshuaChen]

Fixed the following issues:

• SslInfo does not display certificates in TrustStore
• SslInfo does not deduplicate duplicate certificates
• SslInfo cannot display certificates in the following scenarios：

spring:
  ssl:
    bundle:
      pem:
        myBundle:
          key:
            alias: abc
          keystore:
            type: jks
            certificate: |
              -----BEGIN CERTIFICATE-----
              MIICjzCCAhWgAwIBAgIQXIuZxVqUxdJxVt7NiYDMJjAKBggqhkjOPQQDAzCBiDEL
              MAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0plcnNl
              eSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNVBAMT
              JVVTRVJUcnVzdCBFQ0MgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTAwMjAx
              MDAwMDAwWhcNMzgwMTE4MjM1OTU5WjCBiDELMAkGA1UEBhMCVVMxEzARBgNVBAgT
              Ck5ldyBKZXJzZXkxFDASBgNVBAcTC0plcnNleSBDaXR5MR4wHAYDVQQKExVUaGUg
              VVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNVBAMTJVVTRVJUcnVzdCBFQ0MgQ2VydGlm
              aWNhdGlvbiBBdXRob3JpdHkwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAAQarFRaqflo
              I+d61SRvU8Za2EurxtW20eZzca7dnNYMYf3boIkDuAUU7FfO7l0/4iGzzvfUinng
              o4N+LZfQYcTxmdwlkWOrfzCjtHDix6EznPO/LlxTsV+zfTJ/ijTjeXmjQjBAMB0G
              A1UdDgQWBBQ64QmG1M8ZwpZ2dEl23OA1xmNjmjAOBgNVHQ8BAf8EBAMCAQYwDwYD
              VR0TAQH/BAUwAwEB/zAKBggqhkjOPQQDAwNoADBlAjA2Z6EWCNzklwBBHU6+4WMB
              zzuqQhFkoJ2UOQIReVx7Hfpkue4WQrO/isIJxOzksU0CMQDpKmFHjFJKS04YcPbW
              RNZu9YO6bVi9JNlWSOrvxKJGgYhqOkbRqZtNyWHa0V1Xahg=
              -----END CERTIFICATE-----
              -----BEGIN CERTIFICATE-----
              MIIG8TCCBdmgAwIBAgIQC6zY5V7vyZvht91G/T1p4DANBgkqhkiG9w0BAQsFADBZ
              MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMTMwMQYDVQQDEypE
              aWdpQ2VydCBHbG9iYWwgRzIgVExTIFJTQSBTSEEyNTYgMjAyMCBDQTEwHhcNMjUw
              NDE3MDAwMDAwWhcNMjYwNDIzMjM1OTU5WjBhMQswCQYDVQQGEwJVUzETMBEGA1UE
              CBMKQ2FsaWZvcm5pYTESMBAGA1UEBxMJUGFsbyBBbHRvMRMwEQYDVQQKEwpWTXdh
              cmUgTExDMRQwEgYDVQQDDAsqLnNwcmluZy5pbzCCASIwDQYJKoZIhvcNAQEBBQAD
              ggEPADCCAQoCggEBAKXJUa0oYMCwufyhZghhMeP+awEewepZhYUA4LH+LWrcBx0Q
              YfYqQuJb3+pLnJ+mFE5IHD5lUOn8Dr35bm2+PI9U2y0Yjz+ayMkkm+5TtsCzo7r9
              kJyoLgvBuW4Ye2FBFevLcjHbQyrTxwl75LyvJgDi3cvdmQ5yz1NR6vg7QUPpLo0G
              VcrHUwB47wvTlDsWSVilCAdXbv3V9zQrg5EGZTHcc7d650PMrLILxkyUa7tj8ftq
              Nwcyp1wB9TcY7EWSCN5uDZm4DME3YOh+q7jng0b2pmdBFXA7xB/2wUcWdM+aeilb
              WF5CpflFl9Iyv08dROmxg1Mew8gvBFw4MFxVaIsCAwEAAaOCA6swggOnMB8GA1Ud
              IwQYMBaAFHSFgMBmx9833s+9KTeqAx2+7c0XMB0GA1UdDgQWBBTgQRe1jnmDOuQO
              W2OVwO61uw7Z+zA5BgNVHREEMjAwggsqLnNwcmluZy5pb4IJc3ByaW5nLmlvghYq
              LmVudGVycHJpc2Uuc3ByaW5nLmlvMD4GA1UdIAQ3MDUwMwYGZ4EMAQICMCkwJwYI
              KwYBBQUHAgEWG2h0dHA6Ly93d3cuZGlnaWNlcnQuY29tL0NQUzAOBgNVHQ8BAf8E
              BAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMIGfBgNVHR8EgZcw
              gZQwSKBGoESGQmh0dHA6Ly9jcmwzLmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydEdsb2Jh
              bEcyVExTUlNBU0hBMjU2MjAyMENBMS0xLmNybDBIoEagRIZCaHR0cDovL2NybDQu
              ZGlnaWNlcnQuY29tL0RpZ2lDZXJ0R2xvYmFsRzJUTFNSU0FTSEEyNTYyMDIwQ0Ex
              LTEuY3JsMIGHBggrBgEFBQcBAQR7MHkwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3Nw
              LmRpZ2ljZXJ0LmNvbTBRBggrBgEFBQcwAoZFaHR0cDovL2NhY2VydHMuZGlnaWNl
              cnQuY29tL0RpZ2lDZXJ0R2xvYmFsRzJUTFNSU0FTSEEyNTYyMDIwQ0ExLTEuY3J0
              MAwGA1UdEwEB/wQCMAAwggF/BgorBgEEAdZ5AgQCBIIBbwSCAWsBaQB3AA5XlLzz
              rqk+MxssmQez95Dfm8I9cTIl3SGpJaxhxU4hAAABlkJtjzMAAAQDAEgwRgIhAJP1
              h8AgtbTMi6ov5aXgO1cTpIhDGSgHlDiWVII+bbkkAiEAnBFPNwHknz0TwAewHR0g
              NafocNTkHmkC7fqckTEihkYAdgBJnJtp3h187Pw23s2HZKa4W68Kh4AZ0VVS++nr
              Kd34wwAAAZZCbY+CAAAEAwBHMEUCIQCquuyCB6mf27Kme9IqkdAaqSxPaJJtbOwt
              2JZ959VO8gIgKHPwo7UnMuS+9kwAfdNJ91pHwAEM52IzcRfzvr5WF8kAdgDLOPcV
              iXyEoURfW8Hd+8lu8ppZzUcKaQWFsMsUwxRY5wAAAZZCbY9+AAAEAwBHMEUCIQCT
              98htkreIeYI7YPLPzvQB3TkaP4vjKYWelF1Jxg92eAIgYCtHyoEJhf+5q53cBOG3
              R66b7moJh/IHRrcSLH2K0JEwDQYJKoZIhvcNAQELBQADggEBAA6TBH9ijZkvtJ9V
              3WzZHmhu8+8Zi/P8oGkrIbwswxFJwloGRKcyekLmvrbSX24ZtYpm8L6kvEsCzxBw
              EAFs6pr2XdedX38bPwwyGVblaQnqkRwGqCldswVDKx0MiintMDVMJ5C87KpVOmO4
              vdNYLczMV7q/WDiPTcSrQ9NDoTBnFLqpS8tCc8V3NysshgNApmQxufmFPUWb/Yly
              5uNlsGCGenSV7ukQwcTNoFtyqig1r4EMs+UbsdJcx7Vyz+TUDWPQ098YymxUN8oV
              SGaS+Wd5A0Ogv2wUSo8CbLhHzafBh2XmsP+c1dFMjvfh5/Pu7MDxK/2tIh8BUnUP
              AAARw1k=
              -----END CERTIFICATE-----

[jonatan-ivanov]

What is the use-case behind displaying truststores (how would you use this information)?

I feel if this is needed, it should be somehow separated from the keystore certs since they have fundamentally different purpose and mixing them could make things worse during troubleshooting.

[JoshuaChen]

Build a dashboard similar to the one below (this is just an example and not the actual product we’ve chosen).

https://grafana.com/grafana/dashboards/11707-ssl-monitoring-x509-cert/
https://docs.netscaler.com/en-us/netscaler-console-service/networks/ssl-certificate-dashboard/how-to-use-ssl-dashboard.html

What you mentioned makes sense, but here’s a concern: if someone is already using the endpoint to fetch data into a monitoring system, would changing the schema cause the API call to fail? I think this would be a breaking change. Are we really going to do this? If it’s confirmed, I’d be happy to contribute to this change.

[mhalbritter]

Right now, the json contains certificateChains which lists the certificate chains contained in the keystore (i think a certificate chain in Java Keystore speak always has a private key attached). That's also the reason why the certificates in your example don't show up: they are certificates without a private key and therefore not a certificate chain.

Maybe there's a usecase to display the certificates in a truststore, too: e.g. I have imported a certificate from my cluster in my truststore to be able to call them with RestClient etc. and I now want to see when it's time to import the renewed certificate in my app.

But in that scenario wouldn't it make more sense to monitor and display the information from the server part (which holds the certificate and also the private key) instead of displaying truststore contents of the client?

[JoshuaChen]

Certificate chain is another meaning. The reason why the certificate in my provided YAML does not display is that the previous logic lacked the logic to obtain the certificate from keyStore.getCertificate(alias). Public certificates also have their own certificate chain relationships. For example, in the screenshot below:

For truststore, your explanation is correct. I put multiple fixes in the same PR, which caused confusion. I have fixed multiple issues.

[mhalbritter]

I was talking about certificate chains in the context of a Keystore object in Java, not about certificate chains in general.

The JavaDoc for getCertificateChain says:

Returns the certificate chain associated with the given alias. The certificate chain must have been associated with the alias by a call to setKeyEntry, or by a call to setEntry with a PrivateKeyEntry.

So it only works on keys, and not, like in your example, on certificates. But why would you put a certificate in a keystore in the first place? Certficates should be put in a truststore.

[JoshuaChen]

Get your point now.
Do you recommend any code changes I should make?
I read the code of PemSslStoreBundle.createKeyStore() and it doesn't distinguish between different stores.

[mhalbritter]

First we need to decide if we want to see certificates from a truststore in the info endpoint. What's your use case? Why do you want to monitor certificates in a truststore? You linked to two dashboards, but as far as i see they show the expiration of the certificate for which the client has the key and not the certificates in the truststore.

[JoshuaChen]

As mentioned in the comment above, trustStore is used on the client side, such as enabling TLS connections with DB or MQ, or end-to-end TLS links based on HTTP protocol, etc. This is a must for some large companies that enable TLS, and even mandated by compliance laws in different countries.

In my personal understanding, all certificates in use should be managed, regardless of whether they are stored in keyStore or trustStore. They are equally at risk of expiration, which can trigger production incidents.

Regarding the dashboard, I cannot directly share the product we are using as it is a trade secret. But as I mentioned, in large companies, we need to manage all certificates uniformly to prevent expiration. Without it, we would lack a global monitoring view.

[mhalbritter]

So you essentially have a dashboard which monitors the server side (e.g. DB server) and shows the key expiration. That makes sense.

And then you have another dashboard which monitors the client side (your app) which shows the expiration of the certificate of the DB server, which is in the truststore of the client?

[JoshuaChen]

Most of the certificates are displayed on the same dashboard, but we will determine their owner based on extended information such as appName and notify them when the certificate needs to be renewed. For e2e TLS certificates, we need to notify all relevant app owners.

Currently, we do not distinguish which store it is in, but it would be better if there were such a field to mark its store.

In high-level management requirements, it does not matter which store it is in. We only need to know its owner and expiration date. Distinguishing the store helps frontline developers, but does not help high-level management requirements.

BTW, we are using Spring Boot as one of our products, but the technology stack of large companies is much more complex.

[jonatan-ivanov]

My two cents:

Dashboards: if I understand correctly, the dashboards linked above (#45355 (comment)) do not need the truststore and the info endpoint should not be used to build such dashboards (for example aggregating the data is problematic). On the other hand, metrics are also provided so you can build such dashboards on that data.

Breaking change: does not need to be I think but the next version is a major version.

Truststore usage and claiming that it is a "must for some large companies": I feel you are using self-signed certificates. Having a custom truststore is definitely not a must regardless of company size (but you can have a cert that is issued by a trusted CA and the issuer cert is in the default truststore of Java).

In my personal understanding, all certificates in use should be managed, regardless of whether they are stored in keyStore or trustStore. They are equally at risk of expiration, which can trigger production incidents.

Could you please mention a use-case where the server cert chain is not expired but the truststore chain is? Should not the two be the same (since the issuer cert is the same)? If the server cert is expired, it should not matter what is in the truststore or does it?

But as I mentioned, in large companies, we need to manage all certificates uniformly to prevent expiration.

I'm not sure, see my previous point (regardless of company size).

Most of the certificates are displayed on the same dashboard, but we will determine their owner based on extended information such as appName and notify them when the certificate needs to be renewed. For e2e TLS certificates, we need to notify all relevant app owners.

I think this is the cause of the disconnect. It seems you are using the truststore for client-discovery. I feel this is a narrow use-case and I don't think the SslInfoContributor should be used for this purpose (unless I'm missing something); from what I understoond so far, this seem to be quite error-prone, e.g, what if someone:

• Uses -Djavax.net.ssl.trustStore=... and not the ssl bundles?
• Uses the default truststore and disables client-side verification ?
• Only have the issuer chain in the truststore and not the terminating cert?
  (This is the case of the default truststore.)

[JoshuaChen]

you can try to understand the following two cases? Not all scenarios are supported by public CA certificates, and the default truststore of Java is not enough. I think you may have oversimplified the scenario.

https://www.baeldung.com/spring-boot-kafka-ssl
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html

More scenarios are that we have dozens or hundreds of apps linked to DB, MQ, etc. via TLS. When the server's certificate expires and is renewed, we need to know how many client certificates need to be updated at the same time to prevent the server from being updated and the client forgetting to update the certificate, resulting in a mismatch and inability to connect to the server.

We occasionally encounter situations where client certificates are forgotten to be updated because there are too many client services involved. We need a global dashboard to monitor all certificates, regardless of which store they belong to.

Uses -Djavax.net.ssl.trustStore=... and not the ssl bundles?

When we use -Djavax.net.ssl.trustStore=..., it cannot be monitored by the Spingboot ssl endpoint through SslInfoContributor. This is unmanageable, and I don't think this is what we want to see.

Uses the default truststore and disables client-side verification ?
Only have the issuer chain in the truststore and not the terminating cert?

I think we enhance security by enabling TLS (and it is mandated by compliance laws in some countries). Trusting all certificates or only trusting the issuer is unacceptable, as it poses security risks. If this were really possible, we could just use HTTP. Why go through so much "trouble"?

[jonatan-ivanov]

you can try to understand the following two cases? Not all scenarios are supported by public CA certificates, and the default truststore of Java is not enough. I think you may have oversimplified the scenario.

I can try. :) It seems the baeldung post (as every tutorial you can follow in your local dev environment) uses self-signed certs for convenience. This is so that you can try it out locally without getting (paying for) a cert from a trusted CA. This is only for local development purposes, maybe the post should have also called out that in prod self-signed certs are less fortunate.
I'm not sure I get what you want to tell me with the AWS doc. Could you please point to the section you want me to focus on? Amazon is a trusted CA, are you saying they don't issue you a trusted cert for RDS and you need to do tricks with the truststore?

When the server's certificate expires and is renewed, we need to know how many client certificates need to be updated at the same time to prevent the server from being updated and the client forgetting to update the certificate, resulting in a mismatch and inability to connect to the server.

Yes, this is what I referred above as "client-discovery". I still feel this is a narrow use-case. I think knowing the truststore(s) contents is a useful feature in some situations (self-signed certs) and maybe it should be added to SslInfoContributor but I feel that feature is broader than the ssl bundles.

When we use -Djavax.net.ssl.trustStore=..., it cannot be monitored by the Spingboot ssl endpoint through SslInfoContributor. This is unmanageable, and I don't think this is what we want to see.

I think it can and it is manageable, please explain why you think otherwise.

Trusting all certificates or only trusting the issuer is unacceptable, as it poses security risks.

I agree but this does not prevent people doing it.

[JoshuaChen]

Regarding self-signed certificates

I try to answer two questions: one is that the default certificate in the Java JDK cannot meet the required scenarios, and the other is that the requirements I mentioned are not unique to my company.

Let me continue to use AWS as a case. The default certificate in the Java JDK does not include the AWS RDS CA certificate. The JDK only contains the CA certificates mentioned on the page, so when RDS TLS is enabled, we need to additionally add the AWS RDS CA certificate to the truststore.

However, AWS, as a sufficiently generic cloud service provider, does not have its certificate as a self-signed certificate for the company I serve. I believe this case is sufficiently marketable.

Regarding javax.net.ssl.trustStore and "client-discovery".

I think we are starting to turn the topic into "How does Spring Boot define a pattern to manage the certificates in use?".

First, using javax.net.ssl.trustStore to load certificates instead of ssl.bundle cannot be monitored in the Spring Boot ssl endpoint. If I want to know what certificates are being used in the APP, I need to introduce more monitoring products, which complicates the implementation.

I think the ssl endpoint method is a simple and convenient way.

Regarding how to get more people to use ssl.bundle instead of javax.net.ssl.trustStore, we need to add more checkpoints around Spring Boot (I am not saying this is within the scope of Spingboot) to intercept the way that javax.net.ssl.trustStore is not allowed to load certificates so that the certificates can be monitored.

If Spingboot is willing to do more, it may be possible to check whether javax.net.ssl.trustStore is used to load the certificate at startup. If so, give a prompt to suggest that developers use the Spring Boot pattern.