browZer support - 3rd party server certs for controller

[qrkourier]

This enables the controller to handle edge requests from agentless browZer clients which are never pre-configured with a trust bundle, and only trust 3rd party server certs.

It's a feature of the controller to present an alternative server cert when the edge client is requested by a distinctive domain name (SNI selected).

This issue tracks progress on this piece of the browZer puzzle, and is only necessary because no one has yet used browZer with a Ziti network controller hosted in K8s.

Resolving this issue may be as simple as documenting a way to configure the controller's alt certs with Helm input values.

This is related to the router issue to support browZer.

[qmugnier]

Hi, I worked on 3rd party topics for controller edge services, in order to be able to run terraform using the restapi from anywhere (actually my utimate goad is to create a stack using aws service catalog and terraform runtime engine). Long story short, here are the things I have done and what must be changed in the current controller chart to get it working out of the box:

I assme you have already generated a certificate for the edge url using for instance let's encrypt (In my case i am using the edge controller nginx ingress to trigger certifcate generation through certificate manager letsencrypt prod issuer, even though i will eventually use ssl passthough):

1. Add a section n your helm values to mount the certificate related secret to the controller, for instance:

               additionalVolumes:
                - name: zero-controlleredge-tls
                  volumeType: secret
                  secretName: ziti-controlleredge-tls
                  mountPath: /etc/ziti/web-identity-public/

2. As the config map template does not currently support a custom mapping for the web secret you need to update configmap.yaml in the web section to point to the right files, for instance:

    web:
      - name: client
        bindPoints:
          - interface: 0.0.0.0:1280
            address: edge-controller.ziti.mypublicdomain.com:443
        identity:
          cert:        /etc/ziti/web-identity/tls.crt
          server_cert: /etc/ziti/web-identity-public/tls.crt ####Modified part
          key:         /etc/ziti/web-identity-public/tls.key   ####Modified part

It would be great to propose a way to pass these path using helm values with a if block => Can this enhancement be made?

3. Last but not least, if you have installed trust manager using the controller chart, there is no bundle available with the defaultCAs, i had to patch the ctrl-plane-cas bundle to include them, that would be great to have a helm value option for that as well. Here is my modified bundle:

kind: Bundle
metadata:
  generation: 2
  name: ctrl-plane-cas
spec:
  sources:
    - useDefaultCAs: true      ### New line 
    - secret:
        key: ca.crt

[qrkourier]

That sounds interesting, @qmugnier!

Yes, let's improve this controller chart so that it works with browZer. The router chart, too, needs to support wss:// listeners with an alt cert from public CA.

Is this an accurate recap of the customizations you needed to make to the controller chart?

1. Add an item to the existing list additionalVolumes for the alt cert from a public CA, e.g., LetsEncrypt. You used a Cert Manager Issuer or ClusterIssuer to obtain and renew the public cert.
2. You modified the controller's config.yml template to add the identity properties pointing at the mountpoint of the public cert
3. You modified the Trust Manager Bundle resource where the controller's root certs are aggregated to also include some additional CA cert, perhaps the root cert from the public CA?

This is pretty close to how it should work. The main difference from what's needed is that the public cert needs to be configured as an alternative certificate, not the main cert.

The web listener named "client" in your sample has a binding for the client API. That's where the Ziti BrowZer Runtime (ZBR) connects when it's running inside the user's web browser. It needs a public cert because it's not pre-configured to trust any of Ziti's root CAs.

Still, the client API normally presents a server cert from a private CA, e.g., the ziti-controllerweb-root-issuer (CM Issuer) provided by the chart. The public cert is normally configured as an alternative cert like this.

    web:
      - name: client
        bindPoints:
          - interface: 0.0.0.0:1280
            address: edge-controller.ziti.mypublicdomain.com:443
        identity:
          cert:        /etc/ziti/web-identity/tls.crt
          server_cert: /etc/ziti/web-identity/tls.crt
          key:         /etc/ziti/web-identity/tls.key
          ca:          /etc/ziti/web-identity/ca.crt
          alt_server_certs:
              - server_cert: /etc/ziti/web-identity-public/tls.crt
                server_key: /etc/ziti/web-identity-public/tls.key

This avoids the need to add the public CA root cert to the trust Bundle because alternative certs are presumed to be from a CA we don't control. Adding a root CA cert from a CA you don't control to the trust Bundle is a security risk. Anyone who can get a cert from that CA can bypass the outer TLS security layer of the Ziti network!

/etc/ziti/web-identity-public/tls.crt must have a DNS SAN that is distinct from /etc/ziti/web-identity/tls.crt. That's how the controller decides which server cert to present.

EDIT: YAML syntax error

[qmugnier]

This sounds even better. To be clear, is it somethingto be integrated into the chart, or is the alt_server_certs already supported by the chart ? (I cannot see anything related to that in the template).

[qrkourier]

It's a feature of the controller. The chart needs new template values to allow setting alt cert on the web identity, at least.

[qrkourier]

We will also need the alt cert template value in router chart for wss: #114

[qmugnier]

Got it now, it does say in the doc that the web section is powered by Xweb (same in controller and router) and that there is a
alt_server_certs section: https://openziti.io/docs/reference/configuration/conventions#xweb

I tried that but now when I hit the swagger url (https://edge-controller.public.domain.org/edge/management/v1/swagger.json), I am back to the tls error (unkwown CA) and my terraform fails again:

Are you sure this has to be done that way at the controller level?

If I look at the old documentation (Not sure why it was taken down) from archive.org, it says that the certificate must be replaced in the identity section, https://web.archive.org/web/20230908193528/https://openziti.io/docs/guides/alt-server-certs/ and that the router web section must have the alt_server_certs section as you mentionned before.

Any thougths ?

[qmugnier]

I read your answer again I think I did not get this part right:

/etc/ziti/web-identity-public/tls.crt must have a DNS SAN that is distinct from /etc/ziti/web-identity/tls.crt. That's how the controller decides which server cert to present.

I then removed the public address from ziti-controller-web-identity-cert (this would have to be supported by the chart as well). Seems to be working for the swagger url, but when I use my terraform project I get an invalid tls/encrypt error.
In my provider restapi, what shall the cacerts_string be then? Let's encrypt or the ziti native CA?

For now, I am using the archive.org method and it is working fine, but as soon as the public CA solution works I will migrate to it.

[qrkourier]

The main problem with that archived approach was it creates a vulnerability in the network. The CA that issues the primary cert must be trusted by the entire Ziti network, so it's essential to control that CA (not use a public CA).

That's why the alt_server_certs was invented, so we can present a cert from a CA we don't control, like LetsEncrypt.

The controller and router charts still need to be updated to support this. I'm taking a look now to see if I can include the improvement in a batch of changes I'm working. I'll update this issue.

[qrkourier]

For the TF provider, cacerts_string is the PEM bundle of trusted CA certs provided by the Ziti controller to clients during enrollment. Here's a shell example of fetching the bundle.

curl -sSkf https://localhost:1280/.well-known/est/cacerts \
| base64 -d \
| openssl pkcs7 -inform DER -outform PEM -print_certs

[marvkis]

Hi all ;)

I've just studied your discussion and I have some thoughts on it as well.

Regarding the alt-server / "Let's Encrypt cert":
The Let's Encrypt cert could be a regular use case. But the Let's Encrypt cert only has a life of 3 months. Cert-Manager handles this - but how can we update the cert on the controller/router without restarting / redeploying it? AFAIK kubernetes doesn't currently have support for updating mounted secrets like it does for mounted config maps - so just watching the file for changes isn't enough. So we need to use something like https://github.com/stakater/Reloader or a sidecar to watch the secret, and if it changes: export it to a local file and trigger a reload (restart?) on the controller/router?

Alternative approach:
My current understanding is that there is no client cert auth on the wss router port. What do you think of the idea of having the cert be handled by the ingress controller and just forwarding the payload? The router could even provide unencrypted websocket support for this.
But I'm not sure how things work for the communication between Browzer and controller. I've seen the ZITI_CONTROLLER_HOST / ZITI_CONTROLLER_PORT / NODE_EXTRA_CA_CERTS configuration for Browzer. Is all communication with the controller done through Browzer? If so, wouldn't it be possible to set NODE_EXTRA_CA_CERTS to Ziti's internal CA? AFAIK it's not easy to install a client cert on 'the fly' without user interaction. So I'm assuming that even if the client's browser connects directly to the controller, there will be no client cert auth... 🤔

Bye,
Chris

[qmugnier]

Reloader seems the way to go.
I think the client browser with the browZer component communicates with the Browzer router and the controller at any point of time (and the Browzer router and the controller as well). So I assume encryption must be set at all levels.

[qrkourier]

This is important and will be addressed soon.

Here's a recap of the issues related to supporting Ziti BrowZer Runtime (ZBR) clients with K8s-deployed Ziti controllers.

browZer's value proposition is agentless Ziti. This means the user can visit a Ziti-protected web app in a normal web browser without installing additional software. Therefore, any solution mustn't introduce any requirements for additional software or out-of-band configuration, such as adding a root CA to the browser's trust store.

The Ziti controllers and routers must present trusted server certs wherever ZBR clients connect because a normal web browser cannot be configured to trust a different root CA from within the Javascript runtime sandbox.

The Ziti controller and router configurations support alt_server_certs. This is a feature of the conventional identity configuration (link to reference). This is where the trusted server cert must be configured for routers' WebSocket listeners. The router must negotiate TLS directly with ZBR clients for mTLS to succeed.

The controller's trusted server certificate may or may not be bound directly to the client-management API's identity because mTLS is never used by ZBR clients with the client API, only server TLS. Therefore, the client API may be accessed through a reverse proxy that presents a trusted server certificate, e.g., an Ingress Controller w/ Cert Manager issuer.

[qmugnier]

Hi, quick update on the issue.
I followed all the steps explained by @qrkourier :

• added alt_server_certs to the web block

   web:
     - name: client
       bindPoints:
         - interface: 0.0.0.0:1280
           address: edge-controller.ziti.mypublicdomain.com:443
       identity:
         cert:        /etc/ziti/web-identity/tls.crt
         server_cert: /etc/ziti/web-identity/tls.crt
         key:         /etc/ziti/web-identity/tls.key
         ca:          /etc/ziti/web-identity/ca.crt
         alt_server_certs:
             - server_cert: /etc/ziti/web-identity-public/tls.crt
               server_key: /etc/ziti/web-identity-public/tls.key

• Removed the reference to edge-controller.ziti.mypublicdomain.com and regenerated the certificate for /etc/ziti/web-identity/tls.crt as it must have a DNS SAN that is distinct from /etc/ziti/web-identity-public/tls.crt, in other words:
  • /etc/ziti/web-identity-public/tls.crt: DNS san for edge-controller.ziti.mypublicdomain.com:443
  • /etc/ziti/web-identity/tls.crt: DNS san for controller.zitidomain:443 and absolutely NO DNS san for edge-controller.ziti.mypublicdomain.com:443

At first, I thought everything worked fine until I tried to enroll a router.
I got the following error:
[ 1.019] FATAL ziti/router/enroll.(*RestEnroller).Enroll: {cause=[token signature is invalid: crypto/rsa: verification error]} failed to parse JWT

After looking for a root cause I ended up finding this issue that seems very similar:
openziti/ziti/issues/119.

Just in case, I added edge-controller.ziti.mypublicdomain.com:443 to /etc/ziti/ctrl-identity/tls.crt (the certs identity block for the controller), but that did not help, same error.
As I cannot enroll any router, I have to stick to this method.

Am i missing something like an option in the enroll process?

[qrkourier]

The token's signature couldn't be verified by the router during enrollment. It's signed by the key that's configured for the controller's web listener where client-management (i.e., the client API) is bound.

You can inspect the router's enrollment token (type erott) to understand why the signature could not be verified.

The token includes a signed claim iss representing the issuer URL, the URL to the client API. You can fetch the pubkey from the server certificate presented at that URL from the router's perspective and independently verify the token.

Here's a Python script that automates this process. If you run it on the router host that's attempting to enroll you should see that signature verification failed.

https://gist.github.com/qrkourier/b9cacf765b2d62817672bc7e6be6bdc3

ziti-jwt.py /tmp/router1.jwt

1. check the token type is indeed erott (edge router one time token), not ott, which is not a router token

2. check the signature was verified

3. if not verified, troubleshoot by supplying a different pubkey to find the key that did sign the token

   ziti-jwt.py ./router1.jwt ./client-api-pubkey.pem

[marvkis]

Hi,

I got the things up & running. And for the controller, I just added an additional ingress rule:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  name: ziti-controller-ingress-alt-client
  namespace: openziti
spec:
  ingressClassName: nginx
  rules:
  - host: clients.browzer.my.domain
    http:
      paths:
      - backend:
          service:
            name: ziti-controller-client
            port:
              number: 443
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - clients.browzer.my.domain
    secretName: default-nginx-cert

Then I told browzer to use this name for the controller. The controller itself is not aware of it's additional name - but it seems work ;)

[qrkourier]

Yes, it will work for BrowZer (ZBR) clients even if the controller is not aware of the DNS name those clients are using. It's because they never negotiate mTLS with the controller, only routers, so it's perfectly fine to reverse proxy the controller API.

[qmugnier]

@marvkis: this is not exactly my use case. I need ssl passthrough at the ingress level.

[qmugnier]

@qrkourier : I followed the procedure and here are the results.
Result from python:
c:\Users\MSmith>python ziti-jwt.py .\Downloads\test-router.jwt

{
    "header": {
        "alg": "RS256",
        "kid": "b5b45cec76ae94682d4bb5477caf1ea9012c664d",
        "typ": "JWT"
    },
    "payload": {
        "iss": "https://edge-controller.mypublicdomain.com:443",
        "sub": "pmPbC0C30",
        "aud": [
            ""
        ],
        "exp": 1720080190,
        "jti": "3c6ea49d-87ee-47dd-8d8c-35e7b6fb4c31",
        "em": "erott",
        "ctrls": null
    },
    "analysis": {
        "signature_valid": false,
        "enrollment_method": "one-time token for a router",
        "expiration": "2024-07-04T17:03:10"
    }
}

=> Invalid

And then inspected the content of server_cert: /etc/ziti/web-identity/tls.crt (which does not have any reference to edge-controller.mypublicdomain.com as discussed before) to extract the public pem and reran the script:
c:\Users\MSmith>python ziti-jwt.py .\Downloads\test-router.jwt .\pubkey.pem

{
    "header": {
        "alg": "RS256",
        "kid": "b5b45cec76ae94682d4bb5477caf1ea9012c664d",
        "typ": "JWT"
    },
    "payload": {
        "iss": "https://edge-controller.mypublicdomain.com:443",
        "sub": "pmPbC0C30",
        "aud": [
            ""
        ],
        "exp": 1720080190,
        "jti": "3c6ea49d-87ee-47dd-8d8c-35e7b6fb4c31",
        "em": "erott",
        "ctrls": null
    },
    "analysis": {
        "signature_valid": true,
        "enrollment_method": "one-time token for a router",
        "expiration": "2024-07-04T17:03:10"
    }
}

=> Valid
So it seems like it is not hitting the right certificate (Which would make sense as the right dns is now hold by the let's encrypt certificate). Am I supposed to set up something special for the certificates related to token signature?

[qrkourier]

You must configure the controller's client API advertisement to enrollers in edge.api.address to match any DNS SAN from the web-identity server certificate for this to work. At least one web binding for edge-client must have a matching address to allow the controller to determine which key signs the enrollment token.

The alt_server_certs (web-identity-public) cert is for non-enrolling clients that are not configured to trust Ziti's PKI and require a public cert, like BrowZer ZBR clients.

It may be a bug that the controller allowed this configuration because it will never succeed since the issuer URL in the claim always gets the wrong cert.

Clarification: it's the correct cert by SNI selection, but it's not the cert signed by the key that also signed the enrollment token, so enrollment always fails.

[qmugnier]

@qrkourier : makes perfectly sense, thank you.
So I created a new web endpoint dedicated to enrollment, for instance enroll-controller.ziti.mypublicdomain.com:443 and used that address in the edge.api.address and finally deployed a new ingress without a public certificate obviously.
So I think that things are working as expected now.