SSL error when having https enabled with manual certificates in jupyterhub side.

[MarineChap]

Dear,
We have a daskhub server on a supercomputer in our lab. We are based on the latest helm chart ( daskhub-2022.6.0 version 2022.6.1)
https on jupyterhub is enabled through manual certificate and it seems dask cannot success to communicate together.

config :

jupyterhub:
  hub:
    services:
      dask-gateway:
        apiToken: secret1
  proxy:
    https:
      enabled: true
      hosts:
         - 10.86.1.95
      type: secret
      secret:
         name: example-tls
    traefik: 
      networkPolicy: 
        enabled: False


dask-gateway:
  gateway:
    auth:
      jupyterhub:
        apiToken: secret1

Now when I try to create a dask cluster in my jupyterhub session, I am having a ssl error.

from dask_gateway import Gateway
gateway = Gateway()
cluster = gateway.new_cluster() 

Error:

ClientConnectorCertificateError: Cannot connect to host proxy-public:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')]

Note that if I deactivate https in jupyterhub and work only with http everything is running correctly. So, I think there is a bug somewhere when using manual authentification.

Any idea for me ? Thanks

[consideRatio]

I'm always a bit unsure about things related to HTTPS certificates, but here are some guesses.

If you have a HTTPS certificate that you have created yourself (as compared to getting it from commonly trusted Let's Encrypt for example), it is often causing issues like this. Typcially you would need to acquire a HTTPS certificate by a trusted issuers of certificates (Let's Encrypt for example), or, declare that all computers accessing a server (jupyterhub) using HTTPS should trust the self-generated certificate.

I'm not sure there isn't a bug to be fixed or similar, but it seems likeley to me that this failure is expected as the dask-gateway software isn't configured to accept what I understand is a HTTPS certificate - provided by a not commonly trusted source.

---

This is related configuration btw:

helm-chart/daskhub/values.yaml

Lines 21 to 30 in 2e09a8a

	if "PROXY_HTTP_SERVICE_HOST" in os.environ:
	# https is enabled, we want to use the internal http service.
	gateway_address = "http://{}:{}/services/dask-gateway/".format(
	os.environ["PROXY_HTTP_SERVICE_HOST"],
	os.environ["PROXY_HTTP_SERVICE_PORT"],
	)
	print("Setting DASK_GATEWAY__ADDRESS {} from HTTP service".format(gateway_address))
	else:
	gateway_address = "http://proxy-public/services/dask-gateway"
	print("Setting DASK_GATEWAY__ADDRESS {}".format(gateway_address))

---

To conclude, I'm overall confused about the situation etc, and can't say that there isn't a bug or is a bug etc. Logs from various pods and clarifications about what kind of certificate you have provided for use by JupyterHub etc could be relevant.

[MarineChap]

Thanks for your quick answer.
I also not very good with this concept of certificates but from what I understood the certificates are issued by a commonly trusted orga (just without using let's encrypt because of a problem with the name of the computer - space were not accepted something like that) but at the end, it should be the same.

Good you remember this part of the code for me ! I checked before and forget to add this to the issue but it is indeed super important. From the log in my hub, it seems that it does not detect that https is enabled.

Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
Loading extra config: 00-add-dask-gateway-values
Setting DASK_GATEWAY__ADDRESS http://proxy-public/services/dask-gateway
Adding dask-gateway service URL

Maybe I could force it by adding this two variables as extra-config in jupyterhub side but I don't know what value should be expected here. The one from the service proxy-api or another service ?