Configuring Vouch Proxy with self hosted GitLab on FreeBSD (including SSL via LetsEncrypt)

[YetAnotherBugHunter]

Server OS: FreeBSD dev.ccr.net 12.2-RELEASE-p2 FreeBSD 12.2-RELEASE-p2 663e6b09467(HEAD) TRUENAS amd64
Nginx: nginx version: nginx/1.18.0
vouch-proxy:

gist with the config.yaml, vouch-proxy.log and nginx.conf

Describe the problem

Setup two FreeBSD jail, csserver.ccr.net and gitlab.ccr.net (this is an instance of gitlab-ce) within private network ccr.net. Note ccr.net is not the real domain name.

In the gitlab.ccr.net jail I:

• Added an application to gitlab with the callback url set to https://vouch.ccr.net/auth

Within the csserver.ccr.net jail I did the following:

• Built and installed the latest release of vouch-proxy
• Installed nginx 1.18 and added a server block for csserver.ccr.net and vouch.ccr.net using this template as a starting point.
• Installed the protected app, code-server, and configured it to run on port 8080

In testing mode I see the following behavior:

• validate : no jwt found in request
• login http://vouch.ccr.net/login: 400 bad request
• Click on the gitlab link and I'm take to the gitlab's login page, login and the authorise the access and am taken back to the vouch proxy page. Clearly the gitlab authorization is working
• Click on the validate link: no jwt found in request

In normal mode when I go to the https://csserver.ccr.net, I'm taken to the gitlab login page. After successful login I'm presented with the gitlab authorize request page and upon clicking the authorize button, nothing happens other than the url changing to:

https://gitlab.ccr.net/oauth/authorize?client_id=REDACTED&redirect_uri=https%3A%2F%2Fvouch.ccr.net%2Fauth&response_type=code&scope=openid+email+profile&state=ImjII9xwyErrP6VNxaCpP2fmUB9u0kv4

I see the same behavior with firefox, chrome and edge.

Expected behavior
If all went well, I was expecting to see code-server's, the protected application, page displayed.

Desktop (please complete the following information):

• OS: Windows 10
• Browser firefox
• Version 86.0 (64-bit)
• Browser Chrome Version
• Version 88.0.4324.182 (Official Build) (64-bit)
• Browser Edge
• Version 44.18362.449.0

Additional context

None

[bnfinet]

the VP config shows vouch.domains as...
- ccr.net.net

I'm guessing that's a typo. If you set domain you shouldn't need to set cookie.domain.

Your log never shows a request returning the /auth endpoint. Why not?

Without the call to /auth there will never be a VP token/cookie issued.

[bnfinet]

p.s. @YetAnotherBugHunter welcome back :)

Glad to see it running on FreeBSD. Let's get you up and running.

[YetAnotherBugHunter]

Thanks for the prompt reply.

the VP config shows vouch.domains as...
- ccr.net.net
I'm guessing that's a typo.

Yes that is a typo that was introduced when I sanitized the configuration file.

If you set domain you shouldn't need to set cookie.domain.

Removed it from the VP config.yaml and retested with no noticeable difference.

Your log never shows a request returning the /auth endpoint. Why not?

That is the $64K question!
Could it be with the nginx configuration? This is my first exposure to nginx, so it's pretty much greek to me.

[bnfinet]

You're going to run into #309 until #367 is merged.

The Nginx conf looks pretty straight forward.

Is Gitlab setup to return to vouch.ccr.net/auth? Usually that's established when you register Vouch Proxy to get the clientid and clientsecret issued.

https://gitlab.ccr.net/oauth/authorize?client_id=REDACTED&redirect_uri=https%3A%2F%2Fvouch.ccr.net%2Fauth&response_type=code&scope=openid+email+profile&state=ImjII9xwyErrP6VNxaCpP2fmUB9u0kv4

If gitlab is just returning you to gitlab without ever forwarding you to VP then it seems like it's probably a gitlab issue.

[YetAnotherBugHunter]

The Nginx conf looks pretty straight forward.

Do I need to enter a /auth location in the vouch server block?

Gitlab is configure as shown below. The Application ID and Secret have been redacted.
`

Application ID
Secret
Callback URL	https://vouch.cidercreekranch.net/auth
Confidential	Yes
Scopes	openid (Authenticate using OpenID Connect)profile (Allows read-only access to the user's personal information using OpenID Connect)email (Allows read-only access to the user's primary email address using OpenID Connect)
`

[bnfinet]

Do I need to enter a /auth location in the vouch server block?

no, it appears that all requests are being proxied to VP

Are you able to reach https://vouch.cidercreekranch.net/auth in your browser?

It looks like the Nginx service that forwards to VP is running on :80 (http) but your /auth endpoint is configured as https (:443)

[YetAnotherBugHunter]

Are you able to reach https://vouch.cidercreekranch.net/auth in your browser?

Yes. When I enter that url I'm redirected to the GitLab sign in page.

It looks like the Nginx service that forwards to VP is running on :80 (http) but your /auth endpoint is configured as https (:443)

I changed the listen port to 443 and add the requisite cert/key entries and tried again in non-test mode. There is subtle difference in that the authorize button is now grayed-out after clicking on it.

Looking at the VP log I see the following:

{"level":"debug","ts":1614810869.9698768,"msg":"/auth/{state}/"} {"level":"error","ts":1614810875.5880113,"msg":"/auth Error while retreiving user info after successful login at the OAuth provider: Post \"https://gitlab.ccr.net/oauth/token\": x509: certificate signed by unknown authority"}
The authorization succeeded! And the peasants rejoice 👍

But the error message is interesting since the cert is a Let's Encrypt wildcard certificate that is valid until 04/23/2021.

BTW, there appears to be a typo in the error message. The e and i are transposed in retrieving. It's one that I frequently do myself :)

We're making progress ...

[YetAnotherBugHunter]

{"level":"debug","ts":1614810869.9698768,"msg":"/auth/{state}/"} {"level":"error","ts":1614810875.5880113,"msg":"/auth Error while retreiving user info after successful login at the OAuth provider: Post \"https://gitlab.ccr.net/oauth/token\": x509: certificate signed by unknown authority"}

The above error maybe an issue with how FreeBSD manages the CA certificates. I checked the bundle that is installed and they don't contain any of the Let's Encrypt intermediate certificates. I've acquired the intermediate certs and will add them to the supplied bundle and try again. Though doing so may cause maintenance issues since my modified bundle will get overwritten by a package update. But that's a FreeBSD issue.

I found this question on Stack Overflow

// Possible certificate files; stop after finding one.
var certFiles = []string{
"/etc/ssl/certs/ca-certificates.crt", // Debian/Ubuntu/Gentoo etc.
"/etc/pki/tls/certs/ca-bundle.crt", // Fedora/RHEL 6
"/etc/ssl/ca-bundle.pem", // OpenSUSE
"/etc/pki/tls/cacert.pem", // OpenELEC
"/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem", // CentOS/RHEL 7
"/etc/ssl/cert.pem", // Alpine Linux
}

FreeBSD stores the bundle in /etc/ssl/cert.pem and the individual certificates in /etc/ssl/certs/ca-certificates.crt. Before messing with the bundle, I'll try adding the individual cert first.

[bnfinet]

Thanks for documenting your progress.

this is how Go looks for the certs...
https://golang.org/src/crypto/x509/root_linux.go

[YetAnotherBugHunter]

{"level":"debug","ts":1614810869.9698768,"msg":"/auth/{state}/"} {"level":"error","ts":1614810875.5880113,"msg":"/auth Error while retreiving user info after successful login at the OAuth provider: Post \"https://gitlab.ccr.net/oauth/token\": x509: certificate signed by unknown authority"}

Fixed the above error by install the Let's Encrypt intermediate certificates.

I wrote a script to install the certs since it's not as simple as copying them to the appropriate directory. I will post the script to my GitHub account once I've cleaned it up, added a license, a disclaimer, and etc. The script is not specific to Let's Encrypt since it consumes a list of certificates to install.

I now have a working vouch-proxy on FreeBSD. Thanks to all for your help.

[bnfinet]

@YetAnotherBugHunter that's great to hear!

Are you running into any of the #309 / #367 problems with the user? Perhaps they don't apply to your use case.

I wrote a script to install the certs since it's not as simple as copying them to the appropriate directory. I will post the script to my GitHub account once I've cleaned it up, added a license, a disclaimer, and etc. The script is not specific to Let's Encrypt since it consumes a list of certificates to install.

That's popped up a couple times with folks. If you felt that it might be a worthy addition to ./do.sh please do consider submitting a PR. Some other peasant would likely rejoice to have that facility :)

[YetAnotherBugHunter]

@bnfinet

Are you running into any of the #309 / #367 problems with the user? Perhaps they don't apply to your use case.

No, I've has no further issues with vouch-proxy.

That's popped up a couple times with folks. If you felt that it might be a worthy addition to ./do.sh please do consider submitting a PR.

At the moment the script is FreeBSD centric. I would need to do some research on how the various flavours of Linux and other UNIX variants handle adding trusted certificates.

A quick check on an Ubuntu 18.04.5 VM instance I have shows that they do it in a similar way as FreeBSD. Which hopefully means that all Debian based distributions follow the same approach. Fedora seems similar but yet different.

If I can devise a way that works for other Linux/UNIX variants that I'll give it a go. But I do have a few things I need to finish before getting back a 'common' certificate install script.

1. Get the orchard ready for the upcoming season
2. Polish and publish the vouch-proxy FreeBSD RC script I wrote
3. Try to create a FreeBSD binary dist package for vouch-proxy
4. The honey-dos that I can't beg-off from now that I'm retired. :)