OpenID: Internal Server Error #864

Closed
opened 2021-05-15 23:08:45 +00:00 by tpansino · 25 comments

I'm configuring a self-hosted instance of GitLab to serve as the OpenID IdP for my Vikunja instance.

I'm seeing that when the frontend calls /callback however, the API returns a 500 with message: "Internal Server Error"

I've enabled all debug logging, and the only error message I see is:

 ERROR       ▶ openid/HandleCallback 27a3 json: Unmarshal(nil)

Which isn't very helpful... 😅

I'm assuming this is related to this section of the code, but I don't see any obvious bugs.

My config.yml:

---
service:
  jwtsecret: ~REDACTED~
  timezone: ~REDACTED~
  frontendurl: https://vikunja.example.com/
  enableregistration: false
cors:
  origins: vikunja.example.com
  maxage: 600
database:
  host: db
  type: ~REDACTED~
  user: ~REDACTED~
  database: ~REDACTED~
  password: ~REDACTED~
redis:
  enabled: true
  host: ~REDACTED~
  password: ~REDACTED~
cache:
  enabled: true
  type: redis
keyvalue:
  type: redis
auth:
  local:
    enabled: true
  openid:
    enabled: true
    providers:
      - name: gitlab
        authurl: https://gitlab.example.com
        clientid: ~REDACTED~
        clientsecret: ~REDACTED~
I'm configuring a self-hosted instance of GitLab to serve as the OpenID IdP for my Vikunja instance. I'm seeing that when the frontend calls `/callback` however, the API returns a 500 with `message: "Internal Server Error"` I've enabled all debug logging, and the only error message I see is: ``` ERROR ▶ openid/HandleCallback 27a3 json: Unmarshal(nil) ``` Which isn't very helpful... 😅 I'm assuming this is related to [this](https://kolaente.dev/vikunja/api/src/commit/d695681a0e50f7701c3edca22be93241913a0672/pkg/modules/auth/openid/openid.go#L102) section of the code, but I don't see any obvious bugs. My `config.yml`: ```yml --- service: jwtsecret: ~REDACTED~ timezone: ~REDACTED~ frontendurl: https://vikunja.example.com/ enableregistration: false cors: origins: vikunja.example.com maxage: 600 database: host: db type: ~REDACTED~ user: ~REDACTED~ database: ~REDACTED~ password: ~REDACTED~ redis: enabled: true host: ~REDACTED~ password: ~REDACTED~ cache: enabled: true type: redis keyvalue: type: redis auth: local: enabled: true openid: enabled: true providers: - name: gitlab authurl: https://gitlab.example.com clientid: ~REDACTED~ clientsecret: ~REDACTED~ ```
Owner

I can confirm this with gitlab.com. I'll take a look.

I can confirm this with gitlab.com. I'll take a look.
konrad added the
kind/bug
label 2021-05-16 10:21:36 +00:00
Owner

Okay so it looks like gitlab does not send the email of the user with the oauth response.

Okay so it looks like gitlab does not send the email of the user with the oauth response.
Owner

The user authenticating with gitlab has to set their public email address to something, otherwise it is not exposed to the openid connect port. See https://forum.gitlab.com/t/openid-connect-user-info-missing-email-claim/21902/2

I'll update the docs and error messages to clarify this.

The user authenticating with gitlab has to set their public email address to something, otherwise it is not exposed to the openid connect port. See https://forum.gitlab.com/t/openid-connect-user-info-missing-email-claim/21902/2 I'll update the docs and error messages to clarify this.
Owner

Added more docs and better errors in b76ad8efe2

I'd consider this issue solved, please don't hesitate to reopen if you have any issues.

Added more docs and better errors in https://kolaente.dev/vikunja/api/commit/b76ad8efe27d2817452b4b525b3459962b5c7de9 I'd consider this issue solved, please don't hesitate to reopen if you have any issues.
tpansino reopened this issue 2021-05-19 06:42:29 +00:00
Author

Still doesn't seem to be working, unfortunately, but I think I know the cause now.

The initial request to /authorize specifies scope=openid, but based on the OIDC spec that won't return the user's email. For that you would need scope=openid email. And most apps I've seen also want to store the user's first name, last name, and preferred username, which you can get with scope=openid email profile, so perhaps let's try that?

I also just want to confirm you're aware per the spec that email and preferred_username claims are not guaranteed to be unique or consistent over time, as users can potentially change both in the OpenID provider. So you shouldn't use them to look up a user during authentication, you should use the issuer and the subject claims together.

Still doesn't seem to be working, unfortunately, but I think I know the cause now. The initial request to `/authorize` specifies `scope=openid`, but based on the [OIDC spec](https://openid.net/specs/openid-connect-core-1_0.html#ScopeClaims) that won't return the user's email. For that you would need `scope=openid email`. And most apps I've seen also want to store the user's first name, last name, and preferred username, which you can get with `scope=openid email profile`, so perhaps let's try that? I also just want to confirm you're aware [per the spec](https://openid.net/specs/openid-connect-core-1_0.html#ClaimStability) that `email` and `preferred_username` claims are not guaranteed to be unique or consistent over time, as users can potentially change both in the OpenID provider. So you shouldn't use them to look up a user during authentication, you should use the `iss`uer and the `sub`ject claims together.
Author

Oh I supppose I should mention, I did retry with GitLab, but it also took me a few days to respond because I set up a completely different identity provider (Authentik) to do a second test. That one didn't work either, which leads me to believe it is a problem with the scope parameter.

Oh I supppose I should mention, I did retry with GitLab, but it also took me a few days to respond because I set up a completely different identity provider (Authentik) to do a second test. That one didn't work either, which leads me to believe it is a problem with the `scope` parameter.
Owner

This was the config I've tried:

auth:
  openid:
    enabled: true
    providers:
      - name: Gitlab.com
        authurl: https://gitlab.com
        clientid: <client id>
        clientsecret: <secret>

Can you reproduce it with gitlab.com?

Remember, the gitlab user you're authenticating with needs to have one email address which is publicly visible.

So you shouldn't use them to look up a user during authentication, you should use the issuer and the subject claims together.

Yes, I'm doing that. But I need an email address to associate with the newly created account. Subsequent authentications with the same (sub, iss) tuple will check if the email address changed and update if nessecary.

This was the config I've tried: ```yaml auth: openid: enabled: true providers: - name: Gitlab.com authurl: https://gitlab.com clientid: <client id> clientsecret: <secret> ``` Can you reproduce it with gitlab.com? Remember, the gitlab user you're authenticating with needs to have one email address which is publicly visible. > So you shouldn't use them to look up a user during authentication, you should use the issuer and the subject claims together. Yes, I'm doing that. But I need an email address to associate with the newly created account. Subsequent authentications with the same (sub, iss) tuple will check if the email address changed and update if nessecary.
Owner

The scopes used during the authentication are actually openid, email and profile: https://kolaente.dev/vikunja/api/src/branch/main/pkg/modules/auth/openid/providers.go#L140

The scopes used during the authentication are actually `openid`, `email` and `profile`: https://kolaente.dev/vikunja/api/src/branch/main/pkg/modules/auth/openid/providers.go#L140
Author

Yes, I'm doing that. But I need an email address to associate with the newly created account. Subsequent authentications with the same (sub, iss) tuple will check if the email address changed and update if nessecary.

Perfect. My apologies if I came off rude at all, I just know first-hand how tricky OIDC can be to implement. 🙂

Remember, the gitlab user you're authenticating with needs to have one email address which is publicly visible.

I tested just now with GitLab.com, verified my email was public first, same Internal Server Error.

...Which is odd considering I pulled the latest container images. I would have expected to see the new error messages you added, but I'm still seeing the Unmarshal error. Is the latest tag on Docker Hub supposed to be a build of master or am I mistaken?

This also doesn't explain why Authentik isn't working either. That one I think is a result of not passing email in scope, but since I also just got it set up I'll do a little more testing with a different app to confirm.

> Yes, I'm doing that. But I need an email address to associate with the newly created account. Subsequent authentications with the same (sub, iss) tuple will check if the email address changed and update if nessecary. Perfect. My apologies if I came off rude at all, I just know first-hand how tricky OIDC can be to implement. 🙂 > Remember, the gitlab user you're authenticating with needs to have one email address which is publicly visible. I tested just now with GitLab.com, verified my email was public first, same `Internal Server Error`. ...Which is odd considering I pulled the latest container images. I would have expected to see the new error messages you added, but I'm still seeing the `Unmarshal` error. Is the `latest` tag on Docker Hub supposed to be a build of `master` or am I mistaken? This also doesn't explain why [Authentik](https://goauthentik.io/) isn't working either. That one I think is a result of not passing `email` in `scope`, but since I also just got it set up I'll do a little more testing with a different app to confirm.
Owner

Perfect. My apologies if I came off rude at all, I just know first-hand how tricky OIDC can be to implement. 🙂

No worries 🙃

I tested just now with GitLab.com, verified my email was public first, same Internal Server Error.

Since you're on latest, could you turn on debug logging and send the output? Latest always represents the build from the main branch.

Authentik looks interesting, I'll try to reproduce with that.

> Perfect. My apologies if I came off rude at all, I just know first-hand how tricky OIDC can be to implement. 🙂 No worries 🙃 > I tested just now with GitLab.com, verified my email was public first, same Internal Server Error. Since you're on latest, could you turn on debug logging and send the output? Latest always represents the build from the `main` branch. Authentik looks interesting, I'll try to reproduce with that.
Owner

I was able to reproduce this with an Authentik instance, but not sure why it does not work - AFAIU Authentik sends all claims in the id_token as per their settings.
I'll take a look.

I was able to reproduce this with an Authentik instance, but not sure why it does not work - AFAIU Authentik sends all claims in the `id_token` as per their settings. I'll take a look.
Owner

Authentik does not seem to send the email, even when requested through the /userinfo endpoint. That's interesting.

Authentik does not seem to send the email, even when requested through the `/userinfo` endpoint. That's interesting.
Owner

I've asked the Authentik guys on their discord server about this and they don't really have an idea about it either - we're currently figuring out a way to properly debug this.

I've asked the Authentik guys on their discord server about this and they don't really have an idea about it either - we're currently figuring out a way to properly debug this.
Author

@konrad I'm seeing the /authorize URL coming out of both Authentik and GitLab.com looks like this:

https://authentik.example.com/application/o/authorize/?client_id=CLIENT_ID&redirect_uri=https://vikunja.example.com/auth/openid/authentik&response_type=code&scope=openid&state=STATE

Note that the scope only contains openid, despite the fact that you've set it in providers.go. 🤷‍♂️

Can you open your browser Network inspector and confirm email is there on your request to GitLab.com? If not, that rules this out, but if it is I still suspect that's the cause.

@konrad I'm seeing the `/authorize` URL coming out of both Authentik and GitLab.com looks like this: ``` https://authentik.example.com/application/o/authorize/?client_id=CLIENT_ID&redirect_uri=https://vikunja.example.com/auth/openid/authentik&response_type=code&scope=openid&state=STATE ``` Note that the `scope` only contains `openid`, despite the fact that you've set it in `providers.go`. 🤷‍♂️ Can you open your browser Network inspector and confirm `email` is there on your request to GitLab.com? If not, that rules this out, but if it is I still suspect that's the cause.
Owner

I also noticed that and added the scopes in 188c97ae8f - it did not fix the issue with Authentik though. With gitlab it works either way, with Authentik it doesn't work in either case. Gitlab seems to ignore what scope you sent it and just uses whatever you set when creating the application.

I also noticed that and added the scopes in https://kolaente.dev/vikunja/frontend/commit/188c97ae8f5ed07866a655fe0e108fe6eab238e9 - it did not fix the issue with Authentik though. With gitlab it works either way, with Authentik it doesn't work in either case. Gitlab seems to ignore what scope you sent it and just uses whatever you set when creating the application.
Author

Latest always represents the build from the main branch.

I'm pulling latest of both the api and frontend containers. Still seeing scope=openid, and openid/HandleCallback ... json: Unmarshal(nil) errors in the logs.

No sign of either change you've made, what might I be doing wrong? 🤔

Since you're on latest, could you turn on debug logging and send the output?

Only the CRON seems to be outputting anything DEBUG, doesn't look relevant.

> Latest always represents the build from the main branch. I'm pulling `latest` of both the `api` and `frontend` containers. Still seeing `scope=openid`, and `openid/HandleCallback ... json: Unmarshal(nil)` errors in the logs. No sign of either change you've made, what might I be doing wrong? 🤔 > Since you're on latest, could you turn on debug logging and send the output? Only the CRON seems to be outputting anything `DEBUG`, doesn't look relevant.
Owner

Does it work with a cleared browser cache or in a private window?

The frontend should be version 0.17.0+4-fd610d3721.

What version is your api container? (docker run vikunja/api /app/vikunja/vikunja version)

Does it work with a cleared browser cache or in a private window? The frontend should be version `0.17.0+4-fd610d3721`. What version is your api container? (`docker run vikunja/api /app/vikunja/vikunja version`)
Owner

With the help of the people at the Authentik Discord server, we've figured out a workaround: If you unselect the scopes of the provider, save it, then reselect the openid, email and profile scopes and save it again it works. I was able to reproduce it with that.

(You'll need the latest version of the frontend though)

With the help of the people at the Authentik Discord server, we've figured out a workaround: If you unselect the scopes of the provider, save it, then reselect the openid, email and profile scopes and save it again it works. I was able to reproduce it with that. (You'll need the latest version of the frontend though)
Author

Ok, I've given up on getting Authentik working, the whole getting started experience felt half-baked and not well documented. Back to getting Gitlab working.

I have:

  • Pulled the vikunja/api:latest and vikunja/frontend:latest images as of the time of this comment and verified the api container is v0.17.0+11-67f863120e (not sure how to check frontend version?)
  • Configured a self-hosted GitLab instance with a Trusted, Confidential app with openid, email, and profile scopes checked, and a redirect URI of https://vikunja.example.com/auth/openid/gitlab
  • Confirmed my GitLab profile has a Public Email set to what I want.
  • Cleared browser cache AND opened a Firefox private window, and attempted a login with the Network inspector open.

Still doesn't work. For the first time I do see scope=openid email profile in the request, but still Unmarshal in the Docker service logs, not the better error messages you added.

Why am I not seeing those better error messages? Am I still pulling the wrong version of the container?

Ok, I've given up on getting Authentik working, the whole getting started experience felt half-baked and not well documented. Back to getting Gitlab working. I have: * Pulled the `vikunja/api:latest` and `vikunja/frontend:latest` images as of the time of this comment and verified the `api` container is `v0.17.0+11-67f863120e` (not sure how to check frontend version?) * Configured a self-hosted GitLab instance with a Trusted, Confidential app with `openid`, `email`, and `profile` scopes checked, and a redirect URI of `https://vikunja.example.com/auth/openid/gitlab` * Confirmed my GitLab profile has a Public Email set to what I want. * Cleared browser cache _AND_ opened a Firefox private window, and attempted a login with the Network inspector open. Still doesn't work. For the first time I do see `scope=openid email profile` in the request, but still `Unmarshal` in the Docker service logs, not the better error messages you added. Why am I not seeing those better error messages? Am I still pulling the wrong version of the container?
Owner

The API version seems correct, but the error is really weired. Can you reproduce that with gitlab.com?

The API version seems correct, but the error is really weired. Can you reproduce that with gitlab.com?
Author

but the error is really weired.

Found it.

Once you said this, I figured it couldn't be related to the OpenID code anymore. So I did a search for Unmarshal, and I'm pretty certain it's this line that's throwing the unhelpful message now.

If I change my keyvalue type from redis to memory, it works. However cache type set to redis works just fine. 🤔 Mind adding a better log message to that spot so I can figure out why?

Also - I notice the UI sets my full name to my preferred username when I first log in, then sets it to my actual full name on second log in. Doesn't seem like expected behavior, is it? Happy to file another bug if so.

> but the error is really weired. Found it. Once you said this, I figured it couldn't be related to the OpenID code anymore. So I did a search for `Unmarshal`, and I'm pretty certain it's [this line](https://kolaente.dev/vikunja/api/src/commit/c4a71016b8fc46028a2093796c185dbf151c5d33/pkg/modules/keyvalue/redis/redis.go#L58) that's throwing the unhelpful message now. If I change my `keyvalue` type from `redis` to `memory`, it works. However `cache` type set to `redis` works just fine. 🤔 Mind adding a better log message to that spot so I can figure out why? Also - I notice the UI sets my full name to my preferred username when I first log in, then sets it to my actual full name on second log in. Doesn't seem like expected behavior, is it? Happy to file another bug if so.
Owner

That could be the error yeah. I'll take a look. But using memory should work fine, most of the time you'll only ever need redis if you have a lot of concurrent users.

Also - I notice the UI sets my full name to my preferred username when I first log in, then sets it to my actual full name on second log in. Doesn't seem like expected behavior, is it?

Is that with gitlab? I've added some functionality to check if the openid provider provides these fields on the initial request and then does a call to the /userinfo endpoint if they are not there. Maybe gitlab sends different data for these different requests. Needs more debugging I'd say.

But yeah, that'd be a differnt issue.

That could be the error yeah. I'll take a look. But using memory should work fine, most of the time you'll only ever need redis if you have a lot of concurrent users. > Also - I notice the UI sets my full name to my preferred username when I first log in, then sets it to my actual full name on second log in. Doesn't seem like expected behavior, is it? Is that with gitlab? I've added some functionality to check if the openid provider provides these fields on the initial request and then does a call to the `/userinfo` endpoint if they are not there. Maybe gitlab sends different data for these different requests. Needs more debugging I'd say. But yeah, that'd be a differnt issue.
Author

Yeah, that was with GitLab. I filed a separate issue for it, as you can see. 🙂

I'm not blocked on this issue anymore, but still happy to help debug this further if you'd like assistance. In particular that Unmarshal error message - let me know if you add some more logging around it and I'll test it out again.

Yeah, that was with GitLab. I filed a separate issue for it, as you can see. 🙂 I'm not blocked on this issue anymore, but still happy to help debug this further if you'd like assistance. In particular that `Unmarshal` error message - let me know if you add some more logging around it and I'll test it out again.
Owner

I've done some changes to how values are stored in redis in d48aa101cf. Would you mind checking if that fixed the Unmarshal error when using redis?

I've done some changes to how values are stored in redis in https://kolaente.dev/vikunja/api/commit/d48aa101cf1b39790167b3b89c0b716ffeb63eeb. Would you mind checking if that fixed the `Unmarshal` error when using redis?
Author

Apologies for the delay 🙏

I cannot reproduce the Unmarshal error anymore, so I think it's resolved! Thanks very much for your efforts, marking this one closed.

Apologies for the delay 🙏 I cannot reproduce the `Unmarshal` error anymore, so I think it's resolved! Thanks very much for your efforts, marking this one closed.
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: vikunja/vikunja#864
No description provided.