CalDAV not working properly with iOS #753
Labels
No Label
dependencies
duplicate
help wanted
invalid
kind/bug
kind/feature
needs reproduction
question
security
wontfix
No Milestone
No Assignees
14 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: vikunja/vikunja#753
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
[caldav]
I'm trying to add Vikunja to iOS via CalDAV, and it mostly doesn't work. Adding an account works, and listing of lists worked fine once, after that it only showed one list (the last one). Listing of tasks on a list didn't work at all. Adding a task worked, except when it had a specified timestamp. In that case it failed with a timestamp parsing error (bottom of the log).
Everything was tested in native iOS 14.3 Reminders app.
Log from the server is below, there are a lot of failures with PROPPATCH, and some timestamp parsing errors.
Vikunja version v0.15.0+71-be7b53b9bc
The
PROPPATCHmethods don't have a handler, hence the errors in the log. This could be related to the issues you described.I can see some more 500 errors which do not seem to be related to date parsing errors.
There should be more logging available - could you set the log level to
debug, restart vikunja, do again what you did and then send the logs? (obfuscate any private data before sending)Here you go, I hope it's useful. Some
ParseInterrors showed up now.Is this still an issue? Can you reproduce it with the latest version on the demo?
Just tried it out, displaying of lists still doesn't work properly, after the initial load, only the last on shows up. Tasks in lists are still not shown. Adding a task with a timestamp does sort of work, but the timestamp is discarded.
I'm having issues with the iOS Reminders as well. The first time I add the CalDAV account, the list shows up but shows blank. If I add a task on the phone, it shows online. Nothing added online shows on the phone, however. And after a bit of time, all tasks disappear on the phone. Happy to submit any information needed, I just need some guidance on how/where to set the log level to debug. I am using Portainer and running Vikunja in Docker.
@fivetwoseven I guess it would be worth a try if you would set the log level to
debug. For portainer I think the best way to set this would be through an env variableVIKUNJA_LOG_LEVEL=debug(been a while since I've used portainer but iirc you can set env variables there?).So I finally got around testing this with an ios device. Here's what I found.
I first tried to connect it through the local wifi to my dev instance running locally. That did not work. I only got an error "this did not work" and this in the Vikunja api logs:
Status 401 means "unauthorized". That's the status a client gets when they don't send their basic auth credentials. In combination with the very fast response (40µs) it looks like the iOS implementation simply did not send any client credentials.
However, also sends this header in case the authentication credentials were missing:
Www-Authenticate: basic realm=Restricted. That essentially means "hey please come back with proper basic auth credentials". And yet, iOS did not come back, instead it tried a few other urls:As you can see from the status codes, with no success. It just gave up when the first try did not succeed.
Next, I tried connecting to try. And that seemed to have worked, creating the account in the ios settings worked. In the logs I was able to see it also tried to connect without any credentials first but then provided them when that did not work:
I suspect that's due to try being available over https while my local dev instance isn't. That should not be an issue but it looks like it is.
After that it did a bunch of requests to get all lists:
And they all showed up:
But without any tasks. When I tried to create a task that did not work either.
Another interesting thing to note is there's a bunch of failing
PROPPATCHrequests:I suspect these are the actual cause of lists going missing (maybe iOS removes a list when it tried to do something with it and that failed?)
So in conclusion, I'm not sure how to proceed with this problem. I can see there's issues with it but I can't reproduce it all the time (the lists were all there) and I can't look into it bc iOS won't connect to my local dev instance.
Open for ideas.
Is it just a cert issue? I know Nextcloud tasks users were having trouble syncing CalDav to iOS until they figured out that Apple made https mandatory and tightened their cert requirements. Letsencrypt certs work fine though. Could you add a Letsencrypt cert to your dev instance and see if it resolves the issue?
@realcanadrian I guess that would make sense. Try has a Let's Encrypt certificate. Will check with my local dev instance.
I have a valid certificate on mine and I'm still seeing issues with CalDAV on iOS and MacOS.
The lists are retrieved but all show as having 0 tasks (when they do in fact have tasks).
I've been able to add a new tasks to one of the lists which does save into Vikunja but almost immediately after creating it it disappears from the iOS/MacOS app and the list shows 0 tasks again.
I've been testing with a valid certificate and have the same results as @AverageMarcus.
It almost seems as if the iOS client is able to write new tasks but not read.
I also set up an SMTP server and during this behaviour my emails get spammed every few seconds with the following, even though adding new tasks works correctly and the lists are showing up (albeit with 0 tasks still which is not correct) so it absolutely has the correct credentials.
Hmmm that's strange. Can you check the server logs?
Sure! This is what my logs look like when one of those emails is sent. I am using Fastmail as my email hosting provider, I wouldn't be surprised if the
Error closing the mail server connectionis caused by them rate-limiting requests due to the spam.To create the errors above, I 'Enabled' Vikunja in the iOS accounts menu and then opened the iOS Reminders app. I was also sent this email at the same time as the errors above:
If you'd like to see the log with debug mode enabled, here you go: https://pastebin.com/yTyrez3G
@Snuggle Are you using db caching? Does it work if you disable it?
Is this possibly related to Apple's .well-known specification? I ran into this when setting up dockerized Baikal with Traefik — it needed a bunch of middleware to play nice with iOS.
@Skotticus Might be. Do you have any info about this specifically?
About the emails: They will be solved with my PR: vikunja/api#1252
I've been trying out Vikunja lately and it's been great, but the lack of iOS support is stopping me using it full time. Anyway, I decided to dig more into this and I think I've worked out the issue.
iOS seems to send HTTP Auth headers with square brackets:
Authorization:[Basic blahtoken], which Vikunja does not like.You can test this yourself with cURL. The following always returns a 401:
curl -v https://tasks.example.com/dav/lists/1 -H "Authorization:[Basic footoken]"...but the following works fine
curl -v https://tasks.example.com/dav/lists/1 -H "Authorization: Basic footoken"I don't know Go, but happy to try and take a look at a fix if no one is available.
@chrisbanes okay that's interesting. How did you figure that out?
From my own testing it looks like it kind of works initially but then doesn't. If the problem is the brackets, then it wouldn't work at all.
Just looking through the debug logs. It is an assumption based on the cURL requests I used above, I haven't actually got iOS to sync yet. I'm trying to work out if I can modify the request headers with Traefik to verify.
If it's only in the logs, it might be the log format - headers are usually string slices so the
[]around them could just denote that.Ah you're right. That was silly of me 🤦♂️
I would love to see some progress on this bug. It's the only blocker stopping me from actually using Vikunja.
Maybe post a bugbounty or something?
Please, go ahead :)
Where would you prefer one to be posted?
From my limited time debugging this, I was 80% sure this is an issue in https://github.com/samedi/caldav-go which Vikunja uses underneath, but I couldn't find the direct cause.
Oh man, I let this one go, didn't I? I don't know where exactly y'all are at with this, but I can copy what I have in my Traefik config for you.
My fileconfig.yml for Traefik contains this:
For what it's worth, this didn't stop me from setting up Vikunja for my business, and my team seems to be fans.
samedi/caldav-go hasn't been updated since 2019, and seems abandoned. Might be time to switch this out with simething like emersion/go-webdav?
As mentioned earlier, I'd be willing to put up money to get this fixed. Maybe something like US$50? But unless someone can point me towards a service that facilitates transactions like this, all I can do is promise to paypal someone :)
Is there any documentation for go-webdav? I'd be open to switch in hope of some issues going away but not if that would mean to figure out most of the implementation details.
We haven't yet set up a bounty program on a platform so let's say if someone wants to tackle this please comment here and then paypal it I guess? Open for alternatives.
Also happy to chuck some money into a bounty, say NZD$100?
Seems the auto-generated documentation could be quote helpful: https://pkg.go.dev/github.com/emersion/go-webdav@v0.4.0/caldav
Internally, it seems to be using: https://github.com/emersion/go-ical
I really want this to work, so I created a work-around.
Be aware - this work-around is really fragile. It also just syncs one project to one calendar*.
The work-around is to use vdirsyncer to sync a vikunja project (via caldav) to disk, and then sync that up to another caldav-host (I use Baïkal).
In this example-config for vdirsyncer, I sync project 1 to the calendar named vikunja in baikal.
And now you can run
vdirsyncer sync vkandvdirsyncer sync bkto sync a project in vikunja and a calendar in baikal (after having runvdirsyncer discoverwhen first setting this up).As noted, this is really fragile, and I've come across two semi-big problems so far.
The first problem is that vikunja creates a field for
METHOD:which baikal does not support (I getUnknown error occurred for bk: 415 Client Error: Unsupported Media Type), so before runningvdirsyncer sync bkyou have to remove those lines (I just runsed -i '/^METHOD/d' ~/.vdirsyncer/vkbk/* && vdirsyncer sync bk). I don't know if you can expect the same error with other caldav-servers. Also, this problem is why you have to sync to disc before syncing to baikal. If not for this, it would be possible to directly sync baikal and vikunja.The second problem is that vikunja only does VTODO, so if one of your caldav-clients creates a more advanced object, the sync will fail. Just creating a reminder on iOS/macOS works fine. You can even add a due-date. But, if you add a due-time from iOS/macOS, it creates a VCALENDAR-object, and syncing to vikunja will fail.
I'd recommend to keep it as simple as possible (KISS!) when interacting with this calendar from other clients.
[*]: As mentioned, this also just syncs one project to one calendar. You can probably sync more project and calendars using the same config, without duplicating all of it, but you'll have to read up on
collectionsin the vdirsyncer-config, and match up names in vikunja and on the caldav-server.Hello, I'd love to have this feature too, but I have no skill in go. So I tried to analyze where the problem is by using a proxy to check the requests made by iOS to vikunja and a nextcloud server in comparison. The first requests look more or less the same, but then vikunja doesn't return the todo items, which seems like why they don't show up in the reminders app. Hope this helps someone.
Discover all lists
Nextcloud
PROPFIND /remote.php/dav/calendars/{user-id}/Request:
Response:
Vikunja
PROPFIND /dav/projects/Request:
Response:
Get sync token request
Nextcloud
PROPFIND /remote.php/dav/calendars/{user-id}/{list-id-#1}Request:
Response:
Vikunja
PROPFIND /dav/projects/2/Request:
Response:
List reminders todos
Nextcloud
PROPFIND /remote.php/dav/calendars/{user-id}/{list-id-#1}Request:
Response:
Vikunja
PROPFIND /dav/projects/2/Request:
Response:
Get todo items by their id
Nextcloud
REPORT /remote.php/dav/calendars/{user-id}/{list-id-#1}/Request:
Response:
Vikunja
REPORT /dav/projects/2/This request is not made, because there were no todo items returend before
Still want this feature very much. If you decide to put up a bounty for it, I'm willing to add something to it.
Maybe if all the people who want this very useful feature chime in, someone actually fixes it :)
This is a very useful feature. I really hope the author can spend some effort to upgrade it.
Though no "official" bugbounty have been posted, at least two of us have piped up that we want to put money towards a fix [1][2]. I still stand by that.
This workaround was too fragile to be usable.
I believe the simplest solution is to move on to a new library. I've been learning some go, and have been planning on taking a crack at this, but life keeps coming in the way 🙃