Todoist Migration Failure (500 Internal Server Error) #897
Labels
No Label
dependencies
duplicate
help wanted
invalid
kind/bug
kind/feature
needs reproduction
question
security
wontfix
No Milestone
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: vikunja/vikunja#897
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Hello !
I've finally been able to setup Vikunja and reverse proxy it to my domain.
However, when using the Todoist migration I get an error "Error Request failed with status code 500 Internal Server Error"
If this is useful for debugging, I have in Todoist:
In the Vikunja api logs I had this:
Thank you so much !
Could you enable debug logs and try the migration again?
Here you go, hope its useful:
This looks like a bug. Pushed a potential fix which should at least let the import run through in
37718c3282- a new unstable release should be ready and released in ~45min. Could you try again with that?My guess is though that the imported tasks won't contain any reminders or subtasks, maybe neither notes. Not sure why. Could you check that with the new version?
I'll test it tomorrow !
In the meantime, would it help you to look how planner imports Todoist tasks with comments, subtasks, labels etc without any apparent problem (although they only sync to Todoist instead of really importing the tasks to local database...)
Anyway thanks !
Hello @konrad , I updated the docker container and I was partially sucessful. However I got an error (which I wasn't quick enough to take a screenshot) which said something about "bucket end list" or simillar.
At the same time I noticed that both my Inbox and Team Inbox tasks got imported however all my projects (and the subsquent tasks in these projects) didn't get imported, which is basically all my tasks.
Here are the logs from Portainer:
To note that, although the majority of my tasks are in different Projects, the few tasks I had in the Inbox got imported without problems with the respective due dates, tags and comments and even subtasks !
Hope this is helpful and keep up the good work !
Edit:
I was able to see the error before it dissapears:
I've pushed a potential fix for this error in
d7932d2648. Could you check if that lets you import more lists? (New release should be ready in ~45min as usual)Hey @konrad, the server is down again in Matrix..
I've tried the new update and this time besides my Inbox tasks being imported two other projects and their tasks were imported but than it bugged again. However, I think I Know the problem for it. The first project imported had none subprojects, however the second Project imported has 6 subprojects but from what I gather Vikunja does not have the ability to create sub-lists hasn't it ?
So what happened was that, Vikunja imported first my tasks from the Inbox, then it imported the tasks from my project called 2020 (which had none subprojects), however when it moved to my next Project (Personal) after importing the tasks in this project it bugged out when trying to import the subprojects and their tasks from Personal and the migration tasks stopped here with:
Visual aid to help understanding what I'm talking about relating to Todoist subprojects:
Bellow you can see the log's part relevant to the migration task:
Nevertheless, the tasks that were indeed imported had all the attributes correctly imported from Todoist (due date, reminders, comments, sub-tasks relationships) 🙂
Thank you once again for your time and help !
I was able to correctly migrate all tasks from a sub project. So while it may have bugged out while importing a sub project I don't think this was caused by the project being a sub project.
This is an error that happened during the "final" insertion step - does not really make sense because it should have just createde the bucket instead of complaining about it. I've made some adjustments to the logging in
0c5dfe5c48so hopefully we'll get a better insight in why it could not find that bucket. Could you update and try again? A new build should be ready in ~30min, just as the last one.Perfect example of "causation does not mean correlation" 🙂
Here are the logs:
@bolgrov There should be a message starting with
ERROR(rather thanDEBUG) right after the logs you shared. Could you share that one?The logs you shared don't contain any errors - does that mean it worked?
Hey @konrad , I searched for "ERROR" on the logs and didn't find any entry. Plus I updated this morning the container once again, just to be sure, and ran the migration again but the same thing happened. I have to
ERRORentries in the Vikunja API logs and the migration stops exactly in the same step has before.If you need more information please ask me.
Thank you !
Hey @konrad ,
I was thinking again about the question of Projects and subprojects conversion to Vikunja. I can't see a way of creating sublists in Vikunja so I can not see how will subprojects be imported.
From the way I see it, Namespaces seen to be the equivalent of Projects and the lists in these Namespaces to be the equivalent of Subprojects in Todoist.
So if you changed the migration code to create a Namespace for each project and a list for each subproject in the respective Namespace, maybe it would be successful ?
Because, at the moment there is only one Namespace being created called "Migrated From TODOIST" and the imported projects are created has lists inside it, but (and this could be only coincidence) the first Project with a subproject is where it stops importing and bugs out.
Thank you 🙂
I'm a bit confused. Do you have errors in the logs or not?
Could you maybe just send me the the full logs? (Either here or via matrix or per mail to konrad@vikunja.io)
I'd think of namespaces more like a folder of lists. A parent project in Todoist can also have tasks while a namespace cannot. Projects are the Todoist equivalent of Vikunja's lists.
Simply changing the import logic to put all sub projects in a namespace with the name of their parent would open up new problems like where should the tasks of that parent project go, what about projects that don't have sub lists (should they get a namespace as well?)... etc.
Sublists are imported correctly. Vikunja only ignores the hierachy since it does not have that feature - as you correctly pointed out.
My test account in Todoist:
And after the import in Vikunja:
Hey @konrad ,
Sorry about the confusion, I meant to say that I find no ERROR entries in the logs (the two times I tried the migration).
I now understand that what I was suggesting in relation to sub-projects doesn't make sense, it was just mere coincidence it is bugging out the moment it moves to the first subproject in my TOdoist data. Thank you for the throughout explanation.
Nevertheless, if you want I can send you the intire logs.
Thank you for your time once again!
I've just noticed there was an error in the last build so the pipeline didn't actually run through - which means there was no new build for you to test. I've fixed that now and the release pipeline has just finished. Please try that one, it should give you some error log messages (and I'd like to see those).
I've run the migration and now the ERROR entries do appear, albeit there's only 2:
If you want I can send you the rest of the logs but they're the same as before.
Thank you !
This looks like you have a sub task in todoist which is in a different section than any other task and that section does not get created before. When Vikunja then tries to put it in that section, that errors out because it does not exist.
I don't know if that's even possible from Todoist, from my understanding that shouldn't work.
I've pushed a fix for that in
5b825f1cc8, please update and try again.My guess is the migration will run fully through but with some tasks/attributes missing given there are quite some of these log messages:
So if the migration doesn't crash with the new fix, please check the following:
I have a few ideas why this might happening:
Hello @konrad ,
Thank you for the fix ! However, before answering your questions I found a new bug which doesn't let me import all the tasks.
The error it produces is the following:
I went and looked where the migration process stopped and saw that it coincide it some tasks that have a really long title, which is due to the fact that many of my tasks have links in their name and Todoist automatically fetchs the site's title and substitutes the url to
[site title](url)which can sometimes make a really long title. So I experimented with deleting the tasks that had a long title and run the migration again, however the migration stopped with the exact same error after importing a few more projects. I looked where it stopped and it coincides with a shared list I have with a colleague with many tasks that have huge titles because of entered urls.Tasks example:
So then I went to the Vikunja db logs and saw that it reported the same error:
Which I think is associated with PostgresSQL database being set to a limit of 250 characters, see 1 2.
So if you change the code to set the limit of the database of the title's entry to something like 1000 characters it should suffice.
Thank you for your time.
Huh, that's an interesting find. I guess it makes sense to change the column type to
TEXTrather than only increasing the limit of the varchar.I've changed the task title db field to text in
358661e060- please try again with the latest version.Holy ... it finally imported everything !
Everything was imported correctly: tasks, subtasks, comments, the projects, reminders and priorities. There is only one thing that is not imported which, in my case at least, is a big miss: recurrence.
I have a lot of tasks with recurring dates, like every 3 days; every two weeks, every years, every first monday etc...
Is this fixable ? I could fix this the hard way: by hand. But it would take some days..
Nevertheless, you are the best !
Glad it seems to (almost) work now!
Did it import no reminders at all or only a few?
I think you misunderstood me 🙂
The reminders were all imported correctly (with some exceptions which I'll explain further bellow), what was not imported is the recurrence of the tasks which in Vikunja I see that it can be set by "Set a repeating interval" function.
So imagine I have a task name "Vikunja" set to repeat every Friday at 22pm. This tasks is imported into Vikunja from Todoist with the due date for the next Friday at 22pm (this would be now 16 of July at 22pm), however the recurrence is not imported and if I mark this task done it is done forever instead of updating to the next Friday.
A solution would be to manually set again the repeating interval but this would be prohibitively laborious with the amount of recurring tasks I have.
Now, there's another problem which is related with reminders that have recurring dates.
All tasks that had manually set reminders were imported correctly, for example: a tasks with reminders for 4 jun at 9pm, 10 oct at 10am, 11 oct at 17pm. Is imported with all the 3 reminders into Vikunja.
However, if I have a tasks with a reminder set in the following way: every friday at 10pm. It will import only one reminder for the next Friday at 10pm in the same way that happens to the due date.
To conclude, what is missing:
I think the case with reminders is harder to solve because from what I gathered, in Vikunja you can only set recurring due dates but not recurring reminders.
If I missed something or was not clear enough please ask me.
Thank you for your incredible work !
Ah I see 🙃
The problem with todoist recurring stuff is they can set indipentently from each other. That means you can have a recurring due date and a recurring reminder on different schedules. Vikunja, however, only has a concept of recurring tasks not reminders or due dates. If you set a task in Vikunja to repeat at a certain schedule it will repeat all dates (due, start, end, all reminders) in that schedule when the task is marked as done.
Because of all this, my current "strategy" was to simply ignore these when importing the tasks from todoist.
I'm not sure how to solve this in the import though I have a few ideas:
Open to other suggestions.
Ok I was writing a lengthy response but I reached to conclusion that yes, those conditions are the best overall.
One thing I noticed is that the migration API is already importing correctly tasks that have multiple reminders set manually, so please don't modify this. In my opinion, and this could be biased, the most important thing here is to be able to import correctly the static due dates and reminders, but also the recurring due dates, as such it is more important that as you said the recurring due dates take importance over recurring reminders.
If you implement those two conditions as you referred, the API migration will be basically perfect. The only thing missing is sub-lists, which I know might be difficult to deal with in terms of hierarchic. However (and I can open another issue, or talk with you in another place), this could be much simplified by only allowing one level of sub-lists !
As always, thank you for your time!
I've tried to implement that and it looks like there's another problem (and a blocking one). This is how a task with a reccurring due date (same for reminders) looks like in the api response from todoist:
There is the
is_recurringfield which tells me if that task is recurring or not. There's no indication when that task is reccurring other than thestringproperty which Vikunja would need to parse in order to understand when exactly it is reccuring. And that for all languages Todoist supports.I don't think that's something we could implement right now.
Yes, that was my fear when I was looking at the JSON file exported from Todoist.
The only possible solution I see is to limit it to only English queries and implement date parsing like Todoist does into the Vikunja API to be able to read the recurring text fields.
I mean, you already made an wonderful job of date parsing implementation with recent updates so I wonder how difficult would it be to catch up to Todoist level (if you indeed can this would be ironically beautiful how a one person project beats a multi millionaire closed source one). From what I gathered, the parsing part that is missing in Vikunja is recurrence and some embellishments that todoist has like: every can be parsed as ev and Tomorrow can be parsed as tom, and the same for the week days.
So if you could implement recurrence into your magic Quick add function and also abbreviations the migration API would work!
Although maybe this should become another issue of its own (more refined date parsing) ?
Thank you @konrad
Thanks for the kind words 🙂
The main problem with moving the task attribute parsing to the api is it is a frontend feature which I can't really just port over like that (different programming languages). But overall I guess that's the only way to solve that.
Abbreviations for weekdays should work btw, not so sure about the other ones.
I think the original problem of this issue has been resolved so I'm closing this. Feel free to open a new issue for the problem with importing recurring tasks.
Hi @konrad , congratulations on version 0.18.0 and all work put into Vikunja.
I'm sorry to reopen this (close if you see it fit), but now that I've fully migrated into Vikunja I noticed a crucial import lacking feature:
All tasks from Todoist that have a due date but not an hour set are imported to Vikunja without any date set.
This is not a Todoist import error per se, because I just noticed that Vikunja does not allow to set due dates without also setting an hour, so the problem is probably here.
The downside is that my workflow in Todoist is to only set an hour for a task if I know I have to do that task at that specific hour. Otherwise I just set the task for a certain day (without an hour set) but with a priority level set. As this is my typical workflow, most of the imported tasks to Vikunja got their due date erased because they didn't have an hour set.
Thank you !
Hello @konrad how are you doing ?
Do you prefer that I post the last comment as an separate issue on the API ?
Thank you !
@bolgrov Ah yes, totally forgot this issue. I think a new one would be helpful.
I think I did put this in the backlog though.
@bolgrov Ah yes, totally forgot this issue. I think a new one would be helpful.
No worries, making a project this great is very time consuming !
I just checked and you have a task in the Backlog called "API-447 Todoist migration doesn't import dates with no time".
Do you still want me to create an issue ?
I think it should be fine. Thanks for looking it up!
Should be fixed with
fd0d462bf4- please test with the next unstable build (should take ~30 min until the CI released it).Thank you for the work !
But did you fixed this by setting all the tasks without an hour set by setting them all to 23h59 ? 😆 Because I've just migrated all my tasks from Todoist and noticed this.
Can't Vikunja set tasks with dates but not an hour set ? I almost never set an hour on tasks, only a day.
Thank you again for your time !
That's correct. Vikunja can only handle dates with a time. I could've set it to 0:00 or 12:00 or any other time when importing, but it needs to have a time. I'd think if you set a due date to a certain day you usually mean "by the end of the day" so 23:50 seemed appropriate.
I'm used to, in Todoist, when I want to set a task for a certain date but do not know when in that day I'll actually do the tasks I just set:
New task for tomorrowIts quick and I don't have to think about setting hours, in Vikunja I have to do:
New task for tomorrow at 23:59which creates more friction if I'm creating various tasks along the day for different days.Just an opinion 😄
👋 Hello,
I think 23:50 seems be to late. Normally if I have something due on a day I want to do it before the day finishes. If the the date is set to 23:50 I have 10 minutes to finish my task 🤔
I think in Todoist they assign a default time (I think 9:00ish? ) so that you would be reminded when you plan your day so that you are able to finish the task.
Hey @dpschen , actually in Todoist that's not the current behaviour. If you set a task without due date for a certain date you just get a tasks for that day without any hour set. If you want a reminder for that tasks then you have manually set a reminder (or recurring reminder) for it.
This goes back to the discussion about default reminders for tasks I've talked about --> If you set a tasks with due date that has a day and hour specified you should be reminded an amount of time (user-defined in the settings) before that tasks 😀
See bellow an example of some old tasks I have for my Today's Work filter (which filters all tasks projects I consider to be work, that are for today or overdue). In green I marked all tasks that I only set a day (or recurring day) but without any hour set.
There's some code that's primarily used to selecte the next "round" hour when you select a date in the due date setting for a task. Maybe we could use that to set the time when none was provided, only a date?
@bolgrov: You are correct =) So they "kind of" assign a default time, but it's dynamically and based on the user settings (talking here about Todoists automatic remindets).
@konrad:
So you mean changing that function in a way that it creates a date that has no hour and minute values set? And always when the frontend sees one of those we fall back to some kind of user defined hour for that reminder at that date?
@bolgrov:
Btw I love the way you used labels to group your tasks.
I always failed to have a good system for this :D
I think we should create a new issue for this topic.
There is also a topic in the community regarding this: https://community.vikunja.io/t/color-option-on-overdue-done-tomorow-tasks/508
I would be more than happy if you guys want a issue about this ! I just didn't made a new one because I do not want to always be nagging about new feature requests 😅
Thank you for compliment @dpschen !
I started to implement filters through a system of tasks to not be overwhelmed with tasks. I can share my various system with you or the community if people want to.
Matter of fact, one of the features I've been really wanting to ask @konrad but have refrained from (see previous comment) is separation of tasks by their tag name, as in my screenshot above.
In this way, tags work not only to make helpful filters, but at the same time reduce visual clutter by grouping tasks under each label !
hm that could probably work too, yes. I thought more about using that function when creating a task with a date and no time set.
And then doing what?
Setting a time 🤔
So for example when creating a task "lorem ipsum tomorrow" it would create a task due tomorrow at the time specified by that function. Currently the quick add magic uses the current time if none was specified.
The datepicker dropdown for due dates and reminders already use it.
Accepting dates without a time and saving a default one would require quite a bit of effort on the api side to essentially separate the date and time from each other.
Okay I understand now.
What I meant was that we define a date with a set time of 00:00 as "no time set" and by this as "due at some time that day". This would obviously abuse the duedate field a bit.
But on the plus side: as far as I can think we would be able to show these items in the frontend with few changes as a reminder with a time that can be user defined (and has some default time set, e.g. 8:00 in the morning).
I think this could solve the biggest problem with this issue, without creating too much work.
For the future we would still be open to create an extra field for no time set.
If we later add a field for "no time set" we can mark this field as true for items with a time of the due date of 00:00 and free this time from its "defined extra meaning".