diff --git a/config.yml.sample b/config.yml.sample index acd6f1c977..29f518d506 100644 --- a/config.yml.sample +++ b/config.yml.sample @@ -117,3 +117,19 @@ files: # The maximum size of a file, as a human-readable string. # Warning: The max size is limited 2^64-1 bytes due to the underlying datatype maxsize: 20MB + +migration: + # These are the settings for the wunderlist migrator + wunderlist: + # Wheter to enable the wunderlist migrator or not + enable: true + # The client id, required for making requests to the wunderlist api + # You need to register your vikunja instance at https://developer.wunderlist.com/apps/new to get this + clientid: + # The client secret, also required for making requests to the wunderlist api + clientsecret: + # The url where clients are redirected after they authorized Vikunja to access their wunderlist stuff. + # This needs to match the url you entered when registering your Vikunja instance at wunderlist. + # This is usually the frontend url where the frontend then makes a request to /migration/wunderlist/migrate + # with the code obtained from the wunderlist api. + redirecturl: diff --git a/docs/content/doc/development/migrations.md b/docs/content/doc/development/db-migrations.md similarity index 100% rename from docs/content/doc/development/migrations.md rename to docs/content/doc/development/db-migrations.md diff --git a/docs/content/doc/development/migration.md b/docs/content/doc/development/migration.md new file mode 100644 index 0000000000..460107bff8 --- /dev/null +++ b/docs/content/doc/development/migration.md @@ -0,0 +1,91 @@ +--- +date: "2020-01-19:16:00+02:00" +title: "Migrations" +draft: false +type: "doc" +menu: + sidebar: + parent: "development" +--- + +# Writing a migrator for Vikunja + +It is possible to migrate data from other to-do services to Vikunja. +To make this easier, we have put together a few helpers which are documented on this page. + +In general, each migrator implements a migrator interface which is then called from a client. +The interface makes it possible to use helper methods which handle http an focus only on the implementation of the migrator itself. + +### Structure + +All migrator implementations live in their own package in `pkg/modules/migration/`. +When creating a new migrator, you should place all related code inside that module. + +### Migrator interface + +The migrator interface is defined as follows: + +```go +// Migrator is the basic migrator interface which is shared among all migrators +type Migrator interface { + // Migrate is the interface used to migrate a user's tasks from another platform to vikunja. + // The user object is the user who's tasks will be migrated. + Migrate(user *models.User) error + // AuthURL returns a url for clients to authenticate against. + // The use case for this are Oauth flows, where the server token should remain hidden and not + // known to the frontend. + AuthURL() string +} +``` + +### Defining http routes + +Once your migrator implements the migration interface, it becomes possible to use the helper http handlers. +Their usage is very similar to the [general web handler](https://kolaente.dev/vikunja/web#user-content-defining-routes-using-the-standard-web-handler): + +```go +// This is an example for the Wunderlist migrator +if config.MigrationWunderlistEnable.GetBool() { + wunderlistMigrationHandler := &migrationHandler.MigrationWeb{ + MigrationStruct: func() migration.Migrator { + return &wunderlist.Migration{} + }, + } + m.GET("/wunderlist/auth", wunderlistMigrationHandler.AuthURL) + m.POST("/wunderlist/migrate", wunderlistMigrationHandler.Migrate) +} +``` + +You should also document the routes with [swagger annotations]({{< ref "../practical-instructions/swagger-docs.md" >}}). + +### Insertion helper method + +There is a method available in the `migration` package which takes a fully nested Vikunja structure and creates it with all relations. +This means you start by adding a namespace, then add lists inside of that namespace, then tasks in the lists and so on. + +The root structure must be present as `[]*models.NamespaceWithLists`. + +Then call the method like so: + +```go +fullVikunjaHierachie, err := convertWunderlistToVikunja(wContent) +if err != nil { + return +} + +err = migration.InsertFromStructure(fullVikunjaHierachie, user) +``` + +### Configuration + +You should add at least an option to enable or disable the migration. +Chances are, you'll need some more options for things like client ID and secret +(if the other service uses oAuth as an authentication flow). + +The easiest way to implement an on/off switch is to check whether your migration service is enabled or not when +registering the routes, and then simply don't registering the routes in the case it is disabled. + +#### Making the migrator public in `/info` + +You should make your migrator available in the `/info` endpoint so that frontends can display options to enable them or not. +To do this, add an entry to `pkg/routes/api/v1/info.go`. diff --git a/docs/content/doc/development/structure.md b/docs/content/doc/development/structure.md index 46607d0835..8e819b6c48 100644 --- a/docs/content/doc/development/structure.md +++ b/docs/content/doc/development/structure.md @@ -22,6 +22,10 @@ In general, this api repo has the following structure: * `metrics` * `migration` * `models` + * `modules` + * `migration` + * `handler` + * `wunderlist` * `red` * `routes` * `api/v1` @@ -85,7 +89,7 @@ To learn how it works and how to add new metrics, take a look at [how metrics wo This package handles all migrations. All migrations are stored and executed here. -To learn more, take a look at the [migrations docs]({{< ref "../development/migrations.md">}}). +To learn more, take a look at the [migrations docs]({{< ref "../development/db-migrations.md">}}). ### models @@ -97,6 +101,12 @@ Because this package is pretty huge, there are several documents and how-to's ab * [Adding a feature]({{< ref "../practical-instructions/feature.md">}}) * [Making calls to the database]({{< ref "../practical-instructions/database.md">}}) +### modules + +#### migration + +See [writing a migrator]({{< ref "migration.md" >}}). + ### red (redis) This package initializes a connection to a redis server. diff --git a/docs/content/doc/setup/config.md b/docs/content/doc/setup/config.md index b32dc21039..a61931d84d 100644 --- a/docs/content/doc/setup/config.md +++ b/docs/content/doc/setup/config.md @@ -160,4 +160,20 @@ files: # The maximum size of a file, as a human-readable string. # Warning: The max size is limited 2^64-1 bytes due to the underlying datatype maxsize: 20MB + +migration: + # These are the settings for the wunderlist migrator + wunderlist: + # Wheter to enable the wunderlist migrator or not + enable: true + # The client id, required for making requests to the wunderlist api + # You need to register your vikunja instance at https://developer.wunderlist.com/apps/new to get this + clientid: + # The client secret, also required for making requests to the wunderlist api + clientsecret: + # The url where clients are redirected after they authorized Vikunja to access their wunderlist stuff. + # This needs to match the url you entered when registering your Vikunja instance at wunderlist. + # This is usually the frontend url where the frontend then makes a request to /migration/wunderlist/migrate + # with the code obtained from the wunderlist api. + redirecturl: {{< /highlight >}} diff --git a/go.mod b/go.mod index 2b30439190..b1a8ffc57f 100644 --- a/go.mod +++ b/go.mod @@ -59,7 +59,6 @@ require ( github.com/onsi/gomega v1.4.3 // indirect github.com/op/go-logging v0.0.0-20160315200505-970db520ece7 github.com/pelletier/go-toml v1.4.0 // indirect - github.com/pkg/errors v0.8.1 // indirect github.com/prometheus/client_golang v0.9.2 github.com/samedi/caldav-go v3.0.0+incompatible github.com/shurcooL/httpfs v0.0.0-20190527155220-6a4d4a70508b diff --git a/pkg/config/config.go b/pkg/config/config.go index 1016e52ccb..a695ce194e 100644 --- a/pkg/config/config.go +++ b/pkg/config/config.go @@ -89,6 +89,11 @@ const ( FilesBasePath Key = `files.basepath` FilesMaxSize Key = `files.maxsize` + + MigrationWunderlistEnable Key = `migration.wunderlist.enable` + MigrationWunderlistClientID Key = `migration.wunderlist.clientid` + MigrationWunderlistClientSecret Key = `migration.wunderlist.clientsecret` + MigrationWunderlistRedirectURL Key = `migration.wunderlist.redirecturl` ) // GetString returns a string config value diff --git a/pkg/files/files.go b/pkg/files/files.go index 540eab11db..989a994c55 100644 --- a/pkg/files/files.go +++ b/pkg/files/files.go @@ -39,6 +39,9 @@ type File struct { CreatedByID int64 `xorm:"int(11) not null" json:"-"` File afero.File `xorm:"-" json:"-"` + // This ReadCloser is only used for migration purposes. Use with care! + // There is currentlc no better way of doing this. + FileContent []byte `xorm:"-" json:"-"` } // TableName is the table name for the files table diff --git a/pkg/modules/migration/create_from_structure.go b/pkg/modules/migration/create_from_structure.go new file mode 100644 index 0000000000..07164e642e --- /dev/null +++ b/pkg/modules/migration/create_from_structure.go @@ -0,0 +1,97 @@ +// Vikunja is a todo-list application to facilitate your life. +// Copyright 2018-2020 Vikunja and contributors. All rights reserved. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU General Public License for more details. +// +// You should have received a copy of the GNU General Public License +// along with this program. If not, see . + +package migration + +import ( + "bytes" + "code.vikunja.io/api/pkg/models" + "io/ioutil" +) + +// InsertFromStructure takes a fully nested Vikunja data structure and a user and then creates everything for this user +// (Namespaces, tasks, etc. Even attachments and relations.) +func InsertFromStructure(str []*models.NamespaceWithLists, user *models.User) (err error) { + + // Create all namespaces + for _, n := range str { + err = n.Create(user) + if err != nil { + return + } + + // Create all lists + for _, l := range n.Lists { + // The tasks slice is going to be reset during the creation of the list so we rescue it here to be able + // to still loop over the tasks aftere the list was created. + tasks := l.Tasks + + l.NamespaceID = n.ID + err = l.Create(user) + if err != nil { + return + } + + // Create all tasks + for _, t := range tasks { + t.ListID = l.ID + err = t.Create(user) + if err != nil { + return + } + + // Create all relation for each task + for kind, tasks := range t.RelatedTasks { + // First create the related tasks if they does not exist + for _, rt := range tasks { + if rt.ID == 0 { + err = rt.Create(user) + if err != nil { + return + } + } + + // Then create the relation + taskRel := &models.TaskRelation{ + TaskID: rt.ID, + OtherTaskID: t.ID, + RelationKind: kind, + } + err = taskRel.Create(user) + if err != nil { + return + } + } + } + + // Create all attachments for each task + for _, a := range t.Attachments { + // Check if we have a file to create + if len(a.File.FileContent) > 0 { + a.TaskID = t.ID + fr := ioutil.NopCloser(bytes.NewReader(a.File.FileContent)) + err = a.NewAttachment(fr, a.File.Name, a.File.Size, user) + if err != nil { + return + } + } + } + } + } + } + + return nil +} diff --git a/pkg/modules/migration/handler/handler.go b/pkg/modules/migration/handler/handler.go new file mode 100644 index 0000000000..4efef3adb9 --- /dev/null +++ b/pkg/modules/migration/handler/handler.go @@ -0,0 +1,66 @@ +// Vikunja is a todo-list application to facilitate your life. +// Copyright 2018-2020 Vikunja and contributors. All rights reserved. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU General Public License for more details. +// +// You should have received a copy of the GNU General Public License +// along with this program. If not, see . + +package handler + +import ( + "code.vikunja.io/api/pkg/models" + "code.vikunja.io/api/pkg/modules/migration" + "code.vikunja.io/web/handler" + "github.com/labstack/echo/v4" + "net/http" +) + +// MigrationWeb holds the web migration handler +type MigrationWeb struct { + MigrationStruct func() migration.Migrator +} + +// AuthURL is returned to the user when requesting the auth url +type AuthURL struct { + URL string `json:"url"` +} + +// AuthURL is the web handler to get the auth url +func (mw *MigrationWeb) AuthURL(c echo.Context) error { + ms := mw.MigrationStruct() + return c.JSON(http.StatusOK, &AuthURL{URL: ms.AuthURL()}) +} + +// Migrate calls the migration method +func (mw *MigrationWeb) Migrate(c echo.Context) error { + ms := mw.MigrationStruct() + + // Get the user from context + user, err := models.GetCurrentUser(c) + if err != nil { + return handler.HandleHTTPError(err, c) + } + + // Bind user request stuff + err = c.Bind(ms) + if err != nil { + return echo.NewHTTPError(http.StatusBadRequest, "No or invalid model provided: "+err.Error()) + } + + // Do the migration + err = ms.Migrate(user) + if err != nil { + return handler.HandleHTTPError(err, c) + } + + return c.JSON(http.StatusOK, models.Message{Message: "Everything was migrated successfully."}) +} diff --git a/pkg/modules/migration/migrator.go b/pkg/modules/migration/migrator.go new file mode 100644 index 0000000000..bac7dd7160 --- /dev/null +++ b/pkg/modules/migration/migrator.go @@ -0,0 +1,31 @@ +// Copyright 2019 Vikunja and contriubtors. All rights reserved. +// +// This file is part of Vikunja. +// +// Vikunja is free software: you can redistribute it and/or modify +// it under the terms of the GNU General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// Vikunja is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU General Public License for more details. +// +// You should have received a copy of the GNU General Public License +// along with Vikunja. If not, see . + +package migration + +import "code.vikunja.io/api/pkg/models" + +// Migrator is the basic migrator interface which is shared among all migrators +type Migrator interface { + // Migrate is the interface used to migrate a user's tasks from another platform to vikunja. + // The user object is the user who's tasks will be migrated. + Migrate(user *models.User) error + // AuthURL returns a url for clients to authenticate against. + // The use case for this are Oauth flows, where the server token should remain hidden and not + // known to the frontend. + AuthURL() string +} diff --git a/pkg/modules/migration/wunderlist/testimage.jpg b/pkg/modules/migration/wunderlist/testimage.jpg new file mode 100644 index 0000000000..df5f6b91df Binary files /dev/null and b/pkg/modules/migration/wunderlist/testimage.jpg differ diff --git a/pkg/modules/migration/wunderlist/wunderlist.go b/pkg/modules/migration/wunderlist/wunderlist.go new file mode 100644 index 0000000000..3bdbce3d52 --- /dev/null +++ b/pkg/modules/migration/wunderlist/wunderlist.go @@ -0,0 +1,482 @@ +// Vikunja is a todo-list application to facilitate your life. +// Copyright 2018-2020 Vikunja and contributors. All rights reserved. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU General Public License for more details. +// +// You should have received a copy of the GNU General Public License +// along with this program. If not, see . + +package wunderlist + +import ( + "bytes" + "code.vikunja.io/api/pkg/config" + "code.vikunja.io/api/pkg/files" + "code.vikunja.io/api/pkg/log" + "code.vikunja.io/api/pkg/models" + "code.vikunja.io/api/pkg/modules/migration" + "code.vikunja.io/api/pkg/utils" + "encoding/json" + "fmt" + "net/http" + "net/url" + "strconv" + "time" +) + +// Migration represents the implementation of the migration for wunderlist +type Migration struct { + // Code is the code used to get a user api token + Code string `query:"code" json:"code"` +} + +// This represents all necessary fields for getting an api token for the wunderlist api from a code +type wunderlistAuthRequest struct { + ClientID string `json:"client_id"` + ClientSecret string `json:"client_secret"` + Code string `json:"code"` +} + +type wunderlistAuthToken struct { + AccessToken string `json:"access_token"` +} + +type task struct { + ID int `json:"id"` + AssigneeID int `json:"assignee_id"` + CreatedAt time.Time `json:"created_at"` + CreatedByID int `json:"created_by_id"` + DueDate string `json:"due_date"` + ListID int `json:"list_id"` + Revision int `json:"revision"` + Starred bool `json:"starred"` + Title string `json:"title"` + Completed bool `json:"completed"` + CompletedAt time.Time `json:"completed_at"` +} + +type list struct { + ID int `json:"id"` + CreatedAt time.Time `json:"created_at"` + Title string `json:"title"` + ListType string `json:"list_type"` + Type string `json:"type"` + Revision int `json:"revision"` + + Migrated bool `json:"-"` +} + +type folder struct { + ID int `json:"id"` + Title string `json:"title"` + ListIds []int `json:"list_ids"` + CreatedAt time.Time `json:"created_at"` + CreatedByRequestID string `json:"created_by_request_id"` + UpdatedAt time.Time `json:"updated_at"` + Type string `json:"type"` + Revision int `json:"revision"` +} + +type note struct { + ID int `json:"id"` + TaskID int `json:"task_id"` + Content string `json:"content"` + CreatedAt time.Time `json:"created_at"` + UpdatedAt time.Time `json:"updated_at"` + Revision int `json:"revision"` +} + +type file struct { + ID int `json:"id"` + URL string `json:"url"` + TaskID int `json:"task_id"` + ListID int `json:"list_id"` + UserID int `json:"user_id"` + FileName string `json:"file_name"` + ContentType string `json:"content_type"` + FileSize int `json:"file_size"` + LocalCreatedAt time.Time `json:"local_created_at"` + CreatedAt time.Time `json:"created_at"` + UpdatedAt time.Time `json:"updated_at"` + Type string `json:"type"` + Revision int `json:"revision"` +} + +type reminder struct { + ID int `json:"id"` + Date time.Time `json:"date"` + TaskID int `json:"task_id"` + Revision int `json:"revision"` + Type string `json:"type"` + CreatedAt time.Time `json:"created_at"` + UpdatedAt time.Time `json:"updated_at"` +} + +type subtask struct { + ID int `json:"id"` + TaskID int `json:"task_id"` + CreatedAt time.Time `json:"created_at"` + CreatedByID int `json:"created_by_id"` + Revision int `json:"revision"` + Title string `json:"title"` +} + +type wunderlistContents struct { + tasks []*task + lists []*list + folders []*folder + notes []*note + files []*file + reminders []*reminder + subtasks []*subtask +} + +func convertListForFolder(listID int, list *list, content *wunderlistContents) (*models.List, error) { + + l := &models.List{ + Title: list.Title, + Created: list.CreatedAt.Unix(), + } + + // Find all tasks belonging to this list and put them in + for _, t := range content.tasks { + if t.ListID == listID { + newTask := &models.Task{ + Text: t.Title, + Created: t.CreatedAt.Unix(), + Done: t.Completed, + } + + // Set Done At + if newTask.Done { + newTask.DoneAtUnix = t.CompletedAt.Unix() + } + + // Parse the due date + if t.DueDate != "" { + dueDate, err := time.Parse("2006-01-02", t.DueDate) + if err != nil { + return nil, err + } + newTask.DueDateUnix = dueDate.Unix() + } + + // Find related notes + for _, n := range content.notes { + if n.TaskID == t.ID { + newTask.Description = n.Content + } + } + + // Attachments + for _, f := range content.files { + if f.TaskID == t.ID { + // Download the attachment and put it in the file + resp, err := http.Get(f.URL) + if err != nil { + return nil, err + } + defer resp.Body.Close() + buf := &bytes.Buffer{} + _, err = buf.ReadFrom(resp.Body) + if err != nil { + return nil, err + } + + newTask.Attachments = append(newTask.Attachments, &models.TaskAttachment{ + File: &files.File{ + Name: f.FileName, + Mime: f.ContentType, + Size: uint64(f.FileSize), + Created: f.CreatedAt, + CreatedUnix: f.CreatedAt.Unix(), + // We directly pass the file contents here to have a way to link the attachment to the file later. + // Because we don't have an ID for our task at this point of the migration, we cannot just throw all + // attachments in a slice and do the work of downloading and properly storing them later. + FileContent: buf.Bytes(), + }, + Created: f.CreatedAt.Unix(), + }) + } + } + + // Subtasks + for _, s := range content.subtasks { + if s.TaskID == t.ID { + if newTask.RelatedTasks[models.RelationKindSubtask] == nil { + newTask.RelatedTasks = make(models.RelatedTaskMap) + } + newTask.RelatedTasks[models.RelationKindSubtask] = append(newTask.RelatedTasks[models.RelationKindSubtask], &models.Task{ + Text: s.Title, + }) + } + } + + // Reminders + for _, r := range content.reminders { + if r.TaskID == t.ID { + newTask.RemindersUnix = append(newTask.RemindersUnix, r.Date.Unix()) + } + } + + l.Tasks = append(l.Tasks, newTask) + } + } + return l, nil +} + +func convertWunderlistToVikunja(content *wunderlistContents) (fullVikunjaHierachie []*models.NamespaceWithLists, err error) { + + // Make a map from the list with the key being list id for easier handling + listMap := make(map[int]*list, len(content.lists)) + for _, l := range content.lists { + listMap[l.ID] = l + } + + // First, we look through all folders and create namespaces for them. + for _, folder := range content.folders { + namespace := &models.NamespaceWithLists{ + Namespace: models.Namespace{ + Name: folder.Title, + Created: folder.CreatedAt.Unix(), + Updated: folder.UpdatedAt.Unix(), + }, + } + + // Then find all lists for that folder + for _, listID := range folder.ListIds { + if list, exists := listMap[listID]; exists { + l, err := convertListForFolder(listID, list, content) + if err != nil { + return nil, err + } + namespace.Lists = append(namespace.Lists, l) + // And mark the list as migrated so we don't iterate over it again + list.Migrated = true + } + } + + // And then finally put the namespace (which now has all the details) back in the full array. + fullVikunjaHierachie = append(fullVikunjaHierachie, namespace) + } + + // At the end, loop over all lists which don't belong to a namespace and put them in a default namespace + if len(listMap) > 0 { + newNamespace := &models.NamespaceWithLists{ + Namespace: models.Namespace{ + Name: "Migrated from wunderlist", + }, + } + + for _, list := range listMap { + + if list.Migrated { + continue + } + + l, err := convertListForFolder(list.ID, list, content) + if err != nil { + return nil, err + } + newNamespace.Lists = append(newNamespace.Lists, l) + } + + fullVikunjaHierachie = append(fullVikunjaHierachie, newNamespace) + } + + return +} + +func makeAuthGetRequest(token *wunderlistAuthToken, urlPart string, v interface{}, urlParams url.Values) error { + req, err := http.NewRequest(http.MethodGet, "https://a.wunderlist.com/api/v1/"+urlPart, nil) + if err != nil { + return err + } + req.Header.Set("X-Access-Token", token.AccessToken) + req.Header.Set("X-Client-ID", config.MigrationWunderlistClientID.GetString()) + req.URL.RawQuery = urlParams.Encode() + + resp, err := http.DefaultClient.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + + buf := &bytes.Buffer{} + _, err = buf.ReadFrom(resp.Body) + if err != nil { + return err + } + + if resp.StatusCode > 399 { + return fmt.Errorf("wunderlist API Error: Status Code: %d, Response was: %s", resp.StatusCode, buf.String()) + } + + // If the response is an empty json array, we need to exit here, otherwise this breaks the json parser since it + // expects a null for an empty slice + str := buf.String() + if str == "[]" { + return nil + } + + return json.Unmarshal(buf.Bytes(), v) +} + +// Migrate migrates a user's wunderlist lists, tasks, etc. +// @Summary Migrate all lists, tasks etc. from wunderlist +// @Description Migrates all folders, lists, tasks, notes, reminders, subtasks and files from wunderlist to vikunja. +// @tags migration +// @Accept json +// @Produce json +// @Security JWTKeyAuth +// @Param migrationCode body wunderlist.Migration true "The auth code previously obtained from the auth url. See the docs for /migration/wunderlist/auth." +// @Success 200 {object} models.Message "A message telling you everything was migrated successfully." +// @Failure 500 {object} models.Message "Internal server error" +// @Router /migration/wunderlist/migrate [post] +func (w *Migration) Migrate(user *models.User) (err error) { + + log.Debugf("[Wunderlist migration] Starting wunderlist migration for user %d", user.ID) + + // Struct init + wContent := &wunderlistContents{ + tasks: []*task{}, + lists: []*list{}, + folders: []*folder{}, + notes: []*note{}, + files: []*file{}, + reminders: []*reminder{}, + subtasks: []*subtask{}, + } + + // 0. Get api token from oauth user token + authRequest := wunderlistAuthRequest{ + ClientID: config.MigrationWunderlistClientID.GetString(), + ClientSecret: config.MigrationWunderlistClientSecret.GetString(), + Code: w.Code, + } + jsonAuth, err := json.Marshal(authRequest) + if err != nil { + return + } + resp, err := http.Post("https://www.wunderlist.com/oauth/access_token", "application/json", bytes.NewBuffer(jsonAuth)) + if err != nil { + return + } + + authToken := &wunderlistAuthToken{} + err = json.NewDecoder(resp.Body).Decode(authToken) + if err != nil { + return + } + + log.Debugf("[Wunderlist migration] Start getting all data from wunderlist for user %d", user.ID) + + // 1. Get all folders + err = makeAuthGetRequest(authToken, "folders", &wContent.folders, nil) + if err != nil { + return + } + + // 2. Get all lists + err = makeAuthGetRequest(authToken, "lists", &wContent.lists, nil) + if err != nil { + return + } + + for _, l := range wContent.lists { + + listQueryParam := url.Values{"list_id": []string{strconv.Itoa(l.ID)}} + + // 3. Get all tasks for each list + tasks := []*task{} + err = makeAuthGetRequest(authToken, "tasks", &tasks, listQueryParam) + if err != nil { + return + } + wContent.tasks = append(wContent.tasks, tasks...) + + // 3. Get all done tasks for each list + doneTasks := []*task{} + err = makeAuthGetRequest(authToken, "tasks", &doneTasks, url.Values{"list_id": []string{strconv.Itoa(l.ID)}, "completed": []string{"true"}}) + if err != nil { + return + } + wContent.tasks = append(wContent.tasks, doneTasks...) + + // 4. Get all notes for all lists + notes := []*note{} + err = makeAuthGetRequest(authToken, "notes", ¬es, listQueryParam) + if err != nil { + return + } + wContent.notes = append(wContent.notes, notes...) + + // 5. Get all files for all lists + fils := []*file{} + err = makeAuthGetRequest(authToken, "files", &fils, listQueryParam) + if err != nil { + return + } + wContent.files = append(wContent.files, fils...) + + // 6. Get all reminders for all lists + reminders := []*reminder{} + err = makeAuthGetRequest(authToken, "reminders", &reminders, listQueryParam) + if err != nil { + return + } + wContent.reminders = append(wContent.reminders, reminders...) + + // 7. Get all subtasks for all lists + subtasks := []*subtask{} + err = makeAuthGetRequest(authToken, "subtasks", &subtasks, listQueryParam) + if err != nil { + return + } + wContent.subtasks = append(wContent.subtasks, subtasks...) + } + + log.Debugf("[Wunderlist migration] Got all data from wunderlist for user %d", user.ID) + log.Debugf("[Wunderlist migration] Migrating data to vikunja format for user %d", user.ID) + + // Convert + Insert everything + fullVikunjaHierachie, err := convertWunderlistToVikunja(wContent) + if err != nil { + return + } + + log.Debugf("[Wunderlist migration] Done migrating data to vikunja format for user %d", user.ID) + log.Debugf("[Wunderlist migration] Insert data into db for user %d", user.ID) + + err = migration.InsertFromStructure(fullVikunjaHierachie, user) + + log.Debugf("[Wunderlist migration] Done inserting data into db for user %d", user.ID) + log.Debugf("[Wunderlist migration] Wunderlist migration for user %d done", user.ID) + + return err +} + +// AuthURL returns the url users need to authenticate against +// @Summary Get the auth url from wunderlist +// @Description Returns the auth url where the user needs to get its auth code. This code can then be used to migrate everything from wunderlist to Vikunja. +// @tags migration +// @Produce json +// @Security JWTKeyAuth +// @Success 200 {object} handler.AuthURL "The auth url." +// @Failure 500 {object} models.Message "Internal server error" +// @Router /migration/wunderlist/auth [get] +func (w *Migration) AuthURL() string { + return "https://www.wunderlist.com/oauth/authorize?client_id=" + + config.MigrationWunderlistClientID.GetString() + + "&redirect_uri=" + + config.MigrationWunderlistRedirectURL.GetString() + + "&state=" + utils.MakeRandomString(32) +} diff --git a/pkg/modules/migration/wunderlist/wunderlist_test.go b/pkg/modules/migration/wunderlist/wunderlist_test.go new file mode 100644 index 0000000000..17aa6c380c --- /dev/null +++ b/pkg/modules/migration/wunderlist/wunderlist_test.go @@ -0,0 +1,352 @@ +// Vikunja is a todo-list application to facilitate your life. +// Copyright 2018-2020 Vikunja and contributors. All rights reserved. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU General Public License for more details. +// +// You should have received a copy of the GNU General Public License +// along with this program. If not, see . + +package wunderlist + +import ( + "code.vikunja.io/api/pkg/config" + "code.vikunja.io/api/pkg/files" + "code.vikunja.io/api/pkg/models" + "github.com/stretchr/testify/assert" + "gopkg.in/d4l3k/messagediff.v1" + "io/ioutil" + "strconv" + "testing" + "time" +) + +func TestWunderlistParsing(t *testing.T) { + + config.InitConfig() + + time1, err := time.Parse(time.RFC3339Nano, "2013-08-30T08:29:46.203Z") + assert.NoError(t, err) + time2, err := time.Parse(time.RFC3339Nano, "2013-08-30T08:36:13.273Z") + assert.NoError(t, err) + time3, err := time.Parse(time.RFC3339Nano, "2013-09-05T08:36:13.273Z") + assert.NoError(t, err) + time4, err := time.Parse(time.RFC3339Nano, "2013-08-02T11:58:55Z") + assert.NoError(t, err) + + exampleFile, err := ioutil.ReadFile(config.ServiceRootpath.GetString() + "/pkg/modules/migration/wunderlist/testimage.jpg") + assert.NoError(t, err) + + createTestTask := func(id, listID int, done bool) *task { + completedAt, err := time.Parse(time.RFC3339Nano, "1970-01-01T00:00:00Z") + assert.NoError(t, err) + if done { + completedAt = time1 + } + return &task{ + ID: id, + AssigneeID: 123, + CreatedAt: time1, + DueDate: "2013-09-05", + ListID: listID, + Title: "Ipsum" + strconv.Itoa(id), + Completed: done, + CompletedAt: completedAt, + } + } + + createTestNote := func(id, taskID int) *note { + return ¬e{ + ID: id, + TaskID: taskID, + Content: "Lorem Ipsum dolor sit amet", + CreatedAt: time3, + UpdatedAt: time2, + } + } + + fixtures := &wunderlistContents{ + folders: []*folder{ + { + ID: 123, + Title: "Lorem Ipsum", + ListIds: []int{1, 2, 3, 4}, + CreatedAt: time1, + UpdatedAt: time2, + }, + }, + lists: []*list{ + { + ID: 1, + CreatedAt: time1, + Title: "Lorem1", + }, + { + ID: 2, + CreatedAt: time1, + Title: "Lorem2", + }, + { + ID: 3, + CreatedAt: time1, + Title: "Lorem3", + }, + { + ID: 4, + CreatedAt: time1, + Title: "Lorem4", + }, + { + ID: 5, + CreatedAt: time4, + Title: "List without a namespace", + }, + }, + tasks: []*task{ + createTestTask(1, 1, false), + createTestTask(2, 1, false), + createTestTask(3, 2, true), + createTestTask(4, 2, false), + createTestTask(5, 3, false), + createTestTask(6, 3, true), + createTestTask(7, 3, true), + createTestTask(8, 3, false), + createTestTask(9, 4, true), + createTestTask(10, 4, true), + }, + notes: []*note{ + createTestNote(1, 1), + createTestNote(2, 2), + createTestNote(3, 3), + }, + files: []*file{ + { + ID: 1, + URL: "https://vikunja.io/testimage.jpg", // Using an image which we are hosting, so it'll still be up + TaskID: 1, + ListID: 1, + FileName: "file.md", + ContentType: "text/plain", + FileSize: 12345, + CreatedAt: time2, + UpdatedAt: time4, + }, + { + ID: 2, + URL: "https://vikunja.io/testimage.jpg", + TaskID: 3, + ListID: 2, + FileName: "file2.md", + ContentType: "text/plain", + FileSize: 12345, + CreatedAt: time3, + UpdatedAt: time4, + }, + }, + reminders: []*reminder{ + { + ID: 1, + Date: time4, + TaskID: 1, + CreatedAt: time4, + UpdatedAt: time4, + }, + { + ID: 2, + Date: time3, + TaskID: 4, + CreatedAt: time3, + UpdatedAt: time3, + }, + }, + subtasks: []*subtask{ + { + ID: 1, + TaskID: 2, + CreatedAt: time4, + Title: "LoremSub1", + }, + { + ID: 2, + TaskID: 2, + CreatedAt: time4, + Title: "LoremSub2", + }, + { + ID: 3, + TaskID: 4, + CreatedAt: time4, + Title: "LoremSub3", + }, + }, + } + + expectedHierachie := []*models.NamespaceWithLists{ + { + Namespace: models.Namespace{ + Name: "Lorem Ipsum", + Created: time1.Unix(), + Updated: time2.Unix(), + }, + Lists: []*models.List{ + { + Created: time1.Unix(), + Title: "Lorem1", + Tasks: []*models.Task{ + { + Text: "Ipsum1", + DueDateUnix: 1378339200, + Created: time1.Unix(), + Description: "Lorem Ipsum dolor sit amet", + Attachments: []*models.TaskAttachment{ + { + File: &files.File{ + Name: "file.md", + Mime: "text/plain", + Size: 12345, + Created: time2, + CreatedUnix: time2.Unix(), + FileContent: exampleFile, + }, + Created: time2.Unix(), + }, + }, + RemindersUnix: []int64{time4.Unix()}, + }, + { + Text: "Ipsum2", + DueDateUnix: 1378339200, + Created: time1.Unix(), + Description: "Lorem Ipsum dolor sit amet", + RelatedTasks: map[models.RelationKind][]*models.Task{ + models.RelationKindSubtask: { + { + Text: "LoremSub1", + }, + { + Text: "LoremSub2", + }, + }, + }, + }, + }, + }, + { + Created: time1.Unix(), + Title: "Lorem2", + Tasks: []*models.Task{ + { + Text: "Ipsum3", + Done: true, + DoneAtUnix: time1.Unix(), + DueDateUnix: 1378339200, + Created: time1.Unix(), + Description: "Lorem Ipsum dolor sit amet", + Attachments: []*models.TaskAttachment{ + { + File: &files.File{ + Name: "file2.md", + Mime: "text/plain", + Size: 12345, + Created: time3, + CreatedUnix: time3.Unix(), + FileContent: exampleFile, + }, + Created: time3.Unix(), + }, + }, + }, + { + Text: "Ipsum4", + DueDateUnix: 1378339200, + Created: time1.Unix(), + RemindersUnix: []int64{time3.Unix()}, + RelatedTasks: map[models.RelationKind][]*models.Task{ + models.RelationKindSubtask: { + { + Text: "LoremSub3", + }, + }, + }, + }, + }, + }, + { + Created: time1.Unix(), + Title: "Lorem3", + Tasks: []*models.Task{ + { + Text: "Ipsum5", + DueDateUnix: 1378339200, + Created: time1.Unix(), + }, + { + Text: "Ipsum6", + DueDateUnix: 1378339200, + Created: time1.Unix(), + Done: true, + DoneAtUnix: time1.Unix(), + }, + { + Text: "Ipsum7", + DueDateUnix: 1378339200, + Created: time1.Unix(), + Done: true, + DoneAtUnix: time1.Unix(), + }, + { + Text: "Ipsum8", + DueDateUnix: 1378339200, + Created: time1.Unix(), + }, + }, + }, + { + Created: time1.Unix(), + Title: "Lorem4", + Tasks: []*models.Task{ + { + Text: "Ipsum9", + DueDateUnix: 1378339200, + Created: time1.Unix(), + Done: true, + DoneAtUnix: time1.Unix(), + }, + { + Text: "Ipsum10", + DueDateUnix: 1378339200, + Created: time1.Unix(), + Done: true, + DoneAtUnix: time1.Unix(), + }, + }, + }, + }, + }, + { + Namespace: models.Namespace{ + Name: "Migrated from wunderlist", + }, + Lists: []*models.List{ + { + Created: time4.Unix(), + Title: "List without a namespace", + }, + }, + }, + } + + hierachie, err := convertWunderlistToVikunja(fixtures) + assert.NoError(t, err) + assert.NotNil(t, hierachie) + if diff, equal := messagediff.PrettyDiff(hierachie, expectedHierachie); !equal { + t.Errorf("ListUser.ReadAll() = %v, want %v, diff: %v", hierachie, expectedHierachie, diff) + } +} diff --git a/pkg/routes/api/v1/info.go b/pkg/routes/api/v1/info.go index afb39f9e60..3f9096855d 100644 --- a/pkg/routes/api/v1/info.go +++ b/pkg/routes/api/v1/info.go @@ -24,12 +24,13 @@ import ( ) type vikunjaInfos struct { - Version string `json:"version"` - FrontendURL string `json:"frontend_url"` - Motd string `json:"motd"` - LinkSharingEnabled bool `json:"link_sharing_enabled"` - MaxFileSize string `json:"max_file_size"` - RegistrationEnabled bool `json:"registration_enabled"` + Version string `json:"version"` + FrontendURL string `json:"frontend_url"` + Motd string `json:"motd"` + LinkSharingEnabled bool `json:"link_sharing_enabled"` + MaxFileSize string `json:"max_file_size"` + RegistrationEnabled bool `json:"registration_enabled"` + AvailableMigrators []string `json:"available_migrators"` } // Info is the handler to get infos about this vikunja instance @@ -40,12 +41,16 @@ type vikunjaInfos struct { // @Success 200 {object} v1.vikunjaInfos // @Router /info [get] func Info(c echo.Context) error { - return c.JSON(http.StatusOK, vikunjaInfos{ + infos := vikunjaInfos{ Version: version.Version, FrontendURL: config.ServiceFrontendurl.GetString(), Motd: config.ServiceMotd.GetString(), LinkSharingEnabled: config.ServiceEnableLinkSharing.GetBool(), MaxFileSize: config.FilesMaxSize.GetString(), RegistrationEnabled: config.ServiceEnableRegistration.GetBool(), - }) + } + if config.MigrationWunderlistEnable.GetBool() { + infos.AvailableMigrators = append(infos.AvailableMigrators, "wunderlist") + } + return c.JSON(http.StatusOK, infos) } diff --git a/pkg/routes/routes.go b/pkg/routes/routes.go index e5e8f0cc6e..68c1d8c3c6 100644 --- a/pkg/routes/routes.go +++ b/pkg/routes/routes.go @@ -47,6 +47,9 @@ import ( "code.vikunja.io/api/pkg/config" "code.vikunja.io/api/pkg/log" "code.vikunja.io/api/pkg/models" + "code.vikunja.io/api/pkg/modules/migration" + migrationHandler "code.vikunja.io/api/pkg/modules/migration/handler" + "code.vikunja.io/api/pkg/modules/migration/wunderlist" apiv1 "code.vikunja.io/api/pkg/routes/api/v1" "code.vikunja.io/api/pkg/routes/caldav" _ "code.vikunja.io/api/pkg/swagger" // To generate swagger docs @@ -372,6 +375,20 @@ func registerAPIRoutes(a *echo.Group) { } a.PUT("/teams/:team/members", teamMemberHandler.CreateWeb) a.DELETE("/teams/:team/members/:user", teamMemberHandler.DeleteWeb) + + // Migrations + m := a.Group("/migration") + + // Wunderlist + if config.MigrationWunderlistEnable.GetBool() { + wunderlistMigrationHandler := &migrationHandler.MigrationWeb{ + MigrationStruct: func() migration.Migrator { + return &wunderlist.Migration{} + }, + } + m.GET("/wunderlist/auth", wunderlistMigrationHandler.AuthURL) + m.POST("/wunderlist/migrate", wunderlistMigrationHandler.Migrate) + } } func registerCalDavRoutes(c *echo.Group) { diff --git a/pkg/swagger/docs.go b/pkg/swagger/docs.go index 80e202554f..5b10b2ed6d 100644 --- a/pkg/swagger/docs.go +++ b/pkg/swagger/docs.go @@ -1,6 +1,6 @@ // GENERATED BY THE COMMAND ABOVE; DO NOT EDIT // This file was generated by swaggo/swag at -// 2019-12-07 22:54:02.661375666 +0100 CET m=+0.164990732 +// 2020-01-19 16:18:04.294790395 +0100 CET m=+0.176548843 package swagger @@ -1615,6 +1615,83 @@ var doc = `{ } } }, + "/migration/wunderlist/auth": { + "get": { + "security": [ + { + "JWTKeyAuth": [] + } + ], + "description": "Returns the auth url where the user needs to get its auth code. This code can then be used to migrate everything from wunderlist to Vikunja.", + "produces": [ + "application/json" + ], + "tags": [ + "migration" + ], + "summary": "Get the auth url from wunderlist", + "responses": { + "200": { + "description": "The auth url.", + "schema": { + "$ref": "#/definitions/handler.AuthURL" + } + }, + "500": { + "description": "Internal server error", + "schema": { + "$ref": "#/definitions/models.Message" + } + } + } + } + }, + "/migration/wunderlist/migrate": { + "post": { + "security": [ + { + "JWTKeyAuth": [] + } + ], + "description": "Migrates all folders, lists, tasks, notes, reminders, subtasks and files from wunderlist to vikunja.", + "consumes": [ + "application/json" + ], + "produces": [ + "application/json" + ], + "tags": [ + "migration" + ], + "summary": "Migrate all lists, tasks etc. from wunderlist", + "parameters": [ + { + "description": "The auth code previously obtained from the auth url. See the docs for /migration/wunderlist/auth.", + "name": "migrationCode", + "in": "body", + "required": true, + "schema": { + "type": "object", + "$ref": "#/definitions/wunderlist.Migration" + } + } + ], + "responses": { + "200": { + "description": "A message telling you everything was migrated successfully.", + "schema": { + "$ref": "#/definitions/models.Message" + } + }, + "500": { + "description": "Internal server error", + "schema": { + "$ref": "#/definitions/models.Message" + } + } + } + } + }, "/namespace/{id}": { "post": { "security": [ @@ -4463,6 +4540,14 @@ var doc = `{ } } }, + "handler.AuthURL": { + "type": "object", + "properties": { + "url": { + "type": "string" + } + } + }, "models.APIUserPassword": { "type": "object", "properties": { @@ -5527,6 +5612,15 @@ var doc = `{ "type": "string" } } + }, + "wunderlist.Migration": { + "type": "object", + "properties": { + "code": { + "description": "Code is the code used to get a user api token", + "type": "string" + } + } } }, "securityDefinitions": { diff --git a/pkg/swagger/swagger.json b/pkg/swagger/swagger.json index 88ca4f398a..49d341ef90 100644 --- a/pkg/swagger/swagger.json +++ b/pkg/swagger/swagger.json @@ -1597,6 +1597,83 @@ } } }, + "/migration/wunderlist/auth": { + "get": { + "security": [ + { + "JWTKeyAuth": [] + } + ], + "description": "Returns the auth url where the user needs to get its auth code. This code can then be used to migrate everything from wunderlist to Vikunja.", + "produces": [ + "application/json" + ], + "tags": [ + "migration" + ], + "summary": "Get the auth url from wunderlist", + "responses": { + "200": { + "description": "The auth url.", + "schema": { + "$ref": "#/definitions/handler.AuthURL" + } + }, + "500": { + "description": "Internal server error", + "schema": { + "$ref": "#/definitions/models.Message" + } + } + } + } + }, + "/migration/wunderlist/migrate": { + "post": { + "security": [ + { + "JWTKeyAuth": [] + } + ], + "description": "Migrates all folders, lists, tasks, notes, reminders, subtasks and files from wunderlist to vikunja.", + "consumes": [ + "application/json" + ], + "produces": [ + "application/json" + ], + "tags": [ + "migration" + ], + "summary": "Migrate all lists, tasks etc. from wunderlist", + "parameters": [ + { + "description": "The auth code previously obtained from the auth url. See the docs for /migration/wunderlist/auth.", + "name": "migrationCode", + "in": "body", + "required": true, + "schema": { + "type": "object", + "$ref": "#/definitions/wunderlist.Migration" + } + } + ], + "responses": { + "200": { + "description": "A message telling you everything was migrated successfully.", + "schema": { + "$ref": "#/definitions/models.Message" + } + }, + "500": { + "description": "Internal server error", + "schema": { + "$ref": "#/definitions/models.Message" + } + } + } + } + }, "/namespace/{id}": { "post": { "security": [ @@ -4444,6 +4521,14 @@ } } }, + "handler.AuthURL": { + "type": "object", + "properties": { + "url": { + "type": "string" + } + } + }, "models.APIUserPassword": { "type": "object", "properties": { @@ -5508,6 +5593,15 @@ "type": "string" } } + }, + "wunderlist.Migration": { + "type": "object", + "properties": { + "code": { + "description": "Code is the code used to get a user api token", + "type": "string" + } + } } }, "securityDefinitions": { diff --git a/pkg/swagger/swagger.yaml b/pkg/swagger/swagger.yaml index abd41bfa05..6f59419c7c 100644 --- a/pkg/swagger/swagger.yaml +++ b/pkg/swagger/swagger.yaml @@ -13,6 +13,11 @@ definitions: size: type: integer type: object + handler.AuthURL: + properties: + url: + type: string + type: object models.APIUserPassword: properties: email: @@ -860,6 +865,12 @@ definitions: version: type: string type: object + wunderlist.Migration: + properties: + code: + description: Code is the code used to get a user api token + type: string + type: object info: contact: email: hello@vikunja.io @@ -1935,6 +1946,57 @@ paths: summary: Login tags: - user + /migration/wunderlist/auth: + get: + description: Returns the auth url where the user needs to get its auth code. + This code can then be used to migrate everything from wunderlist to Vikunja. + produces: + - application/json + responses: + "200": + description: The auth url. + schema: + $ref: '#/definitions/handler.AuthURL' + "500": + description: Internal server error + schema: + $ref: '#/definitions/models.Message' + security: + - JWTKeyAuth: [] + summary: Get the auth url from wunderlist + tags: + - migration + /migration/wunderlist/migrate: + post: + consumes: + - application/json + description: Migrates all folders, lists, tasks, notes, reminders, subtasks + and files from wunderlist to vikunja. + parameters: + - description: The auth code previously obtained from the auth url. See the + docs for /migration/wunderlist/auth. + in: body + name: migrationCode + required: true + schema: + $ref: '#/definitions/wunderlist.Migration' + type: object + produces: + - application/json + responses: + "200": + description: A message telling you everything was migrated successfully. + schema: + $ref: '#/definitions/models.Message' + "500": + description: Internal server error + schema: + $ref: '#/definitions/models.Message' + security: + - JWTKeyAuth: [] + summary: Migrate all lists, tasks etc. from wunderlist + tags: + - migration /namespace/{id}: post: consumes: