User Data Export and import (#967)

Co-authored-by: kolaente <k@knt.li>
Reviewed-on: vikunja/api#967
Co-authored-by: konrad <k@knt.li>
Co-committed-by: konrad <k@knt.li>

docs/config.yml

@@ -31,10 +31,10 @@ menu:
       url: https://vikunja.io/en/
       weight: 10
     - name: Features
-      url: https://vikunja.io/en/features
+      url: https://vikunja.io/features
       weight: 20
     - name: Download
-      url: https://vikunja.io/en/download
+      url: https://vikunja.io/download
       weight: 30
     - name: Docs
       url: https://vikunja.io/docs

docs/content/doc/development/migration.md

@@ -14,7 +14,17 @@ It is possible to migrate data from other to-do services to Vikunja.
 To make this easier, we have put together a few helpers which are documented on this page.
 
 In general, each migrator implements a migrator interface which is then called from a client.
-The interface makes it possible to use helper methods which handle http an focus only on the implementation of the migrator itself.
+The interface makes it possible to use helper methods which handle http and focus only on the implementation of the migrator itself.
+
+There are two ways of migrating data from another service:
+1. Through the auth-based flow where the user gives you access to their data at the third-party service through an 
+   oauth flow. You can then call the service's api on behalf of your user to get all the data.
+   The Todoist, Trello and Microsoft To-Do Migrators use this pattern.
+2. A file migration where the user uploads a file obtained from some third-party service. In your migrator, you need 
+   to parse the file and create the lists, tasks etc.
+   The Vikunja File Import uses this pattern.
+
+To differentiate the two, there are two different interfaces you must implement.
 
 {{< table_of_contents >}}
 
@@ -23,13 +33,16 @@ The interface makes it possible to use helper methods which handle http an focus
 All migrator implementations live in their own package in `pkg/modules/migration/<name-of-the-service>`.
 When creating a new migrator, you should place all related code inside that module.
 
-## Migrator interface
+## Migrator Interface
 
 The migrator interface is defined as follows:
 
 ```go
 // Migrator is the basic migrator interface which is shared among all migrators
 type Migrator interface {
+	// Name holds the name of the migration.
+	// This is used to show the name to users and to keep track of users who already migrated.
+	Name() string
 	// Migrate is the interface used to migrate a user's tasks from another platform to vikunja.
 	// The user object is the user who's tasks will be migrated.
 	Migrate(user *models.User) error
@@ -37,9 +50,20 @@ type Migrator interface {
 	// The use case for this are Oauth flows, where the server token should remain hidden and not
 	// known to the frontend.
 	AuthURL() string
+}
+```
+
+## File Migrator Interface
+
+```go
+// FileMigrator handles importing Vikunja data from a file. The implementation of it determines the format.
+type FileMigrator interface {
 	// Name holds the name of the migration.
 	// This is used to show the name to users and to keep track of users who already migrated.
 	Name() string
+	// Migrate is the interface used to migrate a user's tasks, list and other things from a file to vikunja.
+	// The user object is the user who's tasks will be migrated.
+	Migrate(user *user.User, file io.ReaderAt, size int64) error
 }
 ```
 
@@ -54,15 +78,26 @@ authUrl, Status and Migrate methods.
 ```go
 // This is an example for the Wunderlist migrator
 if config.MigrationWunderlistEnable.GetBool() {
-    wunderlistMigrationHandler := &migrationHandler.MigrationWeb{
+	wunderlistMigrationHandler := &migrationHandler.MigrationWeb{
 		MigrationStruct: func() migration.Migrator {
 			return &wunderlist.Migration{}
 		},
 	}
-    wunderlistMigrationHandler.RegisterRoutes(m)
+	wunderlistMigrationHandler.RegisterRoutes(m)
 }
 ```
 
+And for the file migrator:
+
+```go
+vikunjaFileMigrationHandler := &migrationHandler.FileMigratorWeb{
+	MigrationStruct: func() migration.FileMigrator {
+		return &vikunja_file.FileMigrator{}
+	},
+}
+vikunjaFileMigrationHandler.RegisterRoutes(m)
+```
+
 You should also document the routes with [swagger annotations]({{< ref "swagger-docs.md" >}}).
 
 ## Insertion helper method
@@ -70,7 +105,8 @@ You should also document the routes with [swagger annotations]({{< ref "swagger-
 There is a method available in the `migration` package which takes a fully nested Vikunja structure and creates it with all relations. 
 This means you start by adding a namespace, then add lists inside of that namespace, then tasks in the lists and so on.
 
-The root structure must be present as `[]*models.NamespaceWithLists`.
+The root structure must be present as `[]*models.NamespaceWithListsAndTasks`. It allows to represent all of Vikunja's 
+hierachie as a single data structure.
 
 Then call the method like so:
 
@@ -85,14 +121,16 @@ err = migration.InsertFromStructure(fullVikunjaHierachie, user)
 
 ## Configuration
 
-You should add at least an option to enable or disable the migration.
+If your migrator is an oauth-based one, you should add at least an option to enable or disable it.
 Chances are, you'll need some more options for things like client ID and secret 
 (if the other service uses oAuth as an authentication flow).
 
 The easiest way to implement an on/off switch is to check whether your migration service is enabled or not when 
-registering the routes, and then simply don't registering the routes in the case it is disabled.
+registering the routes, and then simply don't registering the routes in case it is disabled.
+
+File based migrators can always be enabled.
 
 ### Making the migrator public in `/info` 
 
 You should make your migrator available in the `/info` endpoint so that frontends can display options to enable them or not.
-To do this, add an entry to `pkg/routes/api/v1/info.go`.
+To do this, add an entry to the `AvailableMigrators` field in `pkg/routes/api/v1/info.go`.

pkg/caldav/parsing.go

@@ -26,7 +26,7 @@ import (
 	"github.com/laurent22/ical-go"
 )
 
-func GetCaldavTodosForTasks(list *models.List, listTasks []*models.Task) string {
+func GetCaldavTodosForTasks(list *models.ListWithTasksAndBuckets, listTasks []*models.TaskWithComments) string {
 
 	// Make caldav todos from Vikunja todos
 	var caldavtodos []*Todo

pkg/files/files.go

@@ -22,6 +22,8 @@ import (
 	"strconv"
 	"time"
 
+	"xorm.io/xorm"
+
 	"code.vikunja.io/api/pkg/config"
 	"code.vikunja.io/api/pkg/db"
 	"code.vikunja.io/api/pkg/log"
@@ -77,7 +79,18 @@ func Create(f io.Reader, realname string, realsize uint64, a web.Auth) (file *Fi
 
 // CreateWithMime creates a new file from an FileHeader and sets its mime type
 func CreateWithMime(f io.Reader, realname string, realsize uint64, a web.Auth, mime string) (file *File, err error) {
+	s := db.NewSession()
+	defer s.Close()
 
+	file, err = CreateWithMimeAndSession(s, f, realname, realsize, a, mime)
+	if err != nil {
+		_ = s.Rollback()
+		return
+	}
+	return
+}
+
+func CreateWithMimeAndSession(s *xorm.Session, f io.Reader, realname string, realsize uint64, a web.Auth, mime string) (file *File, err error) {
 	// Get and parse the configured file size
 	var maxSize datasize.ByteSize
 	err = maxSize.UnmarshalText([]byte(config.FilesMaxSize.GetString()))
@@ -96,21 +109,13 @@ func CreateWithMime(f io.Reader, realname string, realsize uint64, a web.Auth, m
 		Mime:        mime,
 	}
 
-	s := db.NewSession()
-	defer s.Close()
-
 	_, err = s.Insert(file)
 	if err != nil {
-		_ = s.Rollback()
 		return
 	}
 
 	// Save the file to storage with its new ID as path
 	err = file.Save(f)
-	if err != nil {
-		_ = s.Rollback()
-		return
-	}
 	return
 }
 

pkg/initialize/init.go

@@ -97,6 +97,7 @@ func FullInit() {
 	user.RegisterTokenCleanupCron()
 	user.RegisterDeletionNotificationCron()
 	models.RegisterUserDeletionCron()
+	models.RegisterOldExportCleanupCron()
 
 	// Start processing events
 	go func() {

pkg/migration/20210829194722.go

@@ -0,0 +1,43 @@
+// Vikunja is a to-do list application to facilitate your life.
+// Copyright 2018-2021 Vikunja and contributors. All rights reserved.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public Licensee as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public Licensee for more details.
+//
+// You should have received a copy of the GNU Affero General Public Licensee
+// along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+package migration
+
+import (
+	"src.techknowlogick.com/xormigrate"
+	"xorm.io/xorm"
+)
+
+type users20210829194722 struct {
+	ExportFileID int64 `xorm:"bigint null" json:"-"`
+}
+
+func (users20210829194722) TableName() string {
+	return "users"
+}
+
+func init() {
+	migrations = append(migrations, &xormigrate.Migration{
+		ID:          "20210829194722",
+		Description: "Add data export file id to users",
+		Migrate: func(tx *xorm.Engine) error {
+			return tx.Sync2(users20210829194722{})
+		},
+		Rollback: func(tx *xorm.Engine) error {
+			return nil
+		},
+	})
+}

pkg/models/events.go

@@ -21,6 +21,16 @@ import (
 	"code.vikunja.io/web"
 )
 
+// DataExportRequestEvent represents a DataExportRequestEvent event
+type DataExportRequestEvent struct {
+	User *user.User
+}
+
+// Name defines the name for DataExportRequestEvent
+func (t *DataExportRequestEvent) Name() string {
+	return "user.export.request"
+}
+
 /////////////////
 // Task Events //
 /////////////////
@@ -257,3 +267,13 @@ type TeamDeletedEvent struct {
 func (t *TeamDeletedEvent) Name() string {
 	return "team.deleted"
 }
+
+// UserDataExportRequestedEvent represents a UserDataExportRequestedEvent event
+type UserDataExportRequestedEvent struct {
+	User *user.User
+}
+
+// Name defines the name for UserDataExportRequestedEvent
+func (t *UserDataExportRequestedEvent) Name() string {
+	return "user.export.requested"
+}

pkg/models/export.go

@@ -0,0 +1,358 @@
+// Vikunja is a to-do list application to facilitate your life.
+// Copyright 2018-2021 Vikunja and contributors. All rights reserved.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public Licensee as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public Licensee for more details.
+//
+// You should have received a copy of the GNU Affero General Public Licensee
+// along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+package models
+
+import (
+	"archive/zip"
+	"encoding/json"
+	"fmt"
+	"io"
+	"os"
+	"strconv"
+	"time"
+
+	"code.vikunja.io/api/pkg/config"
+	"code.vikunja.io/api/pkg/cron"
+	"code.vikunja.io/api/pkg/db"
+	"code.vikunja.io/api/pkg/files"
+	"code.vikunja.io/api/pkg/log"
+	"code.vikunja.io/api/pkg/notifications"
+	"code.vikunja.io/api/pkg/user"
+	"code.vikunja.io/api/pkg/utils"
+	"code.vikunja.io/api/pkg/version"
+	"xorm.io/xorm"
+)
+
+func ExportUserData(s *xorm.Session, u *user.User) (err error) {
+	exportDir := config.ServiceRootpath.GetString() + "/files/user-export-tmp/"
+	err = os.MkdirAll(exportDir, 0700)
+	if err != nil {
+		return err
+	}
+
+	tmpFilename := exportDir + strconv.FormatInt(u.ID, 10) + "_" + time.Now().Format("2006-01-02_15-03-05") + ".zip"
+
+	// Open zip
+	dumpFile, err := os.Create(tmpFilename)
+	if err != nil {
+		return fmt.Errorf("error opening dump file: %s", err)
+	}
+	defer dumpFile.Close()
+
+	dumpWriter := zip.NewWriter(dumpFile)
+	defer dumpWriter.Close()
+
+	// Get the data
+	err = exportListsAndTasks(s, u, dumpWriter)
+	if err != nil {
+		return err
+	}
+	// Task attachment files
+	err = exportTaskAttachments(s, u, dumpWriter)
+	if err != nil {
+		return err
+	}
+	// Saved filters
+	err = exportSavedFilters(s, u, dumpWriter)
+	if err != nil {
+		return err
+	}
+	// Background files
+	err = exportListBackgrounds(s, u, dumpWriter)
+	if err != nil {
+		return err
+	}
+	// Vikunja Version
+	err = utils.WriteBytesToZip("VERSION", []byte(version.Version), dumpWriter)
+	if err != nil {
+		return err
+	}
+
+	// If we reuse the same file again, saving it as a file in Vikunja will save it as a file with 0 bytes in size.
+	// Closing and reopening does work.
+	dumpWriter.Close()
+	dumpFile.Close()
+
+	exported, err := os.Open(tmpFilename)
+	if err != nil {
+		return err
+	}
+
+	stat, err := exported.Stat()
+	if err != nil {
+		return err
+	}
+
+	exportFile, err := files.CreateWithMimeAndSession(s, exported, tmpFilename, uint64(stat.Size()), u, "application/zip")
+	if err != nil {
+		return err
+	}
+
+	// Save the file id with the user
+	u.ExportFileID = exportFile.ID
+	_, err = s.Cols("export_file_id").Update(u)
+	if err != nil {
+		return
+	}
+
+	// Remove the old file
+	err = os.Remove(exported.Name())
+	if err != nil {
+		return err
+	}
+
+	// Send a notification
+	return notifications.Notify(u, &DataExportReadyNotification{
+		User: u,
+	})
+}
+
+func exportListsAndTasks(s *xorm.Session, u *user.User, wr *zip.Writer) (err error) {
+
+	namspaces, _, _, err := (&Namespace{}).ReadAll(s, u, "", -1, 0)
+	if err != nil {
+		return err
+	}
+
+	namespaceIDs := []int64{}
+	namespaces := []*NamespaceWithListsAndTasks{}
+	for _, n := range namspaces.([]*NamespaceWithLists) {
+		if n.ID < 1 {
+			// Don't include filters
+			continue
+		}
+
+		nn := &NamespaceWithListsAndTasks{
+			Namespace: n.Namespace,
+			Lists:     []*ListWithTasksAndBuckets{},
+		}
+
+		for _, l := range n.Lists {
+			nn.Lists = append(nn.Lists, &ListWithTasksAndBuckets{
+				List:             *l,
+				BackgroundFileID: l.BackgroundFileID,
+				Tasks:            []*TaskWithComments{},
+			})
+		}
+
+		namespaceIDs = append(namespaceIDs, n.ID)
+		namespaces = append(namespaces, nn)
+	}
+
+	if len(namespaceIDs) == 0 {
+		return nil
+	}
+
+	// Get all lists
+	lists, err := getListsForNamespaces(s, namespaceIDs, true)
+	if err != nil {
+		return err
+	}
+
+	tasks, _, _, err := getTasksForLists(s, lists, u, &taskOptions{
+		page:    0,
+		perPage: -1,
+	})
+	if err != nil {
+		return err
+	}
+
+	listMap := make(map[int64]*ListWithTasksAndBuckets)
+	listIDs := []int64{}
+	for _, n := range namespaces {
+		for _, l := range n.Lists {
+			listMap[l.ID] = l
+			listIDs = append(listIDs, l.ID)
+		}
+	}
+
+	taskMap := make(map[int64]*TaskWithComments, len(tasks))
+	for _, t := range tasks {
+		taskMap[t.ID] = &TaskWithComments{
+			Task: *t,
+		}
+		listMap[t.ListID].Tasks = append(listMap[t.ListID].Tasks, taskMap[t.ID])
+	}
+
+	comments := []*TaskComment{}
+	err = s.
+		Join("LEFT", "tasks", "tasks.id = task_comments.task_id").
+		In("tasks.list_id", listIDs).
+		Find(&comments)
+	if err != nil {
+		return
+	}
+
+	for _, c := range comments {
+		taskMap[c.TaskID].Comments = append(taskMap[c.TaskID].Comments, c)
+	}
+
+	buckets := []*Bucket{}
+	err = s.In("list_id", listIDs).Find(&buckets)
+	if err != nil {
+		return
+	}
+
+	for _, b := range buckets {
+		listMap[b.ListID].Buckets = append(listMap[b.ListID].Buckets, b)
+	}
+
+	data, err := json.Marshal(namespaces)
+	if err != nil {
+		return err
+	}
+
+	return utils.WriteBytesToZip("data.json", data, wr)
+}
+
+func exportTaskAttachments(s *xorm.Session, u *user.User, wr *zip.Writer) (err error) {
+	lists, _, _, err := getRawListsForUser(
+		s,
+		&listOptions{
+			user: u,
+			page: -1,
+		},
+	)
+	if err != nil {
+		return err
+	}
+
+	tasks, _, _, err := getRawTasksForLists(s, lists, u, &taskOptions{page: -1})
+	if err != nil {
+		return err
+	}
+
+	taskIDs := []int64{}
+	for _, t := range tasks {
+		taskIDs = append(taskIDs, t.ID)
+	}
+
+	tas, err := getTaskAttachmentsByTaskIDs(s, taskIDs)
+	if err != nil {
+		return err
+	}
+
+	fs := make(map[int64]io.ReadCloser)
+	for _, ta := range tas {
+		if err := ta.File.LoadFileByID(); err != nil {
+			return err
+		}
+		fs[ta.FileID] = ta.File.File
+	}
+
+	return utils.WriteFilesToZip(fs, wr)
+}
+
+func exportSavedFilters(s *xorm.Session, u *user.User, wr *zip.Writer) (err error) {
+	filters, err := getSavedFiltersForUser(s, u)
+	if err != nil {
+		return err
+	}
+
+	data, err := json.Marshal(filters)
+	if err != nil {
+		return err
+	}
+
+	return utils.WriteBytesToZip("filters.json", data, wr)
+}
+
+func exportListBackgrounds(s *xorm.Session, u *user.User, wr *zip.Writer) (err error) {
+	lists, _, _, err := getRawListsForUser(
+		s,
+		&listOptions{
+			user: u,
+			page: -1,
+		},
+	)
+	if err != nil {
+		return err
+	}
+
+	fs := make(map[int64]io.ReadCloser)
+	for _, l := range lists {
+		if l.BackgroundFileID == 0 {
+			continue
+		}
+
+		bgFile := &files.File{
+			ID: l.BackgroundFileID,
+		}
+		err = bgFile.LoadFileByID()
+		if err != nil {
+			return
+		}
+
+		fs[l.BackgroundFileID] = bgFile.File
+	}
+
+	return utils.WriteFilesToZip(fs, wr)
+}
+
+func RegisterOldExportCleanupCron() {
+	const logPrefix = "[User Export Cleanup Cron] "
+
+	err := cron.Schedule("0 * * * *", func() {
+		s := db.NewSession()
+		defer s.Close()
+
+		users := []*user.User{}
+		err := s.Where("export_file_id IS NOT NULL AND export_file_id != ?", 0).Find(&users)
+		if err != nil {
+			log.Errorf(logPrefix+"Could not get users with export files: %s", err)
+			return
+		}
+
+		fileIDs := []int64{}
+		for _, u := range users {
+			fileIDs = append(fileIDs, u.ExportFileID)
+		}
+
+		fs := []*files.File{}
+		err = s.Where("created < ?", time.Now().Add(-time.Hour*24*7)).In("id", fileIDs).Find(&fs)
+		if err != nil {
+			log.Errorf(logPrefix+"Could not get users with export files: %s", err)
+			return
+		}
+
+		if len(fs) == 0 {
+			return
+		}
+
+		log.Debugf(logPrefix+"Removing %d old user data exports...", len(fs))
+
+		for _, f := range fs {
+			err = f.Delete()
+			if err != nil {
+				log.Errorf(logPrefix+"Could not remove user export file %d: %s", f.ID, err)
+				return
+			}
+		}
+
+		_, err = s.In("export_file_id", fileIDs).Cols("export_file_id").Update(&user.User{})
+		if err != nil {
+			log.Errorf(logPrefix+"Could not update user export file state: %s", err)
+			return
+		}
+
+		log.Debugf(logPrefix+"Removed %d old user data exports...", len(fs))
+
+	})
+	if err != nil {
+		log.Fatalf("Could not old export cleanup cron: %s", err)
+	}
+}

pkg/models/list.go

@@ -51,12 +51,6 @@ type List struct {
 
 	// The user who created this list.
 	Owner *user.User `xorm:"-" json:"owner" valid:"-"`
-	// An array of tasks which belong to the list.
-	// Deprecated: you should use the dedicated task list endpoint because it has support for pagination and filtering
-	Tasks []*Task `xorm:"-" json:"-"`
-
-	// Only used for migration.
-	Buckets []*Bucket `xorm:"-" json:"-"`
 
 	// Whether or not a list is archived.
 	IsArchived bool `xorm:"not null default false" json:"is_archived" query:"is_archived"`
@@ -85,6 +79,15 @@ type List struct {
 	web.Rights   `xorm:"-" json:"-"`
 }
 
+type ListWithTasksAndBuckets struct {
+	List
+	// An array of tasks which belong to the list.
+	Tasks []*TaskWithComments `xorm:"-" json:"tasks"`
+	// Only used for migration.
+	Buckets          []*Bucket `xorm:"-" json:"buckets"`
+	BackgroundFileID int64     `xorm:"null" json:"background_file_id"`
+}
+
 // TableName returns a better name for the lists table
 func (l *List) TableName() string {
 	return "lists"

pkg/models/listeners.go

@@ -50,6 +50,7 @@ func RegisterListeners() {
 	events.RegisterListener((&TaskCommentUpdatedEvent{}).Name(), &HandleTaskCommentEditMentions{})
 	events.RegisterListener((&TaskCreatedEvent{}).Name(), &HandleTaskCreateMentions{})
 	events.RegisterListener((&TaskUpdatedEvent{}).Name(), &HandleTaskUpdatedMentions{})
+	events.RegisterListener((&UserDataExportRequestedEvent{}).Name(), &HandleUserDataExport{})
 }
 
 //////
@@ -562,3 +563,41 @@ func (s *SendTeamMemberAddedNotification) Handle(msg *message.Message) (err erro
 		Team:   event.Team,
 	})
 }
+
+// HandleUserDataExport  represents a listener
+type HandleUserDataExport struct {
+}
+
+// Name defines the name for the HandleUserDataExport listener
+func (s *HandleUserDataExport) Name() string {
+	return "handle.user.data.export"
+}
+
+// Handle is executed when the event HandleUserDataExport listens on is fired
+func (s *HandleUserDataExport) Handle(msg *message.Message) (err error) {
+	event := &UserDataExportRequestedEvent{}
+	err = json.Unmarshal(msg.Payload, event)
+	if err != nil {
+		return err
+	}
+
+	log.Debugf("Starting to export user data for user %d...", event.User.ID)
+
+	sess := db.NewSession()
+	defer sess.Close()
+	err = sess.Begin()
+	if err != nil {
+		return
+	}
+
+	err = ExportUserData(sess, event.User)
+	if err != nil {
+		_ = sess.Rollback()
+		return
+	}
+
+	log.Debugf("Done exporting user data for user %d...", event.User.ID)
+
+	err = sess.Commit()
+	return err
+}

pkg/models/namespace.go

@@ -187,6 +187,11 @@ type NamespaceWithLists struct {
 	Lists     []*List `xorm:"-" json:"lists"`
 }
 
+type NamespaceWithListsAndTasks struct {
+	Namespace
+	Lists []*ListWithTasksAndBuckets `xorm:"-" json:"lists"`
+}
+
 func makeNamespaceSlice(namespaces map[int64]*NamespaceWithLists, userMap map[int64]*user.User, subscriptions map[int64]*Subscription) []*NamespaceWithLists {
 	all := make([]*NamespaceWithLists, 0, len(namespaces))
 	for _, n := range namespaces {

pkg/models/notifications.go

@@ -302,3 +302,29 @@ func (n *UserMentionedInTaskNotification) ToDB() interface{} {
 func (n *UserMentionedInTaskNotification) Name() string {
 	return "task.mentioned"
 }
+
+// DataExportReadyNotification represents a DataExportReadyNotification notification
+type DataExportReadyNotification struct {
+	User *user.User `json:"user"`
+}
+
+// ToMail returns the mail notification for DataExportReadyNotification
+func (n *DataExportReadyNotification) ToMail() *notifications.Mail {
+	return notifications.NewMail().
+		Subject("Your Vikunja Data Export is ready").
+		Greeting("Hi "+n.User.GetName()+",").
+		Line("Your Vikunja Data Export is ready for you to download. Click the button below to download it:").
+		Action("Download", config.ServiceFrontendurl.GetString()+"user/export/download").
+		Line("The download will be available for the next 7 days.").
+		Line("Have a nice day!")
+}
+
+// ToDB returns the DataExportReadyNotification notification in a format which can be saved in the db
+func (n *DataExportReadyNotification) ToDB() interface{} {
+	return nil
+}
+
+// Name returns the name of the notification
+func (n *DataExportReadyNotification) Name() string {
+	return "data.export.ready"
+}

pkg/models/tasks.go

@@ -129,6 +129,11 @@ type Task struct {
 	web.Rights   `xorm:"-" json:"-"`
 }
 
+type TaskWithComments struct {
+	Task
+	Comments []*TaskComment `xorm:"-" json:"comments"`
+}
+
 // TableName returns the table name for listtasks
 func (Task) TableName() string {
 	return "tasks"

pkg/modules/dump/dump.go

@@ -21,19 +21,15 @@ import (
 	"fmt"
 	"io"
 	"os"
-	"strconv"
 
 	"code.vikunja.io/api/pkg/db"
 	"code.vikunja.io/api/pkg/files"
 	"code.vikunja.io/api/pkg/log"
+	"code.vikunja.io/api/pkg/utils"
 	"code.vikunja.io/api/pkg/version"
 	"github.com/spf13/viper"
 )
 
-// Change to deflate to gain better compression
-// see http://golang.org/pkg/archive/zip/#pkg-constants
-const compressionUsed = zip.Deflate
-
 // Dump creates a zip file with all vikunja files at filename
 func Dump(filename string) error {
 	dumpFile, err := os.Create(filename)
@@ -55,7 +51,7 @@ func Dump(filename string) error {
 
 	// Version
 	log.Info("Start dumping version file...")
-	err = writeBytesToZip("VERSION", []byte(version.Version), dumpWriter)
+	err = utils.WriteBytesToZip("VERSION", []byte(version.Version), dumpWriter)
 	if err != nil {
 		return fmt.Errorf("error saving version: %s", err)
 	}
@@ -68,7 +64,7 @@ func Dump(filename string) error {
 		return fmt.Errorf("error saving database data: %s", err)
 	}
 	for t, d := range data {
-		err = writeBytesToZip("database/"+t+".json", d, dumpWriter)
+		err = utils.WriteBytesToZip("database/"+t+".json", d, dumpWriter)
 		if err != nil {
 			return fmt.Errorf("error writing database table %s: %s", t, err)
 		}
@@ -81,21 +77,12 @@ func Dump(filename string) error {
 	if err != nil {
 		return fmt.Errorf("error saving file: %s", err)
 	}
-	for fid, file := range allFiles {
-		header := &zip.FileHeader{
-			Name:   "files/" + strconv.FormatInt(fid, 10),
-			Method: compressionUsed,
-		}
-		w, err := dumpWriter.CreateHeader(header)
-		if err != nil {
-			return err
-		}
-		_, err = io.Copy(w, file)
-		if err != nil {
-			return fmt.Errorf("error writing file %d: %s", fid, err)
-		}
-		_ = file.Close()
+
+	err = utils.WriteFilesToZip(allFiles, dumpWriter)
+	if err != nil {
+		return err
 	}
+
 	log.Infof("Dumped files")
 
 	log.Info("Done creating dump")
@@ -123,7 +110,7 @@ func writeFileToZip(filename string, writer *zip.Writer) error {
 	}
 
 	header.Name = info.Name()
-	header.Method = compressionUsed
+	header.Method = utils.CompressionUsed
 
 	w, err := writer.CreateHeader(header)
 	if err != nil {
@@ -132,16 +119,3 @@ func writeFileToZip(filename string, writer *zip.Writer) error {
 	_, err = io.Copy(w, fileToZip)
 	return err
 }
-
-func writeBytesToZip(filename string, data []byte, writer *zip.Writer) (err error) {
-	header := &zip.FileHeader{
-		Name:   filename,
-		Method: compressionUsed,
-	}
-	w, err := writer.CreateHeader(header)
-	if err != nil {
-		return err
-	}
-	_, err = w.Write(data)
-	return
-}

pkg/modules/dump/restore.go

@@ -33,6 +33,7 @@ import (
 	"code.vikunja.io/api/pkg/initialize"
 	"code.vikunja.io/api/pkg/log"
 	"code.vikunja.io/api/pkg/migration"
+
 	"src.techknowlogick.com/xormigrate"
 )
 
@@ -194,6 +195,7 @@ func Restore(filename string) error {
 	///////
 	// Done
 	log.Infof("Done restoring dump.")
+	log.Infof("Restart Vikunja to make sure the new configuration file is applied.")
 
 	return nil
 }

pkg/modules/migration/create_from_structure.go

@@ -31,7 +31,7 @@ import (
 
 // InsertFromStructure takes a fully nested Vikunja data structure and a user and then creates everything for this user
 // (Namespaces, tasks, etc. Even attachments and relations.)
-func InsertFromStructure(str []*models.NamespaceWithLists, user *user.User) (err error) {
+func InsertFromStructure(str []*models.NamespaceWithListsAndTasks, user *user.User) (err error) {
 	s := db.NewSession()
 	defer s.Close()
 
@@ -45,7 +45,7 @@ func InsertFromStructure(str []*models.NamespaceWithLists, user *user.User) (err
 	return s.Commit()
 }
 
-func insertFromStructure(s *xorm.Session, str []*models.NamespaceWithLists, user *user.User) (err error) {
+func insertFromStructure(s *xorm.Session, str []*models.NamespaceWithListsAndTasks, user *user.User) (err error) {
 
 	log.Debugf("[creating structure] Creating %d namespaces", len(str))
 
@@ -129,7 +129,7 @@ func insertFromStructure(s *xorm.Session, str []*models.NamespaceWithLists, user
 
 			// Create all tasks
 			for _, t := range tasks {
-				setBucketOrDefault(t)
+				setBucketOrDefault(&t.Task)
 
 				t.ListID = l.ID
 				err = t.Create(s, user)
@@ -221,6 +221,15 @@ func insertFromStructure(s *xorm.Session, str []*models.NamespaceWithLists, user
 					}
 					log.Debugf("[creating structure] Associated task %d with label %d", t.ID, lb.ID)
 				}
+
+				for _, comment := range t.Comments {
+					comment.TaskID = t.ID
+					err = comment.Create(s, user)
+					if err != nil {
+						return
+					}
+					log.Debugf("[creating structure] Created new comment %d", comment.ID)
+				}
 			}
 
 			// All tasks brought their own bucket with them, therefore the newly created default bucket is just extra space

pkg/modules/migration/create_from_structure_test.go

@@ -32,79 +32,95 @@ func TestInsertFromStructure(t *testing.T) {
 	}
 	t.Run("normal", func(t *testing.T) {
 		db.LoadAndAssertFixtures(t)
-		testStructure := []*models.NamespaceWithLists{
+		testStructure := []*models.NamespaceWithListsAndTasks{
 			{
 				Namespace: models.Namespace{
 					Title:       "Test1",
 					Description: "Lorem Ipsum",
 				},
-				Lists: []*models.List{
+				Lists: []*models.ListWithTasksAndBuckets{
 					{
-						Title:       "Testlist1",
-						Description: "Something",
+						List: models.List{
+							Title:       "Testlist1",
+							Description: "Something",
+						},
 						Buckets: []*models.Bucket{
 							{
 								ID:    1234,
 								Title: "Test Bucket",
 							},
 						},
-						Tasks: []*models.Task{
+						Tasks: []*models.TaskWithComments{
 							{
-								Title:       "Task1",
-								Description: "Lorem",
+								Task: models.Task{
+									Title:       "Task1",
+									Description: "Lorem",
+								},
 							},
 							{
-								Title: "Task with related tasks",
-								RelatedTasks: map[models.RelationKind][]*models.Task{
-									models.RelationKindSubtask: {
+								Task: models.Task{
+									Title: "Task with related tasks",
+									RelatedTasks: map[models.RelationKind][]*models.Task{
+										models.RelationKindSubtask: {
+											{
+												Title:       "Related to task with related task",
+												Description: "As subtask",
+											},
+										},
+									},
+								},
+							},
+							{
+								Task: models.Task{
+									Title: "Task with attachments",
+									Attachments: []*models.TaskAttachment{
 										{
-											Title:       "Related to task with related task",
-											Description: "As subtask",
+											File: &files.File{
+												Name:        "testfile",
+												Size:        4,
+												FileContent: []byte{1, 2, 3, 4},
+											},
 										},
 									},
 								},
 							},
 							{
-								Title: "Task with attachments",
-								Attachments: []*models.TaskAttachment{
-									{
-										File: &files.File{
-											Name:        "testfile",
-											Size:        4,
-											FileContent: []byte{1, 2, 3, 4},
+								Task: models.Task{
+									Title: "Task with labels",
+									Labels: []*models.Label{
+										{
+											Title:    "Label1",
+											HexColor: "ff00ff",
+										},
+										{
+											Title:    "Label2",
+											HexColor: "ff00ff",
 										},
 									},
 								},
 							},
 							{
-								Title: "Task with labels",
-								Labels: []*models.Label{
-									{
-										Title:    "Label1",
-										HexColor: "ff00ff",
-									},
-									{
-										Title:    "Label2",
-										HexColor: "ff00ff",
+								Task: models.Task{
+									Title: "Task with same label",
+									Labels: []*models.Label{
+										{
+											Title:    "Label1",
+											HexColor: "ff00ff",
+										},
 									},
 								},
 							},
 							{
-								Title: "Task with same label",
-								Labels: []*models.Label{
-									{
-										Title:    "Label1",
-										HexColor: "ff00ff",
-									},
+								Task: models.Task{
+									Title:    "Task in a bucket",
+									BucketID: 1234,
 								},
 							},
 							{
-								Title:    "Task in a bucket",
-								BucketID: 1234,
-							},
-							{
-								Title:    "Task in a nonexisting bucket",
-								BucketID: 1111,
+								Task: models.Task{
+									Title:    "Task in a nonexisting bucket",
+									BucketID: 1111,
+								},
 							},
 						},
 					},

pkg/modules/migration/handler/common.go

@@ -0,0 +1,40 @@
+// Vikunja is a to-do list application to facilitate your life.
+// Copyright 2018-2021 Vikunja and contributors. All rights reserved.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public Licensee as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public Licensee for more details.
+//
+// You should have received a copy of the GNU Affero General Public Licensee
+// along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+package handler
+
+import (
+	"net/http"
+
+	"code.vikunja.io/api/pkg/modules/migration"
+	user2 "code.vikunja.io/api/pkg/user"
+	"code.vikunja.io/web/handler"
+	"github.com/labstack/echo/v4"
+)
+
+func status(ms migration.MigratorName, c echo.Context) error {
+	user, err := user2.GetCurrentUser(c)
+	if err != nil {
+		return handler.HandleHTTPError(err, c)
+	}
+
+	status, err := migration.GetMigrationStatus(ms, user)
+	if err != nil {
+		return handler.HandleHTTPError(err, c)
+	}
+
+	return c.JSON(http.StatusOK, status)
+}

pkg/modules/migration/handler/handler.go

@@ -84,15 +84,5 @@ func (mw *MigrationWeb) Migrate(c echo.Context) error {
 func (mw *MigrationWeb) Status(c echo.Context) error {
 	ms := mw.MigrationStruct()
 
-	user, err := user2.GetCurrentUser(c)
-	if err != nil {
-		return handler.HandleHTTPError(err, c)
-	}
-
-	status, err := migration.GetMigrationStatus(ms, user)
-	if err != nil {
-		return handler.HandleHTTPError(err, c)
-	}
-
-	return c.JSON(http.StatusOK, status)
+	return status(ms, c)
 }

pkg/modules/migration/handler/handler_file.go

@@ -0,0 +1,79 @@
+// Vikunja is a to-do list application to facilitate your life.
+// Copyright 2018-2021 Vikunja and contributors. All rights reserved.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public Licensee as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public Licensee for more details.
+//
+// You should have received a copy of the GNU Affero General Public Licensee
+// along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+package handler
+
+import (
+	"net/http"
+
+	"code.vikunja.io/api/pkg/models"
+	"code.vikunja.io/api/pkg/modules/migration"
+	user2 "code.vikunja.io/api/pkg/user"
+	"code.vikunja.io/web/handler"
+	"github.com/labstack/echo/v4"
+)
+
+type FileMigratorWeb struct {
+	MigrationStruct func() migration.FileMigrator
+}
+
+// RegisterRoutes registers all routes for migration
+func (fw *FileMigratorWeb) RegisterRoutes(g *echo.Group) {
+	ms := fw.MigrationStruct()
+	g.GET("/"+ms.Name()+"/status", fw.Status)
+	g.PUT("/"+ms.Name()+"/migrate", fw.Migrate)
+}
+
+// Migrate calls the migration method
+func (fw *FileMigratorWeb) Migrate(c echo.Context) error {
+	ms := fw.MigrationStruct()
+
+	// Get the user from context
+	user, err := user2.GetCurrentUser(c)
+	if err != nil {
+		return handler.HandleHTTPError(err, c)
+	}
+
+	file, err := c.FormFile("import")
+	if err != nil {
+		return err
+	}
+	src, err := file.Open()
+	if err != nil {
+		return err
+	}
+	defer src.Close()
+
+	// Do the migration
+	err = ms.Migrate(user, src, file.Size)
+	if err != nil {
+		return handler.HandleHTTPError(err, c)
+	}
+
+	err = migration.SetMigrationStatus(ms, user)
+	if err != nil {
+		return handler.HandleHTTPError(err, c)
+	}
+
+	return c.JSON(http.StatusOK, models.Message{Message: "Everything was migrated successfully."})
+}
+
+// Status returns whether or not a user has already done this migration
+func (fw *FileMigratorWeb) Status(c echo.Context) error {
+	ms := fw.MigrationStruct()
+
+	return status(ms, c)
+}

pkg/modules/migration/microsoft-todo/microsoft_todo.go

@@ -243,15 +243,15 @@ func getMicrosoftTodoData(token string) (microsoftTodoData []*list, err error) {
 	return
 }
 
-func convertMicrosoftTodoData(todoData []*list) (vikunjsStructure []*models.NamespaceWithLists, err error) {
+func convertMicrosoftTodoData(todoData []*list) (vikunjsStructure []*models.NamespaceWithListsAndTasks, err error) {
 
 	// One namespace with all lists
-	vikunjsStructure = []*models.NamespaceWithLists{
+	vikunjsStructure = []*models.NamespaceWithListsAndTasks{
 		{
 			Namespace: models.Namespace{
 				Title: "Migrated from Microsoft Todo",
 			},
-			Lists: []*models.List{},
+			Lists: []*models.ListWithTasksAndBuckets{},
 		},
 	}
 
@@ -262,8 +262,10 @@ func convertMicrosoftTodoData(todoData []*list) (vikunjsStructure []*models.Name
 		log.Debugf("[Microsoft Todo Migration] Converting list %s", l.ID)
 
 		// Lists only with title
-		list := &models.List{
-			Title: l.DisplayName,
+		list := &models.ListWithTasksAndBuckets{
+			List: models.List{
+				Title: l.DisplayName,
+			},
 		}
 
 		log.Debugf("[Microsoft Todo Migration] Converting %d tasks", len(l.Tasks))
@@ -340,7 +342,7 @@ func convertMicrosoftTodoData(todoData []*list) (vikunjsStructure []*models.Name
 				}
 			}
 
-			list.Tasks = append(list.Tasks, task)
+			list.Tasks = append(list.Tasks, &models.TaskWithComments{Task: *task})
 			log.Debugf("[Microsoft Todo Migration] Done converted %d tasks", len(l.Tasks))
 		}
 

pkg/modules/migration/microsoft-todo/microsoft_todo_test.go

@@ -102,57 +102,79 @@ func TestConverting(t *testing.T) {
 		},
 	}
 
-	expectedHierachie := []*models.NamespaceWithLists{
+	expectedHierachie := []*models.NamespaceWithListsAndTasks{
 		{
 			Namespace: models.Namespace{
 				Title: "Migrated from Microsoft Todo",
 			},
-			Lists: []*models.List{
+			Lists: []*models.ListWithTasksAndBuckets{
 				{
-					Title: "List 1",
-					Tasks: []*models.Task{
+					List: models.List{
+						Title: "List 1",
+					},
+					Tasks: []*models.TaskWithComments{
 						{
-							Title:       "Task 1",
-							Description: "This is a description",
-						},
-						{
-							Title:  "Task 2",
-							Done:   true,
-							DoneAt: testtimeTime,
-						},
-						{
-							Title:    "Task 3",
-							Priority: 1,
-						},
-						{
-							Title:    "Task 4",
-							Priority: 3,
-						},
-						{
-							Title: "Task 5",
-							Reminders: []time.Time{
-								testtimeTime,
+							Task: models.Task{
+								Title:       "Task 1",
+								Description: "This is a description",
 							},
 						},
 						{
-							Title:   "Task 6",
-							DueDate: testtimeTime,
+							Task: models.Task{
+								Title:  "Task 2",
+								Done:   true,
+								DoneAt: testtimeTime,
+							},
 						},
 						{
-							Title:       "Task 7",
-							DueDate:     testtimeTime,
-							RepeatAfter: 60 * 60 * 24 * 7, // The amount of seconds in a week
+							Task: models.Task{
+								Title:    "Task 3",
+								Priority: 1,
+							},
+						},
+						{
+							Task: models.Task{
+								Title:    "Task 4",
+								Priority: 3,
+							},
+						},
+						{
+							Task: models.Task{
+								Title: "Task 5",
+								Reminders: []time.Time{
+									testtimeTime,
+								},
+							},
+						},
+						{
+							Task: models.Task{
+								Title:   "Task 6",
+								DueDate: testtimeTime,
+							},
+						},
+						{
+							Task: models.Task{
+								Title:       "Task 7",
+								DueDate:     testtimeTime,
+								RepeatAfter: 60 * 60 * 24 * 7, // The amount of seconds in a week
+							},
 						},
 					},
 				},
 				{
-					Title: "List 2",
-					Tasks: []*models.Task{
+					List: models.List{
+						Title: "List 2",
+					},
+					Tasks: []*models.TaskWithComments{
 						{
-							Title: "Task 1",
+							Task: models.Task{
+								Title: "Task 1",
+							},
 						},
 						{
-							Title: "Task 2",
+							Task: models.Task{
+								Title: "Task 2",
+							},
 						},
 					},
 				},

pkg/modules/migration/migration_status.go

@@ -37,7 +37,7 @@ func (s *Status) TableName() string {
 }
 
 // SetMigrationStatus sets the migration status for a user
-func SetMigrationStatus(m Migrator, u *user