konrad
/
docker-db-backup
2
4
Fork 0
Code Issues 1 Pull Requests Projects Releases Wiki Activity

Backup fails without error #5

New Issue
Closed
opened 2023-11-11 22:47:47 +00:00 by JaCoB1123 · 14 comments
JaCoB1123 commented 2023-11-11 22:47:47 +00:00
Copy Link

Hi,

I noticed another issue, but have no idea how to gain more insights.

One of my containers can not be backed up. I'm guessing that it might have to do with the db size as it's my largest database hovering around 2GB. Another db with about 500MB is getting backed up without issue.

My log looks like this, so it looks like the process just dies:

db-backup-app-1  | 2023/11/11 23:37:00 Found container [/docspell-db-1]
db-backup-app-1  | 2023/11/11 23:37:00 Removing old backup folder 2022-05-10_06-00-00...
db-backup-app-1  | 2023/11/11 23:37:00 Dumping postgres database from container /budgeteer_db_1...
db-backup-app-1  | 2023/11/11 23:37:00 Dumping mysql database from container /photoprism-db-1...
db-backup-app-1  | 2023/11/11 23:37:28 Dumping postgres database from container /docspell-db-1...
db-backup-app-1 exited with code 0

Any idea how I could analyze the issue?

Thanks.

Hi, I noticed another issue, but have no idea how to gain more insights. One of my containers can not be backed up. I'm guessing that it might have to do with the db size as it's my largest database hovering around 2GB. Another db with about 500MB is getting backed up without issue. My log looks like this, so it looks like the process just dies: ``` db-backup-app-1 | 2023/11/11 23:37:00 Found container [/docspell-db-1] db-backup-app-1 | 2023/11/11 23:37:00 Removing old backup folder 2022-05-10_06-00-00... db-backup-app-1 | 2023/11/11 23:37:00 Dumping postgres database from container /budgeteer_db_1... db-backup-app-1 | 2023/11/11 23:37:00 Dumping mysql database from container /photoprism-db-1... db-backup-app-1 | 2023/11/11 23:37:28 Dumping postgres database from container /docspell-db-1... db-backup-app-1 exited with code 0 ``` Any idea how I could analyze the issue? Thanks.
konrad added the
bug
 label 2023-11-13 20:49:59 +00:00
konrad commented 2023-11-13 20:50:05 +00:00
Owner
Copy Link

Is that reproducible?

Is that reproducible?
JaCoB1123 commented 2023-11-13 22:14:45 +00:00
Author
Copy Link

I just checked and I got the same result...

I just checked and I got the same result...
konrad commented 2023-11-15 11:36:08 +00:00
Owner
Copy Link

Do you have a webhook defined?

Can you share the config for the docspell-db container?

Do you have a webhook defined? Can you share the config for the `docspell-db` container?
JaCoB1123 commented 2023-11-15 22:43:45 +00:00
Author
Copy Link

It's actually quite minimal:

db:
  image: postgres:11.7
  volumes:
    - postgres_data:/var/lib/postgresql/data
  ports:
    - "7787:5432"
  environment:
    POSTGRES_USER=user
    POSTGRES_PASSWORD=pass
    POSTGRES_DB=docspell
  restart: always

Just noticed your second question: I do not have a webhook defined.

It's actually quite minimal: ``` db: image: postgres:11.7 volumes: - postgres_data:/var/lib/postgresql/data ports: - "7787:5432" environment: POSTGRES_USER=user POSTGRES_PASSWORD=pass POSTGRES_DB=docspell restart: always ``` Just noticed your second question: I do not have a webhook defined.
konrad commented 2023-11-17 22:52:35 +00:00
Owner
Copy Link

Does this always happen with that container? Or only randomly?

The only other thing I can think of where this would happen is if the container would suddenly go away (when it is restarted or stopped)

Does this always happen with that container? Or only randomly? The only other thing I can think of where this would happen is if the container would suddenly go away (when it is restarted or stopped)
JaCoB1123 commented 2023-11-18 23:08:03 +00:00
Author
Copy Link

I changed the cron expression to execute later, updated the db-container and rebooted my server. Of the last two runs one was successful, although the gitea-backup was created empty without an error showing up in db-backup. So it doesn't always happen,

Here are slightly shortened logs for the last two executions:

2023/11/17 04:15:00 Dumping mysql database from container /gitea_db_1...
2023/11/17 04:15:01 Dumping postgres database from container /docspell-db-1...
2023/11/17 04:22:42 Dumping mysql database from container /photoprism-db-1...
2023/11/17 04:24:57 Done.
2023/11/18 04:15:00 Found container [/photoprism-db-1]
2023/11/18 04:15:00 Removing old backup folder 2023-11-14_07-00-00...
2023/11/18 04:15:00 Dumping postgres database from container /immich-db-1...
2023/11/18 04:15:09 Dumping postgres database from container /n8n-db-1...
2023/11/18 04:21:21 Dumping postgres database from container /docspell-db-1...
2023/11/18 04:21:45 DB backup service started.
I changed the cron expression to execute later, updated the db-container and rebooted my server. Of the last two runs one was successful, although the gitea-backup was created empty without an error showing up in db-backup. So it doesn't always happen, Here are slightly shortened logs for the last two executions: ``` 2023/11/17 04:15:00 Dumping mysql database from container /gitea_db_1... 2023/11/17 04:15:01 Dumping postgres database from container /docspell-db-1... 2023/11/17 04:22:42 Dumping mysql database from container /photoprism-db-1... 2023/11/17 04:24:57 Done. 2023/11/18 04:15:00 Found container [/photoprism-db-1] 2023/11/18 04:15:00 Removing old backup folder 2023-11-14_07-00-00... 2023/11/18 04:15:00 Dumping postgres database from container /immich-db-1... 2023/11/18 04:15:09 Dumping postgres database from container /n8n-db-1... 2023/11/18 04:21:21 Dumping postgres database from container /docspell-db-1... 2023/11/18 04:21:45 DB backup service started. ```
konrad commented 2023-11-20 11:03:04 +00:00
Owner
Copy Link

If you check docker ps after a failed backup attempt, does the uptime seem to indicate the container was restarted around the time when the backup failed?

If you check `docker ps` after a failed backup attempt, does the uptime seem to indicate the container was restarted around the time when the backup failed?
JaCoB1123 commented 2023-11-24 22:59:39 +00:00
Author
Copy Link

Sorry for the delay. I just checked and the backup failed the last 4 days consecutively. The db's uptime is 8 days though.

Sorry for the delay. I just checked and the backup failed the last 4 days consecutively. The db's uptime is 8 days though.
konrad commented 2023-11-28 20:54:22 +00:00
Owner
Copy Link

Is there anything in the db container's logs?

Is there anything in the db container's logs?
JaCoB1123 commented 2024-01-28 19:28:26 +00:00
Author
Copy Link

I just went to the console for the docker server and was greeted by this:
image

So it seems to be a memory issue. Is the output maybe buffered somewhere? Neither db nor db-backup logs show anything helpful.

I just went to the console for the docker server and was greeted by this: ![image](/attachments/dbb0c10b-13ae-4384-aa4e-30a06d7267a1) So it seems to be a memory issue. Is the output maybe buffered somewhere? Neither db nor db-backup logs show anything helpful.
image.png
109 KiB
konrad commented 2024-01-28 19:37:01 +00:00
Owner
Copy Link

It's not buffered anywhere really.

Is it still crashing directly after the docspell-db-1 docker container? How big is that container?

It's not buffered anywhere really. Is it still crashing directly after the `docspell-db-1` docker container? How big is that container?
JaCoB1123 commented 2024-01-28 19:56:11 +00:00
Author
Copy Link

Yes, usually at least. Yesterday the full backup was able to complete. The 5 days before it failed on the docspell container.

The saved .sql file is 3.3GB. The docker volume is 2.4GB in size.

Yes, usually at least. Yesterday the full backup was able to complete. The 5 days before it failed on the docspell container. The saved .sql file is 3.3GB. The docker volume is 2.4GB in size.
konrad commented 2024-01-28 22:05:53 +00:00
Owner
Copy Link

Should be fixed in 5982f1ef07 - please check with the next build if the problem went away.

Should be fixed in https://kolaente.dev/konrad/docker-db-backup/commit/5982f1ef07439eec86b450184e13caafdeb5a78f - please check with the next build if the problem went away.
JaCoB1123 commented 2024-01-29 11:43:40 +00:00
Author
Copy Link

Awesome, it's looking good now. Thanks a lot 👍

Awesome, it's looking good now. Thanks a lot 👍
❤️ 1
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: konrad/docker-db-backup#5
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?