I am trying to setup a restic job to backup my docker stacks, and with half of everything owned by root it becomes problematic. I’ve been wanting to look at podman so everything isn’t owned by root, but for now I want to backup my work I built.

Also, how do you deal with some docker containers having databases. Do you have to create exports for all docker containers that have some form of database?

I’ve spent the last few days moving all my docker containers to a dedicated machine. I was using a mix of NFS and local storage before, but now I am doing everything on local NVME. My original plan was having everything on NFS so I would worry about backups there, and I might go back to that.

  • PaulEngineer-89@alien.topB
    link
    fedilink
    English
    arrow-up
    2
    ·
    1 year ago

    Don’t backup the container!!

    Map volumes with your data to physical storage and then simply backup those folders with the rest of your data. Docker containers are already either backed up in your development directory (if you wrote them) or GitHub so like the operating system itself, no need tk backup anything. The whole idea of Docker is the containers are ephemeral. They are reset at every reboot.

    • doodeoo@alien.topB
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      This is the only correct answer.

      Containers are ephemeral and stateless. If you’re not mounting a volume, think of what happens if the container dies during your process for exporting the data. A failure mode not possible if you mount.

  • McGregorMX@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    I have my config and data volumes mounted to a share on truenas, that share replicates its snapshots to another truenas server. This is likely not ideal for everyone, but it works for me. My friend that also uses docker has it backed up with duplicati.

  • esturniolo@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    KISS method: Script that copy the data on the fly to the /tmp dir, compress it, encrypt it and move it to destination using rclone. Running every hour, 4 hours or 24 hours, depending the container.

    Never fails. The backups nor the restore.

    • atheken@alien.topB
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      I mean, snapshotting and piping it to an rclone mount is arguably simpler than trying to do your own ad hoc file syncronization, also does not require 2x the storage space.

  • Zeal514@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    First, I try not to have it owned by root. But some containers have special privileges that need to be followed.

    So rsync -O will copy the directory retaining permissions and ownership of all files.

  • Do_TheEvolution@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    From my basic selfhosted experience… I run kopia as root , my shit uses bind mounts so all I care about is in that directory.

    And so far it works fine, to just down old, rename the directory, copy from nightly backup back the directory and start container.

    But yeah if there is something I care about I schedule database dumps like here in bookstack or vaultwarden

    To have something more if shit would not work start.

  • MoneyVirus@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    only backup of the data i need to backup (mapped volumes).

    Restore: create fresh container and map volumes again.

  • cbunn81@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    This is one reason I prefer FreeBSD jails. They are each in a separate ZFS filesystem, with a separate filesystem for configuration files. So all I have to do is regular snapshots and send those to a backup pool.

  • SnakeBDD@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    All Docker containers have their persistent data in Docker volumes located on a BTRFS mount. A cronjob takes a snapshot of the BTRFS volume, then calls btrfs send, pipes that through tar and gpg and then directly to AWS S3.

    • froli@alien.topB
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      Great idea. I already do something similar(minus the btrfs part) for Vaultwarden. Mind sharing the script/commands?

      I setup my host with btrfs but I have 0 knowledge of it so I didn’t take advantage of it until now. I already have my docker volumes mapped to /docker/stack so I’m gonna create a sub-volume and move that there.

      I’m mostly interest in your btrfs snapshot and send commands but if you don’t mind sharing the whole thing that would be great.

  • katbyte@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    i don’t, i created a docker VM (and a couple others) and then i backup the VMs (proxmox + PBS make this very easy) with all their data in /home/docker/config/*

    i used to have them run off networked storage but i found it to be to slow/have other issues

    this also means for the primary important services that VM runs in HA and moves to another node when needed

  • techbandits@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    I use a Synology Nas with Active backup for business and back up the VMs that host the containers.

  • SamSausages@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    I do this at the file system level, not the file level, using zfs.

    Unless the container has a database, I use zfs snapshots. If it has a database, my script dumps the database first and then does a ZFS snapshot. Then that snapshot is sent via sanoid to a zfs disk that is in a different backup pool.

    This is a block level backup, so it only backs up the actual data blocks that changed.

  • root-node@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    For backups I use Nautical Backup.

    For the “owned by root” problem, I ensure all my docker compose files have [P]UID and [P]GID set to 1000 (the user my docker runs under). All my 20 containers have no issue running like this.

    How are you launching your containers? Docker compose is the way, I have set the following in all mine:

    environment:
      - PUID=1000
      - PGID=1000
    
    user:
      1000:1000
    
    • Not_your_guy_buddy42@alien.topB
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      Hey, this is where I am stuck just now: I want to keep the docker volumes, as bind mounts, also on my NAS share. If the containers run as a separate non root user (say 1001) then I can mount that share as 1001… sounds good right?

      But somebody suggested running each container from their own user. But then I would need lots of differently owned directories. I wonder if I could keep mounting subdirs of the same NAS share, as different users, so each of them can have their own file access? Perhaps that is overkill.

      (For OP: I’ve been on a selfhosting binge the past week and trying to work my way in at least the general direction of best practice… At least for the container databases I’ve been starting to use tiredofit/docker-db-backup (does database dumps) but also discovered this jdfranel docker backup as well which looks great as well. I save the dumps on a volume mounted from NAS. btrfs and there is a folder replication (snapshots) tool. So far, so good. )

      • root-node@alien.topB
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        …running each container from their own user…

        Ideally this is the perfect option from a security standpoint, this as well as each container having it’s own network too.

        In a homelab it’s not really required unless you are exposing your network to the internet or are better at creating/managing containers.

        If you are just starting out, just keep everything simple.