Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos/borgbackup: init #36927

Merged
merged 1 commit into from Mar 19, 2018
Merged

nixos/borgbackup: init #36927

merged 1 commit into from Mar 19, 2018

Conversation

dotlambda
Copy link
Member

Motivation for this change

I want a secure way to do regular remote backups.
Borg has a nice feature called append-only mode. To use it, I implemented more or less two modules: a server-side and a client-side one.
To read more about forcing append-only mode via SSH, look here: http://borgbackup.readthedocs.io/en/stable/usage/serve.html#examples

I was inspired by @phdoerfler (r-raymond/nixos-mailserver#93) and @bjornfor (https://github.com/bjornfor/nixos-config/blob/a570f4f7e4644520f430b310b4a94bb62decfb40/options/borg-backup.nix).

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

/cc @Mic92 who wrote the previous borg test

@bjornfor
Copy link
Contributor

Good stuff!

A couple of things I would like to see in this PR, which you can pull from my module if you like:

  1. Guarantee that postHook is invoked before exit, even if there is an error during backup. I use pre/postHook for mount and/or LVM snapshotting, and I need to have a way to clean up afterwards.

  2. Have an option for which directory "borg create" is invoked from. This is needed for backing up LVM snapshots without messing up file paths in the backup. (This is "rootDir" in my module.)

  3. Add "UNSUCCESSFUL" or something to archive names in case of error. I like to run "borg list" and know right there whether I'm looking at good or bad backups. (In my module I name archives "${archivename}.UNSUCCESSFUL" and remove the "UNSUCCESSFUL" part only on success.)

${cfg.preHook}
borg init \
--encryption ${cfg.encryption.mode} \
|| true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it difficult to run init only when needed, and fail on error? This seems to ignore errors, which gives me a bad feeling.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is better to touch a file after borg init and only call borg init if this file does not exist.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is better to call e.g. borg list to check if a repo exists. This will also cover the case where the repo has been created manually.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think borg has some safety measures in case somebody creates backup on an external drive/remote filesystem and the mount is currently not active to avoid making backups on the underlying device.
Can we cover this as well?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason is that user might thing their backup works, while they backup the whole time on the same filesystem where the data is located.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know of such measures. Maybe adding a doInit option is a good idea?


startAt = mkOption {
type = with types; either str (listOf str);
example = "hourly";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing default value.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

daily is a good default.

type = types.lines;
description = ''
Shell commands to run after
<command>borg init</command>.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe say that this runs every time the job runs. Or change the backup job script template so that "borg init" is only invoked once, followed by postInit hook. I'd prefer the latter.

@Mic92
Copy link
Member

Mic92 commented Mar 13, 2018

cc @ThomasWaldmann in case you are interested how borg is used.

@dotlambda
Copy link
Member Author

@bjornfor

  1. Done.
  2. You can just add cd ${rootDir} to your preHook.
  3. Do you think that should be the default behaviour? Or should we just add some stuff to make it easier to rename the archive in postCreate, like having a bash variable $archive that can be set as in your script: archive="foobar-$(date +%Y%m%dT%H%M%S)"?

@bjornfor
Copy link
Contributor

@dotlambda
2. Right. (Doh!)
3. Don't know. Personally I would prefer to have it built-in, but no strong opinion. As long as it's possible to do reliably in a hook, that's fine.

@dotlambda
Copy link
Member Author

@bjornfor Are you sure that an archive is created before the backup is finished? I only know of the checkpoints, which are clearly distinguishable by their .checkpoint suffix.

@dotlambda dotlambda force-pushed the borg-module branch 2 times, most recently from 427e4c2 to f764001 Compare March 14, 2018 20:30
@bjornfor
Copy link
Contributor

I guess it's pretty safe to rename the archive in case of error. But in paranoia mode, I feel safer if the archive is renamed to its final name after a successful backup.

@dotlambda
Copy link
Member Author

The thing I'm wondering about is: In which case is an archive created even if borg exits with a non-zero exit code? Afaik, the only possibility is an exit code of 1, i.e. a warning, because in case of a failure, no archive should be created: http://borgbackup.readthedocs.io/en/stable/usage/general.html#return-codes
However, I have no idea what could lead to that exit code.

@bjornfor
Copy link
Contributor

I treat warning as failure in my setup. I've had borg exit with status 1 (warning) on file permission error and inaccessible paths (autofs, remote filesystem down).

Type = "oneshot";
};
wantedBy = [ "multi-user.target" ];
after = [ "multi-user.service" ];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this "after" needed? (I don't see this pattern anywhere in nixpkgs.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It isn't. Thanks for catching that.

@dotlambda
Copy link
Member Author

Warnings should automatically be treated as failures because of the non-zero exit code.

I have added examples to preHook and postCreate showing how renaming on success can be done.

type = with types; either path (nonEmptyListOf path);
description = "Paths to back up.";
example = "/home/user";
apply = x: if isList x then x else [ x ];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to have type just be "nonEmptyListOf path". Reasoning: the option and its documentation is in plural, so not using a list feels weird to me.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just for convenience. I usually back up just one directory and this way I don't have to write the brackets. I can write Path(s) to back up. if you want :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's fine. I'm just talking out loud :-)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thinking out loud

@dotlambda
Copy link
Member Author

@bjornfor Is the way renames can be done now satisfying for you? Any other objections?
I'd rather not do the renaming by default because I do like the ability to use placeholders such as {now}. However, these hinder you from knowing the exact archive name beforehand. And we cannot just rename the latest archive because that could be an earlier one.

${cfg.extraPruneArgs}
${cfg.postPrune}
'' + ''
${cfg.postHook}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't this make postHook run twice?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. Didn't think about that.

mkUsersConfig = name: cfg: {
users.${cfg.user} = {
openssh.authorizedKeys.keyFiles = singleton
(pkgs.writeText "borgbackup.${cfg.user}.authorized_keys" (concatStringsSep "\n"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can pass strings directly to "openssh.authorizedKeys.keys" instead of writing files and passing them to "...keyFiles".

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used keyFiles because I wasn't able to merge multiple keys options using mkMerge. But I tried again and it worked :D

CPUSchedulingPolicy = "idle";
IOSchedulingClass = "idle";
ProtectSystem = "full";
#FIXME:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps do something about this FIXME before merge?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The things necessary to use ProtectSystem = "strict":

  • assert that createHome == true (or don't assert it and just try to create the directory)
  • create $HOME/{.config,.cache}/borg

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went for the assertion, but root doesn't have an entry in config.users by default, so it is treated specially.

Copy link
Member Author

@dotlambda dotlambda Mar 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped the assertion again. Home dirs can also be created in other ways:

system.activationScripts.users = stringAfter [ "stdio" ]
''
install -m 0700 -d /root
install -m 0755 -d /home

I think the error messages should be clear enough though.

type = with types; nullOr str;
description = ''
The passphrase the backups are encrypted with.
Mutually exclusive with <option>passCommand</option>.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a warning that the passphrase will be world readable in the Nix store, and recommend to use the "passCommand" option instead.


exclude = mkOption {
type = with types; listOf str;
description = "Exclude paths matching any of the given patterns";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing dot. Perhaps add "See 'borg help patterns' for pattern syntax"?


extraInitArgs = mkOption {
type = types.str;
description = "Additional arguments for <command>borg init</command>";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing dot. (Also on a couple of other option descriptions.)

restrictedArg = "--restrict-to-repository ${escapeShellArg cfg.path}";
appendOnlyArg = optionalString appendOnly "--append-only";
quotaArg = optionalString (cfg.quota != null) "--storage-quota ${cfg.quota}";
command = "borg serve ${restrictedArg} ${appendOnlyArg} ${quotaArg}";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feature request. Add optional "cd ${cfg.path}" in front of that "borg server" (Inspired by http://borgbackup.readthedocs.io/en/stable/deployment/central-backup-server.html#restrictions.) This allows clients to ignore the server directory layout, and can use repo urls like backup@server:myrepo.

Copy link
Member Author

@dotlambda dotlambda Mar 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice idea!
Btw, the example you linked to uses --restrict-to-path instead of --restrict-to-repository, which I use. Therefore, as it is currently implemented there is no way of using multiple repos with the same key (except if you use multiple users on the storage server).

Copy link
Member Author

@dotlambda dotlambda Mar 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have a use case for --restrict-to-path?
I implemented the working directory change, but noticed the way you have to specify the repo is quite ugly (user@machine:.).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re: --restrict-to-path. I use that in my local setup (not yet pushed to github). I chose that option because it allowed the client to create the repository it needs. I have something like /data/borgbackup/hosts/machine1 and .../machine2, where each machine may have a number of repositories, managed by the client. I'm currently experimenting with the setup, hence it's not pushed yet. I don't have a good answer. I guess I can flip it around and ask "what's the use case of --restrict-to-repo"?

Ah, it should be "cd ${config.services.borgbackup.repos.${name}.path}". (Or whatever expands to the path where repos are stored, e.g. /var/lib/borgbackup.) Then we can specify repo correctly, I think, without the "." (dot).

Btw, I think it's a good idea to "cd dir && borg serve ...", so that borg isn't run if the cd fails. (Unless that's considered a feature in this setup -- I didn't check.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, there's not much use in --restrict-to-repository for restricting repository access. However, I think --restrict-to-path makes --storage-quota useless.

No, we can't leave out the dot:

$ BORG_REPO="machine:" borg create ::archive path
Repository /home/user/machine: does not exist.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the best solution is an option called allowSubRepos. Do you think it should be true or false by default?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point about the --storage-quota. Ok, now I think I understand why there are two --restrict-to-* options :-)

Re: the dot. I was thinking of this syntax: BORG_REPO="machine:repo.borg" borg create ::archive path.

I think the best solution is an option called allowSubRepos. Do you think it should be true or false by default?

No opinion yet. Do you plan to select the --restrict-to-* option based on allowSubRepos?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes exactly. See the third commit.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dude.

ReadWritePaths =
[ "${userHome}/.config/borg" "${userHome}/.cache/borg" ]
# Borg needs write access to repo if it is not remote
++ optional (isPath cfg.repo) cfg.repo;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A thing I just noticed: If you specify a relative path for cfg.repo, this will not trigger and borg won't have access to the repo. I think I should add a fancier test for isPath, e.g. check if no : is contained. It appears like all of

borg init -e none foo:bar
borg init -e none foo\:bar
borg init -e none foo\\:bar

try to use SSH. So I don't see a possibility for using a relative path with a colon. I however did not read completely through borgbackup/borg#1705 and borgbackup/borg#1710.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm stupid. Of course there is a way: You can use ./foo:bar.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's the best I could come up with:

isLocalPath = x:
    builtins.substring 0 1 x == "/"      # Absolute path
    || builtins.substring 0 1 x == "."   # Relative path
    || builtins.match "[.*:.*]" == null; # not machine:path

@dotlambda dotlambda force-pushed the borg-module branch 6 times, most recently from e0444b3 to 0b40dae Compare March 18, 2018 20:07
@bjornfor
Copy link
Contributor

Do you have an example use case for postInit, postCreate, postPrune hooks? We also have global pre/post hooks, which I'll be using. I'm just curious what the other hooks might be used for.

@dotlambda
Copy link
Member Author

dotlambda commented Mar 18, 2018

No, I won't use them for now. I added them initially so you can do renaming etc.
They can still be handy if you want to know which command failed in your postHook or if you want to not prune if some path wasn't mounted but still create the archive or ...
Do you think they should be removed until someone has a need for them?

default = "";
example = ''
# In case you need to know the exact archive name
archiveName="foobar-$(date +%Y%m%dT%H%M%S)"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that in the current version of this PR, "archiveName" is sometimes the base archive name, and sometimes the full archive name (with datetime suffix). On some level, this confuses me. How do you feel about having separate names? (I guess you didn't opt for my "archiveBaseName" before for a reason, but I'd like to hear your thoughts.)

Copy link
Member Author

@dotlambda dotlambda Mar 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separate names are definitely less confusing 👍
The only reason I used archiveName is that archiveBaseName doesn't make sense when using placeholders.

@dotlambda dotlambda force-pushed the borg-module branch 2 times, most recently from 3b3ef3c to 820d6ea Compare March 18, 2018 21:21
type = types.lines;
description = ''
Shell commands to run after <command>borg create</command>. The name
of the created archive is stored in <literal>$achiveName</literal>.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new name is $archive_name.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why change naming style for this variable? (The other variables are lowerCamelCase.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stupid typo: $achiveName instead of $archiveName. Therefore grep didn't find it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought that style was more common for bash variables. But I can indeed change that back if you want.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree about snake_case being more common, and it's what I use myself. But the module currently mixes styles: e.g. $archive_name and $extraInitArgs. That should be fixed, either way.

You know what, I think that since some variables are inherited from the NixOS config, with lowerCamelCase, it feels weird to convert them to $snake_case. How about making all shell variables that are exposed (documented) as part of the module interface $lowerCamelCase?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍
I named them $archiveName, $archiveSuffix and $exitStatus.

@dotlambda dotlambda force-pushed the borg-module branch 3 times, most recently from 66de8a2 to 7ef4e99 Compare March 19, 2018 00:10
@dotlambda
Copy link
Member Author

Can I merge?

@bjornfor
Copy link
Contributor

I wanted to have look at that "dot in repository url". It feels weird to not specify a repo name. But I have little brain bandwidth, so if you feel the interface is OK, you have my ack. Thanks for writing the module! Looking forward to deprecating my own module.

@dotlambda
Copy link
Member Author

Well, it's certainly not wrong, because the working directory is where the repo is located.
I think BorgBackup's regex used to distinguish remote from local might not be perfect (they should recongnize user@machine: as remote) and if that's changed we can update the documentation.
Btw, I added a line in the manual that explains how to access sub-repos: user@machine:repo.

@dotlambda dotlambda merged commit c484079 into NixOS:master Mar 19, 2018
@dotlambda dotlambda deleted the borg-module branch March 19, 2018 19:30
@dotlambda
Copy link
Member Author

Backported in 7a5c7c1.

@dotlambda
Copy link
Member Author

@bjornfor Thanks again very much for your help and review!

@dotlambda dotlambda mentioned this pull request Apr 14, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants