Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

piwik is now matomo #34028

Merged
merged 4 commits into from Feb 26, 2018
Merged

piwik is now matomo #34028

merged 4 commits into from Feb 26, 2018

Conversation

florianjacob
Copy link
Contributor

Motivation for this change

For the 3.3.0 release, piwik was renamed to matomo: https://matomo.org/changelog/matomo-3-3-0/

The release does not contain security fixes it seems, it is mainly the renaming. Not sure what to do when they release subsequent versions with security fixes before NixOS 18.0.3 is out, as they won't be “backported” to the old piwik version.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

@srhb
Copy link
Contributor

srhb commented Jan 27, 2018

Hi Florian. I wasn't able to find an obvious reviewer candidate, and since you're the maintainer and it looks great, I guess we're good to merge. Would you mind fixing the conflict with the release notes? :)

@florianjacob
Copy link
Contributor Author

@srhb conflicts resolved! If you did a basic sanity check on the option renaming / deprecation part it's fine from my side, I just have never done that part before.

Attracting @Alphor 's attention: Maybe you could test the data migration when it's on master, or directly using this branch, before 18.03 is released? It (obviously) worked fine for my data, but having an additional check with an independent setup would be nice.

@ajarara
Copy link

ajarara commented Jan 27, 2018

@florianjacob Yeah, no prob. I'll just test the fork on your branch. Anything you'd like me to check beyond data migration?

@florianjacob
Copy link
Contributor Author

@Alphor thank you! Besides the migration of files from /var/lib/piwik to /var/lib/matomo, the main other point would be whether the upgrade notes in the release changelog are understandable or whether I missed any effects of the upgrade that would need to be mentionend there.

Copy link

@ajarara ajarara left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few hiccups upgrading, primarily due to me not having updated in quite a while. Data is preserved in mysql. Previous piwik front end login was disabled (almost certainly due to the fact that piwik was removed from /etc/shadow and unix auth requires a unix account, although not sure if the piwik front end uses unix auth).

Anyway, upgrade successful, but there were a few notes you might be interested in. I have a clone pre upgrade attempt if you'd like any questions answered/explored further.

# from 127.0.0.1 to localhost.
# unix socket authentication only works with localhost,
# but password-based SQL authentication works with both.
# TODO: is upstream interested in this?
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would think so :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe they're more sceptical, though, as 127.0.0.1 could be considered more failsafe or something. 😆 This requires that localhost resolves properly to ::1 and 127.0.0.1 and such. (It's not too long ago when NixOS itself had no such entries in /etc/hosts as well…) Also this could be bad if MySQL is not listening on IPv6. I'll eventually try what they think, though, the PR would be simple enough. 🙂

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a compile time option, preserving the behavior of 127..1, where --unix-auth or something expects a hosts mapping and changes it to localhost. Although that's a little more complicated :)

<listitem>
<para>
You should probably rename your database to keep things clean,
but this is not strictly required and not enforced.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Matomo should obviously break if this assumption fails in the future, although I'm not suggesting it is Nix's responsibilty to monitor all its software for backwards incompatible changes. It is easy to forget piwik is running, particularly with a stale blog like mine :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 I phrased this too hard, there's no assumption by matomo source code or by the NixOS package how the database is named, and I don't expect there'll be one from the matomo side ever, meaning you're totaly free to name your database banana if you like. This is merely a reminder by me, to reduce suprises with the user's backup scripts or something. I came up with a weaker wording that hopefully gets this across better!

@@ -65,23 +72,27 @@ in {
(import ../web-servers/nginx/vhost-options.nix { inherit config lib; })
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, you implemented this. Impressive! Is the reason you opted for recursiveUpdate instead of attribute set merging to allow for overrides? Not clear on the difference.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, somehow I really managed to get this done. 😄 If you try to merge with // operator it doesn't work as this would discard my custom defaults as soon as the user defines something custom. After a lot of fiddling around, I discovered recursiveUpdate, which does exactly what I need here, together with the locations definitions and the well-dosed mkForce at https://github.com/NixOS/nixpkgs/pull/34028/files#diff-8c50bc76feff6200b2774e17f7f44c52R199.

I also tried to achieve that via mkDefault etc., but I could not find a way that worked. Which does not mean there isn't, but I don't understand enough of the priority attribute merging system to find it…

let
join = hostName: domain: hostName + optionalString (domain != null) ".${domain}";
in join config.networking.hostName config.networking.domain;

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code looks correct. However when setting:

  networking.hostName  = "jarmac"';
  networking.domain = "org";

I received errors from ACME that the machine was trying to register matomo.jarmac. which is clearly incorrect.

Setting networking.hostname to jarmac.org caused an error with the prestart script looking for /var/lib/piwik, but this was resolved with another attempt, and is probably a relic from the first failed ACME attempt.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I designed this I did only think of networking.hostName = "<hostname>.<domain.org>"; and domain = null; (what most people are probably doing today) or, imho preferrable, networking.hostName = "<hostname>"; and networking.domain = "<domain.org>";. But hostName = "jarmac"; and domain = "org"; is nice, should really work and is what I'd like to recommend from now on for people that only want to put a single computer in their domain.

I could not reproduce the issue, though. 😥 The only explanation I have is that somehow the domain was set to "", which would explain the output. Maybe you can recheck whether that still occurs, and if yes, check what server_name landed in your nginx config (systemctl status nginx -> master process ngnix -c /nix/store…nginx-conf), and what landed in the acme service (should be like systemctl status acme-matomo.jarmac..service, and then the service file). If the issue persists, we're hopefully able to track the wrong output down that way.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha! It was an error on my end:

  services.nginx = let
    hostname = config.networking.hostName;
    tld = config.networking.domain;
  in
  {
    enable = true;
    virtualHosts = {
      "${hostname}" = {  # here's the error, no tld splicing in
        forceSSL = true;
        enableACME = true;
        serverAliases = [ "www.${hostname}" ];
        locations."/" = {
          root = "${nginxRootPath}";
        };
      };
    };
  };

Apologies.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I see!

The fqdn = let … in join config.networking.hostName config.networking.domain snippet is actually from my configuration.nix, you might want a copy of that yourself so you can write ${fqdn} everywhere and avoid that error / doing it by hand every time.. On my list of things to upstream next, as config.lib.networking.fqdn helper value or something. 😉 But will be quite a lot of work to find all places in nixpkgs where this should be used but isn't.

</para>

<para>
An automatic setup is not suported by piwik, so you need to configure piwik itself in the browser-based piwik setup.
An automatic setup is not suported by Matomo, so you need to configure Matomo itself in the browser-based Matomo setup.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just explicitly noting that I did not need to go through the set up again.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's as expected, the setup would only come up again if the data was not imported. 🙂

echo "Migrating from ${deprecatedDataDir} to ${dataDir}"
mv -T ${deprecatedDataDir} ${dataDir}
rmdir ${deprecatedDataDir}
fi
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh.. this shouldn't have failed on me.

Systemd output:

warning: the following units failed: matomo_setup_update.service

● matomo_setup_update.service
   Loaded: loaded (/nix/store/9f2hkvllc46vjp3x0h92vfnwljsf381g-unit-matomo_setup_update.service/matomo_setup_update.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2018-01-31 22:59:45 EST; 16s ago
  Process: 29211 ExecStartPre=/nix/store/wmg25rziv6zcm0hcr0q4wd1dzjqw7scj-unit-script/bin/matomo_setup_update-pre-start (code=exited, status=1/FAILURE)

Jan 31 22:59:45 jarmac.org systemd[1]: Starting matomo_setup_update.service...
Jan 31 22:59:45 jarmac.org matomo_setup_update-pre-start[29211]: Migrating from /var/lib/piwik to /var/lib/matomo
Jan 31 22:59:45 jarmac.org matomo_setup_update-pre-start[29211]: rmdir: failed to remove '/var/lib/piwik': No such file or directory
Jan 31 22:59:45 jarmac.org systemd[1]: matomo_setup_update.service: Control process exited, code=exited status=1
Jan 31 22:59:45 jarmac.org systemd[1]: Failed to start matomo_setup_update.service.
Jan 31 22:59:45 jarmac.org systemd[1]: matomo_setup_update.service: Unit entered failed state.
Jan 31 22:59:45 jarmac.org systemd[1]: matomo_setup_update.service: Failed with result 'exit-code'.
warning: error(s) occurred while switching to the new configuration

I personally would not worry about this as it went away on retry, and there's no reason to suspect a race condition here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested it again, the rmdir is superfluous I think. Probably it's a leftover / merge error from before I discovered mv -T. Removed it now.

@florianjacob
Copy link
Contributor Author

@Alphor thank you for your valuable testing experiences! 👍 I'll try to address them next week and get the PR into shape with them.

@fpletz fpletz added this to the 18.03 milestone Feb 22, 2018
@florianjacob
Copy link
Contributor Author

Previous piwik front end login was disabled (almost certainly due to the fact that piwik was removed from /etc/shadow and unix auth requires a unix account, although not sure if the piwik front end uses unix auth).

It does use unix auth when you don't provide a database login password in the setup, and I'm quite sure you tested the unix auth manual part that time back, so your analysis is correct and you were in this case in the release notes. 🙂

I have a clone pre upgrade attempt if you'd like any questions answered/explored further.

If you still have that lying around, you could re-test whether the data directoy move with mv -T now works flawlessly on first try without rmdir for you as well. Otherwise, this is ready to be merged now! 🍻

@fpletz fpletz merged commit 37c009c into NixOS:master Feb 26, 2018
@florianjacob florianjacob deleted the matomo branch February 26, 2018 07:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants