Matomo (module, package) updates #69342

Kiwi · 2019-09-24T08:59:35Z

Motivation for this change

I got tired of seeing red every time I nixos-rebuilt my server so I updated Matomo. While doing so I found a bug in the latest version that is fixed in the next release (3.12.0) (edit: ...and then the b4 ended up breaking it on NixoOS which stalled this PR for a while... and now ~1 month later and 3.12.0 was released 20 hrs ago :| ) which is currently at beta 3. The project releases beta builds frequently and release builds not so often... so I think it'll be good to be able to use them. Forewarning, you can't downgrade (at least not easily, I don't know how to and I didn't try too hard, I'm sure if you had backups you could...) though as it changes the database. But it's been working better for me than the last one so.

Things done

Added a beta build (modeled after the way atom/atom-beta handles it) and fixed deprecation warnings related to phpfpm socket and extraConfig that showed up in nixos-unstable a while ago.

@mmilata added a test for matomo and @aanderse helped me figure out how to make it work on matomo-beta as well. Thanks all!

Notify maintainers

cc @florianjacob

aanderse

Thanks for taking this on!

nixos/modules/services/web-apps/matomo.nix

pkgs/top-level/all-packages.nix

aanderse · 2019-09-25T00:38:30Z

@Kiwi please post configuration you are testing with.

Kiwi · 2019-09-27T11:36:09Z

@Kiwi please post configuration you are testing with.

(Already has it from IRC but for anyone else that may want it) minimal example

florianjacob · 2019-09-27T11:47:13Z

I'll try to have an in-depth look on Sunday. Ahead: Just properly referencing the option instead of the value seems to be a good solution for the phpSocket issue.

aanderse · 2019-09-27T21:17:59Z

nixos/modules/rename.nix

@@ -134,6 +134,7 @@ with lib;
    (mkRenamedOptionModule [ "services" "piwik" "enable" ] [ "services" "matomo" "enable" ])
    (mkRenamedOptionModule [ "services" "piwik" "webServerUser" ] [ "services" "matomo" "webServerUser" ])
    (mkRenamedOptionModule [ "services" "piwik" "phpfpmProcessManagerConfig" ] [ "services" "matomo" "phpfpmProcessManagerConfig" ])


Out of curiosity what happens if you have services.piwik.phpfpmProcessManagerConfig set in your configuration.nix with this branch? (ie. an option which has been renamed and then removed)

Yeah I'm not sure if the way I did that is the right way and I don't know this well enough to even know of what other possible approaches there might be. it seemed logical to me, first attempt that fixed the error, seemed to work, and I didn't explore it much after that.

To answer your question...

# with services.piwik.* 00:54:19 [kiwi@absurd-nixos nixpkgs]$ sudo nixos-rebuild -I nixpkgs=. build building Nix... building the system configuration... trace: warning: The option definition `services.matomo.phpfpmProcessManagerConfig' in `/home/kiwi/nixpkgs/nixos/modules/rename.nix' no longer has any effect; please remove it. Use services.phpfpm.pools.<name>.settings trace: warning: The option `services.piwik.phpfpmProcessManagerConfig' defined in `/etc/nixos/modules/matomo.nix' has been renamed to `services.matomo.phpfpmProcessManagerConfig'. 00:54:34 [kiwi@absurd-nixos nixpkgs]$ sudo vim /etc/nixos/modules/matomo.nix # changed to services.matomo.* 00:54:58 [kiwi@absurd-nixos nixpkgs]$ sudo nixos-rebuild -I nixpkgs=. build building Nix... building the system configuration... trace: warning: The option definition `services.matomo.phpfpmProcessManagerConfig' in `/etc/nixos/modules/matomo.nix' no longer has any effect; please remove it. Use services.phpfpm.pools.<name>.settings 00:55:06 [kiwi@absurd-nixos nixpkgs]$

In that case the message seems confusing so let's modify the mkRenamedOptionModule to something like

(mkRemovedOptionModule [ "services" "piwik" "phpfpmProcessManagerConfig" ] "Use services.phpfpm.pools.<name>.settings")

aanderse · 2019-09-27T21:18:21Z

@Kiwi it looks like you have resolved your issues, correct?

Kiwi · 2019-09-28T01:05:31Z

@Kiwi it looks like you have resolved your issues, correct?

Yes, I believe so and then some. (Improved/corrected the documentation more than I had, which also avoids the phpSocket definition altogether)

mmilata · 2019-10-04T20:46:55Z

@Kiwi Thanks for fixing this! I suck at using github's search and opened duplicate PR ↑. It adds simple NixOS test. Would you be interested in adding it to your PR? Feel free to use it: mmilata@068c0b2. It's missing a maintainer - I'm OK with being one if you're not interested.

nixos/modules/rename.nix

aanderse · 2019-10-12T21:21:24Z

ping @Kiwi

Fixes the phpfpm deprecation warnings about listen and extraConfig by using fpm.socket and settings. Removes phpfpmProcessManagerConfig.

Add a beta version of matomo 3.12 that has recent bug fixes. They release these more frequently so it's a good option to have.

Kiwi · 2019-10-28T05:47:51Z

Apologies this took so long. I hope I've done everything right...

If I didn't mess anything up:

I have reverted the beta back to the last version that worked which was 3.12.0-b3. I have tried the latest versions... they're still broken. It's up to version 3.12.0-rc2 at this point. I'm not sure how to fix it, either. I've looked briefly but I and I think everyone else that's looked at it have been incredibly busy >.> and I'm not on good (or recent) terms with PHP...

I think I have the removed modules lines how they're supposed to be??

I git cherry-picked the test above that @mmilata supplied.
It could probably be added to the beta build too...

I rebased on to master as of a few minutes ago because there were conflicts with the old PR. I've been watching the bot go, as I type, and it looks like it's probably going to pass (2 to go, 10 passed) Aha! there it is. 12 (you saw nothing!) passed.

I wish I could get it building the newest version which I expect will be out soon...but as it is now, the beta 3 I believe is still better than what we have currently, 3.11.0. I know it's at least fixed one annoying bug that was bothering me in 3.11.0, I don't know either way if b3 introduced any bugs, but so far it's been fine... and I can't go back to 3.11.0 (the beta updated the database) on my server so I'd really appreciate if this was merged. :)

Anyway, if this PR is satisfactory to all involved for what it does?? I think it should be merged. After all it does fix multiple problems; known bugs in 3.11.0 if you choose to use the beta, which is now possible, as well as the phpfpm deprecation warnings being resolved. And it has a test...! :) After that we can worry about getting the next version to work in a separate PR that won't conflict with this one.... as soon as anyone that can fix it has time. and add that other test...

florianjacob · 2019-10-28T20:05:26Z

@GrahamcOfBorg test matomo

florianjacob

The port to the new phpfpm options is done well and the reviewer feedback was adressed. We should merge this now and do a backport to 19.09, at least for the new phpfpm options. Finding the cause for the issues with the upcoming matomo version will need more time for research.

@Kiwi volunteered to maintain beta versions, so I'm fine with having multiple versions in nixpkgs. This could also be a handy way to package the new stable versions which do not contain security fixes, i.e. for which we don't want to change the default in stable, but to give users the choice to upgrade if they want without relying on unstable.

@mmilata thanks a lot for the test, did not think of such a simple version as I was busy thinking about the impossibility of matomo automated installs, but even just checking the installer page adds a lot of value to see wether the stack generally works.

aanderse · 2019-10-29T00:26:22Z

Since @florianjacob has approved there isn't too much to say past this point. If this PR suits him it should be good to go.

The only remaining thing for this PR is we might consider testing the beta version of matomo as well so when someone files a PR to update it we can merge more confidently. Sound good @Kiwi?

Kiwi · 2019-10-29T02:37:23Z

(already addressed on IRC but for anyone else watching) @aanderse Yes, that sounds good. I should have it taken care of either tonight or tomorrow.

aanderse · 2019-10-30T00:41:04Z

@GrahamcOfBorg test matomo.matomo matomo.matomo-beta

aanderse · 2019-10-30T03:12:37Z

Thanks everyone for your contributions on this.

In a future PR it would be nice to see this module take advantage of the services.mysql.ensureUsers option to provide a seamless experience for the user. I would also love to see usage of PermissionsStartOnly removed from this module as that feature has been deprecated.

Kiwi · 2019-10-30T04:07:13Z

I would also love to see usage of PermissionsStartOnly removed from this module as that feature has been deprecated.

HAHAHAHHA

What the... I'm floored!

This couldn't be more perfect

Let me make sure I understand this correctly...

You're saying that this PR, the primary purpose of which was to fix deprecation warnings, took me so long to wrap up that something else was deprecated in the meantime? Ha! (and if that's not true, please let me remain ignorant, just this once)

Lest us not forget the number of betas (all broken after b3) of 3.12.0 there were. And indeed, less than 24 hours ago, the official version 3.12.0 was released. Which we still can't use.

and of course who can forget about the 2 subsequent duplicate PRs during this time.

This is the best laugh I've had all week. This has gone far beyond being mostly absurd to entirely. I love it. It's amazing. <3

aanderse · 2019-10-30T10:10:15Z

PermissionsStartOnly was deprecated 12 months ago.

Thanks for your patience on this one. For future reference had this been 3 separate PRs they probably would have been much faster to merge.

Great job everyone! 🎉

Kiwi · 2019-10-30T14:27:21Z

PermissionsStartOnly was deprecated 12 months ago.

Boo. :( I reject your reality! Maybe you mean 12 days? Yes, surely it was deprecated 12 days ago, not months.

Thanks for your patience on this one. For future reference had this been 3 separate PRs they probably would have been much faster to merge.

Yeah, most likely, and in the future I'll keep that in mind... this time it ended up being incredibly amusing and I wouldn't have had to look at how to do the tests if it were 3.

Thanks for all of your help :D

florianjacob · 2019-10-30T22:20:15Z

Thanks alot to everyone! I will try to investigate the issues with the new 3.12 release soon.

For the record: I intentionally did not use ensureUsers in the module due to the lack of automated install for matomo, I wanted the admin to create database and user by themselves so that they know what to enter. Automatic install has come in reach though, if we manage to pack this plugin, the Matomo installer is a thing of the past for NixOS users: https://plugins.matomo.org/ExtraTools . This will be the next big thing for matomo on NixOS when the current 3.12 issues are fixed.

mmilata · 2019-11-18T15:59:48Z

Backport to 19.09: #73692. I've dropped the commit that adds matomo-beta, let me know if you want me to add it.

ofborg bot added 6.topic: nixos 8.has: module (update) 8.has: package (new) labels Sep 24, 2019

ofborg bot requested a review from florianjacob September 24, 2019 09:44

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Sep 24, 2019

aanderse requested changes Sep 24, 2019

View reviewed changes

Kiwi force-pushed the matomo-updates branch from 8b65c45 to 9f956fa Compare September 24, 2019 11:11

aanderse requested changes Sep 24, 2019

View reviewed changes

pkgs/top-level/all-packages.nix Outdated Show resolved Hide resolved

Kiwi force-pushed the matomo-updates branch from 9f956fa to abd437c Compare September 24, 2019 20:33

Kiwi force-pushed the matomo-updates branch from abd437c to 681452b Compare September 25, 2019 00:51

ofborg bot added the 8.has: documentation label Sep 25, 2019

aanderse reviewed Sep 27, 2019

View reviewed changes

Kiwi requested review from edwtjo and FRidh as code owners September 29, 2019 06:28

ofborg bot added 6.topic: python 8.has: changelog 2.status: merge conflict labels Sep 29, 2019

Kiwi force-pushed the matomo-updates branch from a837eb8 to 681452b Compare September 29, 2019 06:33

ofborg bot removed 6.topic: python 8.has: changelog 2.status: merge conflict labels Sep 29, 2019

Kiwi force-pushed the matomo-updates branch 2 times, most recently from ac8494c to 4ce6299 Compare September 29, 2019 09:58

aanderse approved these changes Sep 29, 2019

View reviewed changes

aanderse mentioned this pull request Oct 4, 2019

nixos/matomo: fix phpfpm config, add test #70413

Closed

10 tasks

mmilata approved these changes Oct 4, 2019

View reviewed changes

nixos/modules/rename.nix Show resolved Hide resolved

Kiwi force-pushed the matomo-updates branch from e07c30b to 89ed32f Compare October 28, 2019 04:54

ofborg bot added the 2.status: merge conflict label Oct 28, 2019

Kiwi added 2 commits October 28, 2019 05:07

nixos/matomo: fix deprecation warnings

6ea626d

Fixes the phpfpm deprecation warnings about listen and extraConfig by using fpm.socket and settings. Removes phpfpmProcessManagerConfig.

matomo-beta: init at 3.12.0-b3

cb85d27

Add a beta version of matomo 3.12 that has recent bug fixes. They release these more frequently so it's a good option to have.

Kiwi force-pushed the matomo-updates branch from 89ed32f to b857cf1 Compare October 28, 2019 05:12

ofborg bot removed the 2.status: merge conflict label Oct 28, 2019

florianjacob mentioned this pull request Oct 28, 2019

matomo: Refactor to use phpFpm settings instead of extraConfig #71997

Closed

10 tasks

florianjacob approved these changes Oct 28, 2019

View reviewed changes

nixos/matomo: add test

605379b

Kiwi force-pushed the matomo-updates branch from b857cf1 to 605379b Compare October 29, 2019 21:05

ofborg bot requested a review from florianjacob October 29, 2019 21:18

aanderse merged commit 722b99b into NixOS:master Oct 30, 2019

Kiwi mentioned this pull request Oct 31, 2019

3.12.0 (3.12.0-b4 and up) is broken on NixOS matomo-org/matomo#15087

Closed

mmilata mentioned this pull request Nov 18, 2019

nixos/matomo: backport module fix (19.09) #73692

Merged

10 tasks

Kiwi deleted the matomo-updates branch November 28, 2019 17:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Matomo (module, package) updates #69342

Matomo (module, package) updates #69342

Kiwi commented Sep 24, 2019 •

edited

aanderse left a comment

aanderse commented Sep 25, 2019

Kiwi commented Sep 27, 2019

florianjacob commented Sep 27, 2019

aanderse Sep 27, 2019

Kiwi Sep 28, 2019

aanderse Sep 28, 2019

aanderse commented Sep 27, 2019

Kiwi commented Sep 28, 2019

mmilata commented Oct 4, 2019

aanderse commented Oct 12, 2019

Kiwi commented Oct 28, 2019 •

edited

florianjacob commented Oct 28, 2019

florianjacob left a comment

aanderse commented Oct 29, 2019

Kiwi commented Oct 29, 2019

aanderse commented Oct 30, 2019

aanderse commented Oct 30, 2019

Kiwi commented Oct 30, 2019

aanderse commented Oct 30, 2019

Kiwi commented Oct 30, 2019

florianjacob commented Oct 30, 2019

mmilata commented Nov 18, 2019

Matomo (module, package) updates #69342

Matomo (module, package) updates #69342

Conversation

Kiwi commented Sep 24, 2019 • edited

Motivation for this change

Things done

Notify maintainers

aanderse left a comment

Choose a reason for hiding this comment

aanderse commented Sep 25, 2019

Kiwi commented Sep 27, 2019

florianjacob commented Sep 27, 2019

aanderse Sep 27, 2019

Choose a reason for hiding this comment

Kiwi Sep 28, 2019

Choose a reason for hiding this comment

aanderse Sep 28, 2019

Choose a reason for hiding this comment

aanderse commented Sep 27, 2019

Kiwi commented Sep 28, 2019

mmilata commented Oct 4, 2019

aanderse commented Oct 12, 2019

Kiwi commented Oct 28, 2019 • edited

florianjacob commented Oct 28, 2019

florianjacob left a comment

Choose a reason for hiding this comment

aanderse commented Oct 29, 2019

Kiwi commented Oct 29, 2019

aanderse commented Oct 30, 2019

aanderse commented Oct 30, 2019

Kiwi commented Oct 30, 2019

aanderse commented Oct 30, 2019

Kiwi commented Oct 30, 2019

florianjacob commented Oct 30, 2019

mmilata commented Nov 18, 2019

Kiwi commented Sep 24, 2019 •

edited

Kiwi commented Oct 28, 2019 •

edited