salt: 2019.2.0 -> 3000.2 #85561

Flakebi · 2020-04-19T13:37:09Z

Motivation for this change

salt-minion service failed for me since a while with the following error, so I upgraded salt itself and python from 2 to 3.
The error is gone now and minion and master work as expected.

Process Process-1:
Traceback (most recent call last):
  File "/nix/store/mdnyphb6385xrcx2qw2g43bq81x1p67m-python-2.7.17/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
  File "/nix/store/mdnyphb6385xrcx2qw2g43bq81x1p67m-python-2.7.17/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/scripts.py", line 157, in minion_process
    minion.start()
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/cli/daemons.py", line 348, in start
    self.minion.tune_in()
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/minion.py", line 1053, in tune_in
    self._bind()
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/minion.py", line 953, in _bind
    self.event = salt.utils.event.get_event('minion', opts=self.opts, io_loop=self.io_loop)
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/utils/event.py", line 146, in get_event
    raise_errors=raise_errors)
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/utils/event.py", line 274, in __init__
    self.connect_pub()
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/utils/event.py", line 386, in connect_pub
    io_loop=self.io_loop
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/transport/ipc.py", line 264, in __new__
    client.__singleton_init__(io_loop=io_loop, socket_path=socket_path)
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/transport/ipc.py", line 666, in __singleton_init__
    socket_path, io_loop=io_loop)
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/transport/ipc.py", line 292, in __singleton_init__
    self.unpacker = msgpack.Unpacker(encoding=encoding)
TypeError: __init__() got an unexpected keyword argument 'encoding'
Exception AttributeError: AttributeError("'IPCMessageSubscriber' object has no attribute '_refcount_lock'",) in <bound method IPCMessageSubscriber.__del__ of <salt.transport.ipc.IPCMessageSubscriber object at 0x7a276f80fed0>> ignored

Things done

aneeshusa

Tested this out locally and seems to be working for me! Thanks for this, been waiting a while for Salt to release this update.

pkgs/tools/admin/salt/default.nix

drewrisinger

Diff LGTM, better formatting than before. Minor note on repetition of version in msgpack.

Builds locally with nixpkgs-review
(note: no tests run other than build)

https://github.com/NixOS/nixpkgs/pull/85561
1 package built:
salt

pkgs/tools/admin/salt/default.nix

Flakebi · 2020-04-29T20:17:21Z

Salt 3000.2 and 2019.2.4 are out, fixing two CVEs: https://docs.saltstack.com/en/latest/topics/releases/3000.2.html

drewrisinger · 2020-04-29T21:57:58Z

In that case, I suggest:

this PR should be renamed 2019.2.0 -> 3000.2
After this is merged, create 2x new PR with backports to [19.09] and [20.03] (see CONTIBUTING.md)

I'm not sure if the backports should be of 2019.2.4 or 3000.2. I don't know enough about this package to know how much breaking changes there are b/w 2019 & 3000. @Mic92, thoughts?

Mic92 · 2020-04-30T14:17:06Z

I would backport 2019.2.4. Salt is non-trivial software.

mweinelt · 2020-05-03T11:57:08Z

This is now being exploited in the wild. saltstack/salt#57057

Flakebi · 2020-05-03T13:27:48Z

Thanks for the reference @mweinelt, I’m lucky that I run salt in a vpn only.

I tried to backport 2019.2.4 but it does work neither for 20.03 nor for 19.09:

builder for '/nix/store/zpn8dbhf03rr94lc0n1lcb6mkr7ydp29-salt-2019.2.4.drv' failed with exit code 1; last 10 log lines:
  Finished executing setuptoolsInstallPhase
  installing
  Executing pipInstallPhase
  /build/salt-2019.2.4/dist /build/salt-2019.2.4
  DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
  Processing ./salt-2019.2.4-py2-none-any.whl
  Requirement already satisfied: futures>=2.0 in /nix/store/04wmd8vzgvm9das2hf5sr2c84k30v29s-python2.7-futures-3.2.0/lib/python2.7/site-packages (from salt==2019.2.4) (3.2.0)
  Collecting PyYAML<5.1 (from salt==2019.2.4)
    ERROR: Could not find a version that satisfies the requirement PyYAML<5.1 (from salt==2019.2.4) (from versions: none)
  ERROR: No matching distribution found for PyYAML<5.1 (from salt==2019.2.4)
cannot build derivation '/nix/store/mj2nlpk5w2026z69c2zmdhgb8661m8j6-salt-pepper-0.7.5.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/h12n1kdfniibcsf6hjv419ncnqry2q1c-env.drv': 2 dependencies couldn't be built
[0 built (1 failed), 128 copied (348.4 MiB), 78.1 MiB DL]
error: build of '/nix/store/h12n1kdfniibcsf6hjv419ncnqry2q1c-env.drv' failed
2 packages failed to build:
pepper salt

3000.2 builds and seems to work fine in 19.09 (didn’t test thoroughly).
Does anyone know if the salt-minion in 20.03 and 19.09 actually works?

mweinelt · 2020-05-03T13:31:18Z

Does anyone know if the salt-minion in 20.03 and 19.09 actually works?

Yep, is outdated and seems to work.

❯ git reset --hard release-20.03
HEAD is now at c0137ebba71 Merge pull request #86604 from LibreCybernetics/backport-keybase-upgrades
❯ nix-build -A salt
these paths will be fetched (9.22 MiB download, 47.56 MiB unpacked):
  /nix/store/02ra46sj0kh4alkz76cd3vnfmcdprd5b-salt-2019.2.0
copying path '/nix/store/02ra46sj0kh4alkz76cd3vnfmcdprd5b-salt-2019.2.0' from 'https://cache.nixos.org'...
/nix/store/02ra46sj0kh4alkz76cd3vnfmcdprd5b-salt-2019.2.0
❯ cd result/bin
❯ ls
salt  salt-api  salt-call  salt-cloud  salt-cp  salt-key  salt-master  salt-minion  salt-proxy  salt-run  salt-ssh  salt-syndic  salt-unity  spm
❯ ./salt-minion --version
/nix/store/02ra46sj0kh4alkz76cd3vnfmcdprd5b-salt-2019.2.0/lib/python2.7/site-packages/salt/scripts.py:198: DeprecationWarning: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date.  Salt will drop support for Python 2.7 in the Sodium release or later.
salt-minion 2019.2.0 (Fluorine)

For some reason v2019.2.4 requires an older PyYAML version, ours is at 5.2 already, but I can't see that requirement in their repository.

drewrisinger · 2020-05-03T16:30:44Z

Since this is a standalone application, you can override the pyyaml version to 5.X and build with that. I'm not able to post an example right now, but a search for overridepythonattrs should show an example.

…

On Sun, May 3, 2020, 9:31 AM Martin Weinelt ***@***.***> wrote: Does anyone know if the salt-minion in 20.03 and 19.09 actually works? Yep, is outdated and seems to work. ❯ git reset --hard release-20.03 HEAD is now at c0137eb Merge pull request #86604 from LibreCybernetics/backport-keybase-upgrades ❯ nix-build -A salt these paths will be fetched (9.22 MiB download, 47.56 MiB unpacked): /nix/store/02ra46sj0kh4alkz76cd3vnfmcdprd5b-salt-2019.2.0 copying path '/nix/store/02ra46sj0kh4alkz76cd3vnfmcdprd5b-salt-2019.2.0' from 'https://cache.nixos.org'... /nix/store/02ra46sj0kh4alkz76cd3vnfmcdprd5b-salt-2019.2.0 ❯ cd result/bin ❯ ls salt salt-api salt-call salt-cloud salt-cp salt-key salt-master salt-minion salt-proxy salt-run salt-ssh salt-syndic salt-unity spm ❯ ./salt-minion --version /nix/store/02ra46sj0kh4alkz76cd3vnfmcdprd5b-salt-2019.2.0/lib/python2.7/site-packages/salt/scripts.py:198: DeprecationWarning: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. Salt will drop support for Python 2.7 in the Sodium release or later. salt-minion 2019.2.0 (Fluorine) For some reason v2019.2.4 requires an older PyYAML version, ours is at 5.2 already, but I can't see that requirement in their repository. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#85561 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACNZYI4FXO6SWNFYE2BVBMLRPVW3FANCNFSM4MLZJ46A> .

Flakebi · 2020-05-03T19:35:18Z

Interesting that 2019.2.0 works on 20.03, thanks @mweinelt, I thought it might suffer from the same bug as on unstable.

salt 2019.2.4 requires an old PyYaml as specified here.
I made prs with overrides for 19.09 and 20.03 in #86651 and #86653.

mweinelt · 2020-05-03T19:41:02Z

LGTM.

❯ nixpkgs-review pr 85561
$ git -c fetch.prune=false fetch --force https://github.com/NixOS/nixpkgs master:refs/nixpkgs-review/0 pull/85561/head:refs/nixpkgs-review/1
remote: Enumerating objects: 44, done.
remote: Counting objects: 100% (44/44), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 53 (delta 34), reused 40 (delta 34), pack-reused 9
Unpacking objects: 100% (53/53), 13.36 KiB | 285.00 KiB/s, done.
From https://github.com/NixOS/nixpkgs
   0dc3e485a11..af62dc53559  master               -> refs/nixpkgs-review/0
   06e55a145b3..435713eac22  refs/pull/85561/head -> refs/nixpkgs-review/1
$ git worktree add /home/hexa/.cache/nixpkgs-review/pr-85561/nixpkgs af62dc53559c419d570df5bf9367a2e5d0c2a1d4
Preparing worktree (detached HEAD af62dc53559)
Updating files: 100% (21630/21630), done.
HEAD is now at af62dc53559 kismet: 2016-07-R1 -> 2020-04-R2
$ git merge --no-commit 435713eac2237abc1708497cf71a621fc49cc5ea
Automatic merge went well; stopped before committing as requested
$ nix build --no-link --keep-going --option build-use-sandbox relaxed -f /home/hexa/.cache/nixpkgs-review/pr-85561/build.nix
[4 built, 13 copied (61.0 MiB), 2.2 MiB DL]
https://github.com/NixOS/nixpkgs/pull/85561
1 package built:
salt

$ nix-shell /home/hexa/.cache/nixpkgs-review/pr-85561/shell.nix
these paths will be fetched (0.05 MiB download, 0.28 MiB unpacked):
  /nix/store/d27ihabal1lgajnjj56i0kg02fnavs5a-bash-interactive-4.4-p23-dev
copying path '/nix/store/d27ihabal1lgajnjj56i0kg02fnavs5a-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...

[nix-shell:~/.cache/nixpkgs-review/pr-85561]$ ./results/salt/bin/
salt                  .salt-call-wrapped    .salt-cp-wrapped      .salt-master-wrapped  .salt-proxy-wrapped   .salt-ssh-wrapped     .salt-unity-wrapped
salt-api              salt-cloud            salt-key              salt-minion           salt-run              salt-syndic           .salt-wrapped
.salt-api-wrapped     .salt-cloud-wrapped   .salt-key-wrapped     .salt-minion-wrapped  .salt-run-wrapped     .salt-syndic-wrapped  spm
salt-call             salt-cp               salt-master           salt-proxy            salt-ssh              salt-unity            .spm-wrapped

[nix-shell:~/.cache/nixpkgs-review/pr-85561]$ ./results/salt/bin/salt-minion --version
/nix/store/bghqfj3cdk2s5qwzvl6kf4sq1vab8id9-salt-3000.2/lib/python3.7/site-packages/salt/ext/tornado/httputil.py:107: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  class HTTPHeaders(collections.MutableMapping):
salt-minion 3000.2
    
[nix-shell:~/.cache/nixpkgs-review/pr-85561]$ ./results/salt/bin/salt-master --version
/nix/store/bghqfj3cdk2s5qwzvl6kf4sq1vab8id9-salt-3000.2/lib/python3.7/site-packages/salt/ext/tornado/httputil.py:107: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  class HTTPHeaders(collections.MutableMapping):
salt-master 3000.2

bhipple · 2020-05-03T23:52:28Z

Result of nixpkgs-review pr 85561 1

1 package built:

- salt

ofborg bot requested a review from aneeshusa April 19, 2020 13:43

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 1-10 10.rebuild-linux: 1 labels Apr 19, 2020

aneeshusa reviewed Apr 21, 2020

View reviewed changes

pkgs/tools/admin/salt/default.nix Outdated Show resolved Hide resolved

pkgs/tools/admin/salt/default.nix Outdated Show resolved Hide resolved

Flakebi force-pushed the salt branch from a02469c to 42ed66f Compare April 21, 2020 17:30

ofborg bot requested a review from aneeshusa April 21, 2020 17:39

drewrisinger approved these changes Apr 28, 2020

View reviewed changes

pkgs/tools/admin/salt/default.nix Outdated Show resolved Hide resolved

Flakebi force-pushed the salt branch from 42ed66f to de08daf Compare April 29, 2020 18:45

salt: 2019.2.0 -> 3000.2

435713e

Flakebi force-pushed the salt branch from de08daf to 435713e Compare April 29, 2020 20:15

Mic92 added the 1.severity: security label Apr 29, 2020

Flakebi changed the title ~~salt: 2019.2.0 -> 3000.1~~ salt: 2019.2.0 -> 3000.2 Apr 29, 2020

This was referenced May 3, 2020

[19.09] salt: 2019.2.0 -> 2019.2.4 #86651

Merged

[20.03] salt: 2019.2.0 -> 2019.2.4 #86653

Merged

bhipple approved these changes May 3, 2020

View reviewed changes

bhipple merged commit f2b48ae into NixOS:master May 3, 2020

Flakebi deleted the salt branch May 4, 2020 08:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

salt: 2019.2.0 -> 3000.2 #85561

salt: 2019.2.0 -> 3000.2 #85561

Flakebi commented Apr 19, 2020

aneeshusa left a comment

drewrisinger left a comment •

edited

Flakebi commented Apr 29, 2020

drewrisinger commented Apr 29, 2020

Mic92 commented Apr 30, 2020 •

edited

mweinelt commented May 3, 2020

Flakebi commented May 3, 2020

mweinelt commented May 3, 2020

drewrisinger commented May 3, 2020 via email

Flakebi commented May 3, 2020

mweinelt commented May 3, 2020

bhipple commented May 3, 2020

salt: 2019.2.0 -> 3000.2 #85561

salt: 2019.2.0 -> 3000.2 #85561

Conversation

Flakebi commented Apr 19, 2020

Motivation for this change

Things done

aneeshusa left a comment

Choose a reason for hiding this comment

drewrisinger left a comment • edited

Choose a reason for hiding this comment

Flakebi commented Apr 29, 2020

drewrisinger commented Apr 29, 2020

Mic92 commented Apr 30, 2020 • edited

mweinelt commented May 3, 2020

Flakebi commented May 3, 2020

mweinelt commented May 3, 2020

drewrisinger commented May 3, 2020 via email

Flakebi commented May 3, 2020

mweinelt commented May 3, 2020

bhipple commented May 3, 2020

drewrisinger left a comment •

edited

Mic92 commented Apr 30, 2020 •

edited