Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update all skaware packages (s6, s6-rc, execline etc) #110381

Merged
merged 2 commits into from Jan 23, 2021

Conversation

svanderburg
Copy link
Member

@svanderburg svanderburg commented Jan 21, 2021

Motivation for this change

This pull request updates all skaware software packages, most notably: the s6 supervising system, the underlying libraries and related tools.

I have successfully tested this against my work in progress branch of the very experimental nix-processmgmt framework that adds s6-rc support: https://github.com/svanderburg/nix-processmgmt/tree/s6-rc

In addition to updating all the packages to their latest versions, I have also packaged s6-linux-init that depends on newer versions of execline and skalibs.

I have commit access to Nixpkgs, but since I'm still a relative newbie to s6, I'd like the community to take a look at this update first, just to be sure that I don't break anything important.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@SuperSandro2000
Copy link
Member

This is a semi-automatic executed nixpkgs-review which is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.

Result of nixpkgs-review pr 110381 run on x86_64-linux 1

1 package failed to build and are new build failure:
12 packages built:
  • execline (skawarePackages.execline)
  • nsss (skawarePackages.nsss)
  • s6 (skawarePackages.s6)
  • s6-dns (skawarePackages.s6-dns)
  • s6-linux-init (skawarePackages.s6-linux-init)
  • s6-linux-utils (skawarePackages.s6-linux-utils)
  • s6-networking (skawarePackages.s6-networking)
  • s6-portable-utils (skawarePackages.s6-portable-utils)
  • s6-rc (skawarePackages.s6-rc)
  • sdnotify-wrapper (skawarePackages.sdnotify-wrapper)
  • skalibs (skawarePackages.skalibs)
  • utmps (skawarePackages.utmps)

@svanderburg
Copy link
Member Author

svanderburg commented Jan 22, 2021

@SuperSandro2000 I just tried building fdtools locally, and it seems to work for me. If I look at the output log of the build above, then I see that the log is truncated. This suggests there is an infrastructure problem, not a build problem.

@SuperSandro2000
Copy link
Member

I think this is the error:

Making compile/host/vc-switch
/nix/store/wcz8lsibkw2nikdb0bjl22p75lfwahzr-binutils-2.35.1/bin/ld: ./compile/host/vc-switch.o: in function `main':
vc-switch-linux.c:(.text.startup+0x182): undefined reference to `pathexec0'
collect2: error: ld returned 1 exit status

Making include/fdtools.h

Making library/fdtools.a

@svanderburg
Copy link
Member Author

@SuperSandro2000 Ah I see, the output is indeed a bit weird. I think I can provide a fix for this.

@svanderburg
Copy link
Member Author

@SuperSandro2000 I have patched fdtools to make it compile against the latest skalibs. Some preliminary tests show that the package works.

The upstream maintainer does not seem to offer a newer version, so maybe we can submit the patch.

@Profpatsch I'm not a fdtools user and I have noticed that you are the maintainer. Let me know if there are any objections in getting this pull request integrated.

Copy link
Contributor

@bqv bqv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My testing proves successful - I'm satisfied

@Profpatsch
Copy link
Member

Thanks for doing this, I started last week but was a bit off-put by the execline exec changes. Checking …

@Profpatsch
Copy link
Member

Can we move the updates into one commit with the content of https://www.mail-archive.com/skaware@list.skarnet.org/msg01510.html as the commit description? Then add the s6-linux-init as a commit on top. Makes blaming a lot easier, and they are conceptually one bump.

@Profpatsch
Copy link
Member

Let me do it

Maintainer notes
-----
The execline exec function interface changed quite drastically, and
backwards-compatibility to the old functions was dropped in-between
the last release and this one. Thus, downstream code might break.
At the end of this commit message is a compatibility interface.

-----
Release notes
-----

 Hello,

 Happy New Year to everyone!
 New versions of the skarnet.org packages are available.

 This is a major release. The skalibs major version number has been
bumped, which means that compatibility with previous versions is not
ensured. Other packages have been updated to build against the new
skalibs. If they only had their patch number increased, that's all
the modifications they had (save for possible bugfixes); but some
packages also received significant changes and underwent either a major
(compatibility not ensured) or minor (simple additions) release.

 Support for the 2.9.* branch of skalibs, and associated versions of
the other packages, is still ensured for a while, but users are always
strongly encouraged to upgrade.

 * General
   -------

 - Some rarely-triggered build bugs have been fixed.
 - -fno-stack-protector is not part of the default CFLAGS anymore;
stack protector policy now defaults to the compiler's settings.

* skalibs-2.10.0.0
  ----------------

 - Bugfixes.
 - Significant code cleanup.
 - New sysdep: chroot.
 - Lots of new functions, mostly to optimize the number of needed
fcntl() calls at open() time. Traces should generally be marginally
shorter than they were before.
 - Removal of the DJBUNIX_FLAG_NB and DJBUNIX_FLAG_COE macros, replaced
by the POSIX O_NONBLOCK and O_CLOEXEC macros wherever they were used.
 - Removal of the skalibs/webipc.h header, and better header separation.
 - Complete revamping of the pathexec functions, now separated into
exec_* (simple execution) and mexec_* (execution with merging of the
environment first). In true skalibs fashion, there is a little code,
and 3 pages of convenience macros (the exec.h header).
 - Complete rewrite of the locking functions, with a change of
underlying mechanisms. The skalibs locking primitives are now named
fd_lock(), fd_unlock() and fd_islocked().
 The Unix locks primitive space is a horror show. flock() is not
POSIX and does not have a way to test for a lock without taking it.
The POSIX lockf() only has exclusive locks, not shared ones. The least
bad option is fcntl(), which has shared and exclusive locks *and* a way
to check for a lock without taking it, but does not allow taking a
shared lock via a O_WRONLY file descriptor. Of all inconveniences this
is the most minor one, so now skalibs uses fcntl().

 https://skarnet.org/software/skalibs/
 git://git.skarnet.org/skalibs

 * nsss-0.1.0.0
   ------------

 - New --enable-libc-includes configure option. Without this option,
the pwd.h, grp.h and shadow.h headers are not installed anymore, so
by default installing nsss on a FHS system does not overwrite the
libc headers.

 https://skarnet.org/software/nsss/
 git://git.skarnet.org/nsss

 * utmps-0.1.0.0
   -------------

 - New --enable-libc-includes configure option. Without this option,
the utmpx.h header is not installed anymore, so by default installing
utmps on a FHS system does not overwrite the libc headers.

 https://skarnet.org/software/utmps/
 git://git.skarnet.org/utmps

 * execline-2.7.0.0
   ----------------

 - Bugfixes.
 - The trap program has changed. The "timeout" directive has been
removed; a "default" directive has been added, to handle all signals
for which a specific directive has not been given. Subprograms are
now run with the SIGNAL environment variable set to the signal number
(in addition to ! always being set to the application's pid).
 - The forstdin program has changed. It now exits 0 if it has read at
least one line, and 1 otherwise.
 - The default list of delimiters for backtick, withstdinas, forstdin
and forbacktickx has been set to "\n", so by default those programs
will read and/or split on lines and only lines.
 - The backtick, withstdinas, forstdin, forbacktickx, forx, getpid
and getcwd programs now have a -E option to activate autoimport.
(This saves the user from manually adding "importas var var" after
every use of these programs.)

 https://skarnet.org/software/execline/
 git://git.skarnet.org/execline

 * s6-2.10.0.0
   -----------

 It is imperative to restart your supervision trees, by rebooting if
necessary, after upgrading s6 to the new version. Otherwise, new s6
binaries interacting with service directories maintained by old
s6-supervise binaries may not work.
 If you are using s6-linux-init, it is necessary to upgrade to the
latest version of s6-linux-init at the same time as s6.

 - Bugfixes.
 - Significant code refactoring.
 - The internal locking system of service directories has changed,
allowing for a cleaner permissions model and official support of
relaxed permissions.
 - New binary to implement those relaxed permissions: s6-svperms.
 - The "nosetsid" file is not supported anymore in service directories.
Services are now always started in a new session.
 - s6-supervise now traps SIGINT: before dying, it sends a SIGINT to its
service's process group. This allows correct transmission of ^C when a
supervision tree is running in a terminal, even though every service
runs in its own session.
 - s6-svc -X doesn't exist anymore. s6-supervise now always closes stdin
and stdout on the last execution of the service.
 - The semantics of SIGHUP and SIGQUIT have changed for s6-supervise.
 - The set of commands sent by s6-svscanctl and received by s6-svscan
has been cleaned up and made more logical.
 - When told to exit normally (typically via s6-svscanctl -t), s6-svscan
now first waits for the whole supervision tree to die. The
.s6-svscan/finish script can now assume that all services are completely
down. (s6-svscanctl -b is an exception; it should not be used in normal
circumstances.)
 - The -s and -S options to s6-svscan are not supported anymore. Signal
management in s6-svscan has been streamlined: signals have a default
handler that can be overridden by a corresponding executable
.s6-svscan/SIGfoo file.
 - Default signal handlers for s6-svscan have more intuitive semantics.
 - New binary to help with management of user-owned supervision trees:
s6-usertree-maker.

 https://skarnet.org/software/s6/
 git://git.skarnet.org/s6

s6 now has man pages! Thanks to flexibeast for performing the conversion

work. Please allow some time for the man pages to be updated to reflect
the current HTML documentation. The repository can be found here:

 https://github.com/flexibeast/s6-man-pages

 * s6-linux-init-1.0.6.0
   ---------------------

 It *is necessary* to upgrade s6-linux-init at the same time as s6.
 It *is recommended*, although not strictly necessary, to create your
run-image directory again via a s6-linux-init-maker invocation. Old
images will still boot, as long as you are using an upgraded version
of s6-linux-init; but they may incorrectly handle signals sent to init,
so for instance Ctrl-Alt-Del may not work anymore, until you run
s6-linux-init-maker again.

 - New internal binary: s6-linux-init-nuke. This program is not meant
to be invoked by users directly: it simply removes a dependency to the
'kill' program in a rare case involving containers.

 https://skarnet.org/software/s6-linux-init/
 git://git.skarnet.org/s6-linux-init

 * s6-dns-2.3.4.0
   --------------

 - New library function: s6dns_message_parse_question().

 https://skarnet.org/software/s6-dns/
 git://git.skarnet.org/s6-dns

 * s6-networking-2.4.0.0
   ---------------------

 - Important refactoring of the tls code. The crypto tunnel now runs
as a child of the application, instead of the other way around. It is
now isolated in a s6-tls[cd]-io binary; s6-tlsc is now a simple wrapper
around s6-tlsc-io, and s6-tlsd is a simple wrapper around s6-tlsd-io.
 - New binaries: s6-ucspitlsc and s6-ucspitlsd. Those implement
opportunistic TLS via the UCSPI-TLS protocol.
 - The -K option to the tls binaries has changed semantics: it now
enforces a timeout for the handshake instead of dropping the connection
after some inactivity. Note that this option is only useful with the
bearssl backend: the libtls backend always performs a synchronous
handshake, with no way of interrupting it after a timeout expires.
 - The execline dependency is now optional. Disabling execline, however,
changes the behaviour of s6-tcpserver-access (which cannot support
exec files without it).

 https://skarnet.org/software/s6-networking/
 git://git.skarnet.org/s6-networking

 It is now possible to build the s6-networking package against OpenSSL
instead of LibreSSL, thanks to the libretls project:
 https://git.causal.agency/libretls/about/

 * mdevd-0.1.3.0
   -------------

 - New -C option to the mdevd program. This option makes mdevd
automatically spawn a mdevd-coldplug program when it's ready, allowing
mdevd to be used as a drop-in mdev -d replacement. (Note that the
coldplug is also performed if mdevd restarts after being killed, so
this feature should not be used in place of a proper service startup
sequence with a mdevd-coldplug oneshot depending on the mdevd longrun.
It has only been added for convenience.)

 https://skarnet.org/software/mdevd/
 git://git.skarnet.org/mdevd

 * Other packages
   --------------

 The following packages have received an update so they build with the

latest version of skalibs and other dependencies, but nothing has changed

except possibly some bugfixes, and hopefully not too many bug additions.

 - s6-rc-0.5.2.1. (It is not necessary to recompile your service
database. However, it is necessary to upgrade s6-rc along with s6, and
to reboot the system ASAP after upgrading.)
 - s6-portable-utils-2.2.3.1
 - s6-linux-utils-2.5.1.4
 - bcnm-0.0.1.2

 Enjoy,
 Bug-reports welcome.

--
 Laurent

-----
execline compat interface
-----

    /* Compatibility */

    #define pathexec_run(file, argv, envp) exec_ae(file, argv, envp)
    #define pathexec0_run(file, argv, envp) exec0_ae(file, argv, envp)
    #define xpathexec_run(file, argv, envp) xexec_ae(file, argv, envp)
    #define xpathexec0_run(file, argv, envp) xexec0_ae(file, argv, envp)

    #define pathexec_env(key, value) env_mexec(key, value)
    #define pathexec_fromenv(argv, envp, envlen) mexec_f(argv, envp, envlen)
    #define pathexec(argv) mexec(argv)
    #define pathexec0(argv) mexec0(argv)
    #define xpathexec_fromenv(argv, envp, envlen) xmexec_f(argv, envp, envlen)
    #define xpathexec(argv) xmexec(argv)
    #define xpathexec0(argv) xmexec0(argv)

    #define pathexec_r_name(file, argv, envp, envlen, modif, modiflen) mexec_afm(file, argv, envp, envlen, modif, modiflen)
    #define pathexec_r(argv, envp, envlen, modif, modiflen) mexec_fm(argv, envp, envlen, modif, modiflen)
    #define xpathexec_r_name(file, argv, envp, envlen, modif, modiflen) xmexec_afm(file, argv, envp, envlen, modif, modiflen)
    #define xpathexec_r(argv, envp, envlen, modif, modiflen) xmexec_fm(argv, envp, envlen, modif, modiflen)

    #endif

copied from skarnet/skalibs@18e4356#diff-69efbe5d997280a1430a6af2fa38e3f5105e706076a26fc751885c505ca598c6R140
@Profpatsch
Copy link
Member

@GrahamcOfBorg build skawarePackages

@Profpatsch
Copy link
Member

Everything looks reasonably buildy-looking, let’s merge.

@Profpatsch Profpatsch merged commit 01ea0b5 into NixOS:master Jan 23, 2021
@svanderburg
Copy link
Member Author

@Profpatsch Thanks! Squashing all the package update commits looks fine to me. All that I needed was the s6-linux-init pacakge :P

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants