Fix FreeBSD support #81459

wahjava · 2020-03-01T20:03:50Z

Motivation for this change

I'm trying to make nixpkgs work on FreeBSD, similar to nix-darwin. For that, I attempted to fix FreeBSD stdenv. I'm not super familiar with stdenv, or internals of nixpkgs, so I would appreciate review.

Following is the output from FreeBSD 12.1 amd64 trying to build curl:

https://pastebin.com/0b7cJeHF

For those interested in trying out nix on FreeBSD, there is a FreeBSD port in progress, https://github.com/0mp/freebsd-ports-nix which is how I got it running.

Thanks in advance.

Things done

Able to bootstrap stdenv on FreeBSD by compiling various dependencies using built-in FreeBSD tools so mostly works now

nixos-discourse · 2020-03-06T17:23:31Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/error-installing-python-3-7-nixpkgs-on-freebsd/6136/1

Mic92 · 2020-03-06T17:29:48Z

Hi. So all what I need to test this, is the nix port you referenced?

Mic92 · 2020-03-06T17:33:42Z

@yater do you still have a machine for me to test?

Ericson2314 · 2020-03-06T18:07:17Z

This is cool! I'd like to get to a pure bootstrap from binary bootstrap tools, but we can start here.

Once something is up and running on FreeBSD, it is easier to improve it.

Mic92 · 2020-03-07T09:36:56Z

Could somebody help me out how to boot freebsd in qemu with functional network?

I tried:

$ wget http://ftp.freebsd.org/pub/FreeBSD/snapshots/VM-IMAGES/13.0-CURRENT/amd64/Latest/FreeBSD-13.0-CURRENT-amd64.qcow2.xz
$ xz -d FreeBSD-13.0-CURRENT-amd64.qcow2.xz
$ qemu-system-x86_64 -m 20G -hda FreeBSD-13.0-CURRENT-amd64.qcow2 -smp $(nproc) -enable-kvm -netdev user,id=mynet0,hostfwd=tcp:127.0.0.1:7722-:22 -cpu host -display curses -net nic,netdev=user.0,model=virtio -netdev user,id=user.0 -device virtio-rng-pci

I see an ip address in freebsd configured. However I cannot reach the VM.
This command is more or less what we use in NixOS to get userspace networking with qemu.

melchips · 2020-03-30T12:53:05Z

I see an ip address in freebsd configured. However I cannot reach the VM.
This command is more or less what we use in NixOS to get userspace networking with qemu.

Hello,

Maybe it's not related at all, but I noticed that my windows 10 VM, using virtio for network is not working anymore since my upgrade to 20.03 channel.

As a matter of fact, packets emitted from the host to the VM are correctly received on the VM, but packets from the VM to the host are not received and don't appear with tcpdump on the host.

By switching the virtual network card model to e1000e, networking is working properly again.

Could-you try setting the model to e1000e, maybe "virtio" is not working for you also, or there is a problem between the virtio modules on the VM and the host.

If no model is working for you, your network configuration may lack something...

Regards.

Mic92 · 2020-03-30T13:36:52Z

I just got it working using the following setup virtualisation.libvirtd.enable = true; in /etc/nixos/configuration.nix and by using the following qemu configuration:

#!/usr/bin/env bash

sudo ip tuntap add dev tap0 mode tap
sudo ip link set tap0 up promisc on
sudo ip link set dev virbr0 up
sudo ip link set dev tap0 master virbr0

sudo chmod 666 /dev/net/tun /dev/kvm
qemu-system-x86_64 \
  -m 20G -hda FreeBSD-13.0-CURRENT-amd64.qcow2 -smp $(nproc) -enable-kvm \
  -cpu host -display curses \
  -device virtio-rng-pci \
  -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
  -device virtio-net,id=net0,netdev=net0 \

The FreeBSD Image I got from https://download.freebsd.org/ftp/snapshots/VM-IMAGES/13.0-CURRENT/amd64/Latest/

Mic92 · 2020-03-30T15:00:44Z

Also got the nix port installed now.

Mic92 · 2020-03-30T15:19:07Z

I am currently on FreeBSD freebsd 13.0-CURRENT.
When I run nix-build it fails however:

$ truss -o /tmp/trace -f nix-build -A curl
these derivations will be built:
  /nix/store/24jlg3hb7mljwwy2i6qhx4br0f5v69by-trivial-bootstrap-patch-2.7.6.drv
  /nix/store/dbws3x1b9b4405mlxiy3zdnkzy04hv2g-cpio-2.13.tar.gz.drv
  /nix/store/50w7lcya3rbxi10kw43vakv8asyd337d-trivial-bootstrap-cpio-2.13.drv
  /nix/store/5igv461gh2f5w6as8mrj5mmjxhjmhfjx-trivial-bootstrap-sed-4.8.drv
  /nix/store/99cyj5fg0v403rfaqsjwrmyar6ghhww3-trivial-bootstrap-coreutils-8.31.drv
  /nix/store/bfsp7i0sb50bxmlbkwlz1lwfsywpclfs-trivial-bootstrap-grep-3.4.drv
  /nix/store/c59ip2y9drqqnwi5d410s9r97zkr8chc-trivial-bootstrap-diffutils-3.7.drv
  /nix/store/1pw4s8mx8v6dq7ak51bf2q685lmn70rg-make-4.3.tar.gz.drv
  /nix/store/f36ahifla35vxxh3lsd9s9fkpqpydp75-trivial-bootstrap-make-4.3.drv
  /nix/store/svr2qidcwb01j5jmbhwibcxga0vpzhzj-trivial-bootstrap-gawk-5.0.1.drv
  /nix/store/cfnx34rxcjidnhvzxj28gkpdj7i29zvk-curl-7.68.0.tar.xz.drv
  /nix/store/zl7nrjvp4g3j0mvr0s50yh98j2rkvgby-cacert-2020-01-01.pem.drv
  /nix/store/vlcy5py3mgxqld7mk9dafbq5kf9fymms-trivial-bootstrap-curl-7.68.0.drv
  /nix/store/ykjb58605vyb41pb4r6pqiddyqyxsxcm-trivial-bootstrap-findutils-4.7.0.drv
  /nix/store/2bhfsbvz68jq4pa4qan5g5wbckw1c46r-bash-4.4.18.tar.gz.drv
  /nix/store/zcky3nmy59vbi8bnafk0wwpj08yanylv-trivial-bootstrap-bash-4.4.18.drv
  /nix/store/ywswy4qh8z9271wfv3iqc7a77qcwxlfa-trivial-bootstrap-tools.drv
  /nix/store/pjsvbkmdkg8k4d8xlm35q9mpwiq9qv28-stdenv-freebsd-boot-0.drv
  /nix/store/f0qfc73qrb9vqhsymwg7sk52h1qac36s-stdenv-freebsd-boot-wrapper-3-binutils.drv
  /nix/store/arkv1zm8hl5ndqnfq788f0s86kibjv98-clang-wrapper-9.9.9.drv
  /nix/store/10yn9lhj0wirin3krypqvg6ycj9qknxp-stdenv-freebsd-boot-3.drv
  /nix/store/5456x32bf21r8w4c61h1hnv1ln2izrm8-c-ares-1.15.0.drv
  /nix/store/zsfnn1qddl0xkxd7i5kppy0f0a5lgq1g-gnum4-1.4.18.drv
  /nix/store/6ycz8skwdchfdybp8hxp8nbr2wv7h08w-gmp-6.2.0.drv
  /nix/store/jawpnv1g7z6bhkcgj5ilbnwd31xjdnzc-xz-5.2.4.drv
  /nix/store/s1g06qrlxz7cd90zyjy8yg6zsfli41ki-perl-5.30.1.drv
  /nix/store/8c4b3yql4np7kb63dz5l137rmx6230dd-coreutils-8.31.drv
  /nix/store/6grjg8f2jc6jbc6kds32dp06slip8ay3-openssl-1.1.1d.drv
  /nix/store/a2apysa916fdpl76ygfky5r64kwqg72v-libev-4.31.drv
  /nix/store/im297rwn83cylqdk8kk7px6kx9cay4h9-zlib-1.2.11.drv
  /nix/store/qjdhia4nd0spicknf1d46vb1mlkp81ls-pkg-config-0.29.2.drv
  /nix/store/0anp0fiv95cggm639f1g5c371kqnx249-nghttp2-1.40.0.drv
  /nix/store/2wvlili3ci9sk97is0m4xzin5r36759i-libkrb5-1.18.drv
  /nix/store/h3c1yl7ylmg29pd3rlx7gh4qwr52jwrr-libssh2-1.9.0.drv
  /nix/store/0sanxss5q7xnvad58qj8lycc873apmb3-curl-7.68.0.drv
error: cannot kill processes for uid '1001': Operation not permitted
error: cannot kill processes for uid '1001': failed with exit code 1

trace.txt

I will have a deeper look later.

wahjava · 2020-03-30T15:35:19Z

I am currently on FreeBSD freebsd 13.0-CURRENT.
When I run nix-build it fails however:
$ truss -o /tmp/trace -f nix-build -A curl
trace.txt

I will have a deeper look later.

Seems like following is the culprit:

46107: setuid(0x3e9)				 = 0 (0x0)
46107: kill(-1,SIGKILL)				 ERR#1 'Operation not permitted'

Did you try building it in multi-user/daemon mode ? or single-user ?

I guess you are not able to build stdenv either ? Also could you share the build output (stdout/stderr, or from /nix/var/log/) ?

Kleidukos · 2020-04-11T13:27:28Z

Hi! Are there any news on that front? :)

Mic92 · 2020-04-11T21:53:30Z

I am currently on FreeBSD freebsd 13.0-CURRENT.
When I run nix-build it fails however:
$ truss -o /tmp/trace -f nix-build -A curl
trace.txt
I will have a deeper look later.
Seems like following is the culprit:
46107: setuid(0x3e9)				 = 0 (0x0)
46107: kill(-1,SIGKILL)				 ERR#1 'Operation not permitted'
Did you try building it in multi-user/daemon mode ? or single-user ?

I guess you are not able to build stdenv either ? Also could you share the build output (stdout/stderr, or from /nix/var/log/) ?

It fails both in single-user mode and with nix-daemon:

root@freebsd:~ # nix-daemon
accepted connection from pid <unknown>, user root (trusted)
error: cannot kill processes for uid '1001': Operation not permitted
accepted connection from pid <unknown>, user root (trusted)
error: cannot kill processes for uid '1001': Operation not permitted

wahjava · 2020-04-12T06:29:28Z

I am currently on FreeBSD freebsd 13.0-CURRENT.
When I run nix-build it fails however:
$ truss -o /tmp/trace -f nix-build -A curl
trace.txt
I will have a deeper look later.
Seems like following is the culprit:
46107: setuid(0x3e9)				 = 0 (0x0)
46107: kill(-1,SIGKILL)				 ERR#1 'Operation not permitted'
Did you try building it in multi-user/daemon mode ? or single-user ?
I guess you are not able to build stdenv either ? Also could you share the build output (stdout/stderr, or from /nix/var/log/) ?
It fails both in single-user mode and with nix-daemon:
root@freebsd:~ # nix-daemon
accepted connection from pid <unknown>, user root (trusted)
error: cannot kill processes for uid '1001': Operation not permitted
accepted connection from pid <unknown>, user root (trusted)
error: cannot kill processes for uid '1001': Operation not permitted

This is interesting. I'm testing it in 12.1-RELEASE and had no such issue. I wonder if you have some fancy security setting enabled in 13-CURRENT. Could you maybe try it with 12.1-RELEASE ?

Mic92 · 2020-04-12T11:02:12Z

I am currently on FreeBSD freebsd 13.0-CURRENT.
When I run nix-build it fails however:
$ truss -o /tmp/trace -f nix-build -A curl
trace.txt
I will have a deeper look later.
Seems like following is the culprit:
46107: setuid(0x3e9)				 = 0 (0x0)
46107: kill(-1,SIGKILL)				 ERR#1 'Operation not permitted'
Did you try building it in multi-user/daemon mode ? or single-user ?
I guess you are not able to build stdenv either ? Also could you share the build output (stdout/stderr, or from /nix/var/log/) ?
It fails both in single-user mode and with nix-daemon:
root@freebsd:~ # nix-daemon
accepted connection from pid <unknown>, user root (trusted)
error: cannot kill processes for uid '1001': Operation not permitted
accepted connection from pid <unknown>, user root (trusted)
error: cannot kill processes for uid '1001': Operation not permitted
This is interesting. I'm testing it in 12.1-RELEASE and had no such issue. I wonder if you have some fancy security setting enabled in 13-CURRENT. Could you maybe try it with 12.1-RELEASE ?

I am using the official qemu image without any non-standard modifications.

Ericson2314 · 2020-04-20T05:33:58Z

@Kleidukos You may also be interested in #82131, which we can use to make a pure freebsd stdenv by cross-building the bootstrap tools.

There is plenty to be done for #82131, and no need to set up and qemu or other things to start (let's get things building and then worry about whether they actually run!) so I hope it is relatively easy to contribute to---I certainly can use all the help I can get!

Mic92 · 2020-10-02T09:21:33Z

Now that nix is in ports, can someone else have a look at this again?

AndersonTorres · 2020-11-19T12:58:20Z

I would like to help... what it needs to be done?

deliciouslytyped · 2020-11-20T00:55:11Z

By the way, there is a #freebsd-nix channel on freenode.

Mic92 · 2020-11-25T14:22:56Z

I would like to help... what it needs to be done?

Some maintainer should review/test the patch proposed here.

I tested a trivial program that called kill(-1, SIGKILL), which was run as the only process for an unpriveleged user, on Linux and FreeBSD. On Linux, kill reported success, while on FreeBSD it failed with EPERM. POSIX says: > If pid is -1, sig shall be sent to all processes (excluding an > unspecified set of system processes) for which the process has > permission to send that signal. and > The kill() function is successful if the process has permission to > send sig to any of the processes specified by pid. If kill() fails, > no signal shall be sent. and > [EPERM] > The process does not have permission to send the signal to any > receiving process. My reading of this is that kill(-1, ...) may fail with EPERM when there are no other processes to kill (since the current process is ignored). Since kill(-1, ...) only attempts to kill processes the user has permission to kill, it can't mean that we tried to do something we didn't have permission to kill, so it should be fine to interpret EPERM the same as success here for any POSIX-compliant system. This fixes an issue that Mic92 encountered[1] when he tried to review a Nixpkgs PR on FreeBSD. [1]: NixOS/nixpkgs#81459 (comment)

alyssais · 2021-02-07T14:10:48Z

Fix for @Mic92’s Nix issue: NixOS/nix#4530

alyssais

Tested nix-build -A hello && result/bin/hello; works fine.

Will need to be rebased and have some minor conflicts fixed.

alyssais · 2021-02-07T15:26:51Z

Rebased, fixed up, and pushed. (Couldn’t do that on the PR since the branch disappeared.)

Thanks @wahjava!

wahjava added 2 commits March 2, 2020 00:56

stdenv: Improve/fix FreeBSD support

b62e87a

Able to bootstrap stdenv on FreeBSD by compiling various dependencies using built-in FreeBSD tools so mostly works now

stdenv: Fix FreeBSD stdenv

1bb2836

wahjava requested a review from Ericson2314 as a code owner March 1, 2020 20:03

ofborg bot added 6.topic: stdenv Standard environment 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Mar 1, 2020

0mp mentioned this pull request Mar 7, 2020

Add NetBSD/FreeBSD Support NixOS/nix#3280

Closed

alyssais mentioned this pull request Feb 7, 2021

libutil: EPERM from kill(-1, ...) is fine NixOS/nix#4530

Merged

alyssais approved these changes Feb 7, 2021

View reviewed changes

alyssais closed this in 1c39662 Feb 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix FreeBSD support #81459

Fix FreeBSD support #81459

wahjava commented Mar 1, 2020

nixos-discourse commented Mar 6, 2020

Mic92 commented Mar 6, 2020

Mic92 commented Mar 6, 2020 •

edited

Ericson2314 commented Mar 6, 2020 •

edited

Mic92 commented Mar 7, 2020 •

edited

melchips commented Mar 30, 2020

Mic92 commented Mar 30, 2020

Mic92 commented Mar 30, 2020

Mic92 commented Mar 30, 2020 •

edited

wahjava commented Mar 30, 2020

Kleidukos commented Apr 11, 2020

Mic92 commented Apr 11, 2020

wahjava commented Apr 12, 2020

Mic92 commented Apr 12, 2020

Ericson2314 commented Apr 20, 2020 •

edited

Mic92 commented Oct 2, 2020

AndersonTorres commented Nov 19, 2020

deliciouslytyped commented Nov 20, 2020

Mic92 commented Nov 25, 2020

alyssais commented Feb 7, 2021

alyssais left a comment

alyssais commented Feb 7, 2021

Fix FreeBSD support #81459

Fix FreeBSD support #81459

Conversation

wahjava commented Mar 1, 2020

Motivation for this change

Things done

nixos-discourse commented Mar 6, 2020

Mic92 commented Mar 6, 2020

Mic92 commented Mar 6, 2020 • edited

Ericson2314 commented Mar 6, 2020 • edited

Mic92 commented Mar 7, 2020 • edited

melchips commented Mar 30, 2020

Mic92 commented Mar 30, 2020

Mic92 commented Mar 30, 2020

Mic92 commented Mar 30, 2020 • edited

wahjava commented Mar 30, 2020

Kleidukos commented Apr 11, 2020

Mic92 commented Apr 11, 2020

wahjava commented Apr 12, 2020

Mic92 commented Apr 12, 2020

Ericson2314 commented Apr 20, 2020 • edited

Mic92 commented Oct 2, 2020

AndersonTorres commented Nov 19, 2020

deliciouslytyped commented Nov 20, 2020

Mic92 commented Nov 25, 2020

alyssais commented Feb 7, 2021

alyssais left a comment

Choose a reason for hiding this comment

alyssais commented Feb 7, 2021

Mic92 commented Mar 6, 2020 •

edited

Ericson2314 commented Mar 6, 2020 •

edited

Mic92 commented Mar 7, 2020 •

edited

Mic92 commented Mar 30, 2020 •

edited

Ericson2314 commented Apr 20, 2020 •

edited