Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Various: Add support for raspberry pi 4. #68265

Merged
merged 1 commit into from Sep 27, 2019
Merged

Conversation

tkerber
Copy link
Member

@tkerber tkerber commented Sep 7, 2019

Motivation for this change

Run NixOS on the Raspberry Pi 4B #63720. I'm reopening as the WiFi / 4G issues are resolved in the rpi kernel branch now.

This pull request has breaking changes that need feedback!

Changes Mage:

  • Bumped WiFi and boot firmware for the raspberry pi.
  • Updated the default raspberry pi bootloader to work with the rpi4.
  • Bumped the kernel. This encountered an interesting issue: The raspberry pi currently has a single kernel image: linux_rpi, which uses different defconfig settings depending on architecture. The rpi4 has the same architecture as the rpi3, but should use a different defconfig setting. I cannot see how to modify the current kernel package backwards-compatibly to do this. As a result, I've split it into several. Feedback on this would be good. It will break any existing builds using this kernel.

Outstanding issues:

  • Uboot support is not available upstream, as far as I can tell.
  • The raspberry pi sdcard build is broken, both due to the lack of uboot, and because the issue of selecting the correct kernel.
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Notify maintainers

cc @lopsided98 @dezgeg @tavyc

@samueldr samueldr self-requested a review September 8, 2019 04:02
@samueldr
Copy link
Member

samueldr commented Sep 8, 2019

Uboot support is not available upstream, as far as I can tell.

The last time I checked (a couple weeks ago) u-boot were waiting on the mainline kernel to produce a device tree file, so that u-boot could sync theirs from the mainline kernel. I'm not sure this is the latest patch set. AFAIK it means it's mostly a question of time, and that mainline and u-boot support should coincide nicely.

The raspberry pi sdcard build is broken, both due to the lack of uboot, and because the issue of selecting the correct kernel.

This is a hard situation. The sd_image is not a raspberry pi image. It is a generic AArch64 image, which for convenience has the raspberry pi firmware pre-baked into. It should and always will use the mainline kernel. This means that for as long as the raspberry pi 4 is not supported in the aarch64 defconfig, it will not work with the Rapsberry Pi 4. Though, we should be able to add the right firmware files so that its firmware partition works on both the 3 family and 4 when that happens in u-boot.

Though with that said, we shouldn't make it harder than necessary to build an image for the Raspberry Pi 4. What I was considering, in the past few weeks, is to have a raspberry-pi specific image builder, that doesn't rely on u-boot, but rather on the specific raspberry pi boot chain. The main drawback would be the loss of generation selection at boot, but if we're careful, it may produce images that are easier to deal with the raspberry pi-isms compared to u-boot and its generic handling of things.

Thoughts?

@tkerber
Copy link
Member Author

tkerber commented Sep 8, 2019

That seems reasonable. I didn't look deep enough to realise the sd images were generic, I just did a grep for the firmware, and figured it'd be rpi specific (I also saw sd-image-raspberrypi.nix, but now I realise that is only for pi 0 and 1).

I agree that waiting for a mainline uboot probably makes sense, and an interim rpi4 specific sd card builder may make sense. I may look into how to do that next week.

@tkerber
Copy link
Member Author

tkerber commented Sep 12, 2019

I've added an sd image builder, that builds a sensible-looking image, which is yet untested (I'll check later today). nixos/release.nix has serveral image builders as attributes. I'm not sure what purpose these have, as I can't see them being used in any instructions to actually create the images. Is this for hydra? I've added the rpi4 image there too, but I don't know if this is ideal.

nixos/modules/installer/cd-dvd/sd-image-raspberrypi4.nix Outdated Show resolved Hide resolved
nixos/release.nix Outdated Show resolved Hide resolved
@samueldr
Copy link
Member

To better answer your question, <nixpkgs/nixos/release.nix> is indeed used by Hydra. It is part of what ends up being used to add jobs the to hydra. See the configuration, which uses release-combined.nix, which in turn imports release.nix.

Though this in itself doesn't add it to the tested jobset, which is what is used for channel advances.

@tkerber
Copy link
Member Author

tkerber commented Sep 12, 2019

The current sd image doesn't boot - I'll have to look into why not, at a glance, I can't tell what's different to my known working config.

@tkerber
Copy link
Member Author

tkerber commented Sep 13, 2019

Turns out I should have actually used the right kernel 🙄

The latest push builds an image with boots fine.

@samueldr
Copy link
Member

Yes, just verified, it boots fine.

Do you have X11 running? if so, what else did you need to do? I'm trying to use modesetting / vc4, but can't seem to have drm and vc4 working. I tried a couple things, searched around online about whether the raspberry pi foundation fork should work (it should), but haven't found clues yet. (Though, to be fair, I stopped mostly as I was frustrated by the slowness of the SD card.)

@tkerber
Copy link
Member Author

tkerber commented Sep 15, 2019

I think the only thing I did to get X11 booting was set services.xserver.videoDrivers = [ "modesetting" ];. I'll try to see if I can use the sdcard builder for a minimal graphical boot. Issues with the rpi foundation drivers are that a lot of their work is 32-bit only -- fortunately it does seem like the hardware has some level of upstream support. I got some level of openGLES by doing the following:

{ pkgs, ... }:

{
  hardware.opengl = {
    enable = true;
    setLdLibraryPath = true;
    package = pkgs.mesa_drivers;
  };
  boot.loader.raspberryPi.firmwareConfig = ''
    dtoverlay=vc4-fkms-v3d
    gpu_mem=192
  '';
}

@samueldr
Copy link
Member

samueldr commented Sep 15, 2019

I think the only thing I did to get X11 booting was set services.xserver.videoDrivers = [ "modesetting" ];.

:/ that's what I did, and X11 doesn't start, doesn't see any screen.

Issues with the rpi foundation drivers are that a lot of their work is 32-bit only

Yes, that's a known irritating thing from the ecosystem.

-- fortunately it does seem like the hardware has some level of upstream support. I got some level of openGLES by doing the following:

Here, does "upstream" mean the raspberry pi foundation, or the mainline kernel?


Tested by adding the config fragment from the last comment:

[   209.631] (II) The server relies on udev to provide the list of input devices.
        If no devices become available, reconfigure udev or disable AutoAddDevices.
[   209.631] (II) Loader magic: 0x5f5d10
[   209.631] (II) Module ABI versions:
[   209.631]    X.Org ANSI C Emulation: 0.4
[   209.631]    X.Org Video Driver: 24.0
[   209.631]    X.Org XInput driver : 24.1
[   209.631]    X.Org Server Extension : 10.0
[   209.632] (++) using VT number 7

[   209.633] (II) systemd-logind: logind integration requires -keeptty and -keeptty was not provided, disabling logind integration
[   209.633] (II) no primary bus or device found
[   209.634] (II) "glx" will be loaded by default.
[   209.634] (II) LoadModule: "glx"
[   209.634] (II) Loading /nix/store/7swc8v8vg1mg8agdci4ga94g3ljqjhrb-xorg-server-1.20.5/lib/xorg/modules/extensions/libglx.so
[   209.637] (II) Module glx: vendor="X.Org Foundation"
[   209.637]    compiled for 1.20.5, module version = 1.0.0
[   209.637]    ABI class: X.Org Server Extension, version 10.0
[   209.637] (II) LoadModule: "modesetting"
[   209.637] (II) Loading /nix/store/7swc8v8vg1mg8agdci4ga94g3ljqjhrb-xorg-server-1.20.5/lib/xorg/modules/drivers/modesetting_drv.so
[   209.638] (II) Module modesetting: vendor="X.Org Foundation"
[   209.638]    compiled for 1.20.5, module version = 1.20.5
[   209.638]    Module class: X.Org Video Driver
[   209.638]    ABI class: X.Org Video Driver, version 24.0
[   209.638] (II) modesetting: Driver for Modesetting Kernel Drivers: kms
[   209.638] (WW) Falling back to old probe method for modesetting
[   209.638] (EE) open /dev/dri/card0: No such file or directory
[   209.638] (WW) Falling back to old probe method for modesetting
[   209.638] (EE) open /dev/dri/card0: No such file or directory
[   209.638] (EE) No devices detected.
[   209.638] (EE)
Fatal server error:
[   209.638] (EE) no screens found(EE)
[   209.638] (EE)
Please consult the The X.Org Foundation support
         at http://wiki.x.org
 for help.
[   209.638] (EE) Please also check the log file at "/var/log/X.0.log" for additional information.
[   209.639] (EE)
[   209.644] (EE) Server terminated with error (1). Closing log file.

@tkerber
Copy link
Member Author

tkerber commented Sep 15, 2019

Hmm... I'll look into this. I dimly remember having a similar issue, and fixing it in my own config. I just have to figure out which part is the fix :p

Scratchpad for ideas on what could be going on:

  • Cross reference the overlays available on my working sdcard vs non-working. Possibly it's working due to manual artifacts flying around in the boot partition?
  • The vc4 overlay doesn't seem to get installed in the boot builder -- makes some sense, as it's not required for boot.
  • When I originally posted on this, I suggested using the video driver fbdev -- maybe this one works without vc4, and modesetting requires the latter to be functional?
  • pkgs/os-specific/linux/device-tree/raspberrypi.nix looks like it needs an update, which I missed. What is the purpose of this file?

Update 1:

  • I was testing gdm with fbdev. This doesn't work (permission error). slim and fbdev does!
  • My working sd card does have copies of overlay dtds flying about. Likely I copied the whole firmware over in the early stages when I was trying to get it booting at all.

@tkerber
Copy link
Member Author

tkerber commented Sep 17, 2019

Alright, I misunderstood how overlays are supposed to work in nixos, and the non-uboot builder wasn't set up to deal with them. I've pushed fixes to this, and have been able to get an sdcard booting into xorg both with and without gpu acceleration:

Without GPU

{ pkgs, ... }:

{
  imports = [ <nixpkgs/nixos/modules/installer/cd-dvd/sd-image-raspberrypi4.nix> ];

  networking.wireless.enable = false;
  services.xserver = {
    enable = true;
    displayManager.slim.enable = true;
    desktopManager.gnome3.enable = true;
    videoDrivers = [ "fbdev" ];
  };
}

With GPU

{ pkgs, ... }:

{
  imports = [ <nixpkgs/nixos/modules/installer/cd-dvd/sd-image-raspberrypi4.nix> ];

  networking.wireless.enable = false;
  hardware.opengl = {
    enable = true;
    setLdLibraryPath = true;
    package = pkgs.mesa_drivers;
  };
  hardware.deviceTree = {
    base = pkgs.device-tree_rpi;
    overlays = [ "${pkgs.device-tree_rpi.overlays}/vc4-fkms-v3d.dtbo" ];
  };
  services.xserver = {
    enable = true;
    displayManager.slim.enable = true;
    desktopManager.gnome3.enable = true;
    videoDrivers = [ "modesetting" ];
  };

  boot.loader.raspberryPi.firmwareConfig = ''
    gpu_mem=192
  '';
}

I don't believe either of these should be in this PR directly, although the information should definitely go in the wiki once this is merged.

@@ -11,7 +11,7 @@ stdenvNoCC.mkDerivation {

cp ${raspberrypifw}/share/raspberrypi/boot/bcm*.dtb .

cp bcm2708-rpi-0-w.dtb bcm2835-rpi-zero-w.dtb
cp bcm2708-rpi-zero-w.dtb bcm2835-rpi-zero-w.dtb
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's where the change comes from:

I don't know if we want this to handle both possibilities of an older downstream kernel with the older name for the dtb or not.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intesting. Here the dtb's are grabbed from raspberrypifw however, which just consists of binary blobs. Makes me question if that's the right approach, but for now, the firmware package has only one version in the repo, and it's quite unlikely anyone will override it with an older version, imo.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that was my thought too. I wasn't sure how likely a user would rely on an older kernel revision, while using a newer nixpkgs. Unlikely I guess, and at that point, they may as well patch this package in their overlay.

@willtim
Copy link
Contributor

willtim commented Sep 24, 2019

Great work! Is there anything stopping it getting merged?

Copy link
Member

@0x4A6F 0x4A6F left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed boot on raspberry pi 4. Thanks a lot.

@samueldr samueldr merged commit 95005c1 into NixOS:master Sep 27, 2019
@samueldr
Copy link
Member

Quick note, as a de-facto lead on ARM stuff.

  • The derivation is not expected to be kept and supported when the raspberry pi 4 is going to be supported by mainline kernel+u-boot.
  • Do not backport this to 19.09, and hopefully this can be removed before the 20.03 branch-off.

This is a convenience for the arguably most popular aarch64 platform, and we don't intend to add and keep board-specific builders in Nixpkgs. Generic and mainline support is preferred.

@RonnyPfannschmidt
Copy link
Contributor

thanks for this, just as i'm about to setup my pi4 this hit the repo, so i can go for a nix setup - made my day - you are awesome

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/boot-loader-raspberrypi-deprecated/30694/3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
NixOS on ARM
Awaiting triage
Development

Successfully merging this pull request may close these issues.

None yet

6 participants