Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial implementation of remarkable1 cross-compile #102503

Merged
merged 1 commit into from Nov 24, 2020

Conversation

siraben
Copy link
Member

@siraben siraben commented Nov 2, 2020

Motivation for this change

Initial implementation of reMarkable 1 cross-compile. Doesn't rely on the company-provided toolchain, so if the user wants to run Nix-build binaries on the device they have two options:

  • compile to static binaries
  • install Nix on the device and use nix-copy-closure

Try out a build with

$ nix-build --argstr system 'x86_64-linux' -A pkgs.pkgsCross.remarkable1.hello
$ sha256sum ./result/bin/hello 
ab10b198df6f61efaa98ef1ced6ec94c651489871c556b365138b5a42891820d  ./result/bin/hello
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

Pinging some people who I've talked to on IRC/seen add infra in the commits. @cleverca22 @matthewbauer . Thanks!

lib/systems/parse.nix Outdated Show resolved Hide resolved
@Ericson2314
Copy link
Member

Ericson2314 commented Nov 2, 2020

What exactly is the reMarkable architecture? It is Linux and Arm, does it have a custom libc or something? I would rather GCC from source for this than bundle their pre-built stuff. I know we do just that for Android, but I wish we didn't there, too (#38411).

@siraben
Copy link
Member Author

siraben commented Nov 3, 2020

See https://github.com/toltec-dev/toolchain , which builds the toolchain from source instead of using the pre-built toolchain.

@siraben
Copy link
Member Author

siraben commented Nov 3, 2020

This is how the toolchain is built on Debian; https://github.com/toltec-dev/toolchain/blob/master/base/Dockerfile

@siraben
Copy link
Member Author

siraben commented Nov 3, 2020

@GrahamcOfBorg build pkgs.pkgsCross.remarkable1.hello

@siraben
Copy link
Member Author

siraben commented Nov 3, 2020

@GrahamcOfBorg build pkgsCross.remarkable1.hello

@pl-semiotics
Copy link

pl-semiotics commented Nov 4, 2020

Author of the overlay these were derived from here!

I did look into using an upstream GCC, and I remember running into some issues, but I don't recall precisely what they were, unfortunately. I needed ABI compatible with the software on the device, and using the upstream toolchain was the simplest way to get that. I think their libc is glibc-based---the device is pretty much a GNU userland running on an i.MX6 SoloLite, as far as I'm aware---but I didn't have success trying to build it from scratch.

I'm also happy to maintain this as an overlay (I'll also try to figure out how to add Flakes support) if you'd rather not see the upstream binary toolchain and related hackery added to nixpkgs directly---dealing with the ugliness of that is the main reason that I didn't try to upstream the configuration in the first place.

Ben, if using a built-from-source nixpkgs version of the standard gcc and glibc is working for you, I don't think you need the rmToolchain packages at all---the various wrappers around oecore that I'm setting up there are only really useful for satisfying the constraints that Nixpkgs expects out of a compiler/libc. If you want to remove them here, I'm happy to continue to maintain the upstream-binaries packages out of tree (hopefully with flakes to make using them easy, once I figure out flakes).

@siraben
Copy link
Member Author

siraben commented Nov 4, 2020

Hi @pl-semiotics! The toltec-dev toolchain seems to be able to replace the reMarkable toolchain with one using Crosstool-NG. And importantly the closed-source libqsgepaper Qt library as well. Do you think we could follow their example?

@pl-semiotics
Copy link

@siraben, unfortunately, I have not used their toolchain, so I can't say for sure---I wasn't aware of it (or, possibly, it didn't exist---I'm not sure) when I first set up the nix-remarkable overlay. I could try to take a look at it ~this weekend to audit it and see if it works for my current use cases (unfortunately, I don't believe I still have copies of the experiments that necessitated the switch to the binary toolchain in the first place). It does look promising, though!

If the open source toolchain here works well enough (hopefully!), we should remove the rmToolchain packages in their entirety, since they're just the equivalent of cc-wrapper for the binary toolchain.

@siraben
Copy link
Member Author

siraben commented Nov 10, 2020

I've been recently testing the from-source toolchain and it produces dynamic executables that don't run on the tablet. the toolchain from the company also produces dynamic executables but links and runs properly. @Ericson2314 What do you suggest?

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/remarkable1-cross-compile-toolchain-for-nix/9961/1

@Ericson2314
Copy link
Member

@siraben OK so first a few things:

  1. If the PR isn't ready pleas mark it WIP, so I know what I'm reviewing.

  2. "produces dynamic executables that don't run on the tablet." Thank you for taking the time to do the build, but note that this isn't enough information for anyone to actually debug the issue.

I would like to get to the bottom of this, as having from-source builds is much more maintainable. Perhaps you can paste some errors you get? Or, if there error is very vague, try comparing the readelf output of the final binary or something?

@siraben
Copy link
Member Author

siraben commented Nov 11, 2020

If the PR isn't ready pleas mark it WIP, so I know what I'm reviewing.

Ok, when would the PR be considered ready? It currently cross-compiles many packages.

I would like to get to the bottom of this, as having from-source builds is much more maintainable. Perhaps you can paste some errors you get? Or, if there error is very vague, try comparing the readelf output of the final binary or something?

I have some executables I cross-compiled from various setups; hello was cross-compiled using the cross toolchain based on the company's GCC and linker, hello_new was compiled using the from-source version of the cross toolchain, and hello_new_static was cross-compiled statically via nix-build . -A pkgs.pkgsCross.remarkable1.pkgsStatic.hello. The only binaries that run correctly on the tablet are hello and hello_new_static

root@reMarkable:~# file hello_new
hello_new: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /nix/store/3s2x09bbl1xss1lcx7a6dbw69zmip3gx-glibc-2.32-armv7l-unknown-linux-gnueabihf/lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, not stripped
root@reMarkable:~# ./hello_new
-sh: ./hello_new: No such file or directory
root@reMarkable:~# file hello
.hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=c213556b777ae8e1fb8f644a343a65cc0a83b2e7, not stripped
root@reMarkable:~# file hello_new_static
hello_new_static: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped

@pl-semiotics
Copy link

root@reMarkable:~# file hello_new
hello_new: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /nix/store/3s2x09bbl1xss1lcx7a6dbw69zmip3gx-glibc-2.32-armv7l-unknown-linux-gnueabihf/lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, not stripped

That would be your first problem. You must override the dynamic linker used by the CC wrapper to match the one present on the device. For comparison, this is how I do it with the proprietary toolchain: https://github.com/pl-semiotics/nix-remarkable/blob/master/rM/toolchain-pkgs.nix#L38 . I think since this is intended for inclusion in Nixpkgs the most reasonable thing to do would likely be to extend this if tree: https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/bintools-wrapper/default.nix#L52 .

@Ericson2314
Copy link
Member

Ok, when would the PR be considered ready? It currently cross-compiles many packages.

Sorry! I mistakenly thought you had begun pushing the source version.

@Ericson2314
Copy link
Member

@pl-semiotics Thanks! I think that will work just fine.

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/help-wanted-remarkable1-cross-compile-toolchain-for-nix/9961/7

@siraben
Copy link
Member Author

siraben commented Nov 17, 2020

Per discussion on Discourse, there seems to be an issue with setting the appropriate dynamic linker. @Ericson2314 is there some way we could do the following two commands to set the dynamic linker?

# copied from the default wrapper script since there's no way to
# set this :-(
echo "/lib/ld-linux-armhf.so.3" >$out/nix-support/dynamic-linker
echo "-dynamic-linker /lib/ld-linux-armhf.so.3" > $out/nix-support/libc-ldflags-before

@Ericson2314
Copy link
Member

@siraben I'm pretty sure all you need is the zero-gravitas definition and the remarkable1 example that uses it.

@siraben
Copy link
Member Author

siraben commented Nov 18, 2020

@Ericson2314 Ok. I've followed your suggestions regarding the custom toolchain/depending on on-device dynamic linker.

Build from scratch, statically linked, should be able to drop in
Build from scratch, dynamically linked, can use patchelf to replace libraries with pre-installed versions if desired.

Could you clarify how patchelf could be used to achieve this?

@Ericson2314
Copy link
Member

Could you clarify how patchelf could be used to achieve this?

patchelf has a --set-interpreter flag. --remove-rpath or --set-rpath is also useful the packages are being mashed together.

@siraben
Copy link
Member Author

siraben commented Nov 18, 2020

Using patchelf worked. I might need to note this somewhere in the manual or wiki so users would know to do this.

@siraben
Copy link
Member Author

siraben commented Nov 24, 2020

I got Nix installed on my tablet and did various cross-builds and used nix-copy-closure to send them over, and they all work, these packages include: hello, tmux, chicken, openssh, gcc.

@siraben siraben force-pushed the remarkable-init branch 2 times, most recently from 43f062d to 77e0015 Compare November 24, 2020 15:00
@Ericson2314 Ericson2314 merged commit d19b920 into NixOS:master Nov 24, 2020
@Ericson2314
Copy link
Member

Ofborg error was unrelated, guess something bad got merged at some point.

@siraben siraben deleted the remarkable-init branch November 24, 2020 15:07
@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/help-wanted-remarkable1-cross-compile-toolchain-for-nix/9961/16

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants