Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos/lib/make-ext4-fs: use mkfs.ext4 instead of cptofs #82718

Merged
merged 1 commit into from Jul 2, 2020

Conversation

misuzu
Copy link
Contributor

@misuzu misuzu commented Mar 16, 2020

Motivation for this change

When building big sd-image on armv7l cptofs is failing with this error:

can't open fsimg /nix/store/aal5c4ia7x587kmy0x9swna8bhqd8g92-ext4-fs.img: Value too large for defined data type
21119 Segmentation fault      cptofs -t ext4 -i $out nix-path-registration /

Fix this issue by dropping cptofs from script and using -d argument of mkfs.ext4 instead as lkl is not maintained and takes a very long time to build.

Also make image creation more reproducible.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

cc @samueldr @grahamc

@misuzu
Copy link
Contributor Author

misuzu commented May 4, 2020

This should also fix building on x86_64 using boot.binfmt.emulatedSystems = [ "aarch64-linux" ];:

nix-build nixos -I nixos-config=nixos/modules/installer/cd-dvd/sd-image-aarch64.nix -A config.system.build.sdImage --argstr system aarch64-linux

Before:

building '/nix/store/0ls2xbc6na607bh6lhdnq7zx6j2g95y2-ext4-fs.img.zst.drv'...
Creating an EXT4 image of 2773430272 bytes (numInodes=94009, numDataBlocks=489089)
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
mke2fs 1.45.5 (07-Jan-2020)
Unsupported ioctl: cmd=0xffffffff80200204
Discarding device blocks: done
Creating filesystem with 677107 4k blocks and 169344 inodes
Filesystem UUID: 44444444-4444-4444-8888-888888888888
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

error while reading directory .: Cannot allocate memory
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
error while reading directory ./nix/store: Cannot allocate memory
error while reading directory ./nix: Cannot allocate memory
error while reading directory .: Cannot allocate memory
copying store paths to image...
error while reading directory /nix/store/03nsv3a6a81ms8dsak74gb2jaqrn5llf-bind-9.14.11-lib/lib/named: Cannot allocate memory
error while reading directory /nix/store/03nsv3a6a81ms8dsak74gb2jaqrn5llf-bind-9.14.11-lib/lib: Cannot allocate memory
error while reading directory /nix/store/03nsv3a6a81ms8dsak74gb2jaqrn5llf-bind-9.14.11-lib: Cannot allocate memory
error while reading directory /nix/store: Cannot allocate memory
....
etc, etc

After:

building '/nix/store/0jlr2r3cxjcixp22ii3lgmywzwlc25pb-ext4-fs.img.zst.drv'...
copying store paths for image...
copying files for image...
Creating an EXT4 image of 2681835520 bytes (numInodes=94013, numDataBlocks=466719)
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object 'libfakeroot.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
mke2fs 1.45.5 (07-Jan-2020)
Unsupported ioctl: cmd=0xffffffff80200204
Discarding device blocks: done
Creating filesystem with 654745 4k blocks and 163840 inodes
Filesystem UUID: 44444444-4444-4444-8888-888888888888
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Copying files into the device: done
Writing superblocks and filesystem accounting information: done

e2fsck 1.45.5 (07-Jan-2020)
Unsupported ioctl: cmd=0xffffffff80200204
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
NIXOS_SD: 93759/163840 files (0.2% non-contiguous), 526170/654745 blocks
Resizing to minimum allowed size
resize2fs 1.45.5 (07-Jan-2020)
Unsupported ioctl: cmd=0xffffffff80200204
Resizing the filesystem on temp.img to 630560 (4k) blocks.
The filesystem on temp.img is now 630560 (4k) blocks long.

e2fsck 1.45.5 (07-Jan-2020)
Unsupported ioctl: cmd=0xffffffff80200204
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
NIXOS_SD: 93759/163840 files (0.2% non-contiguous), 526170/630560 blocks
Compressing image
*** zstd command line interface 64-bits v1.4.4, by Yann Collet ***
./temp.img           : 23.13%   (2582773760 => 597319969 bytes, /nix/store/9hf8pz2k4r3b6flh4k6jfsn4q5w41km0-ext4-fs.img.zst)
building '/nix/store/rpx12maw8rqawm1847dja791yfzv1m65-nixos-sd-image-20.09pre-git-aarch64-linux.img.drv'...
Decompressing rootfs image
/nix/store/9hf8pz2k4r3b6flh4k6jfsn4q5w41km0-ext4-fs.img.zst: 2582773760 bytes
Checking that no-one is using this disk right now ... OK

Disk /nix/store/nks2rj2zigpbblr0ar1gwa4zpaw4093m-nixos-sd-image-20.09pre-git-aarch64-linux.img/sd-image/nixos-sd-image-20.09pre-git-aarch64-linux.img: 2.45 GiB, 2622619648 bytes, 5122304 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

>>> Script header accepted.
>>> Script header accepted.
>>> Created a new DOS disklabel with disk identifier 0x2178694e.
/nix/store/nks2rj2zigpbblr0ar1gwa4zpaw4093m-nixos-sd-image-20.09pre-git-aarch64-linux.img/sd-image/nixos-sd-image-20.09pre-git-aarch64-linux.img1: Created a new partition 1 of type 'W95 FAT32' and of size 30 MiB
.
/nix/store/nks2rj2zigpbblr0ar1gwa4zpaw4093m-nixos-sd-image-20.09pre-git-aarch64-linux.img/sd-image/nixos-sd-image-20.09pre-git-aarch64-linux.img2: Created a new partition 2 of type 'Linux' and of size 2.4 GiB.
/nix/store/nks2rj2zigpbblr0ar1gwa4zpaw4093m-nixos-sd-image-20.09pre-git-aarch64-linux.img/sd-image/nixos-sd-image-20.09pre-git-aarch64-linux.img3: Done.

New situation:
Disklabel type: dos
Disk identifier: 0x2178694e

Device                                                                                                                                            Boot Start     End Sectors  Size Id Type
/nix/store/nks2rj2zigpbblr0ar1gwa4zpaw4093m-nixos-sd-image-20.09pre-git-aarch64-linux.img/sd-image/nixos-sd-image-20.09pre-git-aarch64-linux.img1      16384   77823   61440   30M  b W95
/nix/store/nks2rj2zigpbblr0ar1gwa4zpaw4093m-nixos-sd-image-20.09pre-git-aarch64-linux.img/sd-image/nixos-sd-image-20.09pre-git-aarch64-linux.img2 *    77824 5122303 5044480  2.4G 83 Linu

The partition table has been altered.
Syncing disks.
5044480+0 records in
5044480+0 records out
2582773760 bytes (2.6 GB, 2.4 GiB) copied, 25.4228 s, 102 MB/s
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
mkfs.fat 4.1 (2017-01-24)
Copying bootcode.bin
Copying config.txt
Copying fixup.dat
Copying fixup4.dat
Copying fixup4cd.dat
Copying fixup4db.dat
Copying fixup4x.dat
Copying fixup_cd.dat
Copying fixup_db.dat
Copying fixup_x.dat
Copying start.elf
Copying start4.elf
Copying start4cd.elf
Copying start4db.elf
Copying start4x.elf
Copying start_cd.elf
Copying start_db.elf
Copying start_x.elf
Copying u-boot-rpi3.bin
fsck.fat 4.1 (2017-01-24)
Checking we can access the last sector of the filesystem
Boot sector contents:
System ID "mkfs.fat"
Media byte 0xf8 (hard disk)
       512 bytes per logical sector
      2048 bytes per cluster
         4 reserved sectors
First FAT starts at byte 2048 (sector 4)
         2 FATs, 16 bit entries
     30720 bytes per FAT (= 60 sectors)
Root directory starts at byte 63488 (sector 124)
       512 root directory entries
Data area starts at byte 79872 (sector 156)
     15321 data clusters (31377408 bytes)
32 sectors/track, 64 heads
         0 hidden sectors
     61440 sectors total
Checking for unused clusters.
firmware_part.img: 20 files, 12115/15321 clusters
61440+0 records in
61440+0 records out
31457280 bytes (31 MB, 30 MiB) copied, 0.306261 s, 103 MB/s
/nix/store/nks2rj2zigpbblr0ar1gwa4zpaw4093m-nixos-sd-image-20.09pre-git-aarch64-linux.img

@misuzu misuzu requested a review from lovesegfault May 4, 2020 13:54
@Renha Renha mentioned this pull request May 4, 2020
Copy link
Member

@lovesegfault lovesegfault left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome :)

@misuzu misuzu changed the title ext4-fs: fix image creation on armv7l when image is bigger than 2G nixos/lib/make-ext4-fs: use mkfs.ext4 instead of cptofs May 4, 2020
@mebubo
Copy link
Contributor

mebubo commented May 21, 2020

I have successfully used this in conjunction with #86291 to cross-build an aarch64 image for a raspberry 3 via https://github.com/nix-community/nixos-generators (+ enabling binfmt.emulatedSystems = [ "aarch64-linux" ]; on the host x86_64 system):

nixos-generate --format sd-aarch64 --system aarch64-linux -I nixpkgs=[path-to-nixpkgs] --configuration configuration.nix

Can confirm that this change gets rid of the error while reading directory ... Cannot allocate memory errors.

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-in-distress/3604/26

Copy link
Contributor

@jonringer jonringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

diff LGTM

would like to see what @samueldr has to say, especially with this work making mobile NixOS.

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/cant-boot-into-new-configuration-after-nix-rebuild-raspberry-pi3-b/7812/1

nixos/lib/make-ext4-fs.nix Outdated Show resolved Hide resolved
@c00w
Copy link
Contributor

c00w commented Jun 30, 2020

I poked @samueldr on IRC about reviewing this, and they said they would take a look

Copy link
Member

@samueldr samueldr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor change and imo this can be merged to unstable then.

nixos/lib/make-ext4-fs.nix Outdated Show resolved Hide resolved
This fixes image creation on armv7l when image is bigger than 2G.
Also fix some reproducibility issues and other cptofs issues.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants