Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't write asteroid-image-tetra.ext4 to userdata #31

Closed
Espionage724 opened this issue Jun 28, 2017 · 21 comments
Closed

Can't write asteroid-image-tetra.ext4 to userdata #31

Espionage724 opened this issue Jun 28, 2017 · 21 comments

Comments

@Espionage724
Copy link

Espionage724 commented Jun 28, 2017

This happens with the latest nightly. Not sure if it happens with the latest release

sudo fastboot flash userdata '/home/espionage724/asteroid-image-tetra.ext4' 
target reported max download size of 425721856 bytes
Invalid sparse file format at header magi
erasing 'userdata'...
OKAY [  9.291s]
sending sparse 'userdata' 1/1 (369345 KB)...
OKAY [ 16.117s]
writing 'userdata' 1/1...
FAILED (remote: unknown chunk type in sparse image)
finished. total time: 25.457s

The image located here flashes fine (but doesn't work; it reboot loops). Back when I was on Android Wear, I was able to temporary-boot the latest nightly no problem.

This is for the Sony Smartwatch 3.

@FlorentRevest
Copy link
Member

You may need to play around with the alignment of the partition AsteroidOS/meta-tetra-hybris@4e7def9 A value of 4 is needed for all qcom devices but this may be different on the SW3

@fergy
Copy link

fergy commented Jul 25, 2017

SW3 probably need kind of recapitulation of mkbootimg as from MM bootloader. Will be checked out within fay/two

@ghost
Copy link

ghost commented Sep 11, 2017

Theres a warning on the site saying that Real Installation doesn't work while Temp Install does, however temp install just causes the device to get stuck during boot and not load the OS.

@FlorentRevest
Copy link
Member

It looks like @fergy 's "tested" pull request broke tetra AsteroidOS/meta-tetra-hybris#2 could anyone issue a local build with the commits in this PR reverted and confirm whether or not this was the cause of the problems on tetra?

@fergy
Copy link

fergy commented Sep 12, 2017

It wasn't possible to flash before My latest commit but be free to check

@FlorentRevest
Copy link
Member

I was refering to the IaYScI3eB1weQ1uIxeeG's temp installation bug. Flashing is a different problem

@ghost
Copy link

ghost commented Sep 12, 2017

I'd test an older version but I don't know how to use the ". ./prepare-build.sh" command with older versions, anybody know how?

@FlorentRevest
Copy link
Member

IaYScI3eB1weQ1uIxeeG: issue this command once and then cd to src/meta-tetra-hybris and git checkout the commit you want.
However, this shouldn't be needed, nicofossa seems to have proven that the PR was faulty here AsteroidOS/meta-tetra-hybris#2 (comment) and I reverted the commits from the PR. Use . ./prepare-build.sh update before the rest of the usual instructions or wait for the next nightly build.

But again, this has nothing to do with the initial problem here, which is related to the userdata partition size. This is a different problem. I suggest someone should try to define a new OE "partition type" using the usual ext4 partition but then using the img2simg tool.

@ghost
Copy link

ghost commented Sep 21, 2017

Real install still fails on newest nightly. Temp install however does work, that didn't work in the last nightly.

@fergy
Copy link

fergy commented Sep 21, 2017 via email

@tristan-k
Copy link

I'm having the same issue on a Asus Zenwatch 2 (sparrow, wren). The watch is stuck in CSC Fastboot Mode. Any suggestions?

@gmelchett
Copy link

Real install still fails with build from today.

@FlorentRevest
Copy link
Member

Hi, an idea crosses my mind. Could you try to run img2simg on the downloaded image and see if the result can be flashed?

If that fixes the issue, we'd have to close AsteroidOS/meta-asteroid#16 and compile simgs. However, those "simg" files can't be used for the temporary install method... So we would probably need to distribute two different images with two different instructions for tetra. Which is a pain

@gmelchett
Copy link

Nope, it didn't help.

$ img2simg asteroid-image-tetra.ext4 asteroid-image-tetra-converted.ext4
$ file asteroid-image-tetra-converted.ext4
asteroid-image-tetra-converted.ext4: Android sparse image, version: 1.0, Total of 136946 4096-byte output blocks in 86 input chunks.
$ file asteroid-image-tetra.ext4
asteroid-image-tetra.ext4: Linux rev 1.0 ext4 filesystem data, UUID=4fd00966-3de9-4046-9477-1584de6470e5 (extents) (large files) (huge files)
$ fastboot flash userdata asteroid-image-tetra-converted.ext4
target reported max download size of 425721856 bytes
erasing 'userdata'...
OKAY [  9.299s]
sending sparse 'userdata' 1/2 (392793 KB)...
OKAY [ 18.971s]
writing 'userdata' 1/2...
FAILED (remote: unknown chunk type in sparse image)
finished. total time: 28.307s

@FlorentRevest
Copy link
Member

Thank you gmelchett, I'll consider other solutions then.

@schoenbeck
Copy link

Same issue like gmelchett here with today's nightly build

@steckus
Copy link

steckus commented Nov 20, 2017

C:\Users\steck>fastboot flash userdata C:\asteroid-image-tetra.ext4
target reported max download size of 425721856 bytes
Invalid sparse file format at header magi
erasing 'userdata'...
OKAY [  9.236s]
sending sparse 'userdata' (395814 KB)...
OKAY [ 17.529s]
writing 'userdata'...
FAILED (remote: unknown chunk type in sparse image)
finished. total time: 26.796s

there are builds for SW3?
to see how the OS works?

@FlorentRevest
Copy link
Member

Updates regarding this issue as of today:

tl;dr:

  • bitbake android-tools-native
  • asteroid/build/tmp-glibc/sysroots-components/x86_64/android-tools-native/usr/bin/ext2simg asteroid/build/tmp-glibc/deploy/images/tetra/asteroid-image-tetra.ext4 asteroid/build/tmp-glibc/deploy/images/tetra/asteroid-image-tetra.ext4.sparse
  • fastboot flash userdata asteroid/build/tmp-glibc/deploy/images/tetra/asteroid-image-tetra.ext4.sparse
  • enjoy ! (... or report)

Details about the installation methods:

  • The "Temporary installation" method requires an ext4 raw image on every kind of watches because it is stored as a file and directly loop-mounted by the asteroid's ramdisk when necessary. OE already generates ext4 raw image so this method works fine, including on tetra.
  • The "Real installation" method requires flashing a partition using the bootloader mode (aka fastboot mode) of the watch:
    • The "fastboot flash partitionName fileToFlash" command first asks the watch for its "max download size".
    • If the fileToFlash is smaller than the max download size (which was the case at the time of AsteroidOS Alpha 1.0 or with the result of "dd if=/dev/zero of=fileToFlash bs=1024 count=415744")
      • The file is directly sent to the bootloader and it is flashed correctly.
    • If the fileToFlash is larger than the max download size (which is the case with recent nightly builds or with the result of "dd if=/dev/zero of=fileToFlash bs=1024 count=415748")

Details about the sparse format (reference: http://www.2net.co.uk/tutorial/android-sparse-image-format ):

  • sparse images (generated by fastboot or img2simg) are splitted into a couple of large pieces (smaller than the "reported max download size") which are themselves made of lots of "chunks".
  • each "chunk" has a specific "chunk type" which can be one of those 4 types: https://android.googlesource.com/platform/system/core/+/master/libsparse/sparse_format.h#41
  • Android provides a very handy simg_dump script which can be used with a -v verbose parameter to get the full details of every chunk types in use in a sparse image https://android.googlesource.com/platform/system/core/+/master/libsparse/simg_dump.py
  • an usual conpression of a reasonably entropic file (eg: a typical asteroid-image-tetra.ext4) makes use of two chunk types: "Raw" and "Fill". Raw chunks contain uncompressed data and Fill chunks contain patterns to be repeated a certain amount of times (eg: for large blocks of zeroes)

Details about the tetra's bootloader:

  • Most Android devices bootloaders just accept any of those chunk types, but given the observed error message: remote (tetra's bootloader) reports a chunk type as unknown.
  • Judging by the commit log of u-boot's aboot.c implementation (lots of broadcom.com email addresses) and by the (partition layout reported by fastboot get-vars all), it looks like tetra uses an old implementation of u-boot's aboot.
  • Flashing a minimal "Raw" sparse image (1 block of 4096 of random bytes obtained with "dd if=/dev/random of=test1 bs=1024 count=1 iflag=fullblock" compressed with "img2simg test1 test1.sparse") works well. This chunk type is well recognized by tetra's bootloader
  • Flashing a minimal "Fill" sparse image (1 block of 4096 bytes set to 0 obtained with "dd if=/dev/zero of=test2 bs=1024 count=1" compressed with "img2simg test2 test2.sparse") results in an "unknown chunk type in sparse image".

Proposed workaround:

  • In order to be able to flash asteroid's rootfs on the userdata partition, we need to avoid fastboot's own img2simg implementation (which generates Fill chunks) by using a custom img2simg conversion that only uses Raw chunks.
  • bitbaking the android-tools-native recipe generates a few commands in asteroid/build/tmp-glibc/sysroots-components/x86_64/android-tools-native/usr/bin One of those commands is named "ext2simg" and seems to produce sparse images from ext partitions using only Raw chunks. I was able to flash the produced sparse images on my tetra however I haven't done enough testing to be 100% sure it's the right solution, your feedback is welcome.
  • ext2simg can either be built by OE and used to output a new partition file format (we would have to add a new IMAGE_FORMAT depending on android-tools-native, distribute two partitions on https://release.asteroidos.org/nightlies/tetra/ and document the two methods because a raw ext4 image can only be used for the temporary installation method and a sparse image made of raw chunks can only be used for the real installation method)
  • otherwise, ext2simg could also be downloaded from some distributions' repositories (eg: https://packages.debian.org/jessie/arm64/android-tools-fsutils/filelist but https://bugzilla.redhat.com/show_bug.cgi?id=1276447 ) and only the a couple of lines of instructions would need to be modified on the tetra's installation page. However, on other distributions or OSes, flashing asteroid on tetra would still be problematic.

@tristan-k
Copy link

tristan-k commented Jan 3, 2018

Is this issue just solved for the Sony Smartwatch 3 (tetra) or for any other Smartwatch as well? Because I was having the same issue on a Asus Zenwatch 2 (sparrow, wren). Does the latest nightly build for wren incorporates this fixes?

@FlorentRevest
Copy link
Member

tristan-k: The instructions on the website have only been changed for the SW3.

No one else ever reported this problem on the Zenwatch 2. Your messages here don't give enough details to help you, we don't know if you have a wren or a sparrow (those are two different watches). You don't detail what commands you use and what they output. You haven't tested the solution I detailed in my above comment. And if the bug still happens after using ext2simg, you should open a different issue.

@9600
Copy link

9600 commented May 29, 2018

@FlorentRevest just used ext2simg from android-tools-native, as per proposed workaround, and can confirm that this worked fine for me with a Sony Smartwatch 3 and commit f20567e.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants