Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

squashfsTools: 4.4dev_20180612 -> 4.4 #68275

Merged
merged 4 commits into from Sep 20, 2019
Merged

squashfsTools: 4.4dev_20180612 -> 4.4 #68275

merged 4 commits into from Sep 20, 2019

Conversation

ruuda
Copy link
Contributor

@ruuda ruuda commented Sep 7, 2019

Motivation for this change

Squashfs-tools 4.4 has been released upstream. See also the release notes for 4.4.

Welcome to Squashfs 4.4. This is the first release in over 5 years, and
there are substantial improvements: reproducible builds, new compressors,
CVE fixes, security hardening and new options for Mksquashfs/Unsquashfs.

Particularly interesting for us, output is now reproducible, so we no longer need to keep the patches for that. I built a squashfs image twice on different machines with this version, and the sha256sums of the images matched.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date (I don’t think there is any that needs changing.)
  • Fits CONTRIBUTING.md.
Notify maintainers

I am the maintainer of this package.

  • @charles-dyfis-net, you added the 4k alignment patch in f15d9d5. I updated this patch to apply to 4.4, and I verified that mksquashfs still lists the option in -help, but I haven’t verified that it still works, is there anything you would like to test?
  • @matthewbauer, you added Darwin support in 6cd5863. I updated this patch to apply to 4.4, but I don’t use Darwin, so I could not test if it still works. Do you want to have a look at the changes?

A new release has been made upstream. Reproducibility issues were fixed
in that release, so we no longer need those patches. For a full overview
of the changes, see the 4.4-specific readme at [1].

The alignment patch no longer applies cleanly; I disabled it for now,
and I will try to restore it in a follow-up commit.

[1]: https://github.com/plougher/squashfs-tools/blob/52eb4c279cd283ed9802dd1ceb686560b22ffb67/README-4.4
I took the patch, and applied it on top of the previous squashfs-tools
commit that we packaged (which required editing one line in the patch,
as it assumed to be applied on top of the reproducibility patches). Then
I rebased that on top of master, resolved one conflict, and I formatted
a new patch for this.
I took the patch, and applied it on top of the previous squashfs-tools
commit that we packaged. It applied cleanly. Then I rebased that on top
of master, and resolved the conflicts. I'm not sure I resolved them
correctly though, I don't have access to Darwin. Somebody needs to
review this.
@risicle
Copy link
Contributor

risicle commented Sep 8, 2019

re darwin support: macos 10.13 is unhappy:

  configuring
  no configure script, doing nothing
  building
  build flags: SHELL=/nix/store/9jd5ihjm10rwm3m6aj1wwbyx7f3cc902-bash-4.4-p23/bin/bash XZ_SUPPORT=1 ZSTD_SUPPORT=1
  clang -O2  -I. -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DCOMP_DEFAULT=\"gzip\" -Wall -DGZIP_SUPPORT -DXZ_SUPPORT -DZSTD_SUPPORT -DXATTR_SUPPORT -DXATTR_DEFAULT -DREPRODUCIBLE_DEFAULT   -c -o mksquashfs.o mksquashfs.c
  mksquashfs.c:38:10: fatal error: 'sys/sysmacros.h' file not found
  #include <sys/sysmacros.h>
           ^~~~~~~~~~~~~~~~~
  1 error generated.
  make: *** [<builtin>: mksquashfs.o] Error 1

@risicle
Copy link
Contributor

risicle commented Sep 8, 2019

I suspect this may be relevant plougher/squashfs-tools#69

@charles-dyfis-net
Copy link
Contributor

I can confirm that the 4k-padding patch still works as amended.

@ruuda
Copy link
Contributor Author

ruuda commented Sep 9, 2019

re darwin support: macos 10.13 is unhappy

Thanks for checking! That include was indeed part of an conflict and I wasn’t sure on which side of the #ifdef it should go. Thanks for the pointer too, I will try and fix this.

I can confirm that the 4k-padding patch still works as amended.

Thanks for checking!

@lheckemann lheckemann added this to the 19.09 milestone Sep 10, 2019
This new patch is the patch between 4.4, and commit [1], a pull request
at [2] to upsteam Mac and BSD compatibility. That pull request is
itself a rebase of an older pull request for an earlier version of
squashfs-tools. I squashed all commits between 4.4 and [1], apart from
the BSD-specific ones, and exported the new patch from that. Attached
below is the git diff --ignore-space-change between squashfs-tools 4.4
with the previous patch applied, and with the new patch applied.

[1]: 7d31beec53e6245d3405d6ef2b96e9811ae07044
[2]: plougher/squashfs-tools#69

---

diff --git a/squashfs-tools/info.c b/squashfs-tools/info.c
index 8efefe6..5c2f835 100644
--- a/squashfs-tools/info.c
+++ b/squashfs-tools/info.c
@@ -159,7 +159,7 @@ void *info_thrd(void *arg)
 			case EINTR:
 				continue;
 			default:
-				BAD_ERROR("sigwaitfailed "
+				BAD_ERROR("sigwait failed "
 					"because %s\n", strerror(errno));
 			}
 		}
diff --git a/squashfs-tools/mksquashfs.c b/squashfs-tools/mksquashfs.c
index 67d86a5..3607448 100644
--- a/squashfs-tools/mksquashfs.c
+++ b/squashfs-tools/mksquashfs.c
@@ -35,7 +35,10 @@
 #include <stddef.h>
 #include <sys/types.h>
 #include <sys/stat.h>
-#ifdef linux
+#ifndef linux
+#include <sys/sysctl.h>
+#else
+#include <sys/sysinfo.h>
 #include <sys/sysmacros.h>
 #endif
 #include <fcntl.h>
@@ -52,7 +55,6 @@
 #include <sys/wait.h>
 #include <limits.h>
 #include <ctype.h>
-#include <sys/sysinfo.h>

 #ifndef FNM_EXTMATCH /* glibc extension */
 	#define FNM_EXTMATCH 0
@@ -5191,7 +5193,17 @@ int parse_mode(char *arg, mode_t *res)

 int get_physical_memory()
 {
+	/*
+	 * Long longs are used here because with PAE, a 32-bit
+	 * machine can have more than 4GB of physical memory
+	 *
+	 * sysconf(_SC_PHYS_PAGES) relies on /proc being mounted.
+	 * If it fails use sysinfo, if that fails return 0
+	 */
+	long long num_pages = sysconf(_SC_PHYS_PAGES);
+	long long page_size = sysconf(_SC_PAGESIZE);
 	int phys_mem;
+
 #ifndef linux
 	#ifdef HW_MEMSIZE
 		#define SYSCTL_PHYSMEM HW_MEMSIZE
@@ -5221,16 +5233,6 @@ int get_physical_memory()
 	}
   #undef SYSCTL_PHYSMEM
 #else
-	/*
-	 * Long longs are used here because with PAE, a 32-bit
-	 * machine can have more than 4GB of physical memory
-	 *
-	 * sysconf(_SC_PHYS_PAGES) relies on /proc being mounted.
-	 * If it fails use sysinfo, if that fails return 0
-	 */
-	long long num_pages = sysconf(_SC_PHYS_PAGES);
-	long long page_size = sysconf(_SC_PAGESIZE);
-
 	if(num_pages == -1 || page_size == -1) {
 		struct sysinfo sys;
 		int res = sysinfo(&sys);
diff --git a/squashfs-tools/mksquashfs.h b/squashfs-tools/mksquashfs.h
index 88d0b5c..1beefef 100644
--- a/squashfs-tools/mksquashfs.h
+++ b/squashfs-tools/mksquashfs.h
@@ -24,7 +24,6 @@
  * mksquashfs.h
  *
  */
-#include <pthread.h>

 struct dir_info {
 	char			*pathname;
diff --git a/squashfs-tools/pseudo.c b/squashfs-tools/pseudo.c
index f8fd529..48e6b27 100644
--- a/squashfs-tools/pseudo.c
+++ b/squashfs-tools/pseudo.c
@@ -30,7 +30,6 @@
 #include <errno.h>
 #include <string.h>
 #include <stdlib.h>
-#include <sys/stat.h>
 #include <sys/types.h>
 #include <sys/wait.h>
 #include <sys/stat.h>
diff --git a/squashfs-tools/unsquashfs.c b/squashfs-tools/unsquashfs.c
index 00615ce..c1a6183 100644
--- a/squashfs-tools/unsquashfs.c
+++ b/squashfs-tools/unsquashfs.c
@@ -38,7 +38,6 @@
 #include <sys/sysinfo.h>
 #include <sys/sysmacros.h>
 #endif
-
 #include <sys/types.h>
 #include <sys/time.h>
 #include <sys/resource.h>
@@ -1085,7 +1084,7 @@ int create_inode(char *pathname, struct inode *i)
 			break;
 		case SQUASHFS_SYMLINK_TYPE:
 		case SQUASHFS_LSYMLINK_TYPE: {
-			struct timespec times[2] = {
+			struct timeval times[2] = {
 				{ i->time, 0 },
 				{ i->time, 0 }
 			};
@@ -1104,8 +1103,7 @@ int create_inode(char *pathname, struct inode *i)
 				goto failed;
 			}

-			res = utimensat(AT_FDCWD, pathname, times,
-					AT_SYMLINK_NOFOLLOW);
+			res = lutimes(pathname, times);
 			if(res == -1) {
 				EXIT_UNSQUASH_STRICT("create_inode: failed to set time on "
 					"%s, because %s\n", pathname,
@ruuda
Copy link
Contributor Author

ruuda commented Sep 10, 2019

I took the commits from plougher/squashfs-tools#69 except for the BSD-specific ones, and formatted them together in a single patch. The git diff --ignore-space-change between squashfs-tools with the previous patch applied, and with the new patch applied, is as follows:

Diff
diff --git a/squashfs-tools/info.c b/squashfs-tools/info.c
index 8efefe6..5c2f835 100644
--- a/squashfs-tools/info.c
+++ b/squashfs-tools/info.c
@@ -159,7 +159,7 @@ void *info_thrd(void *arg)
 			case EINTR:
 				continue;
 			default:
-				BAD_ERROR("sigwaitfailed "
+				BAD_ERROR("sigwait failed "
 					"because %s\n", strerror(errno));
 			}
 		}
diff --git a/squashfs-tools/mksquashfs.c b/squashfs-tools/mksquashfs.c
index 6d186ea..3607448 100644
--- a/squashfs-tools/mksquashfs.c
+++ b/squashfs-tools/mksquashfs.c
@@ -35,7 +35,12 @@
 #include <stddef.h>
 #include <sys/types.h>
 #include <sys/stat.h>
+#ifndef linux
+#include <sys/sysctl.h>
+#else
+#include <sys/sysinfo.h>
 #include <sys/sysmacros.h>
+#endif
 #include <fcntl.h>
 #include <errno.h>
 #include <dirent.h>
@@ -50,7 +55,6 @@
 #include <sys/wait.h>
 #include <limits.h>
 #include <ctype.h>
-#include <sys/sysinfo.h>
 
 #ifndef FNM_EXTMATCH /* glibc extension */
 	#define FNM_EXTMATCH 0
@@ -5189,7 +5193,17 @@ int parse_mode(char *arg, mode_t *res)
 
 int get_physical_memory()
 {
+	/*
+	 * Long longs are used here because with PAE, a 32-bit
+	 * machine can have more than 4GB of physical memory
+	 *
+	 * sysconf(_SC_PHYS_PAGES) relies on /proc being mounted.
+	 * If it fails use sysinfo, if that fails return 0
+	 */
+	long long num_pages = sysconf(_SC_PHYS_PAGES);
+	long long page_size = sysconf(_SC_PAGESIZE);
 	int phys_mem;
+
 #ifndef linux
 	#ifdef HW_MEMSIZE
 		#define SYSCTL_PHYSMEM HW_MEMSIZE
@@ -5219,16 +5233,6 @@ int get_physical_memory()
 	}
   #undef SYSCTL_PHYSMEM
 #else
-	/*
-	 * Long longs are used here because with PAE, a 32-bit
-	 * machine can have more than 4GB of physical memory
-	 *
-	 * sysconf(_SC_PHYS_PAGES) relies on /proc being mounted.
-	 * If it fails use sysinfo, if that fails return 0
-	 */
-	long long num_pages = sysconf(_SC_PHYS_PAGES);
-	long long page_size = sysconf(_SC_PAGESIZE);
-
 	if(num_pages == -1 || page_size == -1) {
 		struct sysinfo sys;
 		int res = sysinfo(&sys);
diff --git a/squashfs-tools/mksquashfs.h b/squashfs-tools/mksquashfs.h
index 88d0b5c..1beefef 100644
--- a/squashfs-tools/mksquashfs.h
+++ b/squashfs-tools/mksquashfs.h
@@ -24,7 +24,6 @@
  * mksquashfs.h
  *
  */
-#include <pthread.h>
 
 struct dir_info {
 	char			*pathname;
diff --git a/squashfs-tools/pseudo.c b/squashfs-tools/pseudo.c
index f8fd529..48e6b27 100644
--- a/squashfs-tools/pseudo.c
+++ b/squashfs-tools/pseudo.c
@@ -30,7 +30,6 @@
 #include <errno.h>
 #include <string.h>
 #include <stdlib.h>
-#include <sys/stat.h>
 #include <sys/types.h>
 #include <sys/wait.h>
 #include <sys/stat.h>
diff --git a/squashfs-tools/unsquashfs.c b/squashfs-tools/unsquashfs.c
index 00615ce..c1a6183 100644
--- a/squashfs-tools/unsquashfs.c
+++ b/squashfs-tools/unsquashfs.c
@@ -38,7 +38,6 @@
 #include <sys/sysinfo.h>
 #include <sys/sysmacros.h>
 #endif
-
 #include <sys/types.h>
 #include <sys/time.h>
 #include <sys/resource.h>
@@ -1085,7 +1084,7 @@ int create_inode(char *pathname, struct inode *i)
 			break;
 		case SQUASHFS_SYMLINK_TYPE:
 		case SQUASHFS_LSYMLINK_TYPE: {
-			struct timespec times[2] = {
+			struct timeval times[2] = {
 				{ i->time, 0 },
 				{ i->time, 0 }
 			};
@@ -1104,8 +1103,7 @@ int create_inode(char *pathname, struct inode *i)
 				goto failed;
 			}
 
-			res = utimensat(AT_FDCWD, pathname, times,
-					AT_SYMLINK_NOFOLLOW);
+			res = lutimes(pathname, times);
 			if(res == -1) {
 				EXIT_UNSQUASH_STRICT("create_inode: failed to set time on "
 					"%s, because %s\n", pathname,

The new patch indeed guards the sysmacros include. @risicle, could you try again?

@Mic92
Copy link
Member

Mic92 commented Sep 10, 2019

@GrahamcOfBorg build squashfsTools

@ruuda
Copy link
Contributor Author

ruuda commented Sep 20, 2019

Does anything still need to be done here?

@Mic92 Mic92 merged commit 27994da into NixOS:master Sep 20, 2019
dtzWill pushed a commit to dtzWill/nixpkgs that referenced this pull request Sep 21, 2019
squashfsTools: 4.4dev_20180612 -> 4.4
(cherry picked from commit 27994da)
@risicle risicle mentioned this pull request Nov 9, 2019
10 tasks
@ruuda ruuda deleted the squashfs-4.4 branch October 6, 2022 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants