flutter/dart: remove non-stable versions and bump #109477

thiagokokada · 2021-01-15T20:47:39Z

Motivation for this change

This PR removes non-stable versions of Dart and Flutter. The reason is that Flutter/Dart is really old nowadays, specially the non-stable versions. Having to maintain multiple versions makes bumps more difficult. Removing it and focusing in the stable will make it easier to bump versions (at least I hope).

For those that need to use beta or dev versions of Dart/Flutter, there is still a way using the following example:

rec {
  dart-dev = dart.override({
    sources = { "2.12.0-242.0.dev-x86_64-linux" = {...}; }; # supply an attrset with the source so you want to download.
    version = "2.12.0-242.0.dev";
  });
  flutter-dev =  flutterPackages.mkFlutter {
    pname = "flutter-dev";
    version = "1.26.0-12.0.pre";
    src = fetchurl {...}; # supply your own source
    depsSha256 = "..."; # this is the SHA256 of Flutter dependencies, set it to `lib.fakeSha256`, build once and replace with the generated value
    patches = [ ]; # supply your own patches
    dart = dart-dev;
  } {};
}

Bumps:

dart: 2.10.0 -> 2.10.4, 2.10.4 -> 2.10.5
flutter: 1.22.0 -> 1.22.4, 1.22.4 -> 1.22.5

Didn't bump Flutter to version 1.22.5 because it is missing some dependency. I think it is lack of a proper Dart release with this version of Flutter (Google didn't release Dart 2.10.5), so it tries to download the dependency from https://pub.dev/ and fails because we use --offline flag during build.

I don't think this is something that will happen frequently, so I didn't bother trying to fix it, but we may need to create some way to solve dependencies online before building if this is a constantly issue in the future.

Flutter derivation is now build in a 2-pass process: the first step download its dependencies from https://pub.dev, creating a local repository cache, and the second step uses the repository cache from the first step to finally build Flutter. This makes the build process much more robust and future proof, since we don't depend on Google packaging Flutter with all dependencies anymore.

Things done

pkgs/development/interpreters/dart/default.nix

thiagokokada · 2021-01-15T20:55:08Z

@GrahamcOfBorg build dart flutter

pkgs/development/interpreters/dart/default.nix

pkgs/top-level/aliases.nix

SuperSandro2000

diff LGTM

thiagokokada · 2021-01-19T13:39:58Z

@GRBurst Can we also have your review here?

PeterZainzinger · 2021-01-24T08:31:43Z

Didn't bump Flutter to version 1.22.5 because it is missing some dependency. I think it is lack of a proper Dart release with this version of Flutter (Google didn't release Dart 2.10.5), so it tries to download the dependency from https://pub.dev/ and fails because we use --offline flag during build.

@thiagokokada I tried your your diff and it works with 1.22.4 but not with 1.22.5 even after bumping dart to 2.10.5 (which is release as by now) with same described error:

configuring
 no configure script, doing nothing
 building
 Resolving dependencies...
 Because flutter_tools depends on node_preamble any which doesn't exist (could not find package node_preamble in cache), version solving failed.

The same issue occurs when trying to use a beta/dev/master version using mkFlutter. So I don't think this could be merged as is cause it prevents to use any other version than 1.22.4

thiagokokada · 2021-01-24T10:54:45Z

@PeterZainzinger Huh... going to try, but I don't think this PR is the cause of this issue. If we did the same just by updating how master is current is it would have the same issue.

The fault of this problem is that Flutter or Dart is not shipping with all the dependencies needed to build anymore, so we probably need some changes in how Flutter is packaged.

thiagokokada · 2021-01-24T10:57:34Z

If anything, this PR will make easier to fix this issue, because instead of trying to fix it in all channels, now the only channel that matters is stable.

PeterZainzinger · 2021-01-24T11:52:48Z

Ok, I see your point. I agree that it is outside of the scope of this PR, although I think we should try to fix it anyway is a we make a big change to this module.

I investigated a bit further, downloaded both tarballs (1.22.4 vs 1.22.5) and noticed that the 1.22.5 does not ship node_preamble in its .pub_cache/hosted/pub.dartlang.org directory. The version has not changed at all.

The workaround is quite obvious, add the pub package as an source and copy it to cache. But that pretty hacky and maybe there are other packages that are missing as well...

PeterZainzinger · 2021-01-24T12:03:32Z

I created an issue in the flutter repo. The problem in the tarball is pretty obviously on their side.

flutter/flutter#74589

thiagokokada · 2021-01-24T12:15:11Z

@PeterZainzinger Actually I do think the issue is in our side. We need to do the build in two steps, the first one will download the deps and the second one will build the derivation. I gave it a try in this branch: thiagokokada@e2e30e0

It almost works, the dependency resolution now works correctly. But it fails later on with Unable to open file /build/flutter/bin/cache/flutter_tools.snapshot for writing snapshot. The old HOME=../.. trick seems to not work anymore.

If someone knows how to fix this issue it would be welcome.

thiagokokada · 2021-01-24T12:19:55Z

Rebased.

thiagokokada · 2021-01-24T12:34:29Z

@PeterZainzinger Fixed the issue. Now Flutter 1.22.5 builds and runs (at least on my machine).

thiagokokada · 2021-01-24T12:38:04Z

pkgs/development/compilers/flutter/flutter.nix

-
-      rm -rf bin/cache/{artifacts,dart-sdk,downloads}
-      rm -f  bin/cache/*.stamp


Not really needed since those paths were not in $out.

PeterZainzinger · 2021-01-24T12:43:51Z

@PeterZainzinger Fixed the issue. Now Flutter 1.22.5 builds and runs (at least on my machine).

I would like to try. Not sure if the outputHash in deps.nix is stable. Looks like a bit of an hack to be tbh and something that has to be maintained in the future. But if it works....

thiagokokada · 2021-01-24T12:45:00Z

@PeterZainzinger Fixed the issue. Now Flutter 1.22.5 builds and runs (at least on my machine).

I would like to try. Not sure if the outputHash in deps.nix is stable. Looks like a bit of an hack to be tbh and something that has to be maintained in the future. But if it works....

It should be. I took this trick from the official docs: https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/maven.section.md#double-invocation. Yeah, it is for Maven but applied you can apply the idea in other places.

BTW, it is generated from a pubspec.lock that is included in flutter_tools directory. I can't see a reason why this would change (unless Dart is doing something bizarre here).

thiagokokada · 2021-01-24T12:59:51Z

In an perfect world there would be something like pub2nix :) But if flutter_tools is something that doesn't evolve with every version I think we are fine here.

I think how this derivation is currently it could eventually evolve to a more generic build (like buildDartPackage), since it is not that different from other derivations from other languages (like buildGoPackage).

Maybe a possibility to override this hash from mkFlutter would be nice.

Coincidentally had the same idea. Please test it with commit 1b838f969c699cdc4fd09f6b91dfdd300f36e110 (everything is the same so it will not even trigger a build).

Can you review and approve @PeterZainzinger? Also need re-reviews from @GRBurst @ericdallo @babariviere and @SuperSandro2000.

thiagokokada · 2021-01-24T13:14:36Z

Some last cleanups:

Renamed deps.nix to repository.nix (so GitHub don't hide the diffs by default)
No more dependency on $HOME being set, instead use $PUB_CACHE
Add depsSha256 to mkFlutter so we can pass a custom version of Flutter and overwrite its SHA256 from dependencies

pkgs/development/compilers/flutter/flutter.nix

PeterZainzinger · 2021-01-24T14:21:54Z

@thiagokokada other than the note above the diff looks good to me.

pkgs/development/compilers/flutter/default.nix

pkgs/development/compilers/flutter/flutter.nix

pkgs/development/compilers/flutter/repository.nix

pkgs/development/interpreters/dart/default.nix

thiagokokada · 2021-01-25T14:45:22Z

I think this PR is mostly ready to merge, but still would like a review from @GRBurst for the Dart parts of it.

thiagokokada · 2021-01-25T15:24:11Z

pkgs/development/compilers/flutter/flutter.nix

+      export PUB_CACHE="${repository}"
+
+      pushd "$FLUTTER_TOOLS_DIR"
+      ${dart}/bin/pub get --offline


Using pub get here since it works and avoids an offline warning.

Related issue #108606. Nowadays we have multiple outdated versions of Flutter in nixpkgs. Instead, let's focus in having in having stable versions of Flutter working. Users needing to use beta or dev versions of Flutter can use mkFlutter function, that still exists.

For those that need other versions of Dart, they can use the new sources parameter instead.

Needs to do a build in 2-pass now since Google stopped shipping all dependencies needed to build Flutter. This may be an oversight from them since they used to ship everything, but this makes the whole build process more robust. The first step will download all dependencies from pub, and the second step will build Flutter. Since we need to build repository first, we also require a new depsSha256 parameter to be passed, that represents the SHA256 of the resulting derivation of all Flutter dependencies downloaded from https://pub.dev. This commit also makes some changes in mkFlutter, allowing the user to pass src instead of passing version/channel/filename, allowing for more flexibility (i.e: building from a local fork of Flutter).

thiagokokada · 2021-01-27T17:50:57Z

Rebased and fixed conflicts.

@SuperSandro2000 can we have this merged? I think it is in a good state, and while @GRBurst didn't respond it yet it concurs that this is a good idea: #108606 (comment)

SuperSandro2000 · 2021-01-28T08:06:16Z

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.

Result of nixpkgs-review pr 109477 run on x86_64-linux 1

3 packages built:

dart
flutter (flutterPackages.stable)
hover

SuperSandro2000 · 2021-01-28T08:24:25Z

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.

Result of nixpkgs-review pr 109477 run on x86_64-darwin 1

1 package built:

dart

Updating Linux Kernel from 5.9 to 5.10, supporting the latest available Kernel (as of this writing); Emacs 28 with support for Pure GTK and Native compilation enabled, the required snippet from [this Reddit post](https://www.reddit.com/r/emacs/comments/kqb9s9/cannot_recompile_packagess_error_wrong_number_of/gj3cfn3/?utm_source=share&utm_medium=web2x&context=3) was included using a short URL provided by Bitly. flutterPackages.dev was removed by [this](NixOS/nixpkgs#109477), changing to default stable packages.

thiagokokada requested review from ericdallo and GRBurst January 15, 2021 20:47

thiagokokada mentioned this pull request Jan 15, 2021

flutter: 1.22.0 -> 1.22.4, drop non-stable versions #108606

Closed

2 tasks

thiagokokada commented Jan 15, 2021

View reviewed changes

pkgs/development/interpreters/dart/default.nix Show resolved Hide resolved

ericdallo approved these changes Jan 15, 2021

View reviewed changes

ofborg bot added 8.has: clean-up 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Jan 15, 2021

thiagokokada requested a review from babariviere January 15, 2021 21:01

ofborg bot requested a review from ericdallo January 15, 2021 21:03

ofborg bot added the 10.rebuild-darwin: 1 label Jan 15, 2021

SuperSandro2000 reviewed Jan 15, 2021

View reviewed changes

pkgs/development/interpreters/dart/default.nix Outdated Show resolved Hide resolved

SuperSandro2000 reviewed Jan 15, 2021

View reviewed changes

pkgs/development/interpreters/dart/default.nix Outdated Show resolved Hide resolved

SuperSandro2000 reviewed Jan 15, 2021

View reviewed changes

pkgs/top-level/aliases.nix Outdated Show resolved Hide resolved

thiagokokada requested a review from SuperSandro2000 January 16, 2021 03:11

SuperSandro2000 approved these changes Jan 17, 2021

View reviewed changes

thiagokokada commented Jan 24, 2021

View reviewed changes

ofborg bot added the 11.by: package-maintainer label Jan 24, 2021

PeterZainzinger reviewed Jan 24, 2021

View reviewed changes

pkgs/development/compilers/flutter/flutter.nix Outdated Show resolved Hide resolved

ericdallo approved these changes Jan 24, 2021

View reviewed changes

PeterZainzinger approved these changes Jan 24, 2021

View reviewed changes

SuperSandro2000 reviewed Jan 25, 2021

View reviewed changes

ofborg bot requested a review from ericdallo January 25, 2021 14:54

thiagokokada commented Jan 25, 2021

View reviewed changes

thiagokokada added 8 commits January 27, 2021 14:48

dart: remove all versions except the one used by flutterPackages.stable

75326db

For those that need other versions of Dart, they can use the new sources parameter instead.

dart: 2.10.0 -> 2.10.4

c5623e1

flutter: 1.22.0 -> 1.22.4

d6db914

dart: 2.10.4 -> 2.10.5

e19491e

dart/flutter: add thiagokokada as maintainer

d87aba5

dart: add some assertions

577645d

SuperSandro2000 merged commit 7c64ac5 into NixOS:master Jan 28, 2021

thiagokokada mentioned this pull request Jan 28, 2021

Bump nixpkgs-unstable, flutter: 1.22.4 -> 1.22.5 nubank/nixpkgs#26

Merged

thiagokokada deleted the flutter-dart-remove-non-stable-versions branch January 28, 2021 17:17

thiagokokada mentioned this pull request Jan 29, 2021

flutter: 1.22.5 -> 1.22.4 #111187

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

flutter/dart: remove non-stable versions and bump #109477

flutter/dart: remove non-stable versions and bump #109477

thiagokokada commented Jan 15, 2021 •

edited

thiagokokada commented Jan 15, 2021

SuperSandro2000 left a comment

thiagokokada commented Jan 19, 2021

PeterZainzinger commented Jan 24, 2021

thiagokokada commented Jan 24, 2021

thiagokokada commented Jan 24, 2021

PeterZainzinger commented Jan 24, 2021

PeterZainzinger commented Jan 24, 2021 •

edited

thiagokokada commented Jan 24, 2021 •

edited

thiagokokada commented Jan 24, 2021

thiagokokada commented Jan 24, 2021

thiagokokada Jan 24, 2021

PeterZainzinger commented Jan 24, 2021

thiagokokada commented Jan 24, 2021 •

edited

thiagokokada commented Jan 24, 2021 •

edited

thiagokokada commented Jan 24, 2021

PeterZainzinger commented Jan 24, 2021

thiagokokada commented Jan 25, 2021

thiagokokada Jan 25, 2021

thiagokokada commented Jan 27, 2021 •

edited

SuperSandro2000 commented Jan 28, 2021

SuperSandro2000 commented Jan 28, 2021


		rm -rf bin/cache/{artifacts,dart-sdk,downloads}
		rm -f bin/cache/*.stamp

flutter/dart: remove non-stable versions and bump #109477

flutter/dart: remove non-stable versions and bump #109477

Conversation

thiagokokada commented Jan 15, 2021 • edited

Motivation for this change

Things done

thiagokokada commented Jan 15, 2021

SuperSandro2000 left a comment

Choose a reason for hiding this comment

thiagokokada commented Jan 19, 2021

PeterZainzinger commented Jan 24, 2021

thiagokokada commented Jan 24, 2021

thiagokokada commented Jan 24, 2021

PeterZainzinger commented Jan 24, 2021

PeterZainzinger commented Jan 24, 2021 • edited

thiagokokada commented Jan 24, 2021 • edited

thiagokokada commented Jan 24, 2021

thiagokokada commented Jan 24, 2021

thiagokokada Jan 24, 2021

Choose a reason for hiding this comment

PeterZainzinger commented Jan 24, 2021

thiagokokada commented Jan 24, 2021 • edited

thiagokokada commented Jan 24, 2021 • edited

thiagokokada commented Jan 24, 2021

PeterZainzinger commented Jan 24, 2021

thiagokokada commented Jan 25, 2021

thiagokokada Jan 25, 2021

Choose a reason for hiding this comment

thiagokokada commented Jan 27, 2021 • edited

SuperSandro2000 commented Jan 28, 2021

SuperSandro2000 commented Jan 28, 2021

thiagokokada commented Jan 15, 2021 •

edited

PeterZainzinger commented Jan 24, 2021 •

edited

thiagokokada commented Jan 24, 2021 •

edited

thiagokokada commented Jan 24, 2021 •

edited

thiagokokada commented Jan 24, 2021 •

edited

thiagokokada commented Jan 27, 2021 •

edited