Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ruby_3_0: init 3.0.0 #107933

Merged
merged 4 commits into from Feb 23, 2021
Merged

ruby_3_0: init 3.0.0 #107933

merged 4 commits into from Feb 23, 2021

Conversation

marsam
Copy link
Contributor

@marsam marsam commented Dec 30, 2020

Motivation for this change

https://www.ruby-lang.org/en/news/2020/12/25/ruby-3-0-0-released/

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

Copy link
Member

@prusnak prusnak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

untested, code looks fine

@@ -10836,7 +10836,8 @@ in
})
ruby_2_5
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we drop 2.5 anytime soon?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By March end after it hits EOL, maybe: https://www.ruby-lang.org/en/downloads/branches/

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we know it's EOL'd before the next release, it can be done ASAP for unstable. Though I suggest in another PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't see that anyone else had done it, so: #111662.

@SuperSandro2000
Copy link
Member

Please target staging.

@SuperSandro2000
Copy link
Member

/rebase-staging

@github-actions github-actions bot changed the base branch from master to staging January 7, 2021 03:48
@emptyflask
Copy link
Contributor

Is there a problem in the tests? I see some failures, but it looks like a (temporary?) network issue:

SocketError: Failed to open TCP connection to rubygems.org:443 (getaddrinfo: Name or service not known): https://rubygems.org/downloads/minitest-5.14.2.gem

It's been about 2.5 weeks since Ruby 3.0 was released... is there anything else I can do to help push this along?

@emptyflask
Copy link
Contributor

By the way, there's the rubyPackages section in all-packages.nix that looks like it should have the line:

rubyPackages_3_0 = recurseIntoAttrs ruby_3_0.gems;

@SuperSandro2000
Copy link
Member

Is there a problem in the tests? I see some failures, but it looks like a (temporary?) network issue:

There is no network connection in the test phase.

@anna328p
Copy link
Member

is there anything I can do to help bring this about?

@manveru
Copy link
Contributor

manveru commented Feb 2, 2021

Here's a potential fix. The problem seems to be that when using railsexpress, the source is built twice, and once from git, which doesn't include the bundled gems that the release tarball has. So this just injects them. Might need some fiddling to avoid rebuilds of older Ruby versions though.

diff --git a/pkgs/development/interpreters/ruby/bundled-gems.nix b/pkgs/development/interpreters/ruby/bundled-gems.nix
new file mode 100644
index 000000000000..6b7f2f20a16b
--- /dev/null
+++ b/pkgs/development/interpreters/ruby/bundled-gems.nix
@@ -0,0 +1,12 @@
+{
+  "3.0.0" = {
+    "minitest-5.14.2"    = "sha256-m0ATOOKHxQzSNUNTtLeB03ZthjyuQTsqG/WF0jcTHpw=";
+    "power_assert-1.2.0" = "sha256-Qfqz2cpGutN/WXKem77D/XRdLyY9GzBovG8hhjtWsGo=";
+    "rake-13.0.3"        = "sha256-xyizOlvQlTQpCnAP8X3As01tMqriOjT0Y8TP5KooM8Y=";
+    "test-unit-3.3.7"    = "sha256-5cg8h+jMiwjjvbUs9Fr57S+5QntkgfWSHDhFadr6e9k=";
+    "rexml-3.2.4"        = "sha256-A2sx88BSvkK3ouaRTzMi2q7M5GsXKAbzj+pClzibe9Y=";
+    "rss-0.2.9"          = "sha256-oEWHa+qbNUViQdTVe5NA2eOgQiZNa0rqnZOYPA/oP6w=";
+    "rbs-1.0.0"          = "sha256-XOoW2A00VODqpvev72VV3e0YwL0rK/+d+MThKGCr7rE=";
+    "typeprof-0.11.0"    = "sha256-2emPq1+ZN6HlQiSGsqPRXL4C0skAL1UapyRwabFlf/I=";
+  };
+}
diff --git a/pkgs/development/interpreters/ruby/default.nix b/pkgs/development/interpreters/ruby/default.nix
index 7696b716d02a..0812fcd55bfd 100644
--- a/pkgs/development/interpreters/ruby/default.nix
+++ b/pkgs/development/interpreters/ruby/default.nix
@@ -115,13 +115,28 @@ let
           cp -r ${rubygems}/test/rubygems $sourceRoot/test
         '';
 
-        postPatch = ''
+        postPatch = let
+          bundledGems = (import ./bundled-gems.nix);
+          forThisVersion = bundledGems.${ver.majMinTiny};
+          copyGems = if (lib.versionAtLeast ver.majMin "3.0") then
+            lib.mapAttrsToList (name: hash:
+              let
+                src = fetchurl {
+                  name = "${name}.gem";
+                  inherit hash;
+                  url = "https://rubygems.org/downloads/${name}.gem";
+                };
+              in "cp ${src} gems/${name}.gem") forThisVersion
+          else [];
+        in ''
           sed -i configure.ac -e '/config.guess/d'
           cp --remove-destination ${config}/config.guess tool/
           cp --remove-destination ${config}/config.sub tool/
           # Make the build reproducible for ruby <= 2.7
           # See https://github.com/ruby/io-console/commit/679a941d05d869f5e575730f6581c027203b7b26#diff-d8422f096931c58d4463e2489f62a228b0f24f0492950ba88c8c89a0d741cfe6
           sed -i ext/io/console/io-console.gemspec -e '/s\.date/d'
+
+          ${lib.concatStringsSep "\n" copyGems}
         '';
 
         # Force the revision.h generation. Somehow `revision.tmp` is an empty

@jonringer
Copy link
Contributor

is there anything I can do to help bring this about?

Yea, try to validate that common scenarios work as intended. If you're using nixpkgs-update you can do nixpkgs-review pr 107933 and use the ruby from that shell to see if everything is working. However, appears that something is not right from the tests

@anna328p
Copy link
Member

is there anything I can do to help bring this about?

Yea, try to validate that common scenarios work as intended. If you're using nixpkgs-update you can do nixpkgs-review pr 107933 and use the ruby from that shell to see if everything is working. However, appears that something is not right from the tests

I don't use nixpkgs-update. Is there a resource for getting it set up? All I can find is the hackage page which doesn't cover this usecase.

@emptyflask
Copy link
Contributor

I haven't used nixpkgs-update either, but I've attempted nixpkgs-review on its own inside my nixpkgs repo.

nix-shell -p nixpkgs-review --run "nixpkgs-review pr 107933"

Running it takes a very long time (and is checking for each package on every cachix source on my system!), so I re-ran with nixpkgs-review pr -p ruby_3_0 107933, and it hangs on:

[1/12/14 built, 1 copied (1.3/1.3 MiB), 1.3 MiB DL] building ruby-3.0.0 (installPhase): retrying SocketError (Failed to open TCP connection to rubygems.org:443 (getaddrinfo: Name or service not known)) after 25 seconds...

I'll retry after applying @manveru's patch.

@jonringer
Copy link
Contributor

sorry, I meant to say nixpkgs-review not nixpkgs-update, which the update bot. Sorry.

you can do nixpkgs-review pr 107933 --package ruby_3_0 to just build ruby, similarly, if you can use similar cli tools like hub pr checkout 107933 and it will just fetch and checkout the branch. You will probably have to apply @manveru 's patch to get some progress.

@emptyflask
Copy link
Contributor

emptyflask commented Feb 12, 2021

This is the first I've even heard of the railsexpress patches. They're apparently just some sets of changes collected by the creator of RVM -- does anyone really use these, or is it adding complexity here for no reason?

@jdelStrother
Copy link
Contributor

jdelStrother commented Feb 12, 2021

FWIW when I try to build this, it tries to write rdoc files into /homeless-shelter:

builder for '/nix/store/sv7qgkyww7wmdg9vppga3n8phic7ki46-ruby-3.0.0.drv' failed with exit code 2; last 10 log lines:
   Could not be configured. It will not be installed.
     Check ext/win32ole/mkmf.log for more details.
  *** Fix the problems, then remove these directories and try again if you want.
  make[1]: Leaving directory '/private/tmp/nix-build-ruby-3.0.0.drv-0/source'
  Generating RDoc documentation
  uh-oh! RDoc had a problem:
  Operation not permitted @ dir_s_mkdir - /homeless-shelter

  run with --debug for full backtrace
  make: *** [uncommon.mk:567: rdoc] Error 1
cannot build derivation '/nix/store/wjcyhvjzkw08vik0lfrk7da648jis3ky-env.drv': 1 dependencies couldn't be built

I'm using nix on macOS 10.15.7. I checked out nixpkgs, merged @marsam's branch into master and applied @manveru's patch, then ran nix-shell -p nixpkgs-review --run "nixpkgs-review rev --package ruby_3_0 HEAD"

Here's the build log, in case it's any help

If instead I build 2.7 (nix-shell -p nixpkgs-review --run "nixpkgs-review rev --package ruby_2_7 HEAD"), it seems fine - no complaints about trying to write rdoc files to homeless-shelter, at leat.

Update: Ah, if I set docSupport=false, then 3.0 does built ok. I'm going to try running our ruby app against it & see if anything explodes.

@jdelStrother
Copy link
Contributor

^ This has been mostly ok for me since installing with docSupport=false. I did have to tweak some native gem compilation flags in bundlerEnv - eg:

{
    hiredis = attrs: {
      buildFlags = ["--with-cflags=-fdeclspec"];
    };
    byebug = attrs: {
      buildFlags = ["--with-cflags=-fdeclspec" "--with-cppflags=-fdeclspec"];
    };
}

for anything that imports ruby 3's header files. This fixes errors like:

/nix/store/mi2n5bnrshc5m5bhidy78rfqrn3hrgk9-ruby-3.0.0/include/ruby-3.0.0/ruby/assert.h:132:1: error: '__declspec' attributes are not enabled; use '-fdeclspec' or '-fms-extensions' to enable support for __declspec attributes
RBIMPL_ATTR_NORETURN()
^
/nix/store/mi2n5bnrshc5m5bhidy78rfqrn3hrgk9-ruby-3.0.0/include/ruby-3.0.0/ruby/internal/attr/noreturn.h:29:33: note: expanded from macro 'RBIMPL_ATTR_NORETURN'
# define RBIMPL_ATTR_NORETURN() __declspec(noreturn)
                                ^

Not sure if there's a more global way of supplying that fdeclspec flag to all gem compilations?

@thefloweringash
Copy link
Member

The rdoc error seems to be rdoc creating $XDG_CONFIG_DIR at $HOME/.local/share if it doesn't exist, so it can be fixed by setting a HOME.

I think putting -fdeclspec into NIX_CFLAGS_COMPILE in buildRubyGem is the appropriate global place.

diff --git a/pkgs/development/interpreters/ruby/default.nix b/pkgs/development/interpreters/ruby/default.nix
index 0812fcd55bf..1c633d8c918 100644
--- a/pkgs/development/interpreters/ruby/default.nix
+++ b/pkgs/development/interpreters/ruby/default.nix
@@ -161,6 +161,10 @@ let
 
         preConfigure = opString docSupport ''
           configureFlagsArray+=("--with-ridir=$devdoc/share/ri")
+
+          # rdoc creates XDG_DATA_DIR (defaulting to HOME/.local/share) even if
+          # it's not going to be used.
+          export HOME=$TMPDIR
         '';
 
         # fails with "16993 tests, 2229489 assertions, 105 failures, 14 errors, 89 skips"
diff --git a/pkgs/development/ruby-modules/gem/default.nix b/pkgs/development/ruby-modules/gem/default.nix
index 9e64b120af8..78395c88bcf 100644
--- a/pkgs/development/ruby-modules/gem/default.nix
+++ b/pkgs/development/ruby-modules/gem/default.nix
@@ -129,6 +129,12 @@ stdenv.mkDerivation ((builtins.removeAttrs attrs ["source"]) // {
     runHook postUnpack
   '';
 
+  # As of ruby 3.0, ruby headers require -fdeclspec when building with clang
+  # Introduced in https://github.com/ruby/ruby/commit/0958e19ffb047781fe1506760c7cbd8d7fe74e57
+  NIX_CFLAGS_COMPILE = lib.optionals (stdenv.cc.isClang && lib.versionAtLeast ruby.version.major "3") [
+    "-fdeclspec"
+  ];
+
   buildPhase = attrs.buildPhase or ''
     runHook preBuild
 

@marsam marsam marked this pull request as ready for review February 22, 2021 08:41
@marsam
Copy link
Contributor Author

marsam commented Feb 22, 2021

Sorry for the delay, this took WAY longer than I expected.
I didn't have problems building with rdoc, but if you still do, I'll update the PR with HOME=$TMPDIR.

@thefloweringash
Copy link
Member

I didn't have problems building with rdoc, but if you still do, I'll update the PR with HOME=$TMPDIR.

This is still a problem on darwin. The difference seems to be in rdoc's error handling on this line:

https://github.com/ruby/rdoc/blob/d23594ecd781ee1c79ef284eebfdfa2a0fde218a/lib/rdoc.rb#L139

On linux, attempting to create $HOME throws Errno::EACCES but on darwin this throws Errno::EROFS. rdoc ignores the former and propagates the latter, resulting in build failure on darwin.

Copy link
Member

@thefloweringash thefloweringash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested a medium-sized project on x86_64-darwin and x86_64-linux. LGTM.

@marsam marsam merged commit ff05a24 into NixOS:staging Feb 23, 2021
@marsam marsam deleted the init-ruby-3.0 branch February 23, 2021 01:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet