Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gitlab: 12.8.10 -> 12.10.6 #85293

Merged
merged 6 commits into from May 31, 2020
Merged

gitlab: 12.8.10 -> 12.10.6 #85293

merged 6 commits into from May 31, 2020

Conversation

ghost
Copy link

@ghost ghost commented Apr 15, 2020 •
edited by ghost

Motivation for this change

Fixes #83478.
Changes for vgo2nix are here: https://github.com/petabyteboy/vgo2nix
First I still saw vgo2nix failures, then I added "--keep-going" to see if there was more than one failure, and then all failures disappeared.
For ruby I got an error about the bundler version (1.17.3 vs 2.1.4). I changed the ruby version to 2.7, and then got the opposite error. It seems like some of the lockfiles were created with bundler 1.17.3 and some were created with 2.1.4, so I manually changed all to 2.1.4.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@talyz @flokli

@ofborg ofborg bot requested review from fpletz, globin and talyz April 15, 2020 11:38
@ghost
Copy link
Author

ghost commented Apr 15, 2020

IMO we should package 12.8.9 if we don't get this tested, reviewed and merged in the next few days. The GitLab admin dashboard screams "Update ASAP".

@ghost ghost changed the title WIP: gitlab: 12.8.8 -> 12.9.3 gitlab: 12.8.8 -> 12.9.3 Apr 15, 2020
@ghost
Copy link
Author

ghost commented Apr 15, 2020

@GrahamcOfBorg test gitlab

@ghost
Copy link
Author

ghost commented Apr 15, 2020

gitlab-ee fails to build, because webpack runs out of memory (even though there is enough memory left on the system):

/build/source/node_modules/.bin/webpack --config /build/source/config/webpack.config.js --bail

<--- Last few GCs --->

[5820:0x19c2990]   375650 ms: Mark-sweep 1995.7 (2098.8) -> 1987.7 (2101.3) MB, 746.7 / 0.0 ms  (average mu = 0.161, current mu = 0.049) allocation failure scavenge might not succeed
[5820:0x19c2990]   376354 ms: Mark-sweep 1999.0 (2101.3) -> 1990.3 (2104.3) MB, 666.1 / 0.0 ms  (average mu = 0.112, current mu = 0.054) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x30e74175be1d]
Security context: 0x03925769e6c1 <JSObject>
    1: /* anonymous */(aka /* anonymous */) [0x47857182201] [/build/source/node_modules/webpack-sources/lib/applySourceMap.js:~58] [pc=0x30e74387d512](this=0x2ee1ee5026f1 <undefined>,chunk=0x13fa089f6081 <String[1]: {>,middleMapping=0x374f2989d001 <Object map = 0xe070ffdd51>)
    2: SourceNode_walk [0x2f1572cc8989] [/build/source/node_modules/source-map/lib/...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8448f8 node::Abort() [/nix/store/imck490sdmw0vcbxarb9xrki4cfq2q5b-nodejs-10.20.1/bin/node]
 2: 0x844941  [/nix/store/imck490sdmw0vcbxarb9xrki4cfq2q5b-nodejs-10.20.1/bin/node]
 3: 0xa33e32 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/nix/store/imck490sdmw0vcbxarb9xrki4cfq2q5b-nodejs-10.20.1/bin/node]
 4: 0xa340d8 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/nix/store/imck490sdmw0vcbxarb9xrki4cfq2q5b-nodejs-10.20.1/bin/node]
 5: 0xdf2a92  [/nix/store/imck490sdmw0vcbxarb9xrki4cfq2q5b-nodejs-10.20.1/bin/node]
 6: 0xe04419 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/nix/store/imck490sdmw0vcbxarb9xrki4cfq2q5b-nodejs-10.20.1/bin/node]
 7: 0xe04d0c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/nix/store/imck490sdmw0vcbxarb9xrki4cfq2q5b-nodejs-10.20.1/bin/node]
 8: 0xe06db5 v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/nix/store/imck490sdmw0vcbxarb9xrki4cfq2q5b-nodejs-10.20.1/bin/node]
 9: 0xe06e01 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/nix/store/imck490sdmw0vcbxarb9xrki4cfq2q5b-nodejs-10.20.1/bin/node]
10: 0xdd1416 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/nix/store/imck490sdmw0vcbxarb9xrki4cfq2q5b-nodejs-10.20.1/bin/node]
11: 0x106e0ca v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/nix/store/imck490sdmw0vcbxarb9xrki4cfq2q5b-nodejs-10.20.1/bin/node]
12: 0x30e74175be1d
rake aborted!
Command failed with status (): [/build/source/node_modules/.bin/webpack --...]

@talyz
Copy link
Contributor

talyz commented Apr 15, 2020

Try changing the --max_old_space_size argument on line 92 in pkgs/applications/version-management/gitlab/default.nix:

bundle exec rake webpack:compile RAILS_ENV=production NODE_ENV=production NODE_OPTIONS="--max_old_space_size=2048"

@ofborg ofborg bot requested review from krav and talyz April 15, 2020 13:09
@ghost
Copy link
Author

ghost commented Apr 15, 2020

When loading the admin dashboard, I get a message saying "Error loading the statistics. Please try again" and this appears in the gitlab.service log:

Apr 15 14:05:13 lab unicorn[1547]: F, [2020-04-15T14:05:13.518454 #1605] FATAL -- :
Apr 15 14:05:13 lab unicorn[1547]: ArgumentError (wrong number of arguments (given 3, expected 1..2)):
Apr 15 14:05:13 lab unicorn[1547]:
Apr 15 14:05:13 lab unicorn[1547]: lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
Apr 15 14:05:13 lab unicorn[1547]: lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
Apr 15 14:05:13 lab unicorn[1547]: lib/gitlab/metrics/transaction.rb:62:in `run'
Apr 15 14:05:13 lab unicorn[1547]: lib/gitlab/metrics/rack_middleware.rb:17:in `call'
Apr 15 14:05:13 lab unicorn[1547]: lib/gitlab/request_profiler/middleware.rb:17:in `call'
Apr 15 14:05:13 lab unicorn[1547]: ee/lib/gitlab/jira/middleware.rb:19:in `call'
Apr 15 14:05:13 lab unicorn[1547]: lib/gitlab/middleware/go.rb:20:in `call'
Apr 15 14:05:13 lab unicorn[1547]: lib/gitlab/etag_caching/middleware.rb:13:in `call'
Apr 15 14:05:13 lab unicorn[1547]: lib/gitlab/middleware/multipart.rb:122:in `call'
Apr 15 14:05:13 lab unicorn[1547]: lib/gitlab/middleware/read_only/controller.rb:53:in `call'
Apr 15 14:05:13 lab unicorn[1547]: lib/gitlab/middleware/read_only.rb:18:in `call'
Apr 15 14:05:13 lab unicorn[1547]: lib/gitlab/middleware/basic_health_check.rb:25:in `call'
Apr 15 14:05:13 lab unicorn[1547]: lib/gitlab/middleware/request_context.rb:23:in `call'
Apr 15 14:05:13 lab unicorn[1547]: config/initializers/fix_local_cache_middleware.rb:9:in `call'
Apr 15 14:05:13 lab unicorn[1547]: lib/gitlab/metrics/requests_rack_middleware.rb:49:in `call'
Apr 15 14:05:13 lab unicorn[1547]: lib/gitlab/middleware/release_env.rb:12:in `call'

@ghost
Copy link
Author

ghost commented Apr 15, 2020 •
edited by ghost

Pushing to repository gives this:

remote:
remote: ========================================================================
remote:
remote: Internal API error (500)
remote:
remote: ========================================================================
remote:
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

This seems have the same cause as the previous problem:

Apr 15 14:14:26 lab unicorn[1547]: I, [2020-04-15T14:14:26.027059 #1605]  INFO -- : Started POST "/api/v4/internal/allowed" for 127.0.0.1 at 2020-04-15 14:14:26 +0000
Apr 15 14:14:26 lab unicorn[1547]: /nix/store/z3zwr9hfd9n9p90vgb9g5q7wkw2cq9nl-ruby2.7.1-json-1.8.6/lib/ruby/gems/2.7.0/gems/json-1.8.6/lib/json/common.rb:155: warning: Using the last argument as keyword parameters is deprecated
Apr 15 14:14:26 lab unicorn[1547]: F, [2020-04-15T14:14:26.049237 #1605] FATAL -- :
Apr 15 14:14:26 lab unicorn[1547]: ArgumentError (wrong number of arguments (given 3, expected 1..2)):
Apr 15 14:14:26 lab unicorn[1547]:
Apr 15 14:14:26 lab unicorn[1547]: lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
Apr 15 14:14:26 lab unicorn[1547]: lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
Apr 15 14:14:26 lab unicorn[1547]: lib/gitlab/metrics/transaction.rb:62:in `run'
Apr 15 14:14:26 lab unicorn[1547]: lib/gitlab/metrics/rack_middleware.rb:17:in `call'
Apr 15 14:14:26 lab unicorn[1547]: lib/gitlab/request_profiler/middleware.rb:17:in `call'
Apr 15 14:14:26 lab unicorn[1547]: ee/lib/gitlab/jira/middleware.rb:19:in `call'
Apr 15 14:14:26 lab unicorn[1547]: lib/gitlab/middleware/go.rb:20:in `call'
Apr 15 14:14:26 lab unicorn[1547]: lib/gitlab/etag_caching/middleware.rb:13:in `call'
Apr 15 14:14:26 lab unicorn[1547]: lib/gitlab/middleware/multipart.rb:122:in `call'
Apr 15 14:14:26 lab unicorn[1547]: lib/gitlab/middleware/read_only/controller.rb:53:in `call'
Apr 15 14:14:26 lab unicorn[1547]: lib/gitlab/middleware/read_only.rb:18:in `call'
Apr 15 14:14:26 lab unicorn[1547]: lib/gitlab/middleware/basic_health_check.rb:25:in `call'
Apr 15 14:14:26 lab unicorn[1547]: lib/gitlab/middleware/request_context.rb:23:in `call'
Apr 15 14:14:26 lab unicorn[1547]: config/initializers/fix_local_cache_middleware.rb:9:in `call'
Apr 15 14:14:26 lab unicorn[1547]: lib/gitlab/metrics/requests_rack_middleware.rb:49:in `call'
Apr 15 14:14:26 lab unicorn[1547]: lib/gitlab/middleware/release_env.rb:12:in `call'

@flokli
Copy link
Contributor

flokli commented Apr 15, 2020

@petabyteboy can we bump to 12.8.9 for now?

Looks like 12.9.x requires a bit more work.

@ghost
Copy link
Author

ghost commented Apr 15, 2020

Sure! I've never actually done such an update, so how did you do it for 12.8.8? Does the update script support updating to the latest fix release somehow?

@flokli
Copy link
Contributor

flokli commented Apr 20, 2020 •
edited

Sorry, this comment actually fell off my radar.

The ./update.py update-data script has a --rev argument, unfortunately didn't have for update-all, meaning you'd have had to run the individual stages manually without any good reason.

I fixed this and ran the update, PR in #85623.

@flokli
Copy link
Contributor

flokli commented May 5, 2020

@petabyteboy can you update this to the latest release, 12.10.2?

@ghost
Copy link
Author

ghost commented May 5, 2020

Done, there are still errors in vgo2nix though and I had to use --keep-going (in which case it doesn't seem to print any errors).

@ghost ghost changed the title gitlab: 12.8.8 -> 12.9.3 gitlab: 12.8.10 -> 12.10.4 May 5, 2020
@flokli
Copy link
Contributor

flokli commented May 5, 2020

Thanks! So how are things looking with vgo2nix? Did you have to run it with your custom fork, or did it work out of the box? Any update.py fixes necessary?

@ghost
Copy link
Author

ghost commented May 5, 2020 •
edited by ghost

The vgo2nix fixes were merged upstream in nix-community/vgo2nix#37 and an update to include these fixes in nixpkgs is being tracked in #86933.

The additional logging also gives us a clue about the issues that continue to exist even with the updated vgo2nix:

panic: Error processing import path "cloud.google.com/go/bigquery": nix-prefetch-git --fetch-submodules --url https://code.googlesource.com/gocloud --rev v1.0.1 failed:
Initialized empty Git repository in /run/user/1000/git-checkout-tmp-PDsLyn1o/gocloud/.git/
fatal: couldn't find remote ref refs/tags/v1.0.1
[...]
 * [new tag]           bigquery/v1.0.1       -> bigquery/v1.0.1
[...]
fatal: empty string is not a valid pathspec. please use . instead if you meant to match all paths
Unable to checkout refs/tags/v1.0.1 from https://code.googlesource.com/gocloud.

So appearantly this Google gocloud repository uses some strange branch naming scheme that is not correctly picked up by vgo2nix.

To work around this you can add "--keep-going" to all invocations of vgo2nix in the update script. I am not sure how it handles those dependencies that fail, but it doesn't seem to have an effect on the functionality. I am not sure if this hacky workaround should be committed even, but I uploaded a diff you can apply locally here.

I just ran some manual tests in a VM and pushing to a repository as well as looking at the admin dashboard works without errors in this case. I will have to test if upgrading an older GitLab state will cause any issues to say if the issues I had previously seen are still there.

@flokli
Copy link
Contributor

flokli commented May 5, 2020

Okay, thanks for the detailed explanation. Let's keep this PR for now as it is. @globin, @Ma27, can you give this a try?

@ghost
Copy link
Author

ghost commented May 5, 2020 •
edited by ghost

I rebased once more to have the updated vgo2nix included in this branch, so you can just run the update script with NIX_PATH=nixpgks=$PWD and observe the remaining errors.

@ghost ghost changed the title gitlab: 12.8.10 -> 12.10.4 gitlab: 12.8.10 -> 12.10.6 May 18, 2020
@ghost
Copy link
Author

ghost commented May 18, 2020

Bumped to 12.10.6 and added one patch to gitaly to revert a change that prevents the gitlab shell secret from being read correctly. Someone should probably figure out why exactly it breaks but for now this fixes pushing to repositories.

@ghost
Copy link
Author

ghost commented May 19, 2020

Somehow the build for gitaly succeeds with the revert patch, but it does not fix the issue. Reverting the commit in a local clone of gitaly and changing the src of the gitaly package to the "lib.cleanSource /path/to/gitaly" fixes the issue though.

@flokli
Copy link
Contributor

flokli commented May 28, 2020 •
edited

Hmm, I tried bumping to 12.9.8 due to this, but even with vgo2nix --keep-going in the update.py, I ran into this:

fatal: empty string is not a valid pathspec. please use . instead if you meant to match all paths
Unable to checkout refs/tags/v1.0.1 from https://code.googlesource.com/gocloud.

@ghost
Copy link
Author

ghost commented May 28, 2020

There are multiple occurances of vgo2nix, did you add the parameter for all of them? I would even expect the error, but I can not reproduce it with --keep-going added to all occurances.

@flokli
Copy link
Contributor

flokli commented May 28, 2020

Indeed you're right - only added it to two of the occurences 🤦‍♂️.

Let's see if things go through now.

flokli added a commit to flokli/nixpkgs that referenced this pull request May 28, 2020
@flokli
gitlab: add --keep-going to vgo2nix
120201b 
vgo2nix is flaky, and it seems passing `--keep-going` to it gets it to
do the right thing (tm), so let's add this for now instead of not being
able to bump anything at all. See
NixOS#85293 for the full discussion.
@flokli
Copy link
Contributor

flokli commented May 28, 2020

It did succeed, but I'm not entirely sure about which of the other patches you prepared here are also required on that version of GitLab to get pushing to work again. Did that become necessary with 12.10.x, or is it already broken in 12.9.x?

@ghost
Copy link
Author

ghost commented May 28, 2020

Afaict this change which broke stuff was merged for 12.10.

@flokli
Copy link
Contributor

flokli commented May 28, 2020

Sorry, I need to bail out of this. I'm not currently operating any gitlab installation, nor maintaining the nixpkgs packages.

The security issues sound rather scary, @fpletz @globin @krav @talyz, could you take a look at this?

@globin globin self-assigned this May 31, 2020
@globin
Copy link
Member

globin commented May 31, 2020

@GrahamcOfBorg test gitlab

@globin globin merged commit b64205d into NixOS:master May 31, 2020
@ghost
Copy link
Author

ghost commented May 31, 2020

Thanks! I did not see this part.

@flokli flokli mentioned this pull request Jun 3, 2020
Merged
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fpletz fpletz Awaiting requested review from fpletz

@globin globin Awaiting requested review from globin

@krav krav Awaiting requested review from krav

@talyz talyz Awaiting requested review from talyz

Assignees

@globin globin

Labels
6.topic: nixos 8.has: module (update) 10.rebuild-darwin: 0 10.rebuild-linux: 1-10
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

bumping gitlab to 12.9.x fails
3 participants
@talyz @flokli @globin