Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gitlab: fix module and corresponding test #43844

Closed
wants to merge 3 commits into from

Conversation

binarin
Copy link
Contributor

@binarin binarin commented Jul 20, 2018

Motivation for this change

This fixes some things I've encountered while trying to use this module:

  • uploads symlink from store was broken, causing migrations on fresh
    installation to fail
  • Get lib folder from configured package, not from a global gitlab
    one
  • Increase memory limit, so the test will be able to pass
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@infinisil
Copy link
Member

@GrahamcOfBorg test gitlab

@GrahamcOfBorg
Copy link

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: tests.gitlab

Partial log (click to expand)

Cannot nix-instantiate `tests.gitlab' because:
error: attribute 'gitlab' in selection path 'tests.gitlab' not found

@GrahamcOfBorg
Copy link

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: tests.gitlab

Partial log (click to expand)

Cannot nix-instantiate `tests.gitlab' because:
error: attribute 'gitlab' in selection path 'tests.gitlab' not found

@infinisil
Copy link
Member

Ah, you can then uncomment this:

nixpkgs/nixos/release.nix

Lines 292 to 294 in 5f5e343

tests.gdk-pixbuf = callTest tests/gdk-pixbuf.nix {};
#tests.gitlab = callTest tests/gitlab.nix {};
tests.gitolite = callTest tests/gitolite.nix {};
, was disabled since 61816b0

@binarin binarin force-pushed the gitlab-fresh-installation-fixes branch from 8efbaa4 to 3c49ef6 Compare July 20, 2018 16:14
@binarin
Copy link
Contributor Author

binarin commented Jul 20, 2018

@GrahamcOfBorg test gitlab

@infinisil
Copy link
Member

@GrahamcOfBorg test gitlab

Currently only trusted users can run builds through ofborg, see https://github.com/NixOS/ofborg#trusted-users-vs-known-users

@GrahamcOfBorg
Copy link

Failure on x86_64-linux (full log)

Attempted: tests.gitlab

Partial log (click to expand)

gitlab: running command: systemctl --no-pager show "gitlab.service"
gitlab: exit status 0
error: action timed out after -1 seconds at /nix/store/id0qh30c3rms41b1f4gn3vqb76nhf58y-nixos-test-driver/lib/perl5/site_perl/Machine.pm line 238, <__ANONIO__> line 189537.
action timed out after -1 seconds at /nix/store/id0qh30c3rms41b1f4gn3vqb76nhf58y-nixos-test-driver/lib/perl5/site_perl/Machine.pm line 238, <__ANONIO__> line 189537.
cleaning up
killing gitlab (pid 593)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
builder for '/nix/store/wxpxsbvp039b6dcb666c2a999zsg71fk-vm-test-run-gitlab.drv' failed with exit code 255
error: build of '/nix/store/wxpxsbvp039b6dcb666c2a999zsg71fk-vm-test-run-gitlab.drv' failed

@GrahamcOfBorg
Copy link

Failure on aarch64-linux (full log)

Attempted: tests.gitlab

Partial log (click to expand)

gitlab: running command: systemctl --no-pager show "gitlab.service"
gitlab: exit status 0
error: action timed out after -1 seconds at /nix/store/pzwslya6jvbcmki7nznl1h7vv07cz1zg-nixos-test-driver/lib/perl5/site_perl/Machine.pm line 238, <__ANONIO__> line 189502.
action timed out after -1 seconds at /nix/store/pzwslya6jvbcmki7nznl1h7vv07cz1zg-nixos-test-driver/lib/perl5/site_perl/Machine.pm line 238, <__ANONIO__> line 189502.
cleaning up
killing gitlab (pid 627)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
builder for '/nix/store/gil5nva9k1bnsw3llzv2k94n4xq6x9vy-vm-test-run-gitlab.drv' failed with exit code 255
error: build of '/nix/store/gil5nva9k1bnsw3llzv2k94n4xq6x9vy-vm-test-run-gitlab.drv' failed

@binarin
Copy link
Contributor Author

binarin commented Jul 21, 2018

The test fails because systemd for some reason decides to enforce start timeout, despite it being explicitly set to infinity. Didn't saw this myself as I was testing on a faster machine.

So what to do now? Disable the test back again?

@flokli
Copy link
Contributor

flokli commented Jul 21, 2018

@binarin that's not systemd , but a hard-coded timeout in nixos-test-driver:

https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/test-driver/Machine.pm#L231

@Mic92
Copy link
Member

Mic92 commented Jul 21, 2018

I think we had problems with this test inside a VM because the p9 filesystem used to passthrough the nix store is not fast enough to load all ruby gems from gitlab in time.

@binarin
Copy link
Contributor Author

binarin commented Jul 21, 2018

@flokli systemd is also responsible, by killing a service few times and slowing everything even more:

gitlab# [  674.906967] systemd[1]: gitlab.service: Start-pre operation timed out. Terminating.
gitlab# [  674.989551] gitlab-pre-start[1492]: rake aborted!
gitlab# [  675.001382] gitlab-pre-start[1492]: SignalException: SIGTERM

@flokli
Copy link
Contributor

flokli commented Jul 22, 2018

That's really weird, ExecStartPre commands should also use timeout_start_usec

@binarin binarin force-pushed the gitlab-fresh-installation-fixes branch 2 times, most recently from b96767b to 0335a12 Compare July 23, 2018 08:27
@binarin
Copy link
Contributor Author

binarin commented Jul 23, 2018

Searching around the internet showed that infinity was incompatible with older versions of systemd. Looks like this should be already fixed/implemented in the version that is currently used, but as I have no other ideas, I'm changing this timeout to 1day. Can someone please kick-off the tests again?

@flokli
Copy link
Contributor

flokli commented Jul 23, 2018

@GrahamcOfBorg test gitlab

@GrahamcOfBorg
Copy link

Failure on aarch64-linux (full log)

Partial log (click to expand)

Merge failed

@GrahamcOfBorg
Copy link

Failure on x86_64-linux (full log)

Partial log (click to expand)

Merge failed

@flokli
Copy link
Contributor

flokli commented Jul 23, 2018

@binarin can you rebase?

@binarin
Copy link
Contributor Author

binarin commented Jul 23, 2018

@flokli I've tried to do so half an hour ago, but master was broken. Will try one more time.

This fixes some things I've encountered while trying to use this module:
- uploads symlink from store was broken, causing migrations on fresh
  installation to fail
- Get `lib` folder from configured package, not from a global `gitlab`
  one
- Increase memory limit, so the test will be able to pass
Looks like systemd is having some problem with respecting `infinity`
@binarin binarin force-pushed the gitlab-fresh-installation-fixes branch from 0335a12 to 0f30ee6 Compare September 25, 2018 11:59
@binarin
Copy link
Contributor Author

binarin commented Sep 25, 2018

@flokli Just rebased this. On my machine the test works again, took around 1000 seconds to succeed.

@flokli
Copy link
Contributor

flokli commented Sep 25, 2018

@GrahamcOfBorg test gitlab

@flokli
Copy link
Contributor

flokli commented Sep 25, 2018

Uff, 16 minutes is quite a long time.

Do you see what's taking so long here?
If it's I/O on the database, can we mount it with a tmpfs as storage for example?

@Mic92
Copy link
Member

Mic92 commented Sep 25, 2018

@flokli it is loading ruby files from p9 file system which is super slow.

@binarin
Copy link
Contributor Author

binarin commented Sep 25, 2018

The longest part of a test is applying all the migrations. I don't think that applying them on bare-metal was much faster.

@GrahamcOfBorg
Copy link

Failure on x86_64-linux (full log)

Attempted: tests.gitlab

Partial log (click to expand)

gitlab: running command: systemctl --no-pager show "gitlab.service"
gitlab: exit status 0
error: action timed out after -1 seconds at /nix/store/h5g1dpjzjzdk2vk15g18pvsbi5pvrdaq-nixos-test-driver/lib/perl5/site_perl/Machine.pm line 238, <__ANONIO__> line 190443.
action timed out after -1 seconds at /nix/store/h5g1dpjzjzdk2vk15g18pvsbi5pvrdaq-nixos-test-driver/lib/perl5/site_perl/Machine.pm line 238, <__ANONIO__> line 190443.
cleaning up
killing gitlab (pid 597)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
builder for '/nix/store/m1g5jbkcrznqvph06hjih0i65h7rs680-vm-test-run-gitlab.drv' failed with exit code 255
error: build of '/nix/store/m1g5jbkcrznqvph06hjih0i65h7rs680-vm-test-run-gitlab.drv' failed

Because it takes at least 10 minutes even on a good bare-metal server.
@binarin
Copy link
Contributor Author

binarin commented Sep 25, 2018

OK, I've found why systemd was killing a unit while despite of infinity timeout. It's because the test was overriding that infinity to 10 minutes.

I've pushed a change increasing this to 30 minutes, let's try again - maybe it'll be able to finish when not experiencing random kills.

@GrahamcOfBorg
Copy link

Failure on aarch64-linux (full log)

Attempted: tests.gitlab

Partial log (click to expand)

gitlab: running command: systemctl --no-pager show "gitlab.service"
gitlab: exit status 0
error: action timed out after -1 seconds at /nix/store/4959zlsg9mdg454k4232awf55ij9xb0d-nixos-test-driver/lib/perl5/site_perl/Machine.pm line 238, <__ANONIO__> line 190541.
action timed out after -1 seconds at /nix/store/4959zlsg9mdg454k4232awf55ij9xb0d-nixos-test-driver/lib/perl5/site_perl/Machine.pm line 238, <__ANONIO__> line 190541.
cleaning up
killing gitlab (pid 631)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
builder for '/nix/store/icxxgy0wv11sj472162vz9hqk78glxgi-vm-test-run-gitlab.drv' failed with exit code 255
error: build of '/nix/store/icxxgy0wv11sj472162vz9hqk78glxgi-vm-test-run-gitlab.drv' failed

@flokli
Copy link
Contributor

flokli commented Sep 26, 2018

@binarin Thanks for your work so far!

I gave this a try, using nix-build nixos/tests/gitlab.nix -A driver, here are my findings:

While applying migrations, gitlab-sidekiq.service regularily spams console, failing with a permission denied erorr:

gitlab# [   97.200418] systemd[1]: Started gitlab-sidekiq.service.
gitlab# [  113.233420] sidekiq[1265]: Permission denied - bs_fetch:atomic_write_cache_file:open
gitlab# [  113.237428] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/compile_cache/iseq.rb:37:in `fetch'
gitlab# [  113.247379] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/compile_cache/iseq.rb:37:in `load_iseq'
gitlab# [  113.252325] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
gitlab# [  113.256758] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
gitlab# [  113.263135] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
gitlab# [  113.272978] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
gitlab# [  113.290365] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
gitlab# [  113.296948] sidekiq[1265]: /nix/store/ah528qzjgfw8rsbp26ckbh9wchnkm0w8-ruby2.5.1-activesupport-4.2.10/lib/ruby/gems/2.5.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `block in require'
gitlab# [  113.302223] sidekiq[1265]: /nix/store/ah528qzjgfw8rsbp26ckbh9wchnkm0w8-ruby2.5.1-activesupport-4.2.10/lib/ruby/gems/2.5.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency'
gitlab# [  113.307957] sidekiq[1265]: /nix/store/ah528qzjgfw8rsbp26ckbh9wchnkm0w8-ruby2.5.1-activesupport-4.2.10/lib/ruby/gems/2.5.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `require'
gitlab# [  113.315895] sidekiq[1265]: /nix/store/693ms2p8m6d092bzwqv68ij93rlj1yba-ruby2.5.1-sprockets-3.7.2/lib/ruby/gems/2.5.0/gems/sprockets-3.7.2/lib/sprockets/directive_processor.rb:2:in `<main>'
gitlab# [  113.321658] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
gitlab# [  113.327718] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
gitlab# [  113.335953] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
gitlab# [  113.339921] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
gitlab# [  113.344361] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
gitlab# [  113.352667] sidekiq[1265]: /nix/store/ah528qzjgfw8rsbp26ckbh9wchnkm0w8-ruby2.5.1-activesupport-4.2.10/lib/ruby/gems/2.5.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `block in require'
gitlab# [  113.360258] sidekiq[1265]: /nix/store/ah528qzjgfw8rsbp26ckbh9wchnkm0w8-ruby2.5.1-activesupport-4.2.10/lib/ruby/gems/2.5.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency'
gitlab# [  113.365941] sidekiq[1265]: /nix/store/ah528qzjgfw8rsbp26ckbh9wchnkm0w8-ruby2.5.1-activesupport-4.2.10/lib/ruby/gems/2.5.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `require'
gitlab# [  113.370276] sidekiq[1265]: /nix/store/693ms2p8m6d092bzwqv68ij93rlj1yba-ruby2.5.1-sprockets-3.7.2/lib/ruby/gems/2.5.0/gems/sprockets-3.7.2/lib/sprockets.rb:96:in `<module:Sprockets>'
gitlab# [  113.375756] sidekiq[1265]: /nix/store/693ms2p8m6d092bzwqv68ij93rlj1yba-ruby2.5.1-sprockets-3.7.2/lib/ruby/gems/2.5.0/gems/sprockets-3.7.2/lib/sprockets.rb:9:in `<main>'
gitlab# [  113.377880] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
gitlab# [  113.387837] systemd[1]: gitlab-sidekiq.service: Main process exited, code=exited, status=1/FAILURE
gitlab# [  113.395605] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
gitlab# [  113.412835] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
gitlab# [  113.481093] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
gitlab# [  113.498426] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
gitlab# [  113.511131] sidekiq[1265]: /nix/store/ah528qzjgfw8rsbp26ckbh9wchnkm0w8-ruby2.5.1-activesupport-4.2.10/lib/ruby/gems/2.5.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `block in require'
gitlab# [  113.529781] sidekiq[1265]: /nix/store/ah528qzjgfw8rsbp26ckbh9wchnkm0w8-ruby2.5.1-activesupport-4.2.10/lib/ruby/gems/2.5.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency'
gitlab# [  113.545592] sidekiq[1265]: /nix/store/ah528qzjgfw8rsbp26ckbh9wchnkm0w8-ruby2.5.1-activesupport-4.2.10/lib/ruby/gems/2.5.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `require'
gitlab# [  113.565127] sidekiq[1265]: /nix/store/53cablx3lxa4vha1wkfzpd5ijw88ppdy-ruby2.5.1-sprockets-rails-3.2.1/lib/ruby/gems/2.5.0/gems/sprockets-rails-3.2.1/lib/sprockets/railtie.rb:6:in `<main>'
gitlab# [  113.581952] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
gitlab# [  113.593198] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
gitlab# [  113.603560] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
gitlab# [  113.643328] systemd[1]: gitlab-sidekiq.service: Failed with result 'exit-code'.
gitlab# [  113.688595] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
gitlab# [  113.723116] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
gitlab# [  113.730087] sidekiq[1265]: /nix/store/ah528qzjgfw8rsbp26ckbh9wchnkm0w8-ruby2.5.1-activesupport-4.2.10/lib/ruby/gems/2.5.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `block in require'
gitlab# [  113.743557] sidekiq[1265]: /nix/store/ah528qzjgfw8rsbp26ckbh9wchnkm0w8-ruby2.5.1-activesupport-4.2.10/lib/ruby/gems/2.5.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency'
gitlab# [  113.758905] sidekiq[1265]: /nix/store/ah528qzjgfw8rsbp26ckbh9wchnkm0w8-ruby2.5.1-activesupport-4.2.10/lib/ruby/gems/2.5.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `require'
gitlab# [  113.774601] sidekiq[1265]: /nix/store/wyl91x9y20kwdqpgmaz2mwbdbxz04xfi-ruby2.5.1-railties-4.2.10/lib/ruby/gems/2.5.0/gems/railties-4.2.10/lib/rails/all.rb:13:in `block in <main>'
gitlab# [  113.794939] sidekiq[1265]: /nix/store/wyl91x9y20kwdqpgmaz2mwbdbxz04xfi-ruby2.5.1-railties-4.2.10/lib/ruby/gems/2.5.0/gems/railties-4.2.10/lib/rails/all.rb:11:in `each'
gitlab# [  113.808416] sidekiq[1265]: /nix/store/wyl91x9y20kwdqpgmaz2mwbdbxz04xfi-ruby2.5.1-railties-4.2.10/lib/ruby/gems/2.5.0/gems/railties-4.2.10/lib/rails/all.rb:11:in `<main>'
gitlab# [  113.822507] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
gitlab# [  113.838037] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
gitlab# [  113.849994] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
gitlab# [  113.878949] systemd[1]: gitlab-sidekiq.service: Service RestartSec=100ms expired, scheduling restart.
gitlab# [  113.923631] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
gitlab# [  113.941539] sidekiq[1265]: /nix/store/aimlg10brvcfcqlmwvm5d4m3zbay40f4-ruby2.5.1-bootsnap-1.3.1/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
gitlab# [  113.951414] sidekiq[1265]: /nix/store/ah528qzjgfw8rsbp26ckbh9wchnkm0w8-ruby2.5.1-activesupport-4.2.10/lib/ruby/gems/2.5.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `block in require'
gitlab# [  113.967916] sidekiq[1265]: /nix/store/ah528qzjgfw8rsbp26ckbh9wchnkm0w8-ruby2.5.1-activesupport-4.2.10/lib/ruby/gems/2.5.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency'
gitlab# [  113.986702] sidekiq[1265]: /nix/store/ah528qzjgfw8rsbp26ckbh9wchnkm0w8-ruby2.5.1-activesupport-4.2.10/lib/ruby/gems/2.5.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `require'
gitlab# [  114.003268] sidekiq[1265]: /nix/store/89pbkf4ryrqcmpf5w020z1zashbgb3va-gitlab-11.3.0/share/gitlab/config/application.rb:3:in `<top (required)>'
gitlab# [  114.018386] sidekiq[1265]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/cli.rb:264:in `require'
gitlab# [  114.032355] sidekiq[1265]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/cli.rb:264:in `boot_system'
gitlab# [  114.050834] sidekiq[1265]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/cli.rb:53:in `run'
gitlab# [  114.062219] sidekiq[1265]: /nix/store/l12alvdvd2ij76fhpijni8qcjcxwnfz7-gitlab-env-11.3.0/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/bin/sidekiq:12:in `<top (required)>'
gitlab# [  114.073457] sidekiq[1265]: /nix/store/l12alvdvd2ij76fhpijni8qcjcxwnfz7-gitlab-env-11.3.0/bin/sidekiq:18:in `load'
gitlab# [  114.083261] sidekiq[1265]: /nix/store/l12alvdvd2ij76fhpijni8qcjcxwnfz7-gitlab-env-11.3.0/bin/sidekiq:18:in `<main>'
gitlab# [  114.090104] systemd[1]: gitlab-sidekiq.service: Scheduled restart job, restart counter is at 6.

That's might be related to parts of ${cfg.statePath} not yet existing, but used via -P ${cfg.statePath}/tmp/sidekiq.pid. Could you move the sidekiq-related "state dir setup" inside the sidekiq unit?

Afterwards, it seems to recover, but has a problem with the following task:

gitlab# [  758.446692] sidekiq[1406]: 2018-09-26T23:00:21.460Z 1406 TID-ovammezba BackgroundMigrationWorker JID-6048c27ce510bda4befe7cf2 INFO: start
gitlab# [  758.488973] sidekiq[1406]: find: File system loop detected; ‘/var/gitlab/state/uploads/uploads’ is part of the same file system loop as ‘/var/gitlab/state/uploads’.
gitlab# [  758.493137] sidekiq[1406]: 2018-09-26T23:00:21.505Z 1406 TID-ovammezba BackgroundMigrationWorker JID-6048c27ce510bda4befe7cf2 INFO: fail: 0.046 sec
gitlab# [  758.494656] sidekiq[1406]: 2018-09-26T23:00:21.505Z 1406 TID-ovammezba WARN: {"context":"Job raised exception","job":{"class":"BackgroundMigrationWorker","args":["PrepareUntrackedUploads",[]],"retry":3,"queue":"background_migration","jid":"6048c27ce510bda4befe7cf2","created_at":1538002737.9257424,"enqueued_at":1538002821.4594133},"jobstr":"{\"class\":\"BackgroundMigrationWorker\",\"args\":[\"PrepareUntrackedUploads\",[]],\"retry\":3,\"queue\":\"background_migration\",\"jid\":\"6048c27ce510bda4befe7cf2\",\"created_at\":1538002737.9257424,\"enqueued_at\":1538002821.4594133}"}
gitlab# [  758.508607] sidekiq[1406]: 2018-09-26T23:00:21.505Z 1406 TID-ovammezba WARN: RuntimeError: Find command failed
gitlab# [  758.517315] sidekiq[1406]: 2018-09-26T23:00:21.505Z 1406 TID-ovammezba WARN: /nix/store/89pbkf4ryrqcmpf5w020z1zashbgb3va-gitlab-11.3.0/share/gitlab/lib/gitlab/background_migration/prepare_untracked_uploads.rb:84:in `block in each_file_batch'
gitlab# [  758.526272] sidekiq[1406]: /nix/store/42lf7caji1bykgrzmf22a58sqyb9knjj-ruby-2.5.1/lib/ruby/2.5.0/open3.rb:205:in `popen_run'
gitlab# [  758.527563] sidekiq[1406]: /nix/store/42lf7caji1bykgrzmf22a58sqyb9knjj-ruby-2.5.1/lib/ruby/2.5.0/open3.rb:147:in `popen2'
gitlab# [  758.529227] sidekiq[1406]: /nix/store/89pbkf4ryrqcmpf5w020z1zashbgb3va-gitlab-11.3.0/share/gitlab/lib/gitlab/background_migration/prepare_untracked_uploads.rb:81:in `each_file_batch'
gitlab# [  758.538310] sidekiq[1406]: /nix/store/89pbkf4ryrqcmpf5w020z1zashbgb3va-gitlab-11.3.0/share/gitlab/lib/gitlab/background_migration/prepare_untracked_uploads.rb:73:in `store_untracked_file_paths'
gitlab# [  758.547258] sidekiq[1406]: /nix/store/89pbkf4ryrqcmpf5w020z1zashbgb3va-gitlab-11.3.0/share/gitlab/lib/gitlab/background_migration/prepare_untracked_uploads.rb:44:in `perform'
gitlab# [  758.548887] sidekiq[1406]: /nix/store/89pbkf4ryrqcmpf5w020z1zashbgb3va-gitlab-11.3.0/share/gitlab/lib/gitlab/background_migration.rb:49:in `perform'
gitlab# [  758.551226] sidekiq[1406]: /nix/store/89pbkf4ryrqcmpf5w020z1zashbgb3va-gitlab-11.3.0/share/gitlab/app/workers/background_migration_worker.rb:26:in `perform'
gitlab# [  758.556303] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:187:in `execute_job'
gitlab# [  758.568279] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:169:in `block (2 levels) in process'
gitlab# [  758.570047] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
gitlab# [  758.574960] sidekiq[1406]: /nix/store/89pbkf4ryrqcmpf5w020z1zashbgb3va-gitlab-11.3.0/share/gitlab/lib/gitlab/sidekiq_status/server_middleware.rb:5:in `call'
gitlab# [  758.586283] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
gitlab# [  758.587987] sidekiq[1406]: /nix/store/89pbkf4ryrqcmpf5w020z1zashbgb3va-gitlab-11.3.0/share/gitlab/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:6:in `call'
gitlab# [  758.589533] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
gitlab# [  758.598370] sidekiq[1406]: /nix/store/89pbkf4ryrqcmpf5w020z1zashbgb3va-gitlab-11.3.0/share/gitlab/lib/gitlab/sidekiq_middleware/shutdown.rb:52:in `call'
gitlab# [  758.620089] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
gitlab# [  758.634264] sidekiq[1406]: /nix/store/89pbkf4ryrqcmpf5w020z1zashbgb3va-gitlab-11.3.0/share/gitlab/lib/gitlab/metrics/sidekiq_middleware.rb:13:in `block in call'
gitlab# [  758.635768] sidekiq[1406]: /nix/store/89pbkf4ryrqcmpf5w020z1zashbgb3va-gitlab-11.3.0/share/gitlab/lib/gitlab/metrics/transaction.rb:53:in `run'
gitlab# [  758.640712] sidekiq[1406]: /nix/store/89pbkf4ryrqcmpf5w020z1zashbgb3va-gitlab-11.3.0/share/gitlab/lib/gitlab/metrics/sidekiq_middleware.rb:13:in `call'
gitlab# [  758.643499] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
gitlab# [  758.648665] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/server/active_record.rb:16:in `call'
gitlab# [  758.653162] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
gitlab# [  758.657887] sidekiq[1406]: /nix/store/ciz4h49nvzpmf2yw64sqdfbiqklkn7r6-ruby2.5.1-sentry-raven-2.7.2/lib/ruby/gems/2.5.0/gems/sentry-raven-2.7.2/lib/raven/integrations/sidekiq.rb:9:in `call'
gitlab# [  758.660205] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
gitlab# [  758.666882] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:133:in `invoke'
gitlab# [  758.671660] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:168:in `block in process'
gitlab# [  758.676634] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:139:in `block (6 levels) in dispatch'
gitlab# [  758.680901] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_retry.rb:98:in `local'
gitlab# [  758.684892] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:138:in `block (5 levels) in dispatch'
gitlab# [  758.689291] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq.rb:36:in `block in <module:Sidekiq>'
gitlab# [  758.693471] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:134:in `block (4 levels) in dispatch'
gitlab# [  758.699228] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:199:in `stats'
gitlab# [  758.705899] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:129:in `block (3 levels) in dispatch'
gitlab# [  758.710446] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_logger.rb:8:in `call'
gitlab# [  758.714772] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:128:in `block (2 levels) in dispatch'
gitlab# [  758.719352] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_retry.rb:73:in `global'
gitlab# [  758.723544] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:127:in `block in dispatch'
gitlab# [  758.727546] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/logging.rb:48:in `with_context'
gitlab# [  758.731631] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
gitlab# [  758.736119] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:126:in `dispatch'
gitlab# [  758.740291] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:167:in `process'
gitlab# [  758.744241] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:85:in `process_one'
gitlab# [  758.748616] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:73:in `run'
gitlab# [  758.750748] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:16:in `watchdog'
gitlab# [  758.757695] sidekiq[1406]: /nix/store/wsd3h2529w41qwqygqzwmds7jnh6n7sn-ruby2.5.1-sidekiq-5.1.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:25:in `block in safe_thread'

find: File system loop detected; ‘/var/gitlab/state/uploads/uploads’ is part of the same file system loop as ‘/var/gitlab/state/uploads’.

I did check, /var/gitlab/state/uploads/uploads is indeed a symlink to /var/gitlab/state/uploads/uploads, so this can't work.

On initial vm boot, gitlab-workhorse failed to start, it needed a manual restart after gitlab.service was started. This is most likely due to the units working directory being configured to gitlabEnv.HOME, which translates to ${cfg.statePath}/home, which also didn't exist at that time.

I'm not exactly sure about whether we should do parts of the service-specific state directory structure setups on each unit that uses it, or have some gitlab-setup unit doing that part of the setup, and needs to finish before the rest starts.

@flokli
Copy link
Contributor

flokli commented Oct 4, 2018

@binarin ping ;-)

@binarin
Copy link
Contributor Author

binarin commented Oct 4, 2018

@flokli I hope I'll be able to get back to this issue within a few weeks.

@globin
Copy link
Member

globin commented Nov 2, 2018

superseded by #49658

@globin globin closed this Nov 2, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants