Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nixos/hadoop: add hadoop module (hdfs, yarn) #41381

Merged
merged 1 commit into from Jun 30, 2018
Merged

nixos/hadoop: add hadoop module (hdfs, yarn) #41381

merged 1 commit into from Jun 30, 2018

Conversation

aespinosa
Copy link
Contributor

Motivation for this change

Create initial Hadoop modules: HDFS and YARN. Enough to make a small working Hadoop cluster.

The modules are currently minimal that it doesn't include other HDFS, YARN or MapReduce components to make it "production-ish"

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

@aespinosa
Copy link
Contributor Author

just realized that the NixOS tests only works in Hadoop 3.0 onwards. I originally cherry-picked this PR from #40712 (which is Hadoop 3.1.0)

Hadoop 3.0.0 changed the default ports in HDFS and YARN.

@aespinosa aespinosa force-pushed the hadoop-modules branch 2 times, most recently from 2a0cedb to 35eaaf8 Compare June 2, 2018 15:23
@aespinosa
Copy link
Contributor Author

@GrahamcOfBorg test hadoop.hdfs

@aespinosa
Copy link
Contributor Author

@GrahamcOfBorg test hadoop.yarn

@matthewbauer
Copy link
Member

@GrahamcOfBorg test hadoop.yarn hadoop.hdfs

@GrahamcOfBorg
Copy link

Success on aarch64-linux

Attempted: tests.hadoop.yarn, tests.hadoop.hdfs

No partial log is available.

@GrahamcOfBorg
Copy link

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: tests.hadoop.yarn, tests.hadoop.hdfs

Partial log (click to expand)

while evaluating 'fold'' at /home/ofborg/ofborg3/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c3/lib/lists.nix:39:15, called from /home/ofborg/ofborg3/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c3/lib/lists.nix:43:8:
while evaluating 'discoverForSystem' at /home/ofborg/ofborg3/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c3/nixos/release.nix:32:25, called from undefined position:
while evaluating 'mapAttrs' at /home/ofborg/ofborg3/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c3/lib/attrsets.nix:198:17, called from /home/ofborg/ofborg3/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c3/nixos/release.nix:32:33:
while evaluating 'discover' at /home/ofborg/ofborg3/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c3/nixos/release.nix:28:16, called from /home/ofborg/ofborg3/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c3/nixos/release.nix:34:9:
while evaluating 'mapAttrs' at /home/ofborg/ofborg3/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c3/lib/attrsets.nix:198:17, called from /home/ofborg/ofborg3/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c3/nixos/release.nix:30:8:
while evaluating 'filterAttrs' at /home/ofborg/ofborg3/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c3/lib/attrsets.nix:115:23, called from /home/ofborg/ofborg3/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c3/nixos/release.nix:29:18:
while evaluating 'concatMap' at /home/ofborg/ofborg3/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c3/lib/lists.nix:104:18, called from /home/ofborg/ofborg3/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c3/lib/attrsets.nix:116:18:
while evaluating anonymous function at /home/ofborg/ofborg3/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c3/lib/attrsets.nix:116:29, called from undefined position:
value is a boolean while a set was expected, at /home/ofborg/ofborg3/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c3/lib/attrsets.nix:116:62

@matthewbauer
Copy link
Member

@GrahamcOfBorg test hadoop.yarn

@GrahamcOfBorg
Copy link

Success on aarch64-linux

Attempted: tests.hadoop.yarn

No partial log is available.

@GrahamcOfBorg
Copy link

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: tests.hadoop.yarn

Partial log (click to expand)

while evaluating 'fold'' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/lists.nix:39:15, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/lists.nix:43:8:
while evaluating 'discoverForSystem' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/nixos/release.nix:32:25, called from undefined position:
while evaluating 'mapAttrs' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:198:17, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/nixos/release.nix:32:33:
while evaluating 'discover' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/nixos/release.nix:28:16, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/nixos/release.nix:34:9:
while evaluating 'mapAttrs' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:198:17, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/nixos/release.nix:30:8:
while evaluating 'filterAttrs' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:115:23, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/nixos/release.nix:29:18:
while evaluating 'concatMap' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/lists.nix:104:18, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:116:18:
while evaluating anonymous function at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:116:29, called from undefined position:
value is a boolean while a set was expected, at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:116:62

@matthewbauer
Copy link
Member

@GrahamcOfBorg test hadoop.hdfs

@GrahamcOfBorg
Copy link

Success on aarch64-linux

Attempted: tests.hadoop.hdfs

No partial log is available.

@GrahamcOfBorg
Copy link

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: tests.hadoop.hdfs

Partial log (click to expand)

while evaluating 'fold'' at /home/ofborg/ofborg1/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c1/lib/lists.nix:39:15, called from /home/ofborg/ofborg1/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c1/lib/lists.nix:43:8:
while evaluating 'discoverForSystem' at /home/ofborg/ofborg1/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c1/nixos/release.nix:32:25, called from undefined position:
while evaluating 'mapAttrs' at /home/ofborg/ofborg1/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c1/lib/attrsets.nix:198:17, called from /home/ofborg/ofborg1/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c1/nixos/release.nix:32:33:
while evaluating 'discover' at /home/ofborg/ofborg1/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c1/nixos/release.nix:28:16, called from /home/ofborg/ofborg1/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c1/nixos/release.nix:34:9:
while evaluating 'mapAttrs' at /home/ofborg/ofborg1/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c1/lib/attrsets.nix:198:17, called from /home/ofborg/ofborg1/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c1/nixos/release.nix:30:8:
while evaluating 'filterAttrs' at /home/ofborg/ofborg1/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c1/lib/attrsets.nix:115:23, called from /home/ofborg/ofborg1/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c1/nixos/release.nix:29:18:
while evaluating 'concatMap' at /home/ofborg/ofborg1/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c1/lib/lists.nix:104:18, called from /home/ofborg/ofborg1/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c1/lib/attrsets.nix:116:18:
while evaluating anonymous function at /home/ofborg/ofborg1/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c1/lib/attrsets.nix:116:29, called from undefined position:
value is a boolean while a set was expected, at /home/ofborg/ofborg1/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/builder-7c6f434c1/lib/attrsets.nix:116:62

@matthewbauer
Copy link
Member

Looks like there is an odd eval error. Does the test run locally>

@aespinosa
Copy link
Contributor Author

let me check on the fresh rebase. it was building locally before

@aespinosa
Copy link
Contributor Author

@matthewbauer it should evaluate now

@matthewbauer
Copy link
Member

@GrahamcOfBorg test hadoop.yarn hadoop.hdfs

@GrahamcOfBorg
Copy link

Success on aarch64-linux

Attempted: tests.hadoop.yarn, tests.hadoop.hdfs

No partial log is available.

@GrahamcOfBorg
Copy link

Failure on x86_64-linux (full log)

Attempted: tests.hadoop.yarn, tests.hadoop.hdfs

Partial log (click to expand)

namenode: exit status 1
error: action timed out after -1 seconds at /nix/store/h0snxsanpvr5qcf91f3a6wfs31rcrcmy-nixos-test-driver/lib/perl5/site_perl/Machine.pm line 238, <__ANONIO__> line 3263.
action timed out after -1 seconds at /nix/store/h0snxsanpvr5qcf91f3a6wfs31rcrcmy-nixos-test-driver/lib/perl5/site_perl/Machine.pm line 238, <__ANONIO__> line 3263.
cleaning up
killing datanode (pid 636)
killing namenode (pid 593)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
builder for '/nix/store/782rqmafq438yibkilidb79g4649dwhg-vm-test-run-unnamed.drv' failed with exit code 255
error: build of '/nix/store/782rqmafq438yibkilidb79g4649dwhg-vm-test-run-unnamed.drv' failed

@aespinosa
Copy link
Contributor Author

@matthewbauer I think i fixed the last missing part of my rebase. tests should pass now.

@matthewbauer
Copy link
Member

@GrahamcOfBorg test hadoop.yarn hadoop.hdfs

@GrahamcOfBorg
Copy link

Success on aarch64-linux

Attempted: tests.hadoop.yarn, tests.hadoop.hdfs

No partial log is available.

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: tests.hadoop.yarn, tests.hadoop.hdfs

Partial log (click to expand)

nodemanager: running command: sync
nodemanager: exit status 0
test script finished in 82.86s
cleaning up
killing resourcemanager (pid 593)
killing nodemanager (pid 605)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
/nix/store/zfrax2xhppyl9jyjgjqq2z05d4g886lm-vm-test-run-unnamed
/nix/store/8h9wmr51m5044xwrx7sn0gqln9fl4lc3-vm-test-run-unnamed

@matthewbauer matthewbauer merged commit 060a98e into NixOS:master Jun 30, 2018
@aespinosa aespinosa deleted the hadoop-modules branch June 30, 2018 23:26
aszlig added a commit that referenced this pull request Jul 2, 2018
@aszlig
nixos/hadoop: Replace users.extra{Users,Groups}
6e47117 
In fff5923 all occurences of
users.extraUsers and users.extraGroups have been changed tree-wide to
users.users and users.group. In the meantime the hadoop modules were
introduced via #41381 (060a98e).

Unfortunately those modules still use users.extraUsers, which has been
renamed a long time ago (14321ae, about
three years from now), so let's actually rename it accordingly as well.

Signed-off-by: aszlig <aszlig@nix.build>
Cc: @matthewbauer, @aespinosa
aszlig
aszlig reviewed Jul 2, 2018
View reviewed changes
nixos/modules/services/cluster/hadoop/default.nix

config = mkMerge [
(mkIf (builtins.hasAttr "yarn" config.users.extraUsers ||
builtins.hasAttr "hdfs" config.users.extraUsers ) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aespinosa: How come this is still using extraUsers/extraGroups? Do we have some outdated documentation somewhere we need to fix?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aszlig aszlig aszlig left review comments

Assignees
No one assigned
Labels
6.topic: nixos 8.has: module (update) 10.rebuild-darwin: 0 10.rebuild-linux: 0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@aespinosa @matthewbauer @GrahamcOfBorg @aszlig