Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

clickhouse: 18.16.1 -> 19.6.2.11 #61988

Merged
merged 1 commit into from May 26, 2019
Merged

clickhouse: 18.16.1 -> 19.6.2.11 #61988

merged 1 commit into from May 26, 2019

Conversation

teh
Copy link
Contributor

@teh teh commented May 24, 2019

Motivation for this change

Update clickhouse - unfortunately I don't know whether the on-disk representation is backwards compatible. I.e. please check before upgrading on live data.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)

Some binaries not working but didn't work before. The main tools, server and client are working.

  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Assured whether relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@FRidh FRidh requested a review from orivej-nixos May 25, 2019 10:48
orivej
orivej suggested changes May 25, 2019
View reviewed changes
Copy link
Contributor

@orivej orivej left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know whether the on-disk representation is backwards compatible.

It is.

pkgs/servers/clickhouse/default.nix Outdated Show resolved Hide resolved
pkgs/servers/clickhouse/default.nix Outdated Show resolved Hide resolved
pkgs/servers/clickhouse/default.nix Outdated Show resolved Hide resolved
@teh
Copy link
Contributor Author

teh commented May 25, 2019

Thanks for the review!

@aanderse
Copy link
Member

@GrahamcOfBorg test clickhouse

@risicle
Copy link
Contributor

risicle commented May 26, 2019

nox-review builds for me, non-nixos linux x86_64

@aanderse
Copy link
Member

About ~20 minutes in I gave up trying to run the test... anyone able to run the test and report back?

@teh
Copy link
Contributor Author

teh commented May 26, 2019

@aanderse

$ nix-build ./nixos/tests/clickhouse.nix
...
Set max number of file descriptors to 4096 (was 1024).
machine# [   60.889739] clickhouse-server[777]: 2019.05.26 19:06:45.405200 [ 1 ] {} <Debug> Application: Initializing DateLUT.
machine# [   60.896540] clickhouse-server[777]: 2019.05.26 19:06:45.405849 [ 1 ] {} <Trace> Application: Initialized DateLUT with time zone `UTC'.
machine# [   60.900125] clickhouse-server[777]: 2019.05.26 19:06:45.414505 [ 1 ] {} <Debug> Application: Configuration parameter 'interserver_http_host' doesn't exist or exists and empty. Will use 'machine' as replica host.
machine# [   61.179772] clickhouse-server[777]: 2019.05.26 19:06:45.853295 [ 1 ] {} <Debug> ConfigReloader: Loading config `/nix/store/nj2p079qkmlp40dy1n3r10kb8g877j87-clickhouse-19.6.2.11/etc/clickhouse-server/users.xml'
machine# [   61.196542] clickhouse-server[777]: Include not found: networks
machine: running command: nc -z localhost 9000
machine# [   61.544984] clickhouse-server[777]: 2019.05.26 19:06:46.213080 [ 1 ] {} <Information> Application: Uncompressed cache size was lowered to 179.53 MiB because the system has low amount of memory
machine: exit status 1
(0.21 seconds)
machine# [   61.566967] clickhouse-server[777]: 2019.05.26 19:06:46.239934 [ 1 ] {} <Information> Application: Mark cache size was lowered to 179.53 MiB because the system has low amount of memory
machine# [   61.573957] clickhouse-server[777]: 2019.05.26 19:06:46.240211 [ 1 ] {} <Information> Application: Loading metadata from /var/lib/clickhouse/
machine# [   61.902134] clickhouse-server[777]: 2019.05.26 19:06:46.575323 [ 1 ] {} <Debug> Application: Loaded metadata.
machine# [   61.914779] clickhouse-server[777]: 2019.05.26 19:06:46.587836 [ 1 ] {} <Information> BackgroundSchedulePool: Create BackgroundSchedulePool with 16 threads
machine# [   62.043631] capability: warning: `BackgrSchedPool' uses 32-bit capabilities (legacy support in use)
machine# [   61.933534] clickhouse-server[777]: 2019.05.26 19:06:46.607508 [ 1 ] {} <Information> Application: It looks like the process has no CAP_NET_ADMIN capability, 'taskstats' performance statistics will be disabled. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_net_admin=+ep /usr/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems. It also doesn't work if you run clickhouse-server inside network namespace as it happens in some containers.
machine# [   62.133475] clickhouse-server[777]: 2019.05.26 19:06:46.616771 [ 1 ] {} <Information> Application: Listening http://[::1]:8123
machine# [   62.140928] clickhouse-server[777]: 2019.05.26 19:06:46.616939 [ 1 ] {} <Information> Application: Listening tcp: [::1]:9000
machine# [   62.145188] clickhouse-server[777]: 2019.05.26 19:06:46.617029 [ 1 ] {} <Information> Application: Listening interserver http: [::1]:9009
machine# [   62.150796] clickhouse-server[777]: 2019.05.26 19:06:46.617140 [ 1 ] {} <Information> Application: Listening http://127.0.0.1:8123
machine# [   62.156123] clickhouse-server[777]: 2019.05.26 19:06:46.617225 [ 1 ] {} <Information> Application: Listening tcp: 127.0.0.1:9000
machine# [   62.159425] clickhouse-server[777]: 2019.05.26 19:06:46.617299 [ 1 ] {} <Information> Application: Listening interserver http: 127.0.0.1:9009
machine# [   62.162486] clickhouse-server[777]: 2019.05.26 19:06:46.618060 [ 1 ] {} <Information> Application: Available RAM: 359.06 MiB; physical cores: 1; logical cores: 1.
machine# [   62.165544] clickhouse-server[777]: 2019.05.26 19:06:46.618094 [ 1 ] {} <Information> Application: Ready for connections.
machine: running command: nc -z localhost 9000
machine# [   62.604793] clickhouse-server[777]: 2019.05.26 19:06:47.277455 [ 25 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:52410
machine# Connection to localhost 9000 port [tcp/cslistener] succeeded!
machine# [   62.694960] clickhouse-server[777]: 2019.05.26 19:06:47.368849 [ 25 ] {} <Warning> TCPHandler: Client has not sent any data.
machine# [   62.697849] clickhouse-server[777]: 2019.05.26 19:06:47.368988 [ 25 ] {} <Information> TCPHandler: Done processing connection.
machine: exit status 0
(0.15 seconds)
(37.59 seconds)
machine: must succeed: cat /nix/store/hcnkplh37byxpnk8wi0r6smpx6pqr1h8-ddl.sql | clickhouse-client
machine# [   63.609904] clickhouse-server[777]: 2019.05.26 19:06:48.283013 [ 25 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:52412
machine# [   63.619546] clickhouse-server[777]: 2019.05.26 19:06:48.287800 [ 25 ] {} <Debug> TCPHandler: Connected ClickHouse client version 19.6.0, revision: 54418, user: default.
machine# [   63.846372] clickhouse-server[777]: 2019.05.26 19:06:48.519236 [ 25 ] {7f8719d7-2a6a-421c-985b-332b0c72b371} <Debug> executeQuery: (from [::1]:52412) CREATE TABLE `demo` (`value` FixedString(10)) engine = MergeTree PARTITION BY value ORDER BY tuple();
machine# [   64.086419] clickhouse-server[777]: 2019.05.26 19:06:48.760390 [ 25 ] {7f8719d7-2a6a-421c-985b-332b0c72b371} <Information> BackgroundProcessingPool: Create BackgroundProcessingPool with 16 threads
machine# [   64.659699] clickhouse-server[777]: 2019.05.26 19:06:49.328032 [ 21 ] {} <Debug> ConfigReloader: Loading config `/nix/store/nj2p079qkmlp40dy1n3r10kb8g877j87-clickhouse-19.6.2.11/etc/clickhouse-server/config.xml'
machine# [   64.676535] clickhouse-server[777]: Include not found: clickhouse_remote_servers
machine# [   64.681750] clickhouse-server[777]: Include not found: clickhouse_compression
machine# [   64.711413] clickhouse-server[777]: 2019.05.26 19:06:49.385386 [ 25 ] {7f8719d7-2a6a-421c-985b-332b0c72b371} <Debug> default.demo (Data): Loading data parts
machine# [   64.722456] clickhouse-server[777]: 2019.05.26 19:06:49.396479 [ 25 ] {7f8719d7-2a6a-421c-985b-332b0c72b371} <Debug> default.demo (Data): Loaded data parts (0 items)
machine# [   64.746205] clickhouse-server[777]: 2019.05.26 19:06:49.418154 [ 25 ] {7f8719d7-2a6a-421c-985b-332b0c72b371} <Debug> MemoryTracker: Peak memory usage (for query): 1.00 MiB.
machine# [   64.756967] clickhouse-server[777]: 2019.05.26 19:06:49.419995 [ 25 ] {7f8719d7-2a6a-421c-985b-332b0c72b371} <Debug> MemoryTracker: Peak memory usage (total): 1.00 MiB.
machine# [   64.760784] clickhouse-server[777]: 2019.05.26 19:06:49.420131 [ 25 ] {7f8719d7-2a6a-421c-985b-332b0c72b371} <Information> TCPHandler: Processed in 0.902 sec.
machine# [   64.763768] clickhouse-server[777]: 2019.05.26 19:06:49.420265 [ 25 ] {} <Information> TCPHandler: Done processing connection.
machine: exit status 0
(2.32 seconds)
machine: must succeed: cat /nix/store/np3fz4dgkckdnm25kh27wmccsxnhxir4-insert.sql | clickhouse-client
machine# [   65.468015] clickhouse-server[777]: 2019.05.26 19:06:50.141797 [ 42 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:52414
machine# [   65.471164] clickhouse-server[777]: 2019.05.26 19:06:50.145178 [ 42 ] {} <Debug> TCPHandler: Connected ClickHouse client version 19.6.0, revision: 54418, user: default.
machine# [   65.501570] clickhouse-server[777]: 2019.05.26 19:06:50.173777 [ 42 ] {afedda7a-629d-4a5b-9abe-b11fd703a074} <Debug> executeQuery: (from [::1]:52414) INSERT INTO `demo` (`value`) VALUES
machine# [   65.785010] clickhouse-server[777]: 2019.05.26 19:06:50.457606 [ 42 ] {afedda7a-629d-4a5b-9abe-b11fd703a074} <Trace> default.demo (Data): Renaming temporary part tmp_insert_f27f51947020fcef9f951d37c56a3834_1_1_0 to f27f51947020fcef9f951d37c56a3834_1_1_0.
machine# [   65.809992] clickhouse-server[777]: 2019.05.26 19:06:50.469690 [ 42 ] {afedda7a-629d-4a5b-9abe-b11fd703a074} <Debug> MemoryTracker: Peak memory usage (for query): 3.01 MiB.
machine# [   65.817539] clickhouse-server[777]: 2019.05.26 19:06:50.473384 [ 42 ] {afedda7a-629d-4a5b-9abe-b11fd703a074} <Debug> MemoryTracker: Peak memory usage (total): 3.01 MiB.
machine# [   65.820735] clickhouse-server[777]: 2019.05.26 19:06:50.477452 [ 42 ] {afedda7a-629d-4a5b-9abe-b11fd703a074} <Information> TCPHandler: Processed in 0.300 sec.
machine# [   65.824742] clickhouse-server[777]: 2019.05.26 19:06:50.478019 [ 42 ] {} <Information> TCPHandler: Done processing connection.
machine: exit status 0
(0.81 seconds)
machine: must succeed: cat /nix/store/3x4ha5aibcm1snsdskx0rnkkclbrvrdh-select.sql | clickhouse-client | grep foo
machine# [   66.230392] clickhouse-server[777]: 2019.05.26 19:06:50.904149 [ 25 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:52416
machine# [   66.236665] clickhouse-server[777]: 2019.05.26 19:06:50.909997 [ 25 ] {} <Debug> TCPHandler: Connected ClickHouse client version 19.6.0, revision: 54418, user: default.
machine# [   66.264072] clickhouse-server[777]: 2019.05.26 19:06:50.938074 [ 25 ] {a5f3c9b3-e1ae-414c-8904-0295403c4da0} <Debug> executeQuery: (from [::1]:52416) SELECT * from `demo`
machine# [   66.474789] clickhouse-server[777]: 2019.05.26 19:06:51.148878 [ 25 ] {a5f3c9b3-e1ae-414c-8904-0295403c4da0} <Debug> default.demo (SelectExecutor): Key condition: unknown
machine# [   66.493438] clickhouse-server[777]: 2019.05.26 19:06:51.163193 [ 25 ] {a5f3c9b3-e1ae-414c-8904-0295403c4da0} <Debug> default.demo (SelectExecutor): MinMax index condition: unknown
machine# [   66.499814] clickhouse-server[777]: 2019.05.26 19:06:51.163494 [ 25 ] {a5f3c9b3-e1ae-414c-8904-0295403c4da0} <Debug> default.demo (SelectExecutor): Selected 1 parts by date, 1 parts by key, 1 marks to read from 1 ranges
machine# [   66.512449] clickhouse-server[777]: 2019.05.26 19:06:51.167986 [ 25 ] {a5f3c9b3-e1ae-414c-8904-0295403c4da0} <Trace> default.demo (SelectExecutor): Reading approx. 8192 rows with 1 streams
machine# [   66.531537] clickhouse-server[777]: 2019.05.26 19:06:51.181245 [ 25 ] {a5f3c9b3-e1ae-414c-8904-0295403c4da0} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
machine# [   66.542245] clickhouse-server[777]: 2019.05.26 19:06:51.182483 [ 25 ] {a5f3c9b3-e1ae-414c-8904-0295403c4da0} <Debug> executeQuery: Query pipeline:
machine# [   66.548124] clickhouse-server[777]: Expression
machine# [   66.552875] clickhouse-server[777]:  Expression
machine# [   66.555660] clickhouse-server[777]:   MergeTreeThread
machine# [   66.559953] clickhouse-server[777]: 2019.05.26 19:06:51.204126 [ 25 ] {a5f3c9b3-e1ae-414c-8904-0295403c4da0} <Information> executeQuery: Read 1 rows, 18.00 B in 0.260 sec., 3 rows/sec., 69.18 B/sec.
machine# [   66.566580] clickhouse-server[777]: 2019.05.26 19:06:51.204524 [ 25 ] {a5f3c9b3-e1ae-414c-8904-0295403c4da0} <Debug> MemoryTracker: Peak memory usage (for query): 1.13 MiB.
machine# [   66.575242] clickhouse-server[777]: 2019.05.26 19:06:51.207817 [ 25 ] {a5f3c9b3-e1ae-414c-8904-0295403c4da0} <Debug> MemoryTracker: Peak memory usage (total): 1.13 MiB.
machine: exit status 0
(0.75 seconds)
(71.39 seconds)
collecting coverage data
machine: running command: test -e /sys/kernel/debug/gcov
machine# [   66.594697] clickhouse-server[777]: 2019.05.26 19:06:51.207907 [ 25 ] {a5f3c9b3-e1ae-414c-8904-0295403c4da0} <Information> TCPHandler: Processed in 0.271 sec.
machine# [   66.599868] clickhouse-server[777]: 2019.05.26 19:06:51.208007 [ 25 ] {} <Information> TCPHandler: Done processing connection.
machine: exit status 1
(0.01 seconds)
(0.01 seconds)
syncing
machine: running command: sync
machine: exit status 0
(0.11 seconds)
(0.11 seconds)
test script finished in 71.52s
cleaning up
killing machine (pid 594)
(0.00 seconds)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
/nix/store/jvkq52yhv2qgswkv4wlkqbhxrxhcj1wd-vm-test-run-clickhouse

@aanderse aanderse merged commit 39adff9 into NixOS:master May 26, 2019
@teh teh deleted the clickhouse branch May 27, 2019 08:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@orivej orivej orivej approved these changes

@orivej-nixos orivej-nixos Awaiting requested review from orivej-nixos

Assignees
No one assigned
Labels
10.rebuild-darwin: 0 10.rebuild-linux: 1-10
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@teh @aanderse @risicle @orivej