Skip to content

matrix-synapse: prepare for python3 switch #48695

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

matrix-synapse: prepare for python3 switch #48695

wants to merge 2 commits into from
+29 −26

Conversation

peterhoeg
Copy link
Member

Motivation for this change

Performance on a low-specced machine is not great, so move to py3 which upstream recommends.

I have not benchmarked it.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

Sorry, something went wrong.

@peterhoeg peterhoeg requested a review from FRidh as a code owner October 19, 2018 04:47
@peterhoeg peterhoeg changed the title matrix-synapse: move from python2 to python3 [WIP] matrix-synapse: prepare for python3 switch Oct 23, 2018
@peterhoeg
Copy link
Member Author

Slight change of plans.

With this commit, we can swap python2 for python3 to run synapse using python3 instead.

The reason for not making the switch is that a number of CLI tools provided with synapse do not yet work under py3 despite synapse running fine.

So this doesn't actually do anything on its own except to prepare for the upcoming py3 switch.

Cc: @Ralith @roblabla @Ekleog

@Ekleog
Copy link
Member

Ekleog commented Oct 23, 2018

This, in its current state, looks like a rather harmless refactoring of synapse. OTOH I wonder, why is an override of python2 required? I can't really see the point over directly defining the packages in the let binding?

Anyway, it'll be great when python3 support will be complete!

@GrahamcOfBorg test matrix-synapse

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: tests.matrix-synapse

Partial log (click to expand)

serverpostgres# [   63.169897] synapse[1081]: synapse.storage.background_updates: [background_updates-0] Starting update batch on background update 'access_tokens_device_index'
serverpostgres# [   63.208254] synapse[1081]: synapse.metrics: [] Collecting gc 0
serversqlite: exit status 0
test script finished in 64.66s
cleaning up
killing serverpostgres (pid 597)
killing serversqlite (pid 609)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
/nix/store/zj98d4vak1jbildnmci6ky4nh7m990p7-vm-test-run-matrix-synapse

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: tests.matrix-synapse

Partial log (click to expand)

serverpostgres: running command: sync
serverpostgres# [  177.583488] synapse[1028]: synapse.metrics: [] Collecting gc 1
serverpostgres: exit status 0
test script finished in 178.94s
cleaning up
killing serversqlite (pid 631)
killing serverpostgres (pid 643)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
/nix/store/q5d77flbzhvrh2xy2r5j995d19yipyql-vm-test-run-matrix-synapse

@peterhoeg
Copy link
Member Author

OTOH I wonder, why is an override of python2 required?

Because it makes it super easy to build either with python2 or python3 and I was trying to actually override the affinity derivation to make it work with py3 but that turned out not to be necessary admittedly.

@Ekleog
Copy link
Member

Ekleog commented Oct 23, 2018

Well, it somehow feels a bit like unneeded complexity, then?

Apart from that (feel free to ignore, it's marginal), r+ from me :)

Also I'm wondering, you say:

The reason for not making the switch is that a number of CLI tools provided with synapse do not yet work under py3 despite synapse running fine.

Do you think it'd be possible to add a test / enrich the matrix-synapse test so that the error you initially made could be detected? (I must say I didn't check which tools don't work yet under py3)

@Ralith
Copy link
Contributor

Ralith commented Oct 23, 2018

Even without using an override, everything still keys off the same import, no?

dotlambda
dotlambda requested changes Oct 29, 2018
View reviewed changes
pkgs/servers/matrix-synapse/default.nix Outdated
matrix-synapse-ldap3 = python2Packages.buildPythonPackage rec {
pname = "matrix-synapse-ldap3";
version = "0.1.3";
src = fetchurl {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fetchPypi

pkgs/servers/matrix-synapse/default.nix Outdated
};
in python2Packages.buildPythonApplication rec {

in with py.pkgs; buildPythonApplication rec {
name = "matrix-synapse-${version}";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name = "matrix-synapse-${version}";
pname = "matrix-synapse";

pkgs/servers/matrix-synapse/default.nix Outdated

checkInputs = with python2Packages; [ ldaptor mock ];
propagatedBuildInputs = with super; [ service-identity ldap3 twisted ];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
propagatedBuildInputs = with super; [ service-identity ldap3 twisted ];
propagatedBuildInputs = with self; [ service-identity ldap3 twisted ];

pkgs/servers/matrix-synapse/default.nix Outdated
propagatedBuildInputs = with super; [ service-identity ldap3 twisted ];

# ldaptor is not ready for py3 yet
doCheck = !super.isPy3k;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
doCheck = !super.isPy3k;
doCheck = !self.isPy3k;

even though it doesn't matter in this case

pkgs/servers/matrix-synapse/default.nix Outdated

propagatedBuildInputs = with python2Packages; [ service-identity ldap3 twisted ];
src = fetchFromGitHub {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not fetchPypi? Are the tests missing?

pkgs/servers/matrix-synapse/default.nix Outdated
version = "0.6.8";
py = python2.override {
packageOverrides = self: super: {
matrix-angular-sdk = super.buildPythonPackage rec {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
matrix-angular-sdk = super.buildPythonPackage rec {
matrix-angular-sdk = self.buildPythonPackage rec {

pkgs/servers/matrix-synapse/default.nix Outdated
rev = "v${version}";
sha256 = "0ss7ld3bpmqm8wcs64q1kb7vxlpmwk9lsgq0mh21a9izyfc7jb2l";
};
matrix-synapse-ldap3 = super.buildPythonPackage rec {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
matrix-synapse-ldap3 = super.buildPythonPackage rec {
matrix-synapse-ldap3 = self.buildPythonPackage rec {

@peterhoeg peterhoeg force-pushed the f/synapse branch 2 times, most recently from ec9ea75 to 69bf8e4 Compare November 7, 2018 05:52
@peterhoeg
matrix-synapse: prepare for python3 switch

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode
74a9f06 
With this commit, we *can* swap python2 for python3 to run synapse using python3
instead.

The reason for not making the switch is that a number of CLI tools provided with
synapse do not yet work under py3 despite synapse running fine.

So this doesn't actually do anything on its own except to prepare for the
upcoming py3 switch.
@peterhoeg
Copy link
Member Author

So here's the updated version that fetches from Pypi and no longer does any overriding. Instead we choose the py2 or py3 from all-packages.nix.

@peterhoeg peterhoeg closed this Nov 7, 2018
@peterhoeg peterhoeg deleted the f/synapse branch November 7, 2018 06:01
@peterhoeg peterhoeg restored the f/synapse branch November 7, 2018 06:02
@peterhoeg
Copy link
Member Author

Something super strange is going on - GH isn't showing the latest commit after a force push.

@peterhoeg peterhoeg mentioned this pull request Nov 7, 2018
Merged
9 tasks
@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: matrix-synapse

Partial log (click to expand)

wrapping `/nix/store/3s2i6fk7vp1rvcdqnpbqz8hry7g6jd4a-matrix-synapse-0.33.8/bin/synctl'...
wrapping `/nix/store/3s2i6fk7vp1rvcdqnpbqz8hry7g6jd4a-matrix-synapse-0.33.8/bin/register_new_matrix_user'...
wrapping `/nix/store/3s2i6fk7vp1rvcdqnpbqz8hry7g6jd4a-matrix-synapse-0.33.8/bin/homeserver'...
running install tests

-------------------------------------------------------------------------------

PASSED
pytestcachePhase
/nix/store/3s2i6fk7vp1rvcdqnpbqz8hry7g6jd4a-matrix-synapse-0.33.8

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: matrix-synapse

Partial log (click to expand)

wrapping `/nix/store/f81rsdr39nwnlxjl730zj52jdyaj2f78-matrix-synapse-0.33.8/bin/move_remote_media_to_new_store.py'...
wrapping `/nix/store/f81rsdr39nwnlxjl730zj52jdyaj2f78-matrix-synapse-0.33.8/bin/register_new_matrix_user'...
wrapping `/nix/store/f81rsdr39nwnlxjl730zj52jdyaj2f78-matrix-synapse-0.33.8/bin/synapse_port_db'...
wrapping `/nix/store/f81rsdr39nwnlxjl730zj52jdyaj2f78-matrix-synapse-0.33.8/bin/synctl'...
running install tests

-------------------------------------------------------------------------------

PASSED
pytestcachePhase

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dotlambda dotlambda

@FRidh FRidh

Assignees
No one assigned
Labels
6.topic: python 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@peterhoeg @Ekleog @GrahamcOfBorg @Ralith @dotlambda