Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

matrix-synapse: 1.19.3 -> 1.20.1 #98476

Merged
merged 1 commit into from Sep 26, 2020
Merged

matrix-synapse: 1.19.3 -> 1.20.1 #98476

merged 1 commit into from Sep 26, 2020

Conversation

Ma27
Copy link
Member

@Ma27 Ma27 commented Sep 22, 2020 •
edited

Motivation for this change

https://github.com/matrix-org/synapse/releases/tag/v1.20.0
https://github.com/matrix-org/synapse/releases/tag/v1.20.1

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@Ma27 Ma27 added the 9.needs: port to stable A PR needs a backport to the stable release. label Sep 22, 2020
Ekleog
Ekleog approved these changes Sep 22, 2020
View reviewed changes
Copy link
Member

@Ekleog Ekleog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading the changelog, I can't find mention of security fixes, and there's this backwards-compatibility warning.

So… as I guess synapse' retro-compat story in the federation is still as bad as before, I'd say maybe let's backport to 20.09 but not 20.03, thus keeping 20.03 stable?

@Ma27 Ma27 changed the title matrix-synapse: 1.19.3 -> 1.20.0 matrix-synapse: 1.19.3 -> 1.20.1 Sep 25, 2020
@Ma27 Ma27 requested a review from Ekleog September 25, 2020 15:49
@Ma27 Ma27 merged commit 1a4a66e into NixOS:master Sep 26, 2020
@Ma27 Ma27 deleted the bump-matrix-synapse branch September 26, 2020 22:48
@Ma27
Copy link
Member Author

Ma27 commented Sep 26, 2020

Ported to stable as e7c26a7.

@Ma27 Ma27 added 8.has: port to stable A PR already has a backport to the stable release. and removed 9.needs: port to stable A PR needs a backport to the stable release. labels Sep 26, 2020
@dali99
Copy link
Member

dali99 commented Oct 2, 2020

1.20 rc1 has a fix for matrix-org/synapse#8220
without which logs are spammed with user directory errors. And crashing the user directory because a server sent an invalid display name.

That could count as a partial denial of service vulnerability 🤷

@Ma27
Copy link
Member Author

Ma27 commented Oct 2, 2020

cc @Ekleog what do you think of it?

@Ekleog
Copy link
Member

Ekleog commented Oct 2, 2020 •
edited

@dali99 What do you mean by “crashing the user directory”? I can't find this in the issue you linked.

As for the part about synapse spamming logs… my synapse instance, on which we are fewer than 5 users, logged ~375k log lines yesterday using more or less the nixos-default configuration, without this bug happening (tested by grepping for InvalidTextRepresentation). Would this bug make the behavior significantly worse? Having tracebacks in the logs is something I remember synapse regularly doing (though I guess it happens less nowadays), so unless it'd increase by an order of magnitude I'm not sure it's (by itself and not counting that “crashing the user directory” issue) worth risking backwards-compatibility breakage so close to 20.03 EOL?

Also anyway, thank you for the report, regardless of the end result it's great you reported this!

@dali99
Copy link
Member

dali99 commented Oct 3, 2020 •
edited

The user directory background process, crashes, and immediately retries, not processing any updates, of course immediately crashing again. repeat multiple times per second. When the user directory crashes that means new users the server sees won't be processed, which means the user directory won't show them to you when you search for their usernames when you want to invite them or otherwise.

@Ekleog in the past 24 hours my synapse has logged 10 027 118 lines. And that is with a log level changed up from nixos default INFO to WARN. With this specific traceback occuring 2-4 times per second at the error log level.

Which is two orders of magnitude worse.

I have a suspicion that this bug is also what's causing my synapse to idle at 70% CPU.

As for where this is described in the issue I linked:

User directory gets stuck

The main issue is that the error path seems to continue throwing errors and stops making progress.

@Ekleog
Copy link
Member

Ekleog commented Oct 3, 2020

Thank you for your feedback!

I… guess this is worth backporting to 20.03, then? My problem with this is, reading the backwards-compatibility issue, it looks like clients that are even just 5 months old are currently incompatible with the new synapse release…

I guess I'll just yell at synapse in my head again and we can backport the update to 20.03, as FluffyChat doesn't look widely-used… if that's also the way you feel, @Ma27?

@Ma27
Copy link
Member Author

Ma27 commented Oct 3, 2020

Tbh I deferred the debate to this thread since I don't have a strong opinion about it (all of my systems are already using a recent version of release-20.09).

However we have people here experiencing this issue and I actually established a backport-by-default policy for matrix-synapse about a year ago, so I'm fine with it and would take care of it now.

While some clients seem to have issues with v1.20.0 now, we shouldn't forget that (1) the client violated the protocol and (2) the most famous example (fluffychat) has fixed the issue already[1].

[1] matrix-org/synapse#6766 (comment)

@Ma27
Copy link
Member Author

Ma27 commented Oct 3, 2020

Ported to 20.03 as 760c6ec now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Ekleog Ekleog Ekleog approved these changes

@ajs124 ajs124 ajs124 approved these changes

@mguentner mguentner Awaiting requested review from mguentner

@pacien pacien Awaiting requested review from pacien

@Ralith Ralith Awaiting requested review from Ralith

@fadenb fadenb Awaiting requested review from fadenb

Assignees
No one assigned
Labels
8.has: port to stable A PR already has a backport to the stable release. 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 1-10 10.rebuild-linux: 1 11.by: package-maintainer
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@Ma27 @dali99 @Ekleog @ajs124