You should add assertions for the new optional features. withExtraLibs should probably be removed.

Good catch, thank you. I added an assert for withDBI now, which is the only new feature which needs one, I think.

Regarding withExtraLibs, I now reconstructed its intention: Some community modules have additional dependencies not stated in the package, like luapam or lualdap, which can be injected via withExtraLibs = [ luapam ] when withCommunityModules = [ auth_pam ] was added. I added a comment to withExtraLibs now.

@orivej Or is there another way to add those extra dependencies when configuring the package, can you do the same with overrideAttrs or something? (I don't know, never needed that so far.) Then I'm quite sure it can be removed.

Tried to do some research on the topic. With withExtraLibs, you only need this to add community modules:

nixpkgs.config.packageOverrides = pkgs: {
  prosody = pkgs.prosody.override {
    withCommunityModules = [ "auth_pam" ];
    withExtraLibs = [ lua51Packages.luaposix ];
  };
};

Without it, you need to use overrideAttrs to create a custom package to add the new libs on top of the default ones,
as libs is no package option (if I understood it right). And then assign that custom package to the prosody service.

environment.systemPackages = [
  (pkgs.emacs.overrideAttrs (oldAttrs: {
    name = "prosody-with-mods";
    withCommunityModules = [ "auth_pam" ];
    libs = oldAttrs.libs ++ lua51Packages.luapam;
  }))
];
# hypothetical module option which does not exist yet
services.prosody.pkg = pkgs.prosody-with-mods;

So it could be made to work without that extra option, I guess.
But from a user's perspective, the first one looks much easier to me. But again, I have no experience with customizing packages.