@shlevy worked with Shea to figure this issue out.

There are real cases that need it, it's perfectly sound if you use it carefully, and if you make a mistake it will fail at compile time. What's the issue?

Also, it's not clear to me why nixpkgs should be making this decision in any case. The haskell ecosystem has decided it should be a warning, why should we overrule that here? If it's the wrong call we should change it upstream... It's not like nix itself makes this situation any worse than it would be otherwise.

if you make a mistake it will fail at compile time

Is there any technical documentation to back that up? Where does this assertion come from?

This post from Well Typed has a good overview. The relevant bits:

This can work ok but only if packages B and C do not expose types defined in D in their interfaces. If they do then package A will not be able to use functions from B and C together because they will not be working with the same type. That is you'll get a type error.

From Haskell's point of view, the "same" definition in two different packages is actually two totally separate definitions that are namespaced implicitly. If you try to use a type from one version of a package in a function that expects the other, you will get a type error:

Couldn't match expected type `bytestring-0.9.0.4:Data.ByteString.ByteString'
against inferred type `bytestring-0.9.0.1:Data.ByteString.ByteString'

From the language's point of view, this is no different from using ByteString from Data.ByteString.Lazy in a context that expects ByteString from Data.ByteString, except it happens because the package is different rather than the module.

This is the failure case for linking two different versions of the same dependency: code that looks like it should compile won't because the implicit package namespace of two types is different. This behavior is confusing for less experienced Haskellers because you normally don't have to think about different versions of the same package as different namespaces—which is why there is a warning—but the resulting behavior is consistent and is manageable if you understand what's going on.

Also, let me spell out my usecase here a bit more clearly than I did in the PR description.

I am developing Haskell packages locally using Nix and Nixpkgs. Our team has one tool (let's call it dsl-a) that uses megaparsec-6.5.0 for parsing and another tool (dsl-b) that uses megaparsec-7.0.4. Since megaparsec is an internal implementation detail of how the tools parse text into domain-specific types, it is not exposed in their public interfaces.

The problem comes up when we have packages that require both dsl-a and dsl-b. If we can only link against a single version of megaparsec, there is no way to depend on both simultaneously. If we can link against different transitive dependencies, the result works exactly as expected. Moreover, if something changes in the future—say dsl-a exposes a value directly from megaparsec and it's passed into a function from dsl-b—the failure case would be a compile-time type error from Haskell.

As it stands right now, we can't build tools that depend on both dsl-a and dsl-b using the standard Nixpkgs Haskell infrastructure only because of the extra check this PR removes.

I really don't understand this.

First, again I ask why this is a nixpkgs issue? If you think Cabal is wrong here then shouldn't we fix it there?

Second, If it's possible to compile and link an executable that will break at runtime or behave strangely at runtime is way too high a bar! If package foo has a type Foo with a Show and Read instance, could I show a Foo from foo-1.0.0 and try to read the resulting string back into a foo-2.0.0 Foo and have issues? Of course,but that's true for any number of otherwise reasonable widely used features. Can I accidentally pass a Foo from foo-1.0.0 to a function expecting a foo-2.0.0? No, and that's not based on "someone's feelings", that's based on how type namespacing actually works. What kind of "deep technical analysis" would satisfy you, why don't you think there's "deep understanding of the problem", what specific question would a ghc developer be able to help us answer? As things currently stand this seems like a very vague objection that we can't possibly address no matter how big or small of an issue this actually is in practice.

Third, there are real use cases that are doing a completely reasonable thing here that are broken by this. If this approach is not OK, what do you suggest we do?

Why this is a nixpkgs issue? If you think Cabal is wrong here then shouldn't we fix it there?

This is a Nixpkgs issue, because the PR is directed to a repository called "nixpkgs". I have no opinion on whether Cabal is doing the right thing or not. I just have an opinion on what I want the Nixpkgs generic Haskell builder to do.

Second, If it's possible to compile and link an executable that will break at runtime or behave strangely at runtime is way too high a bar!

That is the bar we've had for the last couple of years and I see no reason to lower it.

If package foo has a type Foo with a Show and Read instance, could I show a Foo from foo-1.0.0 and try to read the resulting string back into a foo-2.0.0 Foo and have issues? Of course.

There you go. Fortunately, we don't have any of that crap in Nixpkgs because our generic builder makes sure that you cannot do such a thing.

Can I accidentally pass a Foo from foo-1.0.0 to a function expecting a foo-2.0.0?

I don't know for sure. I haven't thought about these problems long enough to feel comfortable having an opinion. I don't know how these things interact with FFI or inlineable class methods or any other of the advanced type machinery modern ghc versions offer. I am not an expert in these things.

What specific question would a ghc developer be able to help us answer?

"When we compile a Haskell executable that references multiple versions of the same library in its dependency tree, will the compiler guarantee that the resulting binary works properly? Or rather, can we rely on the compiler (or linker) to fail if it won't?"

Third, there are real use cases that are doing a completely reasonable thing here that are broken by this. If this approach is not OK, what do you suggest we do?

1. Fix you code to work with the same version of megaparsec consistently.
2. Create your own derived version of the generic Haskell builder and use that to build binaries that the Nixpkgs version won't let you build.
3. Don't use the generic Haskell builder at all.

This is a Nixpkgs issue, because the PR is directed to a repository called "nixpkgs". I have no opinion on whether Cabal is doing the right thing or not. I just have an opinion on what I want the Nixpkgs generic Haskell builder to do.

Let me rephrase: Why should nixpkgs be opinionated here at all? I have a package that builds just fine with the normal build tools/build system of my language/ecosystem, why is nixpkgs telling me it's wrong if it's not breaking any Nix invariants?

That is the bar we've had for the last couple of years and I see no reason to lower it.

We've never had that bar and it's fundamentally impossible. No language is free of potentially confusing runtime behavior and nothing Nix can do will fix that. Should we ban the partial prelude functions? Those can easily lead to breaking/strange behavior at runtime. What about lazy IO? The C FFI? Maybe we should restrict ourselves to Safe Haskell and write a static analyzer to ensure we upper bound our stack space as runtime?

If package foo has a type Foo with a Show and Read instance, could I show a Foo from foo-1.0.0 and try to read the resulting string back into a foo-2.0.0 Foo and have issues? Of course.

There you go. Fortunately, we don't have any of that crap in Nixpkgs because our generic builder makes sure that you cannot do such a thing.

OK, this was meant to be an example of why your demand was too strong. Yes, you can get weird behavior if you bypass the type system, but that's not at all unique to this situation. But sure let's spell it out.

1. I can show from one program and read it into another, which could be running mismatched versions. If we could prohibit that in nixpkgs, would you?
2. I can show a Foo and try to read it into a Bar. If we could prohibit that in nixpkgs, would you?
3. I can show a Foo.A and try to read it into a Bar.A. If we could prohibit that in nixpkgs, would you?
4. I can show a Foo.A from package foo and try to read it into a Foo.A from package bar. If we could prohibit that in nixpkgs, would you? Note that this one would actually be pretty easy to prevent, at least a significant subset of it, by disabling PackageImports in ghc.

If you're fine with 1-4, all of which our generic builder allow without a blink, why is it suddenly a problem now?

Can I accidentally pass a Foo from foo-1.0.0 to a function expecting a foo-2.0.0?

I don't know for sure. I haven't thought about these problems long enough to feel comfortable having an opinion. I don't know how these things interact with FFI or inlineable class methods or any other of the advanced type machinery modern ghc versions offer. I am not an expert in these things.

If you're not comfortable having an opinion, why are you pushing back against those who do, especially when we can back it up? I do know for sure that this is safe: To the typechecker and to the runtime, Foo.Foo from foo 1.0.0 and Foo.Foo from foo 2.0.0 are completely different types. They will never be unified with each other, and for the relevant symbols in object files/shared libraries different symbol mangling will be used. If you get into FFI, sure if the types under the hood have the same representation you could possibly get yourself into trouble, but the same is true for e.g. newtypes.

Again, to the compiler nothing here is meaningfully different from Foo.Foo from foo 1.0.0 vs Foo.Foo from bar 1.0.0, which we do nothing to prevent.

"When we compile a Haskell executable that references multiple versions of the same library in its dependency tree, will the compiler guarantee that the resulting binary works properly? Or rather, can we rely on the compiler (or linker) to fail if it won't?"

Again, "resulting binary works properly" is an impossible bar. No, ghc won't magically tell us that we wrote the program we wanted to right. No, ghc won't prevent us from logic errors, specification errors, etc. No, ghc is not dependently typed, linearly typed, etc. But allowing multiple versions of the same package doesn't change any of this.

Fix you code to work with the same version of megaparsec consistently.

The code is not broken and it is sheer coincidence that the two DSLs in question in our example happen to be in-house projects.

Create your own derived version of the generic Haskell builder and use that to build binaries that the Nixpkgs version won't let you build.

Well yes obviously we can fork if our use case can't be supported. Are you uninterested in supporting it?

I agree that if it's allowed by Cabal, it should be allowed by nixpkgs.

Why should nixpkgs be opinionated here at all?

Executables that link multiple, mutually incompatible versions of their dependencies are wasteful and, more importantly, dangerous. Now, there's a chance that such an executable is going to work fine, but there's also a chance that it's going to misbehave or crash at run-time. And if it does, then those errors are extremely hard to debug. I am not aware of a single distribution that exposes its users to that kind of risk.

I have a package that builds just fine with the normal build tools/build system of my language/ecosystem, why is nixpkgs telling me it's wrong if it's not breaking any Nix invariants?

Your package does not build fine. Your build system prints a big, fat warning telling you that you are doing something that it thinks is a bad idea. Furthermore, if you'll try to run the same build with cabal new-build, then you'll find that the tool outright refuses to run that build at all.

It is mind boggling to me that you don't mind exposing our users to that kind of risk.

We've never had that bar and it's fundamentally impossible. No language is free of potentially confusing runtime behavior and nothing Nix can do will fix that.

What are you talking about? The bar is "we don't distribute binaries that link inconsistent versions of their dependencies". That quality standard has existed forever, and it's enforced by the very configure-time check that you apparently wish to drop. What can i tell you? If you want a loaded gun pointed at your foot, then please go ahead and do it. But please don't make Nixpkgs point that gun at all the feet of all its users, too, just because you personally happen to enjoy that kind of situation.

1. I can show from one program and read it into another, which could be running mismatched versions. If we could prohibit that in nixpkgs, would you?

Why are we talking about this hypothetical situation? What does this have to do with anything? Nixpkgs can detect builds that link multiple inconsistent versions of their dependencies and it does prohibit those kinds of builds from succeeding.

2. I can show a Foo and try to read it into a Bar. If we could prohibit that in nixpkgs, would you?

3. I can show a Foo.A and try to read it into a Bar.A. If we could prohibit that in nixpkgs, would you?

4. I can show a Foo.A from package foo and try to read it into a Foo.A from package bar. If we could prohibit that in nixpkgs, would you?

Honestly, I don't see how this is relevant for this PR. read (show (a::Foo)) :: Bar could be perfectly legitimate code for all I know. That depends entirely on the definitions of Foo and Bar. Why are we talking about that stuff? I feel like are wasting my time lightening up smoke screens and discussing straw mans.

If you're fine with 1-4, all of which our generic builder allow without a blink, why is it suddenly a problem now?

(1) to (4) are hypothetical problems that you've made up and which are not the subject of this PR. The subject of this PR is that you want to remove a check which detects inconsistent build configurations. There is no relationship between (1) to (4) and this PR. Your argument here is: "Since you cannot detect my made up issues (1) to (4), why should we bother detecting inconsistent builds, which we can detect and have been detecting for years?" That is not a very convincing argument. The technical term is "non sequitur".

If you're not comfortable having an opinion, why are you pushing back against those who do, especially when we can back it up?

Nutritionists from all over the world ague about the perfect diet. Should we eat sugar? Or rather not? How much fat is healthy? And so on and so forth. Now, if someone comes to me asks me what my opinion on this matter is, then I'll say that I don't have an opinion because I am not a nutritionist. I don't know enough about the subject to talk about it with any kind of authority.

However, when my 4 year old daughter comes to me and suggests that she wants to eat nothing but chocolate, then I'll push back on that idea because clearly my opinion -- which is naive and uninformed -- is still a hell of a lot more informed than hers.

I do know for sure that this is safe: [...]

Ab, lbh qba'g. Lbh qba'g haqrefgnaq gur vzcyvpngvbaf bs gur punatr lbh ner cebcbfvat. Lbh qba'g haqrefgnaq gur pbafrdhraprf guvf punatr jvyy unir sbe bhe hfref, naq lbh frrz bqqyl qvfvagrerfgrq va svaqvat bhg jung gurl ner. Lbh unir snyyra va ybir jvgu n arj srngher naq abj lbh'yy fnl nalguvat naq qb nalguvat gb znxr vg unccra jvgubhg pbafvqrevat gur vzcyvpngvbaf sbe bgure crbcyr.

Are you uninterested in supporting [our use case]?

What we could do is to make the consistency check configurable, i.e. we could add an argument to the generic builder, like allowInconsistentDependencies, that disables the configure-time check for that particular build, but defaults to false for everyone who doesn't set it otherwise explicitly.

I think you are wrong on the technical merits and are continually moving the goal posts. If we can get back to a purely technical discussion, I'm happy to try to clarify my points here or on a call. But I don't see how that can be productive given that you've brought the discussion to the level of personal attacks, condescension, and dismissiveness. If you want to have a conversation between mutually respected peers each with their own expertise and understandable technical concerns, let me know.

To be absolutely clear, this is what I'm objecting to:

However, when my 4 year old daughter comes to me and suggests that she wants to eat nothing but chocolate, then I'll push back on that idea because clearly my opinion -- which is naive and uninformed -- is still a hell of a lot more informed than hers.

No, you don't. You don't understand the implications of the change you are proposing. You don't understand the consequences this change will have for our users, and you seem oddly disinterested in finding out what they are. You have fallen in love with a new feature and now you'll say anything and do anything to make it happen without considering the implications for other people. [SL I have un-ROT13'd this paragraph. I struggle to understand why it was ever ROT-13'd in the first place]

I think comments like this are part of a pattern, both in this conversation and in broader interactions, but for the sake of this discussion I'm happy to limit my concerns to these two comments.

Ab, lbh qba'g. Lbh qba'g haqrefgnaq gur vzcyvpngvbaf bs gur punatr lbh ner cebcbfvat. Lbh qba'g haqrefgnaq gur pbafrdhraprf guvf punatr jvyy unir sbe bhe hfref, naq lbh frrz bqqyl qvfvagrerfgrq va svaqvat bhg jung gurl ner. Lbh unir snyyra va ybir jvgu n arj srngher naq abj lbh'yy fnl nalguvat naq qb nalguvat gb znxr vg unccra jvgubhg pbafvqrevat gur vzcyvpngvbaf sbe bgure crbcyr.

this is just being childish. Rise above.

I am happy to admit that i am NOT an expert on the technical details here... The point I’d like to make though, is that I believe this really isn’t an issue on technical details and implications - which this thread went into discussing.

I think that what it boils down to is: Should nixpkgs incorporate modifications to potentially improve/secure or generally augment the way haskell packages are built even if that means it will then differ from upstream/everyone else.

“This works just fine with vanilla cabal but not via nix” doesn’t seem like something desirable?

Implemented in #52257

Closed by #52257