Maybe it's simpler to make builtins.getEnv return an empty string for IN_NIX_SHELL?

That would break the lib function completely, it assumes builtins.getEnv does return something when nix-shell is called.

https://github.com/NixOS/nixpkgs/blob/b246b3211964843bca04b0a2a8580247918da6c7/lib/trivial.nix#L169

It's used like this frequently in by the haskell infrastructure, instead of using a shell.nix and default.nix.

{ pkgs ? import <nixpkgs> {} }:

let
  package = ...;
in
  if pkgs.lib.inNixShell then packge.env else package

Is this usage something that should be fixed or do you prefer providing an alternative like a builtin instead (#2796 (comment))?

I prefer the idea of #3168 over using a magical environment variable.