I now separated the orthogonal changes into #82751

I don't think attrValueType is necessary if you merge in one go at the root of the submodule and then recursiveUpdate the unstructured values back into config.
This puts less burden on the types and, if we don't hardcode recursiveUpdate, it allows the module author to keep the unstructured attributes separate if desired.

@roberth I don't think this would work without big changes, because currently it's not known upfront which values are unstructured and which aren't, and this could get complicated with nested submodules and attrsOf and such.

@infinisil mergeModules could return the option values and unstructured attrs separately. Doing so should also fix the laziness problem caused by generating fake option names for the unstructured definitions.

Quote for when the commit is lost:

TODO: This leads to infinite recursion in NixOS if it's always turned on, so some use cases won't be supported anymore because of unstructured attrs. Figure out what they are.

One feature I suspect may be lost due to this change is the ability to check for the existence of option declarations before defining them, which I think will be important when modules move into flakes and need to adapt a bit at runtime to their now variable environment.
I've used this to optionally define NixOps deployment keys in a NixOS-compatible module.
I'll make sure we have a test for this laziness property.

@infinisil test available: #82802

@roberth I think I see what you mean, this might be possible, I'll give it a try soon. Implementation would be a bit like: Do a recursive mergeModules' for declared options first. After that, determine which definitions haven't been matched with a declaration, and merge those with a single unstructured type. Then combine the results of the mergeModules' and the unstructured merge with something like recursiveUpdate (it will need to insert the fake option declarations). I don't think it needs to be configurable though, as it will essentially only allow unstructured values under option sets, which makes a lot of sense, and is also the behavior of the current implementation.

@roberth Done that! Commits need some cleanup, and other things are still left to do, but now the code is looking much nicer. I also did some very sweet cleanups related to this. Also I'm now not generating fake options anymore: Only .config will now contain unstructured values, .options doesn't, which should then also not get in the way of what you mentioned in #82802.

Unfortunately without attrValueType, a specific error message is significantly worse: With something like

{
  config._module.unstructuredType = attrsOf int;
  options.a.b = mkOption {};
  config.a.c = "foo";
}

it now throws

The option value `a' in `example.nix' is not of type `signed integer'.

and if you change it to

{ config.a = 0; }

you get (as expected)

error: The option path `a' is an attribute set of options,
  but it is defined to not be an attribute set in `example.nix'.
  Did you define its value at the correct and complete path?

Instead of the previous (code now in https://github.com/Infinisil/nixpkgs/tree/partially-typed-v2-old)

The option `a.c' defined in `example.nix' does not exist
  and the unstructured type `signed integer' of `a' does not support attribute sets.

I could get back the original error message if I add attrValueType again, but I'm not sure if it's worth it. Maybe that's better left for the future.

error message

This can probably be improved quite a bit by rethrowing the error with an extra message.
"The module at ${prefix} allows undeclared definitions to be mixed with known options, but a definition did not satisfy the type for undeclared definitions: ${caughtMsg}. This may also be the result of mistyping an otherwise known option."

@roberth Hm it does work with builtins.tryEval and a deepSeq on the unstructuredType.merge ..., but that's not very nice. I'd have to do some changes to how types throw errors to make it work without tryEval.

Some more instances of this change finding mistakes/hacks:

• @evanjs reported newly getting an internal module system error because of an invalid option declaration. The code/fix: evanjs/nixos_cfg@8471f10
• @alexarice reported their kind of hacky module evaluation newly failing and their fix: alexarice/nixmacs@84f0dae. After discussing it with them, it looks like freeform modules are actually exactly what could make it work non-hackily.
• @samueldr reported this change detecting an option assignment for an option that didn't exist anymore, the fix: NixOS/mobile-nixos@1ba61f1

I have opened #95932 to add release note entries for the feature and the backwards incompatibility this introduces.

There doesn't seem to be any way to define typing for an optional setting, I was looking to set default = unset in some way, since some configuration handle null and "not set" differently.

@dali99 Indeed, that's something that could be improved in the future. See also #63553 which is related.