Adding here for the moment, as closely related:

• We should give users a way to scan for untracked files and report them. These can indicate that we've got incomplete metadata, or simply show the presence of manually installed mods.
• We should provide a repair function for files which returns them to their installed state. This wouldn't touch preserved files by default (as that's dangerous), but can restore regular files. We'd likely have an option to allow dangerous operations (deleting files that should be removed on upgrade, or replacing preserved files with their original versions).
• We might want to give users a way to associate unclaimed failes with particular mods. This functionality probably shouldn't be too accessible, because there are lots of ways user can screw this up.

I know a lot of people who would benefit immensely from saner overwriting of autodetected mods. It gives so much headaches.

How difficult would it be to do a CRC or SHA check on an AD, then compare to the same of a CKAN managed? If same, an option to convert AD to CKAN could be provided? It could reduce a barrier to usage by those who diligently maintain updates.

Also, would it be difficult to just provide a option to complete convert? If a mod is detected as not compatible with 1.0.x, then it is left AD. Perhaps provide this as a CLI option only; that way if someone wants to do it they can do it, but it's all on the user (I'd use this.)

The big challenge in identifying manually installed mods properly is that currently CKAN doesn't know anything about the actual files installed by a mod until it gets downloaded and installed. registry.json contains an installed_files hash that records each file as it's unpacked and copied into GameData, but before that happens, all we have is the hash of the download file (which isn't useful for manually installed mods) plus any install stanzas that are set up for it (which just explain which folders to look at and where to install them). In effect, the roughly 14384 mod ZIP files out there form an extended database for CKAN, of which only a small fraction is captured in our stored metadata.

To be able to look at your GameData and spit out an accurate list of manually installed mods would require extracting all of that info out of the ZIP files and storing it somewhere. It could be a lot of new data to store and download (I'd ballpark it at an additional 58 MB using the below data model, where today's registry.json is 19 MB, so we'd be quadrupling load times and refresh times).

Performance could be challenging. Suppose I see a manually installed file at GameData/ABCD/EFGH.cfg, and I want to figure out which mod it belongs to. Do I search all 14384 mods one by one to see whether they have that file and whether its checksums match? And then repeat that for all of the potentially hundreds of manually installed files? That could take minutes or hours; we'd need some kind of optimized data structure like:

    "installable_files": {
        "GameData/ABCD/EFGH.cfg": {
            "9C793B4F6E3352BF0387BD77CB7BED61150F8FB403D90D1F825ABA379EE3D57D":  [
                {
                    "name":    "MyCoolMod",
                    "version": "v1.2.3"
                },
                ...
            ]
        },
        ...
    },

Then there's the challenge of choosing between multiple available modules that install the same version of the same file (which is why my example above has an array associated with the hash rather than one object). Presumably we would have to try to use clues from the other installed files to determine which one is really installed, but that sounds like one of those complex AI-level tasks to me. In the end we might have to prompt the user to explain which one they really have installed.

Assuming those problems could be solved, there are others. Users might choose to manually install mods that conflict with one another, and the registry would end up in an inconsistent state if they were simply imported as-is. We'd have to write a UI flow to resolve those conflicts.

Seems like a lot of extra work just to let people skip the step of uninstalling mods manually.

A few more observations:

• This would generally be a one-time process; typical users would convert mods from unmanaged to managed when they first start using CKAN and then install with CKAN from that point on
• Probably a minority of users would use such a conversion feature, with many starting from an unmodded install
• Therefore the data is useless 95%+ of the time
• The data would be the same for all users
• The inputs and outputs to the algorithm are much smaller than the data

To me, these factors all point toward centralizing this logic in a web service. The client could generate a JSON summary of what it sees in GameData, send it off to some ksp-ckan.space URL, and receive back the list of mods we think are present. This would remove all the excess data from the client where it's usually not needed, and the response times should still be about the same.

Obviously there are downsides to that as well, such as having to run a web service.

In my case I just hit this issue through having hit ctrl-c halfway through a ckan scan (i meant ckan search but accidentally typed scan so just ctrl-c'd it) which left a corrupted half-written registry.json file, which crashed ckan doing anything and which was not repairable with ckan repair registry. Blowing it away and then doing a ckan scan restores the list of mods, but now every single one of them are unmanaged and will not update. I actually have in my scrollback the old list of mods with all the versions. I don't need some snazzy list of checksums via a webservice.

This kind of workflow is also necessary when dealing with development, and its painful to go from git installs of stuff like RealismOverhaul, RP-0, etc to wanting to follow CKAN after they are released.

And generally this is wanted by people who have some degree of skill at it, but want an easier workflow than manually deleting all their mods and reinstalling from "scratch". The userbase which is asking for this is probably the kind of userbase who understands what state they're coming from and where they want to get to.

hit ctrl-c halfway through a ckan scan (i meant ckan search but accidentally typed scan so just ctrl-c'd it) which left a corrupted half-written registry.json file, which crashed ckan doing anything and which was not repairable with ckan repair registry.

Could you please file a new issue for this, @lamont-granquist? That sounds like something we could and should fix.

This is showing up in GitHub searches for open issues despite being closed.
Re-opening and re-closing to see if that makes a difference.