Don't merge yet, depends on #43362

Ready for merging again. Having a service makes this package much more useful.
cc @Mic92 @matthewbauer

I would like to keep the granularity of the options currently present, if possible.
I feel it makes for easier and more "clear" configuration.

So you're currently using such a config? I'm asking because I really don't know how these options could be useful in the long run. They seem to be something one would run once in a terminal, not every time the service starts.

addTags and removeTags seem to be tag white and blacklist, which seems to be needed because setting fetchTags = true will override your local db with the fetched one? This should be handled at the application level and not by a NixOS module. I suspect that eventually this will be implemented in hydron.

import and importPaths seem weird, because they only import when the service starts. This should really be either some directory watcher service (in hydron itself optimally), or a systemd timer that runs regularly.

removeIds seems really weird and something that should definitely be handled by just running a command. I wouldn't want to rebuild my whole system just to remove an id from my library.

In my opinion, above mentioned options should be removed, they don't really fit into a declarative configuration. Note that users can still have such functionality by customizing the service as described above.

A systemd timer that regularly imports files from specified directories seems fine though, if you want to add that.

They seem to be something one would run once in a terminal, not every time the service starts.

I know, and it's something I would like to fix upstream, but unfortunately because hydron doesn't check for new or changed images, so it needs to check every start via import, or perhaps like you suggest, on a timer so it doesn't slow down the boot process.

addTags and removeTags I can probably remove without a problem. fetchTags just fetches tags for images their hash from gelbooru. fetching tags should be done after every (re)import.

directory watcher service

As I mentioned, that's an upstream issue and I'll remove the periodic imports once it is. For now, let's go with a systemd timer.

removeIDs seems really weird...

Good point, thanks for the tip. I'll remove it.

@infinisil How does this look?

@infinisil Glad to see you like hydron, I've been using it and it's great. Especially being able to drag and drop images.
Should be ready for building and merging again.

@infinisil

addTags and removeTags seem to be tag white and blacklist, which seems to be needed because setting fetchTags = true will override your local db with the fetched one?

No, addTags and removeTags only affects user-specified tags. Hydron maintains a separate set of tags depending on who set them (user, Gelbooru, Hydrus, etc.). fetch_tags only affects gelbooru-sourced tags. All tags are intersected on data request. Only the tag source can modify tags set by it.

import

That's a one time copy-based CLI commnad. It copies the matched file(s) to an internal database on request and populates metadata for them.

@Chiiruno

fetching tags should be done after every (re)import.

It really should not. You should import with the -f flag. Import + fetch_tags only makes sense on the initial import, which tends to be very large.

@bakape

-f    Fetch tags from gelbooru.com for imported files.
        NB: This will notably slow down importing large amounts of files.
        Consider using import, followed by fetch_tags!

I thought it was the opposite?
I'll fix it, then.

large amounts of files

That means large amounts of files that are not imported (thousands). Files that you try to import, but are already imported, will be hash-compared and ignored.

Okay, that should do it.

Hmm, but I mean, if I were to try out hydron, I'd first start the service, it would create the dataDir/images, and touch the once file (in your current implementation). And then I will put my 2000 files in there, it imports it via -f and is slow because of that.

So this current solution certainly isn't very good. Why does it support 2 ways of doing that anyways (import -f and import; fetch seem to do the same)? If it's really the case that the user has to decide what to use based on amount of imports, maybe it would be best to choose one of them based on how many new files the import directory contains?

The current solution has hydron check if the once file exists, and then if it doesn't, it uses fetch_tags and creates the once file, it it does, it incrementally fetches tags via the -f flag.

Also, the hydron-fetch service runs every hour, or per specified interval, so that's not a problem since we're not having the main hydron service do this.
If you want to import and fetch tags right after adding your importPaths, but before the hour chimes, just start hydron-fetch.

Yeah, but the folder hydron uses to import only exists once it has started, so the chance of the once file technique actually working as intended is rather small, since it probably always exists once the user actually imports files. And when the user later adds his own directories with importPaths, they won't get imported via import; fetch (the fast way?) because the once file already exists.

As @bakape said, the fetch_tags is really only for the first time, after that, -f is fine.

@infinisil
fetch_tags fetches tags for ALL of the images in your DB. -f only does this for the freshly imported ones.

@Chiiruno
Honestly, just always use -f. import+fetch_tags is an optimisation that can be made at the user's discretion and should not be done automatically, imo.

hydron-fetch service runs every hour

Why the fuck? Why would you ever DoS gelbooru like that? Imagine you have 10k images. Why would you refetch tags for all of them every hour?

DoS

Well, that wasn't really my intention. Perhaps daily instead?

Also why is this a problem if we're using -f?

But I mean for initial runs, -f is the same as fetch, because your database was empty, so added pictures = all pictures.

So I also feel like always using -f should be fine, will make the service simpler as well.

Edit: And less stateful and more predictable

Yeah, using -f would be preferred.
What about the "DoS" problem? Is this still a problem if we're using -f? @bakape

Imo that shouldn't be part of our problem: hydron should expect users to import a lot of files in one swoop and handle (if it has that) gelboorus rate limits, or implement rate limiting itself. E.g. nix-index handles this by limiting the number of active requests to be always <100.

@Chiiruno

Well, that wasn't really my intention. Perhaps daily instead?

Weekly would be more apt.

Also why is this a problem if we're using -f?

It's not a problem, if you use -f. -f only fetches for freshly imported files.

@infinisil
Yes, it is. The only difference is that your imports block the limited amount of workers on the HTTP fetch from Gelbooru. This makes importing not only bound to CPU and disk I/O, but also to the Gelbooru JSON API. It could make a huge difference, if you have many thousands of images to import.

Imo that shouldn't be part of our problem: hydron should expect users to import a lot of files in one swoop and handle (if it has that) gelboorus rate limits, or implement rate limiting itself. E.g. nix-index handles this by limiting the number of active requests to be always <100.

That is not exactly what I meant. Hydron already limits to 8 concurrent API requests. What I meant is, if I'm the Gelbooru admin and see that every hour some faggot makes 10k API requests, I will block him. I might even have scripts to flag this as abuse and block him automatically. The end user should not be aware of this possibility.

@bakape I see, but that won't be a problem with -f, right? If so, I think always using -f (even on the initial run) should be preferred.

@infinisil
Yes, that is my stance too. If the user wants to, they can optimise manually.

@infinisil Ready again.

Looking good to me! One last thing: Commit message for the module should be "nixos/hydron: init"

@bakape Also looking good to you?

Classic ;)

@infinisil Am I finally free?

You are finally free, thanks a lot!