@ralsina @Kwpolska, line 301 of nikola/plugins/task/sitemap/init.py is probably a performance regression. Not sure how to generate the tasks before having scanned all of the directories?

@Aeyoun you can make your task depend scan_locs_task like write_sitemap does:

"calc_dep": ["_scan_locs:sitemap"],

If I understand correctly, here is what we need to do.

1. Run a bunch of tasks that generate files
2. Run scan_locs that creates a list of all files in a folder, including those generated by those tasks
3. From that list, find folders with > 200 files, and for each of those create a task that will generate
   a local sitemap
4. Run those tasks, generate all the small local sitemaps
5. Done

I have looked at the code, and I don't know how to make this work correctly:

1. Just running scan_locs before generating the tasks is not good enough, because it runs before the other tasks run, so you will miss files.

2. Using scan_locs as a calc_dep is not good enough, because you won't know how many or what tasks to create.

So, I can see 2 ways to make this work:

1. A "task that generates more tasks", so that we could make taskA have a calc_dep on scan_locs, which returns a list of sitemaps to generate, then taskA takes that list and yields more tasks, one for each sitemap

2. Using doit's getargs in targets, and have one task that uses scan_locs data to decide what sitemaps we need, and then generates exactly those sitemaps.

@schettino72 any of those things possible?

With the earlytask_impl branch, this is a piece of cake. Just use a task with a high enough stage number, its tasks will only be created when all other tasks with lower stage numbers are already done.

The easiest solution is to handle this logic yourself inside the task.

On step 4 "Run those tasks, generate all the small local sitemaps".
Instead of trying to dynamically create tasks only for folders that has more than 200 files,
create tasks for all folders. In these tasks' action check the logic (200 files or more) to see
if a local sitemap should really be created or not.

Remember that inside your action you can manipulate the task metadata.
http://pydoit.org/tasks.html#keywords-on-actions passing task.
You might need to change targets at run time in the action
(and maybe more stuff to keep the folders with less than 200 files from running all the time).

And as @felixfontein said, early_task might be another way to solve this.

Ahhh changing targets inside one task that generates all the sitemaps looks like the easy way out.

Yes, early tasks would fix this but that's a backwards-incompatible change for us so it has to wait for v8 :-(

Well, we actually don’t know what all the folders will be either.

Will investigate these new ideas. Thanks guys!

@Aeyoun just have one task, calc_dep on scan_locs, decide there what folders need sitemaps, and have it set its own targets list accordingly.

Hm. updating targets after running actions doesn’t really work. nikola.gen_tasks() as used by check.py is unaware of metadata that is updated from within a task’s action. So check.py will break. Other than that, it seem to work to update the metadata as part of running the task. Though it also introduced some build instability as scan_locs() and tasks that depend on it is unaware of any sitemap files by design.

Changing the metadata later probably also break nikola run <name of target> for the later inserted targets.

So, @Aeyoun is this stalled? Impossible?

I can’t figure out how to do it, at least. The task is always created too early to know what other files will exist to accurately say which sitemaps will be generated when. I’ve tried some variations from the original idea but they all run into the same problem at some time or another.

This would have to generate it’s tasks and run only after everything else have already finished.

Ok, so basically this never worked, and it's for a minor corner-feature (if that's a thing). I say we close it.