[WIP] Conversion plugin for Nikola's #2761: conversion plugin for pre-v8 tag-style metadata #271

felixfontein · 2018-05-02T14:23:21Z

Companions #3068.

felixfontein · 2018-05-02T15:37:46Z

Ok, first working version.

felixfontein · 2018-05-02T15:41:23Z

(requires getnikola/nikola#3068 to work)

Kwpolska · 2018-05-02T16:19:31Z

v8/upgrade_metadata_v8/upgrade_metadata_v8.py

+                            if extractor.name == 'nikola':
+                                final_str = meta_str + '\n\n' + content_str
+                            elif extractor.name == 'yaml':
+                                final_str = meta_str + '\n---\n' + content_str


Is there a way not to hardcode this?

We could extend the metadata extractors so that they offer a recombine function which does this. That would avoid this. If this is ok for you, I'll start a PR for that.

We’ve got utils.write_metadata for that. Also handles adding  where necessary.

But that only writes metadata, right? And doesn't include the post's content?

Yes, but then you can just fd.write(content) without worrying about anything, cf. any compiler plugin.

Aaah, I missed that part :)

Done. Two drawbacks are:

might use another metadata format (which screws up metadata transformations);

adds HTML comments everywhere, even if they haven't been there before and aren't needed.

You could fix 2. by checking if the format is reST.

I don't understand how you mean this.

For 1., how about adding an argument like force_format to write_metadata which, if set to True, bails out (by returning None or raising an exception) if the given / selected metadata format cannot be used.

Kwpolska · 2018-05-02T16:20:47Z

v8/upgrade_metadata_v8/upgrade_metadata_v8.py

+                                L.warn('Cannot convert {0} (language {1}): a metadata value mapping is defined for "{2}"!'.format(fname, lang, meta_key))
+
+                        # Update metadata
+                        updated = False


Feature request:

if section is present, but category isn’t: set category to section value and remove section
if both are present: show a warning (as is done on the nuke-sections branch of the main repo)

Should this be part of the main functionality, or should it be possible to select one of the two functionalities (kill special tags, and kill sections)? I would make it possible to do only one of them, since (at least for sections) you could still use them with a custom taxonomy plugin. So some people might not want to remove them (but still convert special tags).

Main functionality. Plan for the 99%, the rest can handle it on their own.

Sure. Disabling the corresponding code isn't hard, either.

Implemented in 685be9c.

ralsina · 2018-05-02T18:04:06Z

+1, not familiar with the section killing, so going with Kw's opinion there.

felixfontein · 2018-05-03T04:43:06Z

Now the tests also no longer fail, thanks to 42d6e2b.

felixfontein · 2018-05-04T21:08:44Z

@Kwpolska how do you want to proceed with this one?

felixfontein · 2018-05-04T21:09:06Z

(I guess for a v8 beta, this one should be available.)

Kwpolska · 2018-05-04T21:12:25Z

v8/upgrade_metadata_v8/upgrade_metadata_v8.py

+                flagged.append(post)
+        if flagged:
+            if len(flagged) == 1:
+                L.info('1 post (and/or its translations) contains old-style metadata or have section metadata:')


you mixed up have/has in both cases

Thanks! Fixed.

Kwpolska · 2018-05-04T21:14:12Z

v8/upgrade_metadata_v8/upgrade_metadata_v8.py

+
+                        # Recombine metadata with post text if necessary, and write back to file
+                        meta_str = utils.write_metadata(meta, metadata_format=extractor.name, compiler=post.compiler,
+                                                        comment_wrap=True, site=self.site)


comment_wrap=(post.compiler.name != 'rest')

Ah, ok. I'm using that now. Doesn't make me 100% happy, but I guess it is fine for > 99% of the time ;)

Kwpolska · 2018-05-04T21:26:22Z

Did you test this with YAML or other tricky edge cases?

felixfontein · 2018-05-04T21:45:41Z

Tags as list work now. What doesn't work are two-file posts, it seems. They don't work, because no extractor is stored for them. get_metadata_from_meta_file simply doesn't return it. Is this intentional?

felixfontein · 2018-05-04T21:46:07Z

(YAML and TOML work fine, at least outside two-files.)

Kwpolska · 2018-05-04T21:49:02Z

Fixed on master (getnikola/nikola@c2044a2e)

felixfontein · 2018-05-04T21:50:23Z

LOL, you were faster. I just pushed a branch for that :)

felixfontein · 2018-05-04T21:54:14Z

Now two-file posts work as well. METADATA_MAPPING support also seems to work.

felixfontein · 2018-05-04T21:54:28Z

I think it should be fine now.

Kwpolska · 2018-05-05T13:30:28Z

There were two bugs I found while fixing chriswarrick.com:

saving as UTF-8 with BOM is evil
posts without translations crash:

Traceback (most recent call last):
  File "/Users/kwpolska/virtualenvs/nikola/lib/python3.6/site-packages/doit/doit_cmd.py", line 172, in run
    return command.parse_execute(args)
  File "/Users/kwpolska/virtualenvs/nikola/lib/python3.6/site-packages/doit/cmd_base.py", line 127, in parse_execute
    return self.execute(params, args)
  File "/Users/kwpolska/git/nikola/nikola/plugin_categories.py", line 147, in execute
    return self._execute(options, args)
  File "/Users/kwpolska/website/plugins/upgrade_metadata_v8/upgrade_metadata_v8.py", line 105, in _execute
    with io.open(fname, "r", encoding="utf-8-sig") as meta_file:
FileNotFoundError: [Errno 2] No such file or directory: 'posts/2017/09/02/spawning-subprocesses-smartly-and-securely.pl.rst'

I fixed both in commit f79b5c5.

felixfontein · 2018-05-05T21:18:55Z

Ah, that's what -sig is for. I copied the open command from post.py (and changed it from read to write), and forgot to think about the encoding specified there... Sorry for that!

Thanks for fixing this.

Adding copy of old plugin; will be changed for v8 metadata conversion.

94555d5

felixfontein added the new plugin label May 2, 2018

felixfontein mentioned this pull request May 2, 2018

[WIP] Conversion plugin for #2761 getnikola/nikola#3068

Merged

First version of plugin.

c94cff1

flake8

4e045cc

felixfontein requested a review from Kwpolska May 2, 2018 15:40

Kwpolska reviewed May 2, 2018

View reviewed changes

felixfontein added 3 commits May 2, 2018 21:19

Using nikola.utils.write_metadata.

3881184

Added section -> category conversion.

685be9c

Merge branch 'master' into plugin-for-2761

8b327b9

Kwpolska reviewed May 4, 2018

View reviewed changes

felixfontein added 2 commits May 4, 2018 23:19

Fixing grammar h/t @Kwpolska

86ed7f1

Don't wrap comments for ReST.

2555789

Bad typo...

946459e

Kwpolska merged commit c0918c7 into master May 5, 2018

Kwpolska deleted the plugin-for-2761 branch May 5, 2018 13:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Conversion plugin for Nikola's #2761: conversion plugin for pre-v8 tag-style metadata #271

[WIP] Conversion plugin for Nikola's #2761: conversion plugin for pre-v8 tag-style metadata #271

felixfontein commented May 2, 2018

felixfontein commented May 2, 2018

felixfontein commented May 2, 2018

Kwpolska May 2, 2018

felixfontein May 2, 2018

Kwpolska May 2, 2018

felixfontein May 2, 2018

Kwpolska May 2, 2018

felixfontein May 2, 2018

felixfontein May 2, 2018

Kwpolska May 3, 2018

felixfontein May 3, 2018

Kwpolska May 2, 2018

felixfontein May 2, 2018

Kwpolska May 2, 2018

felixfontein May 2, 2018

felixfontein May 2, 2018

ralsina commented May 2, 2018

felixfontein commented May 3, 2018

felixfontein commented May 4, 2018

felixfontein commented May 4, 2018

Kwpolska May 4, 2018

felixfontein May 4, 2018

Kwpolska May 4, 2018

felixfontein May 4, 2018

Kwpolska commented May 4, 2018

felixfontein commented May 4, 2018

felixfontein commented May 4, 2018

Kwpolska commented May 4, 2018

felixfontein commented May 4, 2018

felixfontein commented May 4, 2018

felixfontein commented May 4, 2018

Kwpolska commented May 5, 2018

felixfontein commented May 5, 2018

[WIP] Conversion plugin for Nikola's #2761: conversion plugin for pre-v8 tag-style metadata #271

[WIP] Conversion plugin for Nikola's #2761: conversion plugin for pre-v8 tag-style metadata #271

Conversation

felixfontein commented May 2, 2018

felixfontein commented May 2, 2018

felixfontein commented May 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ralsina commented May 2, 2018

felixfontein commented May 3, 2018

felixfontein commented May 4, 2018

felixfontein commented May 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kwpolska commented May 4, 2018

felixfontein commented May 4, 2018

felixfontein commented May 4, 2018

Kwpolska commented May 4, 2018

felixfontein commented May 4, 2018

felixfontein commented May 4, 2018

felixfontein commented May 4, 2018

Kwpolska commented May 5, 2018

felixfontein commented May 5, 2018