Extending and refactoring WordPress importer #1867

felixfontein · 2015-07-06T20:19:14Z

Beginning with some refactoring and extending of the WordPress importer.
Done so far:

Feeding all posts together with post format through transform_content, and let it decide extension.
No longer using extension .wp for WordPress-to-markdown converted posts.
Exporting wp-status and excerpt (if available).
Allowing to exclude private items and include empty items.
Exporting attachment-post/page relation as JSON file.
Exporting categories (if there's only one per post) including category hierarchy.
Exporting comments.
Add support for WordPress page compiler plugin (both to make new site use it, or to use it to convert WordPress posts to .html files).

Kwpolska · 2015-07-07T08:08:16Z

Add support for WordPress page compiler plugin (both to make new site use it, or to use it to convert WordPress posts to .html files).

Should be simple to implement by just talking to the plugin plugin and loading the new plugin (can you call collectPlugins twice without everything going down in flames?). Make sure to add a command-line option to pick the usage mode, and if there is no value specified, make sure to ask — please include a note about the compiler’s license for anti-GPL zealots.

Kwpolska · 2015-07-07T08:49:52Z

nikola/plugins/command/import_wordpress.py

+            if out_folder_slug:
+                self.posts_pages[post_id] = (post_type, out_folder_slug[0], out_folder_slug[1])
+
+    def write_attachments_info(self, path, attachments):


Who would use those files? Would it actually be useful to give Nikola some real post attachments support, btw?

These files could be used by plugins for the WordPress page compiler. For example, if you include attachments into a post as a gallery, you have something like
[gallery columns="3" link="file" ids="52,53,54,55,56,57,58"]
inside the post -- here 52 up to 58 are IDs for attachments. If you have this .json file, you can map the IDs to actual images.

About giving Nikola proper post attachments functionality: I don't really care enough about that feature to implement/want it.

Kwpolska · 2015-07-07T08:51:13Z

Reviewed 1 of 2 files at r1.
Review status: 1 of 2 files reviewed at latest revision, 1 unresolved discussion, all commit checks successful.

Comments from the review on Reviewable.io

Kwpolska · 2015-07-07T08:53:15Z

^{nikola/plugins/command/import_wordpress.py, line 443 [r1] (raw file):}
Is there any other post format available? Would that be HTML, or what?

Comments from the review on Reviewable.io

felixfontein · 2015-07-07T19:12:40Z

Regarding the other post formats: I don't know. That was already implemented before I touched it; apparently there's a markdown plugin for WordPress. (I looked it up: #484)

felixfontein · 2015-07-07T19:27:50Z

Ok, I now looked at the WordPress plugin which allows to use different text formats, and added more explicit support, including an error when an unsupported format is encountered.

felixfontein · 2015-07-09T20:48:23Z

Regarding calling collectPlugins() more than once: the documentation doesn't explicitly disallow this, and the source seems to have no problem with it. Yapsy has a map for every category in which is indexed by the plugin info file's filename (PluginManager._category_file_mapping), so if you call the function again, it will go through all plugins again, load them, go through all categories to which they fit, and ignore them (assuming neither the plugin nor the categories changed since the last call of collectPlugins().

So installing a plugin and then calling collectPlugins() a second time will not hurt, but find the new plugin.

Kwpolska · 2015-07-10T07:52:01Z

nikola/plugins/command/import_wordpress.py

+            for cat, path in cat_map.items():
+                self._category_paths[cat] = utils.join_hierarchical_category_path(path)
+
+    def _adjust_config_template(self, channel, rendered_template):


huge 👎

Instead, you should change the input to the template and the template itself, just like I did in 0f4b1e2 (relevant lines: 57/216). Don’t mess with regex. (It was done before, this code is probably ancient)

Yes, I took the old code and modified it. I've removed it now and did it the way you wanted. (Actually I prefer it that way, too :) )

felixfontein · 2015-07-11T20:55:37Z

Ok, I think I'm done for now.

Kwpolska · 2015-07-12T08:17:12Z

CHANGES.txt

+  * Allowing to use WordPress page compiler on imported site instead of
+    converting posts to markdown with --use-wordpress-compiler
+  * Allowing to automatically install the WordPress page compiler when
+    needed with --install_wordpress_compiler


use - instead of _
also, you should advertise the compiler if the user does not ask for it

Sorry, that was a copy'n'paste typo. Fixed.

Kwpolska · 2015-07-12T08:52:10Z

Merge master (or rebase and force-push this branch).
Put your changes at the top of the Features section.
Test with some wordpress dumps if it works properly, if you haven’t already.

Kwpolska · 2015-07-12T09:13:47Z

docs/manual.txt

@@ -1908,13 +1908,23 @@ the following:

  This is also useful for DISQUS thread migration!

+* Allows you to export your comments with each post
+* Exports information on attachments per post
 * Will try to convert the content of your posts. This is *not* error free, because


put info about your compiler here, drop this paragraph (you should really push the wordpress compiler)

I rewrote the documentation a bit. Can you take another look?

…, and let it decide extension. No longer using extension .wp for WordPress-to-markdown converted posts. Exporting wp-status and excerpt (if available). Allowing to exclude private items and include empty items.

felixfontein · 2015-07-12T12:44:33Z

Ok, all tests successfull (with two of my WordPress dumps and different settings, and automatic ones). Squashing a bit, and merging.

Also supporting category hierarchies.

@Kwpolska

…tions (suggested by @Kwpolska).

@felixfontein

cc @felixfontein Signed-off-by: Chris Warrick <kwpolska@gmail.com>

…mport.

…from all actions.

… either locally (when transforming posts right away) or into the new site.

…l be called conf.py.import_wordpress-xxx)

Not showing 'assuming --user' message when output_dir is forced.

Also supplying attachment info to WordPress page compiler when directly converting to HTML.

…ss page compiler won't find shortcode plugins coming in the package.

Extending and refactoring WordPress importer.

felixfontein added the enhancement label Jul 6, 2015

felixfontein self-assigned this Jul 6, 2015

Kwpolska reviewed Jul 7, 2015
View reviewed changes

Kwpolska reviewed Jul 10, 2015
View reviewed changes

felixfontein added this to the v7.6.1 milestone Jul 11, 2015

Kwpolska reviewed Jul 12, 2015
View reviewed changes

felixfontein added 2 commits July 12, 2015 14:04

Feeding all posts together with post format through transform_content…

79d590d

…, and let it decide extension. No longer using extension .wp for WordPress-to-markdown converted posts. Exporting wp-status and excerpt (if available). Allowing to exclude private items and include empty items.

Exporting attachment-post/page relation as JSON file.

dd05a2a

felixfontein force-pushed the import-wordpress-refactoring branch from d36ced3 to 4cbf1aa Compare July 12, 2015 12:08

felixfontein and others added 9 commits July 12, 2015 14:47

Fixed typo and tests.

b03827b

Improved support for post formats.

4a3516f

Fixed wrong line breakups.

4414b6e

Added support for exporting categories as categories (instead of tags).

0aacf78

Also supporting category hierarchies.

Added support for exporting comments.

f59b1bf

Using templatized conf.in.py instead of post-rendering regex modifica…

5e418dc

…tions (suggested by @Kwpolska).

Add new variables to init and handle bools better

0480910

cc @felixfontein Signed-off-by: Chris Warrick <kwpolska@gmail.com>

Allowing to use WordPress page compiler to compile posts to HTML on i…

0ddd073

…mport.

Allowing to use WordPress page compiler to render .wp posts.

5982a5a

felixfontein added 12 commits July 12, 2015 14:50

Allowing to overwrite output directory, and returning boolean values …

5ba8005

…from all actions.

Added option which automatically installs the WordPress page compiler…

ee7402e

… either locally (when transforming posts right away) or into the new site.

Avoiding creating new_site directory too early (otherwise conf.py wil…

877c028

…l be called conf.py.import_wordpress-xxx)

Allowing to hide installation notes.

ec8ccea

Not showing 'assuming --user' message when output_dir is forced.

Less output cluttering when installing the WordPress compiler plugin.

928257a

Informing about changes. Also updated documentation a bit.

9196083

Optimizing WordPress plugin finding.

6265cd1

Using the plugin manager to get hold of the 'plugin' plugin.

a902537

Added support to pass additional data to WordPress plugin.

e368975

Also supplying attachment info to WordPress page compiler when directly converting to HTML.

Making sure to also rescan for compiler extensions. Otherwise WordPre…

662db83

…ss page compiler won't find shortcode plugins coming in the package.

Fixed test.

67d9f13

Fixed detection of no parent (for attachments).

190573a

felixfontein force-pushed the import-wordpress-refactoring branch from ae11c0e to 190573a Compare July 12, 2015 12:52

felixfontein added a commit that referenced this pull request Jul 12, 2015

Merge pull request #1867 from getnikola/import-wordpress-refactoring

1a7c983

Extending and refactoring WordPress importer.

felixfontein merged commit 1a7c983 into master Jul 12, 2015

felixfontein deleted the import-wordpress-refactoring branch July 12, 2015 12:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extending and refactoring WordPress importer #1867

Extending and refactoring WordPress importer #1867

felixfontein commented Jul 6, 2015

Kwpolska commented Jul 7, 2015

Kwpolska Jul 7, 2015

felixfontein Jul 7, 2015

felixfontein Jul 7, 2015

Kwpolska commented Jul 7, 2015

Kwpolska commented Jul 7, 2015

felixfontein commented Jul 7, 2015

felixfontein commented Jul 7, 2015

felixfontein commented Jul 9, 2015

Kwpolska Jul 10, 2015

felixfontein Jul 11, 2015

felixfontein commented Jul 11, 2015

Kwpolska Jul 12, 2015

felixfontein Jul 12, 2015

Kwpolska commented Jul 12, 2015

Kwpolska Jul 12, 2015

felixfontein Jul 12, 2015

felixfontein commented Jul 12, 2015

Extending and refactoring WordPress importer #1867

Extending and refactoring WordPress importer #1867

Conversation

felixfontein commented Jul 6, 2015

Kwpolska commented Jul 7, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kwpolska commented Jul 7, 2015

Kwpolska commented Jul 7, 2015

felixfontein commented Jul 7, 2015

felixfontein commented Jul 7, 2015

felixfontein commented Jul 9, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

felixfontein commented Jul 11, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kwpolska commented Jul 12, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

felixfontein commented Jul 12, 2015