Ignore sliced multibyte characters when reading metadata for sitemaps

getnikola · Aug 9, 2015 · 4d859e0 · 4d859e0 · muxuezi · Aug 17, 2015
1 parent 0860058
commit 4d859e0
Show file tree

Hide file tree

Showing 2 changed files with 2 additions and 1 deletion.
diff --git a/CHANGES.txt b/CHANGES.txt
@@ -12,6 +12,7 @@ Features
 Bugfixes
 --------
 
+* Ignore sliced multibyte characters when reading metadata for sitemaps
 * Handle folders without trailing slashes in nikola auto (Issue #1933)
 * Set a base element to aid relative URL resolution, stripped on-the-fly
   when using the auto or serve command to view site locally. (Issue #1922)

diff --git a/nikola/plugins/task/sitemap/__init__.py b/nikola/plugins/task/sitemap/__init__.py
@@ -188,7 +188,7 @@ def scan_locs():
                                                  b'<meta content=none name=robots',
                                                  b'<meta name=robots content=noindex',
                                                  b'<meta name=robots content=none']
-                            lowquothead = filehead.lower().decode('utf-8').replace('"', '').encode('utf-8')
+                            lowquothead = filehead.lower().decode('utf-8', 'ignore').replace('"', '').encode('utf-8')
                             if any([robot_directive in lowquothead for robot_directive in robots_directives]):
                                 continue