Pass language to slugify and unslugify. (WIP) #2284

felixfontein · 2016-03-06T17:15:23Z

First try on implementing passing of language to slugify and unslugify; see discussion in #2273.

Kwpolska · 2016-03-06T17:18:10Z

nikola/nikola.py

+                        if _tag_slugified in slugged_tags[lang]:
+                            if tag not in self.posts_per_tag:
+                                # Tags that differ only in case
+                                other_tag = [existing for existing in self.posts_per_tag.keys() if utils.slugify(existing) == _tag_slugified][0]


missing , lang here.

ralsina · 2016-03-06T17:19:35Z

Since slugify could be used by plugins, unless you want this to have to
wait until v8, you should give lang a default value.

El dom., mar. 6, 2016 14:15, Felix Fontein notifications@github.com
escribió:

First try on implementing passing of language to slugify and unslugify;

see discussion in #2273 #2273.

You can view, comment on, or merge this pull request online at:

#2284
Commit Summary

First try to pass language to slugify and unslugify (see Replace umlauts with correct vokals #2273).

File Changes

M docs/extending.txt
https://github.com/getnikola/nikola/pull/2284/files#diff-0 (2)

M nikola/nikola.py
https://github.com/getnikola/nikola/pull/2284/files#diff-1 (27)

M nikola/plugins/command/import_wordpress.py
https://github.com/getnikola/nikola/pull/2284/files#diff-2 (7)

M nikola/plugins/command/new_post.py
https://github.com/getnikola/nikola/pull/2284/files#diff-3 (4)

M nikola/plugins/task/authors.py
https://github.com/getnikola/nikola/pull/2284/files#diff-4 (12)

M nikola/plugins/task/galleries.py
https://github.com/getnikola/nikola/pull/2284/files#diff-5 (2)

M nikola/plugins/task/tags.py
https://github.com/getnikola/nikola/pull/2284/files#diff-6 (175)

M nikola/post.py
https://github.com/getnikola/nikola/pull/2284/files#diff-7 (11)

M nikola/utils.py
https://github.com/getnikola/nikola/pull/2284/files#diff-8 (12)

M tests/base.py
https://github.com/getnikola/nikola/pull/2284/files#diff-9 (1)

M tests/test_slugify.py
https://github.com/getnikola/nikola/pull/2284/files#diff-10 (16)

M tests/test_utils.py
https://github.com/getnikola/nikola/pull/2284/files#diff-11 (6)

Patch Links:

https://github.com/getnikola/nikola/pull/2284.patch

https://github.com/getnikola/nikola/pull/2284.diff

—
Reply to this email directly or view it on GitHub
#2284.

Kwpolska · 2016-03-06T17:20:20Z

I’m 👎 for this. You just wasted your time implementing a tiny obscure feature.

felixfontein · 2016-03-06T17:25:41Z

@ralsina: I added a warning in that case. This warning will be useful while this is WIP.

@Kwpolska: I really want to have this feature for my blog, for me it's neither tiny nor obscure. And at least @davidak and @akrist seem to agree since they bothered to open an issue resp. write code for this.

davidak · 2016-03-07T00:38:34Z

It is important for a german site since it is the grammatically correct representation.

Wordpress supports it since Version 3.6:
http://www.perun.net/2013/04/17/wordpress-3-6-umlaute-im-titel-ohne-plugin-entschaerfen/

before, most people used a plugin.

To implement this as a plugin would be ok for me.

there is a python module that does what we want:
https://pypi.python.org/pypi/translitcodec/

>>> import translitcodec
>>> import codecs
>>> codecs.encode(u'Schöne Grüße aus Köln', 'translit/long')
'Schoene Gruesse aus Koeln'

felixfontein · 2016-03-07T06:27:54Z

@davidak: this Python module cannot be used for all languages since for most it would be simply wrong. Also, I wonder how well it works for languages such as Chinese and Japanese.

My goal is to not change Nikola's default behavior, but to just enable this to be changed by plugins without writing a really insane plugin (which does post preprocessing etc.). Such a plugin could use this Python module for the German language, though.

About WordPress: I wonder why they only support de_DE and not also de_CH and de_AT. Also, the current version of WordPress uses similar translations for Danish (da_DK).

davidak · 2016-03-07T07:26:40Z

Such a plugin could use this Python module for the German language, though.

that was my intention.

ralsina · 2016-03-07T12:33:01Z

This looks ok to me, and changes no behaviour. If noone sees a bug in the next 24 hours, I'll merge.

felixfontein · 2016-03-07T12:44:47Z

@ralsina: maybe we should leave it open a bit longer, I'd like to do some more intense tests (like with a real blog :) ). I'm not sure yet when I'll have time for this, but probably not during the next 24h.

Kwpolska · 2016-03-07T15:50:35Z

(Note we currently use unidecode which is a module that does the same as translitcodec, just with different opinions, and I’m pretty sure CJK text is complete garbage with both)

Kwpolska · 2016-03-07T15:51:32Z

tests/test_utils.py

@@ -26,6 +26,7 @@ def test_getting_metadata_from_content(self):
        post = dummy()
        post.source_path = 'file_with_metadata'
        post.metadata_path = 'file_with_metadata.meta'
+        post.default_lang = 'en'


Would be nicer to make a special post_dummy object with default_lang = 'en' already.

I'm using dummy now, since that's precisely used in these situations.

ralsina · 2016-03-07T16:44:31Z

@felixfontain ok, let's give it a few days. I want to do a release before friday because I will then be unavailable for a few weeks. Then again, if it doesn't go in this release, it will go in the next one.

felixfontein · 2016-03-08T22:32:46Z

Seems to work fine. I created a mini plugin for testing, for those who are interested, which can be put into the plugins/ folder of a blog:
The description file (t.plugin):

[Core]
Name = t
Module = t

The code (t.py):

# -*- coding: utf-8 -*-

from nikola.plugin_categories import SignalHandler
from nikola import utils
from unidecode import unidecode
import re


_slugify_strip_re = re.compile(r'[^+\w\s-]')
_slugify_hyphenate_re = re.compile(r'[-\s]+')


def my_slugify(value, lang=None, force=False):
    if not isinstance(value, utils.unicode_str):
        raise ValueError("Not a unicode object: {0}".format(value))
    value = value.replace('ä', 'ae').replace('ö', 'oe').replace('ü', 'ue').replace('ß', 'ss')
    value = value.replace('Ä', 'ae').replace('Ö', 'oe').replace('Ü', 'ue')
    value = utils.unicode_str(unidecode(value))
    value = _slugify_strip_re.sub('', value, re.UNICODE).strip().lower()
    return 'slugified-' + _slugify_hyphenate_re.sub('-', value, re.UNICODE)


class T(SignalHandler):  # Could also be any other plugin type which is initialized early enough.
    def set_site(self, site):
        super(T, self).set_site(site)
        utils.slugify = my_slugify

I recommend to remove the 'slugified-' prefix for production use ;-)

So from my side, merging is fine.

felixfontein · 2016-03-10T15:29:52Z

@ralsina: do you want to merge, or should I just do it myself?

Pass language to slugify and unslugify. (WIP)

Kwpolska · 2016-03-10T16:41:58Z

Merged, let's hope it works.

felixfontein · 2016-03-10T19:06:05Z

Thanks for merging!

First try to pass language to slugify and unslugify (see #2273).

361ca29

felixfontein added enhancement in progress i18n labels Mar 6, 2016

Kwpolska reviewed Mar 6, 2016
View reviewed changes

felixfontein added 2 commits March 6, 2016 18:20

Fixed missing lang.

cd17877

Warning instead of failing when (un)slugify is called without language.

f5816a8

Improving warnings about missing language for slugify* variants.

0b98214

Kwpolska reviewed Mar 7, 2016
View reviewed changes

Moving default_lang into dummy.

b35f8e1

Kwpolska added a commit that referenced this pull request Mar 10, 2016

Merge pull request #2284 from getnikola/pass-lang-to-slugify

4db06f1

Pass language to slugify and unslugify. (WIP)

Kwpolska merged commit 4db06f1 into master Mar 10, 2016

Kwpolska deleted the pass-lang-to-slugify branch March 10, 2016 16:42

felixfontein mentioned this pull request Mar 11, 2016

Added German slugifying plugin. getnikola/plugins#145

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pass language to slugify and unslugify. (WIP) #2284

Pass language to slugify and unslugify. (WIP) #2284

felixfontein commented Mar 6, 2016

Kwpolska Mar 6, 2016

ralsina commented Mar 6, 2016

see discussion in #2273 #2273.

Kwpolska commented Mar 6, 2016

felixfontein commented Mar 6, 2016

davidak commented Mar 7, 2016

felixfontein commented Mar 7, 2016

davidak commented Mar 7, 2016

ralsina commented Mar 7, 2016

felixfontein commented Mar 7, 2016

Kwpolska commented Mar 7, 2016

Kwpolska Mar 7, 2016

felixfontein Mar 8, 2016

ralsina commented Mar 7, 2016

felixfontein commented Mar 8, 2016

felixfontein commented Mar 10, 2016

Kwpolska commented Mar 10, 2016

felixfontein commented Mar 10, 2016

Pass language to slugify and unslugify. (WIP) #2284

Pass language to slugify and unslugify. (WIP) #2284

Conversation

felixfontein commented Mar 6, 2016

Kwpolska Mar 6, 2016

Choose a reason for hiding this comment

ralsina commented Mar 6, 2016

see discussion in #2273 #2273.

Kwpolska commented Mar 6, 2016

felixfontein commented Mar 6, 2016

davidak commented Mar 7, 2016

felixfontein commented Mar 7, 2016

davidak commented Mar 7, 2016

ralsina commented Mar 7, 2016

felixfontein commented Mar 7, 2016

Kwpolska commented Mar 7, 2016

Kwpolska Mar 7, 2016

Choose a reason for hiding this comment

felixfontein Mar 8, 2016

Choose a reason for hiding this comment

ralsina commented Mar 7, 2016

felixfontein commented Mar 8, 2016

felixfontein commented Mar 10, 2016

Kwpolska commented Mar 10, 2016

felixfontein commented Mar 10, 2016