Fix: use __iconv on FreeBSD to skip invalid bytes #3379
Conversation
|
@zarianu can you test if it works for you, now? |
|
Yes! Works like a charm now, merci! As a FreeBSD user I can check on other 3 failing specs if you want. |
|
Just looked at failing spec at spec/std/string_spec.cr:1904. Looks like FreeBSD's iconv interface is pretty limited according to manuals. GNU iconv can detect invalid multibyte sequences by setting errno = EILSEQ, while FreeBSD iconv/__iconv (POSIX legacy) returns number of invalid characters. So maybe Crystal's iconv API should handle iconv.convert return values other than -1 on FreeBSD platform. |
|
@ysbaddaden As always, feel free to merge this whenever you think it's ready. I didn't test it on FreeBSD but I trust you. I'll make a release this Friday, mostly to have the atomic instructions support in the compiler to continue working on that more easily. By the way, if you release a new version of shards before that I can also include it in the next Crystal release, if you have time or if there are new features or bug fixes in shards. |
The GNU extensions from "libiconv" aren't supported on FreeBSD. We must use
__iconvto skip invalid bytes instead of adding//IGNOREto the charsets.There are now only 3 specs failing. One is minor (EINVAL is reported instead of EILSEQ), the 2 others should raise, but they don't... so maybe there is another issue (maybe in the way we should report invalids with
__iconv)?refs #3271