Add MIME registry #5765

straight-shoota · 2018-03-03T22:39:49Z

This adds a MIME type which works as registry for mime types and allows to look up the mime type registered for each file name extension.

By default, a limited set of mime types is always registered, including those that have previously been used in HTTP::StaticFileHandler. Additionally, there is an implementation in Crystal::System::MIME that tries do load a mime database from the operating system. On unix systems the following paths are looked up:

/etc/mime.types
/etc/apache2/mime.types
/etc/apache/mime.types
# only freebsd
/usr/local/etc/mime.types
# only openbsd
/usr/share/misc/mime.types

For windows this is not yet implemented because we're missing access to the registry.

Applications (or shards) can also register custom mime types and extensions.

Closes #4632

src/mime.cr

Sija · 2018-03-04T13:38:38Z

src/mime.cr

+#
+# ```
+# require "mime"
+# MIME[".html"] # => "text/html"


Is MIME.[] implemented?

Ah, no. I was unsure about that. Should we have MIME.[] and/or MIME.fetch? Or a different name?

Personally, I'm finding Klass.[] aliased to Klass.fetch as a quite nice syntactic sugar, but of course there's the Crystal stand on aliasing...

No aliases please. Let's just name these methods normally and leave aside the unnecessary syntax sugar.

Yeah, I figure it doesn't make much sense here anyway.

straight-shoota · 2018-03-07T13:43:40Z

I've added .jpeg entry and removed the doc reference to MIME.[].

Anything else?

woodruffw · 2018-03-07T14:27:03Z

src/mime.cr

+    ".htm"  => "text/html; charset=utf-8",
+    ".html" => "text/html; charset=utf-8",
+    ".jpg"  => "image/jpeg",
+    ".jepg" => "image/jpeg",


Small typo 🙂

src/crystal/system/mime.cr

RX14 · 2018-03-07T17:10:28Z

src/crystal/system/unix/mime.cr

+  ]
+
+  {% if flag?(:freebsd) %}
+  MIME_SOURCES << "/usr/local/etc/mime.types"


macro indent.

RX14 · 2018-03-07T17:14:19Z

src/mime.cr

+    mediatype = parse_media_type(type) || raise Error.new "invalid media type: #{type}"
+
+    @@types[extension] = type
+    @@types_lower[extension.downcase] = type


But you always downcase on line 99.

Yeah, it shouldn't be downcased before.

Why bother keeping two hashes, and not just downcase-d one?

@Sija Resolution should be case-sensitive by default and only fall back to case-insensitive if there is no match.

RX14 · 2018-03-07T17:42:40Z

src/mime.cr

+    @@types_lower[extension.downcase] = type
+
+    if mediatype
+      array = @@extensions.fetch(mediatype) do


@@extensions[mediatype] ||= [] of String?

Sija · 2018-04-05T16:02:46Z

Merge 🕦?

RX14 · 2018-04-06T09:43:53Z

spec/std/mime_spec.cr

+      MIME.extensions("text/custom-type").should eq [".Custom-Type", ".custom-type2"]
+
+      MIME.register(".custom-type", "text/custom-type-lower")
+      MIME.fetch(".custom-type").should eq "text/custom-type-lower"


what happens when you fetch .Custom-Type here?

The same as before. I can add a spec.

RX14 · 2018-04-06T09:44:59Z

src/crystal/system/win32/mime.cr

+module Crystal::System::MIME
+  # Load MIME types from operating system source.
+  def self.load
+    # stub


Replace this with a comment saying that windows supplies no mime types.

Well it does, You'll just need to be able to access the registry to get them. But I will clarify that and add a TODO.

RX14 · 2018-04-06T09:48:46Z

src/mime.cr

+  #
+  # A case sensitive search is tried first, if this yields not result, it it
+  # matched case-insensitive. Returns *default* if *extension* is not registered.
+  def self.fetch(extension : String, default) : String


Perhaps rename fetch to from_extension? Or make extension required? MIME.fetch(extension: ".foo"). Having a MIME.from_filename("index.html") or a MIME.fetch(filename: "index.html") could be really nice (it just calls extname for you).

Comment could be improved too: if this yields not result, it it matched case-insensitive.

The problem with a required named argument is that the default argument would be weird. And I don't think named argument should be used to distinguish whether the method receives an extension or filename argument.

from_filename and from_extension sounds good.

Although I really like fetch because it's short ;)

Sija · 2018-04-06T19:47:31Z

src/mime.cr

+  #
+  # *extension* must start with a dot (`.`) and must not contain any null bytes.
+  def self.register(extension : String, type : String) : Nil
+    raise ArgumentError.new("extension does not start with a dot: #{extension.inspect}") unless extension.starts_with?('.')


Please use Sentence case for exception messages.
ditto all below.

RX14

Just fix the specs and we're good to go.

ysbaddaden

I believe a few things could be done better. Especially the MIME <-> Crystal::System::MIME relationship.

Last but not leat: I'm really sick of seeing return nil. I have goosebumps every single time I see these, and I really want to burn them with a f****** flamethrower.

ysbaddaden · 2018-04-09T08:48:09Z

src/mime.cr

+  private TSPECIAL_CHARACTERS = {'(', ')', '<', '>', '@', ',', ';', ':', '\\', '"', '/', '[', ']', '?', '='}
+
+  # :nodoc:
+  def self.parse_media_type(type : String) : String?


Must be private. It's only public for testing purposes, but internal implementation detail musn't be tested directly. Test MIME.register with various cases instead.

I'd want to avoid speccing this through MIME.register for it's side effects. It's easier to just spec the method directly.

Easier != what should be done.

I don't deny the tests were helpful to design the feature, but either #parse_media_type as a value by itself, and thus should be public & tested, or it's an implementation detail and associated tests can be dropped.

ysbaddaden · 2018-04-09T08:52:21Z

src/crystal/system/unix/mime.cr

+    MIME_SOURCES.each do |path|
+      next unless ::File.exists?(path)
+      ::File.open(path) do |file|
+        ::MIME.load_mime_database file


That should work the other way around: Crystal::System::MIME should provide a method that returns (or yields) a mime.types file that MIME would try to parse.

That would avoid the circular dependency that MIME depends on Crystal::System::MIME which itself depends on MIME.

I don't like the circular dependency either. But the fact that Crystal::System::MIME.load utilizes MIME.load_mime_database is already a platform specific implementation.

It could certainly be reversed if load yields if it needs to load a file. On windows it just wouldn't yield, but load the database differently. But that's baking platform-specifics into the abstraction API.

That's how I chose to implement it and I think it should stay this way. But I won't hurt about changing if requested.

It's similar with Crystal::System::Time.load_localtime btw., the Unix implementation calls the non-platform-specific implementation Time::Location.read_zoneinfo.

I think what makes me wary is that something from Crystal::System is initializing an external namespace; and ::MIME should initialize itself using Crystal::System. I wish we could find a better way.

We could pull the platform-dependent initialization implementations out of Crystal::System into the main namespace. But that doesn't change anything, really.

ysbaddaden · 2018-04-09T08:57:06Z

src/mime.cr

+  end
+
+  # :nodoc:
+  def self.load_mime_database(io : IO) : Nil


Either :

make it public —so we can load whatever mime.types file we want, which would be quite useful (especially on Windows);

make it private —and have Crystal::System::MIME merely tell where the mime.types files are located;

move it to Crystal::System::MIME that will return/yield mime types that MIME can register (compatible with Windows' registry, no circular dependency).

Yes, it should be public.

ysbaddaden · 2018-04-09T12:00:47Z

src/mime.cr

+      array << extension
+    end
+
+    nil


Redundant noise: the method's signature is already Nil.

Sija · 2018-04-09T13:41:40Z

I'm really sick of seeing return nil. I have goosebumps every single time I see these, and I really want to burn them with a f****** flamethrower.

@ysbaddaden wow, you just become my personal holy warrior of code style nit-picking 😲

straight-shoota · 2018-04-26T12:41:12Z

@ysbaddaden Are your concerns satisfied?

However, browsers prompt for download since they're not coming down with a relevant mimetype (they have application/octet-stream). See: crystal-lang/crystal#5765

straight-shoota · 2018-06-05T13:36:55Z

Can this be merged?

straight-shoota · 2018-11-08T15:24:40Z

@asterite I think it should be more flexible than that, for example for loading the MIME database from a network resource. That's what init(false) is for.

bcardiff

initialize_types can be just init if init is guarded with a return if @@initalized. Since it's wrong to call it twice, I think that is a good thing to do.

But is not mandatory.

Also I would've called reset_initialized just reset. Again, not mandatory. 👍on my side.

straight-shoota · 2018-11-08T16:05:06Z

It's not actually wrong. init can be called multiple times. I intended to mention that in the docs but forgot it.

But I'll rename reset_initialized.

straight-shoota · 2018-11-08T16:09:59Z

@Sija: Named arguments for what?

Sija · 2018-11-08T20:26:22Z

@straight-shoota MIME.init(false)

RX14 · 2018-11-10T16:01:10Z

I think a squash + rebase to 2 commits and we're done

Sija · 2018-11-10T20:33:32Z

src/mime.cr

+    ".html" => "text/html; charset=utf-8",
+    ".jpg"  => "image/jpeg",
+    ".jpeg" => "image/jpeg",
+    ".js"   => "application/x-javascript",


Adding ; charset=utf-8 IMO would be a good failsafe (same as in .css).

The charset parameter is only relevant for text/* types.

Nope, see this or this for instance. Btw, official type for javascript atm is application/javascript (without x- prefix).

application/javascript is the official MIME type for JavaScript as per RFC 4329, but application/x-javascript is still considered a de-factor standard.
This list was originally taken from the Go mime package but they've switched to application/javascript since this PR was created and MDN suggest using that as well. So I guess it's fine to change this.

The RFC also specifies a charset parameter for application/javascript, so then it would be okay to use charset=utf8 as well.

straight-shoota · 2018-11-11T11:40:44Z

Changed application/x-javascript to application/javascript; charset=utf-8.

Rebased & squashed.

src/mime.cr

RX14 · 2018-11-13T14:28:37Z

🎉

straight-shoota force-pushed the jm/feature/mime branch 2 times, most recently from bbea07c to c8ad4eb Compare March 3, 2018 22:43

woodruffw reviewed Mar 4, 2018

View reviewed changes

src/mime.cr Show resolved Hide resolved

Sija reviewed Mar 4, 2018

View reviewed changes

straight-shoota force-pushed the jm/feature/mime branch 2 times, most recently from 633ba09 to a4b5703 Compare March 7, 2018 13:42

woodruffw reviewed Mar 7, 2018

View reviewed changes

straight-shoota force-pushed the jm/feature/mime branch from a4b5703 to d9da99a Compare March 7, 2018 15:24

RX14 reviewed Mar 7, 2018

View reviewed changes

straight-shoota force-pushed the jm/feature/mime branch from d9da99a to da3e5ac Compare March 7, 2018 19:33

straight-shoota mentioned this pull request Mar 23, 2018

Remove BakedFile#mime_type schovi/baked_file_system#17

Closed

straight-shoota mentioned this pull request Apr 5, 2018

Support parsing Content-Disposition: file headers #5686

Closed

RX14 reviewed Apr 6, 2018

View reviewed changes

straight-shoota force-pushed the jm/feature/mime branch from daa809c to 49c1543 Compare April 6, 2018 19:20

Sija reviewed Apr 6, 2018

View reviewed changes

RX14 approved these changes Apr 7, 2018

View reviewed changes

jhass approved these changes Apr 9, 2018

View reviewed changes

ysbaddaden requested changes Apr 9, 2018

View reviewed changes

ysbaddaden reviewed Apr 9, 2018

View reviewed changes

straight-shoota mentioned this pull request Apr 13, 2018

Remove BakedFile#mime_type schovi/baked_file_system#22

Merged

RX14 approved these changes Jun 6, 2018

View reviewed changes

RX14 requested review from ysbaddaden and bcardiff June 6, 2018 11:00

bcardiff approved these changes Nov 8, 2018

View reviewed changes

RX14 approved these changes Nov 10, 2018

View reviewed changes

RX14 added this to the 0.27.1 milestone Nov 10, 2018

RX14 added kind:feature topic:stdlib labels Nov 10, 2018

Sija reviewed Nov 10, 2018

View reviewed changes

straight-shoota added 2 commits November 11, 2018 12:39

Add MIME registry

c022585

Use MIME in HTTP::StaticFileHandler

e4a128f

straight-shoota force-pushed the jm/feature/mime branch from adb6608 to e4a128f Compare November 11, 2018 11:39

Sija reviewed Nov 11, 2018

View reviewed changes

src/mime.cr Show resolved Hide resolved

RX14 approved these changes Nov 13, 2018

View reviewed changes

RX14 merged commit 5387248 into crystal-lang:master Nov 13, 2018

straight-shoota deleted the jm/feature/mime branch November 13, 2018 15:25

This was referenced Nov 13, 2018

Add .json (application/json) mime type #7079

Merged

Add .wasm (application/wasm) mime type #7080

Merged

bcardiff added topic:stdlib:networking and removed topic:stdlib labels Jan 28, 2019

Sija mentioned this pull request Feb 5, 2019

Use Crystal MIME registry kemalcr/kemal#516

Merged

Add MIME registry #5765

Add MIME registry #5765

Conversation

straight-shoota commented Mar 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

straight-shoota commented Mar 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Sija commented Apr 5, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Sija Apr 6, 2018 • edited

Choose a reason for hiding this comment

RX14 left a comment

Choose a reason for hiding this comment

ysbaddaden left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Sija commented Apr 9, 2018 • edited

straight-shoota commented Apr 26, 2018

straight-shoota commented Jun 5, 2018

straight-shoota commented Nov 8, 2018

bcardiff left a comment

Choose a reason for hiding this comment

straight-shoota commented Nov 8, 2018

straight-shoota commented Nov 8, 2018

Sija commented Nov 8, 2018

RX14 commented Nov 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Sija Nov 10, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

straight-shoota commented Nov 11, 2018 • edited

RX14 commented Nov 13, 2018

Sija commented Apr 5, 2018 •

edited

Sija Apr 6, 2018 •

edited

Sija commented Apr 9, 2018 •

edited

Sija Nov 10, 2018 •

edited

straight-shoota commented Nov 11, 2018 •

edited