I figure we'll probably move all the encoding/decoding formats into a subfolder at some point.

We also cannot zero-copy if we encounter a standard escape code (e.g. \n) which needs to be transformed.

This was a mildly confusing thing I came across when I had the specific value variants (ObjectSeparator, ValueEnd) not at the start of the enum. Printing the state names gave unexpected results.

Would indexing based on enumeration value make more sense?

Ah right. @tagName was around before enum members could have explicit values. I'll open an issue. #976

#489 may be nice here (more so on the stack overflow checks). Second transition is only required on numbers followed directly by }, ].

Want a has_escape flag set here as in the u unicode case.

Forgot these were still std.debug.warn outputs. Custom formatting functions would be useful here instead.

If this function is for debugging - there's a pattern of using foo.dump() as a method that returns void and prints a bunch of stuff to stderr about foo to aid in debugging.

What happens here if we pass an allocator derived from an ArenaAllocator and the use that as an input to our own ArenaAllocator? Are there other wrapping allocators which don't handle this correctly?

The worst that can happen is inefficient space usage. Allocators that wrap other allocators respect the API that Allocator provides and so everything will work correctly.

This should be a separate function such as mp.end(&mt1). The only token that is possible here would be to terminate a number value. We can never get multiple.

We could expose some helper methods on Value to make accessing nested objects slightly easier. Non-essential for the moment.

May be more idiomatic/nicer to return a double-token struct here.

I wonder if we could make it like the zig tokenizer, and make it always return exactly 1 token? Here's the signature:

    pub fn next(self: &Tokenizer) Token {

Here's what's nice about it:

• It can't fail. But it can return Token.Invalid. The invalid token has start and end of the invalid character(s), and then normal tokens can follow.
• It always returns 1 token.
  • If you get to the end it will always return Token.Eof.
  • If it needs to emit 2 tokens, it saves the one token and feeds it to the caller for next time, making the API simpler for the caller.

I just realized, the difference between the zig tokenizer and this json one is that the json one is streaming. So that makes sense.

If the caller already has to handle possibly greater than 1 token though, then would it make more sense for the feed function to accept a buf: []const u8 instead of c: u8, and then return N tokens?

...oh then it would need to be a callback, because if we needed to return > N tokens, and the return token slice was too small, we would lose data.

Alright, you're 2 steps ahead of me :-)

You could still change this to accept a slice. You instead could take a slice of tokens for output and fill until the input slice runs out or you run out of usable token space. If you run out of token space, return an error.NeedMoreTokens and save the current position in the buffer until the caller can use filled tokens and call again.

More complicated with the only real benefit being a saved function call cost. Technically you could perform faster matching of quoted-strings, ends of numbers etc by skipping some state changes but that would be special-casing longer buffer sizes and may have the reverse effect in other cases.

If this function is for debugging - there's a pattern of using foo.dump() as a method that returns void and prints a bunch of stuff to stderr about foo to aid in debugging.

Ah right. @tagName was around before enum members could have explicit values. I'll open an issue. #976

I wonder if we could make it like the zig tokenizer, and make it always return exactly 1 token? Here's the signature:

    pub fn next(self: &Tokenizer) Token {

Here's what's nice about it:

• It can't fail. But it can return Token.Invalid. The invalid token has start and end of the invalid character(s), and then normal tokens can follow.
• It always returns 1 token.
  • If you get to the end it will always return Token.Eof.
  • If it needs to emit 2 tokens, it saves the one token and feeds it to the caller for next time, making the API simpler for the caller.

max_stack_size should probably be a runtime parameter to init and then we would allocate stack on the heap. Because we may want to decide how much nesting to allow in the json data at runtime - or at least in the API of std.json

I was thinking if it was worthwhile to make the max_stack_size parameter in the streaming parser a comptime variable for this specific case. For the value decoder, I'd actually change this to just use an ArrayList and not be dependent on the streaming parsers stack size. The rationale being that the common case will likely not be nested so reduce memory usage for a few extra dynamic allocations/copies in the worst case.

The worst that can happen is inefficient space usage. Allocators that wrap other allocators respect the API that Allocator provides and so everything will work correctly.

you could build this with "[" ** 100000

you could build this with "[{\"\":" ** 100000

you could build this with ("[" ** 500) ++ ("]" ** 500)

is the u5 meant to save space? zig will use the minimum tag size if you do not specify. This feature is more meant for packed structs where you want to control the bit layout.

it would be idiomatic to rename TokenId to Id and nest it inside Token. Then it can be referred to with Token.Id