I recommend splitting this into some smaller tests like:

test "string_utils.is_ascii_letter" {
    assert(is_ascii_letter('C'));
    assert(is_ascii_letter('e'));
    assert(!is_ascii_letter('2'));
}

Yes of course, I just wanted to get a basic testing suite up first :).

See #888

Fair enough I'll remove it :), I shouldn't be calling 'git add .' anyways.

I think you should be able to use git add .. Everything to ignore falls into 2 categories: system-wide ignore, or project-specific ignore. If both are correct then git add . works just fine.

The decision to put these things in something called "memory" rather than "string" is intentional.

A "string" would be something like []u32, where each item is a unicode point. Zig std lib does not have a string data type yet - and it looks like this PR does not introduce one. Although we do have some utf8 decoding and encoding code in std/unicode.zig.

String utilities would be a welcome addition to the zig standard library, but they should operate on unicode points, not encoded bytes. Utilities that operate on encoded bytes should be clear about that, using parameter names such as "bytes" and explaining the difference between encoded strings and actual strings in the docs.

I'm fairly confident that we should not have asciiUpper and asciiLower in the standard library for these reasons:

• it's too easy to misuse, when actual strings should be used instead
• if it really is the odd case where you need ascii upper/ascii lower, writing out the function body is easy enough. it should be enough tedium to get people to consider using the actual string functions.

Do you have an actual use case for ascii upper/lower right now?

In the randomizer I use a few ascii functions(They take &const u8 because I use them as fn ptrs to generic functions), because I know the strings I get from the nds headers are ascii. I don't mind not having it in std though.

Hm, that's fair. Maybe it's not so bad to have them. We should definitely put a bunch of noticeable warning signs near them and avoid the use of the word "string".

They could live in ascii.zig. Then you sign up for ascii problems by importing that.

I'm ok with that.

note that "ascii" is different from []u8. ascii requires the top bit of every byte to be 0, so if you're doing ascii string processing, you might consider validating that top bit.

those ascii functions you linked are correct, because they check for specific ranges of values, but something like ascii_to_upper() should perhaps assert the top bit is 0.

One could also just check if it was within 127? That avoids the bit masking operations (which are more expensive and remove clarity)? But yes it probably does have to be checked, though I would rather verify data through the locale viewer (which would check just once) rather than an indeterminate amount of times :).

@andrewrk, @Hejsil
This is an idiom that I've done a lot since returning arrays from functions ain't pretty due to the arrays being allocated within the function stack, is there a better way to get an array output, and furthermore is there a better way to assign a slice to a new slice size without doing this 'trick'.

In most languages there is a syntax for creating a new array on the stack and assigning its value to something else, currently this requires a hidden memcpy presumably; which I think we could optimise away.

This is returning stack-allocated data from within the function to the caller and isn't valid.

Consider the following example:

const std = @import("std");
const warn = std.debug.warn;

fn example(out: &[]const u8, s: u8) void {
    var a: [4]u8 = undefined;
    a[0] = s;
    a[1] = s + 1;
    a[2] = s + 2;
    a[3] = s + 3;
    *out = a;
}

pub fn main() void {
    var a: []const u8 = undefined;
    example(&a, 0);

    warn("a: "); for (a) |e| { warn("{} ", e); } warn("\n");

    var b: []const u8 = undefined;
    example(&b, 1);

    warn("b: "); for (b) |e| { warn("{} ", e); } warn("\n");
    warn("a: "); for (a) |e| { warn("{} ", e); } warn("\n");
}

Gives the output:

$ zig run example.zig
a: 0 1 2 3 
b: 1 2 3 4 
a: 1 2 3 4

That is the best case that is happening, there are no guarantees about the data returned since everything within the function call is out of scope on exit unless it was dynamically allocated.

I would suggest instead taking a raw slice and simply asserting that the user has provided a sufficiently sized buffer. This allows the user to use their own buffer which they may want and provides the most flexibility.

Fixing up that previous example:

const std = @import("std");
const warn = std.debug.warn;

fn example(out: []u8, s: u8) void {
    std.debug.assert(out.len >= 4);
    out[0] = s;
    out[1] = s + 1;
    out[2] = s + 2;
    out[3] = s + 3;
}

pub fn main() void {
    var a: [4]u8 = undefined;
    example(a[0..], 0);

    warn("a: "); for (a) |e| { warn("{} ", e); } warn("\n");

    var b: [4]u8 = undefined;
    example(b[0..], 1);

    warn("b: "); for (b) |e| { warn("{} ", e); } warn("\n");
    warn("a: "); for (a) |e| { warn("{} ", e); } warn("\n");
}

If you really wanted to return fixed data you could create a new struct and wrap the array. This requires bounding the array size to the maximum needed. This is less flexible and I wouldn't implement it this way.

const std = @import("std");
const warn = std.debug.warn;

const Encoded = struct { out: [4]u8 };

fn example(s: u8) Encoded {
    var a: Encoded = undefined;
    a.out[0] = s;
    a.out[1] = s + 1;
    a.out[2] = s + 2;
    a.out[3] = s + 3;
    return a;
}

pub fn main() void {
    const a = example(0);

    warn("a: "); for (a.out) |e| { warn("{} ", e); } warn("\n");

    const b = example(1);

    warn("b: "); for (b.out) |e| { warn("{} ", e); } warn("\n");
    warn("a: "); for (a.out) |e| { warn("{} ", e); } warn("\n");
}

To return the length I'd just return it as the return value.

The only problem with it is that this is used for encoding bytes back to a byte[] which means the size isn't known prior for example, this means that it gets a bit finicky. The way I think may be better for this unknown size thing could either be requiring them to run a function to get how many bytes the code point will be OR;

fn example(s: [4]const u8) []const u8 {
    // ... do stuff with 's'
    return s[0..sizeWanted];
}

Perhaps? Or just result to a full buffer based system and return the length of that byte? Giving them the opportunity to either slice it or leave it? Tricky problem this one is.

I agree, I don't like the struct way; it feels icky.

You could do this, and get a static guarantee that the buffer is a certain size:

fn example(out: &[4]u8, s: u8) void {
    (*out)[0] = s;
    (*out)[1] = s + 1;
    (*out)[2] = s + 2;
    (*out)[3] = s + 3;
}

Related: #863

Oooh I do like that too.

One issue with requiring an array of a specific size is that you are forcing the caller to use an array of this type. The caller will need to memcpy the bytes if they want to move the bytes into any other sized array. The static guarantees are nice, but if the data is copied anywhere else you will be using a slice interface anyway so will simply be performing the implied check later down the track.

Also, does anywhere else in std take a fixed array argument?

This is why I want #863, which will allow slice -> array pointer casts.

It is true that the check has to be performed some other place than the call, but isn't this the same as the maybe type. Giving you a static guarantee that something is true, requiring the caller to ensure the truth.

I suppose the main point I'm trying to say is that since a slice is a view into a backing array so in my view it doesn't make sense to limit the output type to an array of a specific size, when you could use a slice and take an arbitrary sized array.

Your proposal looks nice and handles exactly this scenario. For now, I'd stick with taking a slice and if down the track we get better guarantees we can update any of these occurences.

I think mem.eql should be used when you don't care about encoding and only about memory, while something like unicode.eql should take into account that a "letter" can be a combination of many unicode code points, and that the order of these code points, don't always matter.

I did not realise that, I'll look into it and build a proper solution :0

I would expect that slicing a unicode string would be from start_code_point to end_code_point, and not just slice the memory and validate that the substring still is valid.

But then again, the string could be big so maybe this is better

Yes that check is not needed :), as said in my other response. This slice will be renamed because that is a valid point it mainly is used in my split function for example which doesn't need a concept of end points; how about sliceRaw? I'll do a proper slice one since I can just check the first bit to detect when to iterate the index though it'll be max O(N) { N := the raw point corresponding to the code point index } just to detect the range which is unavoidable.

Somehow I missed your second point, it isn't always needed :). So I'll probably add it as 'sliceRawUnchecked' for example

I don't think the things in mem should use these "string iterators". When I use mem.split I only care about memory, not encoding, views, interators and all that.

The next fn in here is a good reason, why mem.split shouldn't use these iterators. Iterating over memory can't fail with an error, because the elements don't need to be validated to decoded.

I disagree with both points because of a few reasons :).

1. We need split in a few cases (just in std); first for string splitting in windows which is Utf16 with a few weird oddities, with my example you can just create a Utf16Win view and corresponding iterator and you don't have to create a unique function for it :), we also need a generic Utf8 one for 'general use' and a raw Ascii one for memory or when you know your just dealing with bland non encoded data.

Thus if we just pretend that iterators and views don't exist it becomes combersome :).

2. You generally don't have to care about views and iterators :) as shown in the test example there exists a type under stringUtils called string_it which maps the iterators and views for you :). That is instead of calling split("A/B/C/", "/") you call stringUtils.string_it.init("A/B/C/", "/"), currently 'next' returns a view however I'm also gonna add a function that returns the raw type that you specify which in the case of the string one is []const u8, this should solve your issue of views as you'll never need to touch one. Views are typically a good thing as they generalise.

3. Your second statement is also false, Iterators don't EVER fail with an error :), just look at both the Utf8 Iterator and Ascii Iterator they have type ?[]u8 not !?[]u8 because of course they don't fail, HOWEVER the views can fail if you give them invalid data, I can change it to just set the data regardless however this would then ensure that the data you originally give it has to be valid which it doesn't necessarily have to be :), however technically you are right next doesn't have to be error checked since I can validate that the slice can use initUnchecked.

So overall I can make next() not return an error ever and it kinda is just due to me wanting to be 100% safe and that kinda eroded the practicality of it :). So with all that said I don't particularly get why you don't like it?

My only real problem is that I don't think it should be in mem. Memory splitting and string splitting is different. When we split memory, there is no concept of valid code points, encoding and all that. Therefore I think the interface should be simpler.

I agree, that for splitting paths, the string splitting functions you provide should be used, as paths are strings. However, there are cases where you just have some memory ([]u8, []16, []32, []64) and want to split on some values. Btw, stringUtils.string_it only works on ascii, so you can't split any arbitrary sequence of bytes.

I guess you could provide a memory view and iterator though, and that might work out. I really think split should not be able to fail not next for these though :)

Yeh it won't fail for next I already finished that, currently it can only fail upon initialisation of the iterator. Also you can get it to return the raw data instead of a view.

And yes it only works for ascii that is just due to me not bothering naming it properly. I think I'll name it stringUtils.split_ascii_it, and have stringUtils.split_it for Utf8 and have stringUtils.split_win_it for the windows special Utf16 (that is super annoying why can't windows do anything right lel).

Again I completely agree next will not fail for anything after my next set of changes :).

One last thing; so you would prefer mem.split to use the old system and have the new split exist under stringUtils :)?

Really appreciate you having an in-depth look at my code by the way, your points are really valid and are helpful 😄 .

I think so. I think mem should only concern itself with slices of data. Not a more complicated iterator/view system. Ofc, I don't have the final say, as I'm not a "designer" of Zig or it's std library.

One note: paths are not strings in linux. They are null-terminated arrays of bytes. You can have invalid utf-8 as a linux file path.

Oh yes I'll make sure that it allows for null-terminated arrays of bytes :). Pretty sure that path sanitises it to a slice anyway but I'll double check (then converts it back at the end).

representations is an ambiguous word. it asks the question, "representing what?"

Ahh fair, I'll work on a better name :).

it looks to me that every place you replaced mem.split with asciiSplit is incorrect. Here we are quite intentionally splitting on utf8-encoded bytes.

Yes :), mem.split only worked on ascii bytes so I've replaced it to only work with ascii then I was aiming to go through and replace it with utf8 where needed :).

Also I want to note that the fact you can just change it to string.utf8Split(...) and you suddenly use unicode correctly kinda shows how 'cool' it is.

why add u8 ?

Haven't gone through and removed little needless changes sorry.

std.os.path.join as it is currently implemented is correct, fully implemented, finished, and passes all tests. Why are we adding workarounds for anything?

Just converting a character into a string since the new join takes a string separator since I found it rarer to want a character than a string i.e. ", " is very common. If you know of a way to convert a character into a string without this little workaround then I would be overjoyed :).

Currently, I've decided to have a function to handle that, it literally does just convert it into an array like []u8 { sep } so functionally its the same but atleast its a bit nicer :). So you can just call string.joinCharSep() and give a single character sep rather than a string :).

I haven't changed my mind about this since the last PR comments

I need more clarification about what specifically you dislike :). Is it the call name ? Like is it just because it is not called string.split or is it the fact it uses generics behind the scene?

Again i'm happy to remove the type independent stuff if need be :)

this hash map is hashing bytes, not strings.

I think this shows a problem with the way its done because the hash_slice_u8 for example and hashStr would be identical and that leads to a singular name meaning two different things, I guess how would you approach this because in situations where it is hashing strings not bytes it leads to this same problem? I would either declare aliases or just live with one of them.

instead, callers should use slicing, and then call mem.copy.

Yes, I used that but then I realised slicing is soo much easier then just slipped my mind to remove it, even when you go through all your commits it is hard to find little things like this haha.

what is this comment communicating?

That I forgot to remove it :P

this is guaranteed by the type system. any time a type is casted to an enum there is a runtime safety check.

Oh sweet, still learning things

these should not be errors that are possible for ChildProcess.spawn to emit.

Why? They are possible now due to it now using Utf8? In saying that it should only return one of them, I believe InvalidCharacter, so I'll remove the other one.