Add fuzz test #1157

isaachier · 2018-06-24T12:08:26Z

So far it is really easy to generate an error. A compiler that doesn't crash implies it shouldn't choke on bad input. Once the compiler is mature enough, this might make sense as part of the standard testing in Travis CI.

isaachier · 2018-06-24T12:09:09Z

CMakeLists.txt

@@ -52,14 +52,14 @@ find_package(clang)

 if(NOT MSVC)
    find_library(LIBXML2 NAMES xml2 libxml2)
-    if(${LIBXML2} STREQUAL "LIBXML2-NOTFOUND")


One of the worst parts of the CMake language is that all *-NOTFOUND strings evaluate to false.

bnoordhuis · 2018-06-24T13:03:05Z

src/fuzz_test.cpp

@@ -0,0 +1,26 @@
+#include <unistd.h>
+
+#include "main.hpp"


You can #include "main.cpp" here. That avoids a lot of churn.

Not a bad idea but I'd have to conditionally rename main when fuzz test is selected.

You mean this?

#define main not_real_main #include "main.cpp" #undef main

Seems acceptable to me.

bnoordhuis

main.hpp can be dropped again now, can't it?

bnoordhuis · 2018-06-24T21:06:56Z

CMakeLists.txt

+if(BUILD_TESTING AND CMAKE_CXX_COMPILER_ID MATCHES "Clang")
+    add_executable(fuzz_test src/fuzz_test.cpp ${ZIG_SOURCES} ${ZIG_CPP_SOURCES})
+    target_compile_options(fuzz_test PRIVATE "-fsanitize=address,undefined,fuzzer")
+    target_link_libraries(fuzz_test PRIVATE "-fsanitize=address,undefined,fuzzer"


This is currently failing with the xcode build, presumably because xcode doesn't have libFuzzer enabled:

/Applications/Xcode.app/Contents/Developer/usr/bin/g++ -I/Users/travis/build/ziglang/zig/deps/lld/include -I/usr/local/Cellar/llvm/6.0.0/include -I/usr/local/opt/llvm@6/include -I/Users/travis/build/ziglang/zig/deps/SoftFloat-3e/source/include -I/Users/travis/build/ziglang/zig -I/Users/travis/build/ziglang/zig/build -I/Users/travis/build/ziglang/zig/src -I/Users/travis/build/ziglang/zig/deps/SoftFloat-3e-prebuilt -I/Users/travis/build/ziglang/zig/deps/SoftFloat-3e/source/8086 -g -Wall -fsanitize=address,undefined,fuzzer -o CMakeFiles/fuzz_test.dir/src/fuzz_test.cpp.o -c /Users/travis/build/ziglang/zig/src/fuzz_test.cpp clang: error: unsupported argument 'fuzzer' to option 'fsanitize='

Don't know about xcode 10 but with xcode 9, neither clang++ nor g++ understand -fsanitize=fuzzer.

Mac aliases g++ to clang++ so they are actually the same compiler. Ironically, despite Apple's support for clang, Mac seems to be incapable of running the sanitizers/fuzzer by default. I know all about this because I have a Mac laptop.

bnoordhuis · 2018-06-24T21:12:45Z

src/fuzz_test.cpp

+        perror("Cannot create temporary file");
+        return 0;
+    }
+    const int num_written = write(fd, data, size);


I'd use fdopen() and fwrite() since the latter takes care of retrying, but if you want to use write(), handle EINTR:

ssize_t n; do { n = write(fd, data, size); } while (n == -1 && errno == EINTR);

Good to know.

isaachier · 2018-06-24T21:38:51Z

Yes I must have forgotten to delete it. Updated now.

isaachier · 2018-06-24T23:34:07Z

Why is Mac the only build in Travis CI?

tiehuis · 2018-06-25T00:18:52Z

The .1 target is actually using the linux build script. Not sure why it specifies XCode as the compiler, though. Seems like it is using the osx_image yaml value on both builds, not too important.

bnoordhuis

LGTM with some comments. Nice work!

bnoordhuis · 2018-06-25T10:04:07Z

CMakeLists.txt

@@ -727,7 +726,7 @@ if(MSVC)
 elseif(MINGW)
  set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -Wall -Werror -Wno-error=format= -Wno-error=format -Wno-error=format-extra-args")
 else()
-  set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -Werror -Wall")
+  set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -Wall")


Just curious, why is it necessary to drop -Werror?

My bad. This was a local issue I needed to ignore momentarily. Will fix now.

bnoordhuis · 2018-06-25T10:04:58Z

CMakeLists.txt

+set(CMAKE_REQUIRED_FLAGS "${fuzzer_flags}")
+check_cxx_source_compiles("
+#include <stdint.h>
+#include <stdlib.h>


This is for size_t, right? Technically, that's from stddef.h.

OK ya the two headers are for size_t and uint8_t. Will try to replace stdlib with stddef.

bnoordhuis · 2018-06-25T10:06:03Z

src/fuzz_test.cpp

+
+    char arg0[] = "zig";
+    char arg1[] = "build-obj";
+    char* argv[] = { arg0, arg1, tmp_file_name };


This should have a nullptr at the end.

Good catch.

bnoordhuis · 2018-06-25T10:06:26Z

src/fuzz_test.cpp

+        perror("Cannot open file handle");
+        return 0;
+    }
+    const int num_written = fwrite(data, 1, size, f);


Use size_t here.

Thanks will do.

isaachier · 2018-06-25T12:57:38Z

OK all-in-all the fuzz test isn't exactly useful at the moment. Had to remove address sanitizer to avoid crashing due to memory leaks (even -detect_leaks=0 doesn't seem to fix that). The options I am using when running this are ./fuzz_test -only_ascii=1 -detect_leaks=0, but seem to get an error each time. Here is the message:

INFO: Seed: 3131641867
INFO: Loaded 1 modules   (147862 inline 8-bit counters): 147862 [0x339b8fc, 0x33bfa92), 
INFO: Loaded 1 PC tables (147862 PCs): 147862 [0x293bdd0,0x2b7d730), 
INFO:        1 files found in corpus
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: seed corpus: files: 1 min: 21b max: 21b total: 21b rss: 134Mb
/home/ihier/proj/zig/zig/build/lib/zig/std/fmt/index.zig:106:23: error: expected type 'TypeId', found 'TypeId'
        builtin.TypeId.Int, builtin.TypeId.Float => {
                      ^
/home/ihier/proj/zig/zig/build/lib/zig/std/fmt/index.zig:52:35: note: called from here
                    try formatType(args[next_arg], fmt[0..0], context, Errors, output);
                                  ^
/home/ihier/proj/zig/zig/build/lib/zig/std/io.zig:214:34: note: called from here
            return std.fmt.format(self, Error, self.writeFn, format, args);
                                 ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:129:17: note: called from here
    stderr.print(format ++ "\n", args) catch os.abort();
                ^
/home/ihier/proj/zig/zig/build/lib/zig/std/special/panic.zig:18:33: note: called from here
            std.debug.panicExtra(error_return_trace, first_trace_addr, "{}", msg);
                                ^
/home/ihier/proj/zig/zig/build/lib/zig/std/fmt/index.zig:106:23: error: expected type 'TypeId', found 'TypeId'
        builtin.TypeId.Int, builtin.TypeId.Float => {
                      ^
/home/ihier/proj/zig/zig/build/lib/zig/std/fmt/index.zig:52:35: note: called from here
                    try formatType(args[next_arg], fmt[0..0], context, Errors, output);
                                  ^
/home/ihier/proj/zig/zig/build/lib/zig/std/io.zig:214:34: note: called from here
            return std.fmt.format(self, Error, self.writeFn, format, args);
                                 ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:64:21: note: called from here
        stderr.print("Unable to dump stack trace: Unable to open debug info: {}\n", @errorName(err)) catch return;
                    ^
/home/ihier/proj/zig/zig/build/lib/zig/std/fmt/index.zig:106:23: error: expected type 'TypeId', found 'TypeId'
        builtin.TypeId.Int, builtin.TypeId.Float => {
                      ^
/home/ihier/proj/zig/zig/build/lib/zig/std/fmt/index.zig:71:35: note: called from here
                    try formatType(args[next_arg], fmt[s..i], context, Errors, output);
                                  ^
/home/ihier/proj/zig/zig/build/lib/zig/std/io.zig:214:34: note: called from here
            return std.fmt.format(self, Error, self.writeFn, format, args);
                                 ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:216:37: note: called from here
                try out_stream.print("???:?:?: " ++ DIM ++ ptr_hex ++ " in ??? (???)" ++ RESET ++ "\n    ???\n\n", address);
                                    ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:159:33: note: called from here
        try printSourceAtAddress(debug_info, out_stream, return_address);
                                ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:67:20: note: called from here
    writeStackTrace(stack_trace, stderr, getDebugInfoAllocator(), debug_info, stderr_file.isTty()) catch |err| {
                   ^
/home/ihier/proj/zig/zig/build/lib/zig/std/fmt/index.zig:106:23: error: expected type 'TypeId', found 'TypeId'
        builtin.TypeId.Int, builtin.TypeId.Float => {
                      ^
/home/ihier/proj/zig/zig/build/lib/zig/std/fmt/index.zig:52:35: note: called from here
                    try formatType(args[next_arg], fmt[0..0], context, Errors, output);
                                  ^
/home/ihier/proj/zig/zig/build/lib/zig/std/io.zig:214:34: note: called from here
            return std.fmt.format(self, Error, self.writeFn, format, args);
                                 ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:222:37: note: called from here
                try out_stream.print(WHITE ++ "{}:{}:{}" ++ RESET ++ ": " ++ DIM ++ ptr_hex ++ " in ??? ({})" ++ RESET ++ "\n", line_info.file_name, line_info.line, line_info.column, address, compile_unit_name);
                                    ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:159:33: note: called from here
        try printSourceAtAddress(debug_info, out_stream, return_address);
                                ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:67:20: note: called from here
    writeStackTrace(stack_trace, stderr, getDebugInfoAllocator(), debug_info, stderr_file.isTty()) catch |err| {
                   ^
/home/ihier/proj/zig/zig/build/lib/zig/std/fmt/index.zig:106:23: error: expected type 'TypeId', found 'TypeId'
        builtin.TypeId.Int, builtin.TypeId.Float => {
                      ^
/home/ihier/proj/zig/zig/build/lib/zig/std/fmt/index.zig:52:35: note: called from here
                    try formatType(args[next_arg], fmt[0..0], context, Errors, output);
                                  ^
/home/ihier/proj/zig/zig/build/lib/zig/std/io.zig:214:34: note: called from here
            return std.fmt.format(self, Error, self.writeFn, format, args);
                                 ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:222:37: note: called from here
                try out_stream.print(WHITE ++ "{}:{}:{}" ++ RESET ++ ": " ++ DIM ++ ptr_hex ++ " in ??? ({})" ++ RESET ++ "\n", line_info.file_name, line_info.line, line_info.column, address, compile_unit_name);
                                    ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:159:33: note: called from here
        try printSourceAtAddress(debug_info, out_stream, return_address);
                                ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:67:20: note: called from here
    writeStackTrace(stack_trace, stderr, getDebugInfoAllocator(), debug_info, stderr_file.isTty()) catch |err| {
                   ^
/home/ihier/proj/zig/zig/build/lib/zig/std/fmt/index.zig:106:23: error: expected type 'TypeId', found 'TypeId'
        builtin.TypeId.Int, builtin.TypeId.Float => {
                      ^
/home/ihier/proj/zig/zig/build/lib/zig/std/fmt/index.zig:71:35: note: called from here
                    try formatType(args[next_arg], fmt[s..i], context, Errors, output);
                                  ^
/home/ihier/proj/zig/zig/build/lib/zig/std/io.zig:214:34: note: called from here
            return std.fmt.format(self, Error, self.writeFn, format, args);
                                 ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:222:37: note: called from here
                try out_stream.print(WHITE ++ "{}:{}:{}" ++ RESET ++ ": " ++ DIM ++ ptr_hex ++ " in ??? ({})" ++ RESET ++ "\n", line_info.file_name, line_info.line, line_info.column, address, compile_unit_name);
                                    ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:159:33: note: called from here
        try printSourceAtAddress(debug_info, out_stream, return_address);
                                ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:67:20: note: called from here
    writeStackTrace(stack_trace, stderr, getDebugInfoAllocator(), debug_info, stderr_file.isTty()) catch |err| {
                   ^
/home/ihier/proj/zig/zig/build/lib/zig/std/fmt/index.zig:106:23: error: expected type 'TypeId', found 'TypeId'
        builtin.TypeId.Int, builtin.TypeId.Float => {
                      ^
/home/ihier/proj/zig/zig/build/lib/zig/std/fmt/index.zig:71:35: note: called from here
                    try formatType(args[next_arg], fmt[s..i], context, Errors, output);
                                  ^
/home/ihier/proj/zig/zig/build/lib/zig/std/io.zig:214:34: note: called from here
            return std.fmt.format(self, Error, self.writeFn, format, args);
                                 ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:241:41: note: called from here
                    try out_stream.print(ptr_hex ++ " in ??? ({})\n", address, compile_unit_name);
                                        ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:159:33: note: called from here
        try printSourceAtAddress(debug_info, out_stream, return_address);
                                ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:67:20: note: called from here
    writeStackTrace(stack_trace, stderr, getDebugInfoAllocator(), debug_info, stderr_file.isTty()) catch |err| {
                   ^
/home/ihier/proj/zig/zig/build/lib/zig/std/fmt/index.zig:106:23: error: expected type 'TypeId', found 'TypeId'
        builtin.TypeId.Int, builtin.TypeId.Float => {
                      ^
/home/ihier/proj/zig/zig/build/lib/zig/std/fmt/index.zig:52:35: note: called from here
                    try formatType(args[next_arg], fmt[0..0], context, Errors, output);
                                  ^
/home/ihier/proj/zig/zig/build/lib/zig/std/io.zig:214:34: note: called from here
            return std.fmt.format(self, Error, self.writeFn, format, args);
                                 ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:241:41: note: called from here
                    try out_stream.print(ptr_hex ++ " in ??? ({})\n", address, compile_unit_name);
                                        ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:159:33: note: called from here
        try printSourceAtAddress(debug_info, out_stream, return_address);
                                ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:67:20: note: called from here
    writeStackTrace(stack_trace, stderr, getDebugInfoAllocator(), debug_info, stderr_file.isTty()) catch |err| {
                   ^
/home/ihier/proj/zig/zig/build/lib/zig/std/fmt/index.zig:106:23: error: expected type 'TypeId', found 'TypeId'
        builtin.TypeId.Int, builtin.TypeId.Float => {
                      ^
/home/ihier/proj/zig/zig/build/lib/zig/std/fmt/index.zig:52:35: note: called from here
                    try formatType(args[next_arg], fmt[0..0], context, Errors, output);
                                  ^
/home/ihier/proj/zig/zig/build/lib/zig/std/io.zig:214:34: note: called from here
            return std.fmt.format(self, Error, self.writeFn, format, args);
                                 ^
/home/ihier/proj/zig/zig/build/lib/zig/std/debug/index.zig:68:21: note: called from here
        stderr.print("Unable to dump stack trace: {}\n", @errorName(err)) catch return;
                    ^
==28870== ERROR: libFuzzer: fuzz target exited
    #0 0xafed03 in __sanitizer_print_stack_trace (/home/ihier/proj/zig/zig/build/fuzz_test+0xafed03)
    #1 0xad4852 in fuzzer::PrintStackTrace() (/home/ihier/proj/zig/zig/build/fuzz_test+0xad4852)
    #2 0xa7ac26 in fuzzer::Fuzzer::ExitCallback() (.part.250) (/home/ihier/proj/zig/zig/build/fuzz_test+0xa7ac26)
    #3 0xab28fd in fuzzer::Fuzzer::StaticExitCallback() (/home/ihier/proj/zig/zig/build/fuzz_test+0xab28fd)
    #4 0x7f6af6de0830  (/lib/x86_64-linux-gnu/libc.so.6+0x37830)
    #5 0x7f6af6de0929 in exit (/lib/x86_64-linux-gnu/libc.so.6+0x37929)
    #6 0xd6173d in report_errors_and_maybe_exit(CodeGen*) /home/ihier/proj/zig/zig/src/codegen.cpp:5623:9
    #7 0xd2b396 in gen_root_source(CodeGen*) /home/ihier/proj/zig/zig/src/codegen.cpp:7089:5
    #8 0xd295e5 in codegen_build(CodeGen*) /home/ihier/proj/zig/zig/src/codegen.cpp:7600:5
    #9 0xb12be5 in zig_main(int, char**) /home/ihier/proj/zig/zig/src/main.cpp:903:17
    #10 0xb16c61 in LLVMFuzzerTestOneInput /home/ihier/proj/zig/zig/src/fuzz_test.cpp:29:5
    #11 0xab2fca in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/ihier/proj/zig/zig/build/fuzz_test+0xab2fca)
    #12 0xabb6d0 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool*) (/home/ihier/proj/zig/zig/build/fuzz_test+0xabb6d0)
    #13 0xabc9f6 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, fuzzer::fuzzer_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (/home/ihier/proj/zig/zig/build/fuzz_test+0xabc9f6)
    #14 0xabee12 in fuzzer::Fuzzer::Loop(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, fuzzer::fuzzer_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (/home/ihier/proj/zig/zig/build/fuzz_test+0xabee12)
    #15 0xaadd53 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/ihier/proj/zig/zig/build/fuzz_test+0xaadd53)
    #16 0xa92092 in main (/home/ihier/proj/zig/zig/build/fuzz_test+0xa92092)
    #17 0x7f6af6dcaa86 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21a86)
    #18 0xa9fc19 in _start (/home/ihier/proj/zig/zig/build/fuzz_test+0xa9fc19)

SUMMARY: libFuzzer: fuzz target exited
MS: 0 ; base unit: 0000000000000000000000000000000000000000
0x70,0x75,0x62,0x20,0x66,0x6e,0x20,0x66,0x6f,0x6f,0x28,0x29,0x20,0x76,0x6f,0x69,0x64,0x20,0x7b,0x7d,0xa,
pub fn foo() void {}\x0a
artifact_prefix='./'; Test unit written to ./crash-a4c3914ab418cb1696b1ed88ae2b3f89c280d42b
Base64: cHViIGZuIGZvbygpIHZvaWQge30K

andrewrk · 2018-06-26T17:29:04Z

This is really neat. Thanks for showing me how to use LLVM's fuzz testing capabilities.

That said, I don't think it's time to fuzz stage1 (the c++ compiler) yet. The syntax isn't fully locked in, and stage1 is only used to build the self hosted compiler. We have 88 open bugs, all more important than anything the fuzz tester finds in the stage1 tokenizer or parser.

On the other hand, I'm going to start getting a well-defined Zig IR syntax going in the self hosted compiler and start creating IR-to-IR test cases. This would be a high impact area to do fuzz testing on. But we're not there yet.

isaachier · 2018-06-26T17:49:13Z

I agree because the fuzz test doesn't work much at the moment anyway. I'm glad I could demonstrate this, but I understand the issues. Closing until further notice.

Add fuzz test

8d8b291

isaachier commented Jun 24, 2018

View reviewed changes

bnoordhuis reviewed Jun 24, 2018

View reviewed changes

isaachier added 2 commits June 24, 2018 12:40

Include main.cpp in fuzzer to avoid code in header

0f158c1

Use build-obj instead of build-exe to avoid lack of main function

f288f0a

bnoordhuis reviewed Jun 24, 2018

View reviewed changes

Remove main.hpp

b1aae01

Use fwrite to avoid EINTR, check for compiler fuzzer support

20c9f87

bnoordhuis approved these changes Jun 25, 2018

View reviewed changes

Minor fixes to fuzz test, remove address sanitizer

81048fc

isaachier closed this Jun 26, 2018

isaachier deleted the fuzz-test branch June 26, 2018 23:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add fuzz test #1157

Add fuzz test #1157

isaachier commented Jun 24, 2018

isaachier Jun 24, 2018

bnoordhuis Jun 24, 2018

isaachier Jun 24, 2018

bnoordhuis Jun 24, 2018

bnoordhuis left a comment

bnoordhuis Jun 24, 2018

isaachier Jun 24, 2018

bnoordhuis Jun 24, 2018

isaachier Jun 24, 2018

isaachier commented Jun 24, 2018

isaachier commented Jun 24, 2018

tiehuis commented Jun 25, 2018

bnoordhuis left a comment

bnoordhuis Jun 25, 2018

isaachier Jun 25, 2018

bnoordhuis Jun 25, 2018

isaachier Jun 25, 2018

bnoordhuis Jun 25, 2018

isaachier Jun 25, 2018

bnoordhuis Jun 25, 2018

isaachier Jun 25, 2018

isaachier commented Jun 25, 2018

andrewrk commented Jun 26, 2018

isaachier commented Jun 26, 2018

Add fuzz test #1157

Add fuzz test #1157

Conversation

isaachier commented Jun 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bnoordhuis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

isaachier commented Jun 24, 2018

isaachier commented Jun 24, 2018

tiehuis commented Jun 25, 2018

bnoordhuis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

isaachier commented Jun 25, 2018

andrewrk commented Jun 26, 2018

isaachier commented Jun 26, 2018