This is an archive of the discontinued LLVM Phabricator instance.

[libFuzzer] Diff 4 - Split FuzzerUtil implementation for Posix and Windows.
ClosedPublic

Authored by mpividori on Nov 29 2016, 3:27 PM.

Details

Summary

Implement utils functions for Windows, using vectored exception handlers as equivalent of POSIX signals. Implementation of timers is incomplete.

Diff Detail

Repository
rL LLVM

Event Timeline

mpividori updated this revision to Diff 79657.Nov 29 2016, 3:27 PM
mpividori retitled this revision from to [libFuzzer] Diff 4 - Split FuzzerUtil implementation for Posix and Windows..
mpividori updated this object.
mpividori added reviewers: kcc, zturner.
mpividori set the repository for this revision to rL LLVM.
mpividori added a subscriber: llvm-commits.

My comments are mostly just things to be aware of. No changes required.

lib/Fuzzer/FuzzerUtil.cpp
205 ↗(On Diff #79657)

Implicit conversion from unsigned to signed. Probably not an issue until processors get a lot bigger.

Relying on implicit conversions makes it hard to turn on conversion warnings, which might help detect real bugs in other parts of the code.

lib/Fuzzer/FuzzerUtilPosix.cpp
56 ↗(On Diff #79657)

Be aware that this name clashes with the Windows function SetTimer that's in scope here. You're OK, since this is a C++ file and so it'll be name-mangled and thus distinct. If this had been a C source file instead, though, this would be a problem.

lib/Fuzzer/FuzzerUtilWindows.cpp
159 ↗(On Diff #79657)

Be aware that this converts from an unsigned to a signed type. This is not a big deal with modern Windows, since it's extremely unlikely you'll ever see a PID approaching 2^31, but it's at least theoretically possible. If some logging somewhere prints this as a signed int, it may not look like the value you'd see in other Window stools.

Raymond Chen mentions seeing PIDs approaching 4 billion, but I suspect that's when they were actually pointers in disguise. http://stackoverflow.com/a/17868515/1386054

186 ↗(On Diff #79657)

I'm not sure if libfuzzer has its own coding style guidelines, but I'd probably use nullptr rather than NULL.

mpividori updated this revision to Diff 79692.Nov 29 2016, 6:54 PM

Update Diff.

zturner added inline comments.Nov 30 2016, 9:09 AM
lib/Fuzzer/FuzzerUtil.cpp
199 ↗(On Diff #79692)

I noticed that in the SleepSeconds function, there is a comment that they couldn't use stl because they wanted to avoid coverage of instrumented libc++. You might wait for comments from kcc@ about whether this is relevant here as well. If so, we might need to go back to pure C implementations for each platform.

lib/Fuzzer/FuzzerUtilWindows.cpp
169 ↗(On Diff #79692)

This function appears almost entirely the same except _popen vs popen. Perhaps this could be hidden behind a function called OpenProcessPipe and then ExecuteCommandAndReadOutput could be shared.

173–174 ↗(On Diff #79692)

This looks wrong to me. fread does not indicate an error by returning a value less than 0, it indicates an error by returning a value different from the requested number of bytes. Then, you check feof and ferr to determine whether it was an error or an end of file. This code doesn't seem to handle the case where there was an error reading the process output.

mpividori added inline comments.Nov 30 2016, 9:39 AM
lib/Fuzzer/FuzzerUtilWindows.cpp
169 ↗(On Diff #79692)

@zturner thanks. Yes, I agree. I will add that modification.

173–174 ↗(On Diff #79692)

@zturner, Yes you are right. This code was moved from the previous implementation for Posix. Anyway, I will fix this.

mpividori added inline comments.Nov 30 2016, 1:24 PM
lib/Fuzzer/FuzzerUtil.cpp
205 ↗(On Diff #79657)

@amccarth Ok, I will make an explicit cast.

lib/Fuzzer/FuzzerUtilPosix.cpp
56 ↗(On Diff #79657)

@amccarth thanks. Anyway, I remove that function in the diff 7.

lib/Fuzzer/FuzzerUtilWindows.cpp
159 ↗(On Diff #79657)

@amccarth Ok. I will add that changes in a new diff.

186 ↗(On Diff #79657)

@amccarth it doesn't have own style guidelines... I think we should follow LLVM guidelines. But this requires many changes... If @kcc agrees I can update the code to follow LLVM code guidelines in a new diff.

@amccarth suggestions were included in the diff: https://reviews.llvm.org/D27281

kcc added inline comments.Nov 30 2016, 2:08 PM
lib/Fuzzer/FuzzerUtil.h
64 ↗(On Diff #79692)

No such ifdefs, please.
Can this be moved to the win-specific file?

zturner added inline comments.Nov 30 2016, 2:10 PM
lib/Fuzzer/FuzzerUtil.h
64 ↗(On Diff #79692)

It is called directly from non platform specific code. The alternative would be to add a function like SearchMemory(), and in Posix file it just calls memmem, and in Win32 it does its own algorithm. Then update callsites to use SearchMemory() instead.

kcc added inline comments.Nov 30 2016, 2:18 PM
lib/Fuzzer/FuzzerUtil.h
64 ↗(On Diff #79692)

Yes, that would be fine.

mpividori added inline comments.Nov 30 2016, 2:43 PM
lib/Fuzzer/FuzzerUtil.h
64 ↗(On Diff #79692)

@kcc @zturner Actually I can remove that #ifdef since it is only a declaration and the type matches the type of the memmem() function in Posix.

mpividori updated this revision to Diff 79822.Nov 30 2016, 2:48 PM

Added changes to include @zturner's suggestion.

zturner accepted this revision.Nov 30 2016, 2:52 PM
zturner edited edge metadata.

lgtm, will wait for kcc to also lgtm before committing.

This revision is now accepted and ready to land.Nov 30 2016, 2:52 PM
mpividori added inline comments.Nov 30 2016, 2:53 PM
lib/Fuzzer/FuzzerUtil.cpp
199 ↗(On Diff #79692)

@kcc what do you think about this?

kcc added inline comments.Nov 30 2016, 2:56 PM
lib/Fuzzer/FuzzerUtil.cpp
217 ↗(On Diff #79822)

why this change from >0 to == sizeof(Buff)

In general, please avoid mixing refactoring and functionality changes.

lib/Fuzzer/FuzzerUtil.h
66 ↗(On Diff #79822)

This sounds wrong. This declares "fuzzer::memmem", while on Linux we actually use "memmem"

zturner added inline comments.Nov 30 2016, 2:58 PM
lib/Fuzzer/FuzzerUtil.cpp
217 ↗(On Diff #79822)

That change was my suggestion. The code is incorrect as written because fread doesn't use <= 0 to indicate an error, it uses != request_size, then you check ferror. In any case, your point about mixing functionality and refactoring changes is still valid, so it might be worth doing this in a followup as you suggest.

kcc added inline comments.Nov 30 2016, 3:02 PM
lib/Fuzzer/FuzzerUtil.cpp
217 ↗(On Diff #79822)

hmmm. am I reading "man fread" wrong?
On success, fread() and fwrite() return the number of items read or written. This number equals the number of bytes transferred only when size is 1. If an error occurs, or the end of

the file is reached, the return value is a short item count (or zero).
zturner added inline comments.Nov 30 2016, 3:08 PM
lib/Fuzzer/FuzzerUtil.cpp
217 ↗(On Diff #79822)

It's a little bit ambiguous, but it doesn't explicitly state that 0 is for errors and the less specific condition of < request_size is eof. Indeed, the very next line in the man page is "fread() does not distinguish between end-of-file and error, and callers must use feof(3) and ferror(3) to determine which occurred."

So it seems like the modified version is slightly more correct.

zturner added inline comments.Nov 30 2016, 3:09 PM
lib/Fuzzer/FuzzerUtil.cpp
217 ↗(On Diff #79822)

Also, the Microsoft documentation says similarly "fread returns the number of full items actually read, which may be less than count if an error occurs or if the end of the file is encountered before reaching count. Use the feof or ferror function to distinguish a read error from an end-of-file condition." So again, it seems you can't use the return value to distinguish feof and ferr, and it seems theoretically possible that a value greater than 0 but less than the number of bytes requested could still be an error, which would only be determinable through ferror.

kcc added inline comments.Nov 30 2016, 3:11 PM
lib/Fuzzer/FuzzerUtil.cpp
217 ↗(On Diff #79822)

Ok.

mpividori updated this revision to Diff 79826.Nov 30 2016, 3:22 PM
mpividori edited edge metadata.

+ Remove memmem() and add SearchMemory() as suggested.
+ Remove changes to fread. I will add that changes in a new diff as suggested.

zturner added inline comments.Nov 30 2016, 4:03 PM
lib/Fuzzer/FuzzerUtil.cpp
199 ↗(On Diff #79826)

Kostya, can you confirm that the call to std::thread::hardware_concurrency() here is ok? The call to sleep earlier has a comment explicitly stating that stl can't be used. Is the same thing going to be true here?

kcc accepted this revision.Nov 30 2016, 4:25 PM
kcc edited edge metadata.

LGTM (I did not read the widows code)
As for every other patch, please ensure check-fuzzer still work on Linux.

lib/Fuzzer/FuzzerUtil.cpp
199 ↗(On Diff #79826)

Yep, should be fine.
We should only avoid heavy STL around the calls to the user callback.
This one is happening at init time.

This revision was automatically updated to reflect the committed changes.