Add eye-fuzz visualization tool #123
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for the contribution! Sounds interesting. Will take a look. |
|
I've been trying to build locally using the instructions on your README, but I get the following test errors when running
And many more similar errors. Do you have an idea of what's going on? |
|
Those are expected - an explanation of the cause is under the first error (although it might be kind of hard to see) The heart of the problem is basically that I'm tracking program locations as essentially a pair of That said, in most cases I don't think this should be an issue - should I just suppress the warning/have it only appear once? |
|
Ohh, I didn't realize it was just a silent warning. I saw the red text and mistakenly assumed it was the cause of the build failure. I can ignore these warnings. Scrolling down, I see that my build failed due to a different test failure: |
|
Huh, I've seen that before but I thought I fixed it - I'll see if I can
reproduce it.
…On Mon, Dec 28, 2020 at 2:55 PM Rohan Padhye ***@***.***> wrote:
Ohh, I didn't realize it was just a silent warning. I saw the red text and
mistakenly assumed it was the cause of the build failure. I can ignore
these warnings. Scrolling down, I see that my build failed due to a
different test failure:
...
Acknowledged completion of work request for RERUN_GENERATOR on thread Thread[main,5,main]
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.033 sec <<< FAILURE!
testLittleEndianLoad(EdgeCaseTest) Time elapsed: 0.024 sec <<< FAILURE!
org.junit.ComparisonFailure: expected:<-5590[3873]7> but was:<-5590[7457]7>
at org.junit.Assert.assertEquals(Assert.java:115)
at kotlin.test.junit.JUnitAsserter.assertEquals(JUnitSupport.kt:32)
at kotlin.test.AssertionsKt__AssertionsKt.assertEquals(Assertions.kt:57)
at kotlin.test.AssertionsKt.assertEquals(Unknown Source)
at kotlin.test.AssertionsKt__AssertionsKt.assertEquals$default(Assertions.kt:56)
at kotlin.test.AssertionsKt.assertEquals$default(Unknown Source)
at EdgeCaseTest$testLittleEndianLoad$1.invoke(EdgeCaseTest.kt:54)
at EdgeCaseTest$testLittleEndianLoad$1.invoke(EdgeCaseTest.kt:11)
at TestServerKt.testServer(TestServer.kt:23)
at EdgeCaseTest.testLittleEndianLoad(EdgeCaseTest.kt:41)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:172)
at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:104)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:70)
Results :
Failed tests: testLittleEndianLoad(EdgeCaseTest): expected:<-5590[3873]7> but was:<-5590[7457]7>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#123 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFFY4GTYFK6JT7R55KVCQATSXEEEJANCNFSM4U63IYKA>
.
|
|
It looks like there were some parallelism bugs that came up about half the time via |
|
Hey I tried this again but there doesn't seem to be any new commit since December 16. The last commit I see is noloerino/eye-fuzz@54b8cc04f, which still gives me the same errors. |
|
Looks like I forgot to push, can you check again? |
|
@noloerino I just tried cloning a fresh version of your repo, and ran What I see when |
|
Sorry for the long turnaround, this got lost in my inbox. I just rebased to latest JQF master and the failure seems to disappear, can you check if that's the case for you? If it still fails, can you also try cloning the eye-fuzz repo independently and running |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds
eye-fuzz, a tool for visualizing the relationship between a source of randomness and generators that I've been working on under the supervision of @carolemieux. See the project repository for how to build and run the tool - the tl;dr is that it provides a javascript interface that lets a user modify the input bytes being fed to a generator class, and to observe its effects on the generator output and produced code coverage. The hope is that this ability to keep a human in the fuzzing loop will make it easier to write better, more robust generators.I've for now put it as a submodule under
examples(the build process is somewhat different from JQF since a significant portion of the code is in Kotlin, and I'm not quite sure how to add another nested maven project within theexamplesdirectory), but I'd be happy to copy my source files into whatever directory structure works best.To test things out: run
git submodule update --init --recursive, thencd eye-fuzzand follow the instructions listed in the README. Comments and suggestions are welcome (though I'm not sure how Github comments work on pull requests to add a submodule - again, if it's easier I can copy in the source files instead of making a submodule if that's easier to review).