Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add command line options support for ant in the compile command #471

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

andre15silva
Copy link

This PR introduces a new option in the compile command of defects4j.

This option enables manually setting additional ant options when compiling through defects4j.

@monperrus
Copy link
Contributor

@rjust WDYT?

@rjust
Copy link
Owner

rjust commented Sep 6, 2024

I am little hesitant to allow build system options to be set in the compile command.

The compile command is supposed to abstract away the actual build system being used. While Defects4J is currently using Ant, it is possible that this will change to mvn, gradle, or plain javac invocations.

What is the use case for changing the default behavior of defects4j compile?

@andre15silva
Copy link
Author

Hi @rjust,

We used this to study fault localization tools under different Java versions.

For this, we needed to enforce specific target versions, which can be done by passing that option to the compile command.

See RQ1 https://arxiv.org/pdf/2111.12513 and https://github.com/ASSERT-KTH/flacoco_experiment/blob/9f5a0bac1fe564b2ff622423773dee3b79117e54/flacoco/scripts/runAllBugs.py#L114-L329

@mernst
Copy link
Collaborator

mernst commented Sep 23, 2024

Unfortunately, forcing a specific version of Defects4J to use a different version of Java than it is designed for will lead to spurious failures. We have observed many such failures, so this is not a hypothetical concern.

My recommendation is:
When using Java 1.7, use Defects4J version 1.
When using Java 1.8, use Defects4J version 2.
When using Java 11, use Defects4J version 3.

We are currently preparing Defects4J version 4, which will support (only) Java 17.

@andre15silva
Copy link
Author

Unfortunately, forcing a specific version of Defects4J to use a different version of Java than it is designed for will lead to spurious failures. We have observed many such failures, so this is not a hypothetical concern.

Yes, we observed the same actually.

In case is useful to you, we found 191 bugs from D4J v2.0 to be compilable and correctly executable (i.e. same sets of executed and failing tests as the original version) under Java 7,8, 11, and 17. List is here: https://github.com/ASSERT-KTH/flacoco_experiment/blob/main/flacoco/results_faultlocalization/all_bugs.csv

We are currently preparing Defects4J version 4, which will support (only) Java 17.

That's cool, thanks :)

@mernst
Copy link
Collaborator

mernst commented Sep 23, 2024

Thanks for the list of bugs that work under all 4 Java versions without change. Have you tried Java 21, which is the current version of Java?

Our approach has been to adjust tests in order to adapt to different execution environments. On the plus side, this means far more than 191 usable bugs. However, this means that a given test may be slightly different between versions of Defects4J. For example, handling of locales changed between Java 8 and Java 11, which forced us to revise tests that depended on locales.

@andre15silva
Copy link
Author

Thanks for the list of bugs that work under all 4 Java versions without change. Have you tried Java 21, which is the current version of Java?

I have not. We did these experiments more than 2 years ago, before Java 21 was released. At the time those were the LTS versions available.

Our approach has been to adjust tests in order to adapt to different execution environments. On the plus side, this means far more than 191 usable bugs. However, this means that a given test may be slightly different between versions of Defects4J. For example, handling of locales changed between Java 8 and Java 11, which forced us to revise tests that depended on locales.

Are you also planning to collect new bugs that would natively be Java 17/21 compatible?

@mernst
Copy link
Collaborator

mernst commented Sep 23, 2024

Are you also planning to collect new bugs that would natively be Java 17/21 compatible?

Yes. That is an ongoing process. Help is always appreciated!

@andre15silva
Copy link
Author

Are you also planning to collect new bugs that would natively be Java 17/21 compatible?

Yes. That is an ongoing process. Help is always appreciated!

Cool! On this topic, you might be interested in our recent tool GitBug-Actions (paper, repo). The core principle is to use the existing CI scripts to reproduce bugs in isolated environments inside docker containers, ensuring reproducibility.

We have recently collected a Java benchmark using it, GitBug-Java (paper, repo).

@mernst
Copy link
Collaborator

mernst commented Sep 23, 2024

That's nice (and useful!) work. Thanks for pointing it out to me. I will check it out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants