Build C++ against JavaCPP, runtime API to replace mvn plugin

[cypof]

New version of the old PR about having include files and versioned libs (symlinks) in the jars. I added an example of how a project could build C++ code at runtime against them. This should allow presets to build against each other instead of having to build all dependencies cppbuild folders, and hopefully get rid of the settings in pom files.

As a motivation for this change, I think this project is very powerful but also very time consuming to work with in practice. I hope this can be a small step toward simplifying things. It's still early, and this code uses the old repo code, coordinates etc. If you are interested we can discuss how to port it to your master.

[saudet]

I could take some time today to look at that! Thanks for the ideas! Though, there are some non-portable hacks in these changes that would need to be generalized a bit. Anything that refers to "linux" explicitly, compiler-specific flags for debugging, etc should be moved to properties, etc. Anyway, I am going to consider this PR mostly as a design proposal. The main question I would have at this point is about the Runtime/JavaCPP class. AFAIU, it's basically meant as a replacement for the Maven plugin, to have more flexibility, more simplicity at development time, but what about using Gradle with a Gradle plugin? Have you considered that? IMO, if we could integrate this into an existing build system, it would make for a much more powerful solution...

[saudet]

@Mistobaan What about H2O, which build system are you planning to use mainly?

[cypof]

Integrating with Gradle might be slightly less bad than with Maven, but why do that? Grade like Maven can easily invoke a Java task, so JavaCPP doesn't have any dependency. It seems to me that would make it simpler, and usable in any other environment, build systems, or at runtime.

[saudet]

Because it allows us to create JAR files and install them in the repository. How do you propose to do that at runtime?

[saudet]

Ah I think I understand what you're proposing. We would basically be taking over the whole build process away from Maven or Gradle. Not sure that's a good idea. I get that you want to do things at runtime, that's fine, but the build process is also important. It's not something we can just dismiss. I think we should focus on that first, no?

[cypof]

Not the whole build, just the javacpp step. Instead of having a plugin invoked with some settings, we could have a java exec step that calls some javacpp method with almost no settings as the preset should have enough info. E.g. the include and lib paths can point to the repo etc.

[saudet]

If we want to do everything with no settings, it means that we have to take over the resources plugin, the copy the resources we need for parsing and compilation, the compiler plugin, to compile the Java source generated from the header files (like you are already doing in the code in this PR), and the dependency plugin, to copy whatever we want to put in the JAR files, and JAR files plugin to put everything in JAR files, and I'm sure we can think of many other variants here. If you have a concrete proposal to make this simpler, I would love to hear about it.

[cypof]

OK, thanks for the list. I have to think about each step. It seems some are necessary for libraries, but maybe not for end applications. I'm only thinking about this last case, not changing everything for the presets, where you already did a great job at making everything work.

Just random thoughts for now, but why not. Let's say for a small app, if we make the parse and compile fast enough, it might not be actually necessary to package in advance. As you know too well, doing it for each platform etc is so much complexity and work for the devs. It might be worth it to parse and compile on the first run instead, when the home cache is empty. Afterward it would be the same as today.

We could accelerate the parse and compile steps by parallelizing, and by serializing your compiler tokens in the preset jars. For me, every time I change something in my code, javacpp needs to re-parse opencv and caffe, which takes a long time. Then it reaches my code, and parses it instantly as it is small. If the preset jars contained serialized token lists for dependencies, the whole thing would be instant. Same for the compile step, if we package pre-compiled headers it might be so fast it doesn't matter to do it at runtime.

If not packaging at all is a problem, we could also have the option to support one platform for devs. If they build and deploy only on linux, they could have the option to just generate the java and libs in default maven locations like src and resources, and let maven or gradle put everything in the default jar?

[saudet]

Yes, there are a lot of issues, but let's focus on one thing at a time: Simplifying the build process. It sounds that what you're looking for would be an external script:
https://docs.gradle.org/current/userguide/organizing_build_logic.html#sec:configuring_using_external_script
That way we could reuse the plugins we need from Gradle, provide some default config, and let the users customize as needed. Can you see anything wrong with that approach?

[cypof]

You don't think that switching to a runtime solution is realistic for end user apps? It would remove enough build steps that we might avoid the new dependency.

Otherwise it's fine, but maybe we can split the dependencies in separate projects? We could have a parent javacpp, a javacpp/maven for the presets and people who use maven, and a javacpp/gradle. The external script solution seems good.

[saudet]

I got what you mean, just didn't have time to reply :) So, a lot of Java development is either for Android or on Mac or Windows, where most developers don't have Visual Studio or Xcode installed. So a runtime solution isn't realistic for a lot of cases, no.

Not saying it isn't useful, I'd do it if it were my job, but it doesn't look like this is going to happen...

Anyway, Gradle, sure it would be a different project. There is already something for sbt here, for example:
https://github.com/bytedeco/sbt-javacpp

[cypof]

You have an LLVM preset, maybe we can use that as a runtime compiler :)

[saudet]

Sure, we can do some interesting stuff with LLVM, but it doesn't support Android or Visual Studio very well, so again, sure, but I don't have the time to do everything, and on my own time I'd rather work on things that work on Android and Windows as well as Linux.

[saudet]

I was able to take some time to work on that this weekend: 909e878

Basically, Loader.cacheResource() can now extract a whole directory into the cache, but we can't really use generic names like "include" for the resource as it might not extract it from the right JAR file. That still needs some work... ideas?

Anyway, I did it without using the FileSystem API because although Android has migrated to OpenJDK, it looks like they are removing these parts of the API that they do not like:
https://developer.android.com/reference/java/nio/package-summary.html

Also, instead of trying to hack support for symbolic links into JAR files, I added a Loader.createLibraryLink() method that creates the necessary links for all versioned libraries extracted in the cache. Is that sufficient? (Of course this isn't useful on Android so we can use Java 7 for that.)

[saudet]

Actually, I think I've figured out exactly how we should do this. I added in commit de43b82 a cacheResources() method that basically extracts all resources with the given name. So, if we have an "include" resource directory in multiple JAR files on the class path, they will all get extracted. Which JAR files end up on the class path depends on the build tool (Maven, Gradle, etc). So the idea would be to let that tool put in the class path the required dependencies, taken from the same repository as any other Java library, and then we just need to call Loader.cacheResources() and pass on these paths to the C++ compiler. Of course, we can also call the API of Maven and do that from Java at runtime.

@cypof I think we now have all the missing pieces to refactor your "Runtime API" without modifying Loader or Builder any further. Do you see anything missing?

[cypof]

Nope, I think it makes sense!

[saudet]

I think I've added the last piece we needed with commit ee96dc6, so please feel free to refactor this PR around the new features and let me know if there's anything missing! Thanks

[saudet]

BTW, it looks like Android is finally going to support NIO.2:
https://developer.android.com/reference/java/nio/file/Files.html

[saudet]

BTW, I created a build plugin for Gradle here:
https://github.com/bytedeco/gradle-javacpp
The workflow that way is looking pretty slick to me.
Can you give it a try and give me your impressions?