cygwin support

[SimonMarquis]

Hi Jake,

I'm trying to use this tool with a cygwin instance (on Windows).
I'm able to run the command, but the process fails when it searches for classes on temporary folders.

Error:  class not found: /tmp/tmp7qET7J/my/package/MyClass1.class
Error:  class not found: /tmp/tmp7qET7J/my/package/MyClass2.class
Error:  class not found: /tmp/tmpthFtVb/my/package/MyClass1.class
Error:  class not found: /tmp/tmpthFtVb/my/package/MyClass2.class

When I try it on the regular Windows cmd prompt, it fails in a different way:

Traceback (most recent call last):
  File "jardiff.py", line 102, in <module>
    _main(sys.argv[1], sys.argv[2])
  File "jardiff.py", line 91, in _main
    old_data = process_archive(temp_folder, old_archive)
  File "jardiff.py", line 77, in process_archive
    infos = _split_info_into_infos(info)
  File "jardiff.py", line 34, in _split_info_into_infos
    original = re.findall(r'Compiled from ".*?\.java"\n.*?\n(?:  .*\n)*}\n', info, flags=re.MULTILINE)
  File "C:\Python33\lib\re.py", line 201, in findall
    return _compile(pattern, flags).findall(string)
TypeError: can't use a string pattern on a bytes-like object

Is there a way to run it on a Windows environment?

(diff and javap are available on both terminals)

[JakeWharton]

The regular expression stuff seems like a Python 2 vs Python 3 thing. I'll take a look. Not sure there's much I can do about the cygwin errors. I'm just using normal os.* and os.path.* functions.

[SimonMarquis]

I'm actually using Python 3.2.3.
I'll try on a unix system.

[JakeWharton]

I tried to code Python 3 style but have been running on Python 2 locally. I think the result of the call to regex just needs a str() around it.

[JakeWharton]

f291c08 fixes the bytes problem on Python 3 but the folders are empty so I'll have to dig more.

[SimonMarquis]

I've tried again on a clean Ubuntu VM with python, python2, python3 and they all remains silent... I may do something wrong but I can't figure it out.

[ghost]

@SimonMarquis Yes, it works very well with Cygwin, I've tested 😄, but you must perform some changes. You need download "diffutils" Cygwin's package for enable "diff" command in Windows because "diff" is typical of UNIX systems. Make sure that Cygwin is in your PATH. Now, the next change is in code. Look at https://github.com/JakeWharton/jardiff/blob/master/jardiff.py#L37, it works perfectly in UNIX systems, but in Windows, you must invoke "javap" differently.

[ghost]

@SimonMarquis In Windows, you may use the following command "javap -public -classpath [here, enter the base path of the *.class files contained in the *.jar file] [class without *.class extension]"

[ghost]

@SimonMarquis @JakeWharton Testing the solution above with Square's Retrofit:

After extracting the *.class files contained in the retrofit.jar, it happens before _javap_public(files) method call, you should have this file tree:

"C:\Users\HoracioCavalcanti\AppData\Local\Temp\tmp09qay1" is your classpath.

Inside the folder "retrofit" you will see something like this:

Now, we are able to call correctly "javap" command-line.

Try use this command: "javap -public -classpath C:\Users\HoracioCavalcanti\AppData\Local\Temp\tmp09qay1 retrofit.Callback" for one class and "javap -public -classpath C:\Users\HoracioCavalcanti\A
ppData\Local\Temp\tmp09qay1 retrofit.Callback retrofit.Endpoint retrofit.Platfor
m$Android retrofit.Platform$Android$2" for a chunk of classes.

The problem seemingly is resolved 😄 😸, since "diffutils" Cygwin's package x86 or x64 versions is installed, all the code will work correctly 😄

[SimonMarquis]

@HoracioFilho It's working manually, but I don't figure out how to process it automatically with the script. Here with Windows cmd it finishes silently without doing nothing. And with cygwin cmd, it doesn't find the .class files.
It seems that the script is looking at the wrong folder name. It indeed creates temp folder with .class inside /tmp/tmpBXNjQ8 for instance, but in the script, it says it is searching somewhere else /tmp/tmpOd96k. Any ideas?

[ghost]

@SimonMarquis After solved the problem with the invocation of "javap". I found the last problem that is with regex expression.

Look at:
In Linux, the result of javap is something like this:
b'Compiled from "RequestInterceptorTape.java"\nabstract class retrofit.RequestInterceptorTape$Command extends java.lang.Enum<retrofit.RequestInterceptorTape$Command> {\n public static final retrofit.RequestInterceptorTape$Command ADD_HEADER;\n public static final retrofit.RequestInterceptorTape$Command ADD_PATH_PARAM;\n public static final retrofit.RequestInterceptorTape$Command ADD_ENCODED_PATH_PARAM;\n public static final retrofit.RequestInterceptorTape$Command ADD_QUERY_PARAM;\n public static final retrofit.RequestInterceptorTape$Command ADD_ENCODED_QUERY_PARAM;\n public static retrofit.RequestInterceptorTape$Command[] values();\n public static retrofit.RequestInterceptorTape$Command valueOf(java.lang.String);\n}\n

The problem is here https://github.com/JakeWharton/jardiff/blob/master/jardiff.py#L42, Windows uses different line separator.

Unix's line separator is "\n"
Windows's line separator is "\r\n"

The regex expression always fails, because of that you find the folder empty.

[ghost]

@SimonMarquis Fixed in pull-request 😄

[SimonMarquis]

It's awsome ! Really, you did a great job. 👍
Things are almost perfect.
Unfortunately, I'm still unable to run this script

It seems that take the relative file path of the wrong argument...

On the _javap_public method, I think it should be:

results.append(str(subprocess.check_output(['javap', '-classpath', os.path.relpath(unjar), '-public'] + chunk)))
# instead of
# results.append(str(subprocess.check_output(['javap', '-classpath', unjar, '-public'] + chunk)))

By doing this, I was able to not get the Error: class not found:
I've also moved the script to /tmp/jardiff to avoid potential conflicts.

Now, the script run smoothly, but it won't spit anything out :) It remains silent. I will try to investigate why.

Update: It seems that the generated folders are simply empty... so diff is correct.

Update 2: The _split_info_into_infos returns an empty array... (Regex result is empty)

[ghost]

@SimonMarquis Thanks a lot, God is wonderful and helps us a lot, I am very grateful to you for the opportunity to help you. I found one mistake with the code, I'll fix it when I come back to home 😄

[ghost]

@SimonMarquis What is your JDK version? Running on what operation system? 😄

[ghost]

@SimonMarquis Please, try the last version 😃

[SimonMarquis]

My JDK is 1.7.0_45.
Your latest version gave me the same result:

original = re.findall(r'Compiled from ".*?\.java"' + os.linesep + '.*?' + os.linesep + '(?:  .*' + os.linesep + ')*}' + os.linesep, info, flags=re.MULTILINE)
# original is an empty array

Despite the info being correct

[ghost]

Hi @SimonMarquis The result is a little weird, here the script is running well on both Windows 8.1 and Ubuntu 14.04. Could you send me the files that have you used for testing the script? I would want to test they 😄 My email is [email protected] 😄

[ghost]

What is your Python version? 2.7.8 and 3.4.3 versions are more stable 😃

[SimonMarquis]

For both tests I'm using tape
python jardiff2.py tape-1.0.0.jar tape-1.1.0.jar

When using cygwin;
python 2.7.8
javap 1.7.0_45

The app finishes silently...
The info variable is:

Compiled from "FileException.java"
public class com.squareup.tape.FileException extends java.lang.RuntimeException {
  public com.squareup.tape.FileException(java.lang.String, java.io.IOException, java.io.File);
  public java.io.File getFile();
}
Compiled from "FileObjectQueue.java"
class com.squareup.tape.FileObjectQueue$1 implements com.squareup.tape.QueueFile$ElementReader {
  public void read(java.io.InputStream, int) throws java.io.IOException;
}
Compiled from "FileObjectQueue.java"
public interface com.squareup.tape.FileObjectQueue$Converter<T> {
  public abstract T from(byte[]) throws java.io.IOException;
  public abstract void toStream(T, java.io.OutputStream) throws java.io.IOException;
}
Compiled from "FileObjectQueue.java"
class com.squareup.tape.FileObjectQueue$DirectByteArrayOutputStream extends java.io.ByteArrayOutputStream {
  public com.squareup.tape.FileObjectQueue$DirectByteArrayOutputStream();
  public byte[] getArray();
}
Compiled from "FileObjectQueue.java"
public class com.squareup.tape.FileObjectQueue<T> implements com.squareup.tape.ObjectQueue<T> {
  public com.squareup.tape.FileObjectQueue(java.io.File, com.squareup.tape.FileObjectQueue$Converter<T>) throws java.io.IOException;
  public int size();
  public final void add(T);
  public T peek();
  public final void remove();
  public void setListener(com.squareup.tape.ObjectQueue$Listener<T>);
}
Compiled from "InMemoryObjectQueue.java"
public class com.squareup.tape.InMemoryObjectQueue<T> implements com.squareup.tape.ObjectQueue<T> {
  public com.squareup.tape.InMemoryObjectQueue();
  public void add(T);
  public T peek();
  public int size();
  public void remove();
  public void setListener(com.squareup.tape.ObjectQueue$Listener<T>);
}
Compiled from "ObjectQueue.java"
public interface com.squareup.tape.ObjectQueue$Listener<T> {
  public abstract void onAdd(com.squareup.tape.ObjectQueue<T>, T);
  public abstract void onRemove(com.squareup.tape.ObjectQueue<T>);
}
Compiled from "ObjectQueue.java"
public interface com.squareup.tape.ObjectQueue<T> {
  public abstract int size();
  public abstract void add(T);
  public abstract T peek();
  public abstract void remove();
  public abstract void setListener(com.squareup.tape.ObjectQueue$Listener<T>);
}
Compiled from "QueueFile.java"
class com.squareup.tape.QueueFile$1 implements com.squareup.tape.QueueFile$ElementReader {
  public void read(java.io.InputStream, int) throws java.io.IOException;
}
Compiled from "QueueFile.java"
class com.squareup.tape.QueueFile$Element {
  public java.lang.String toString();
}
Compiled from "QueueFile.java"
final class com.squareup.tape.QueueFile$ElementInputStream extends java.io.InputStream {
  public int read(byte[], int, int) throws java.io.IOException;
  public int read() throws java.io.IOException;
}
Compiled from "QueueFile.java"
public interface com.squareup.tape.QueueFile$ElementReader {
  public abstract void read(java.io.InputStream, int) throws java.io.IOException;
}
Compiled from "QueueFile.java"
public class com.squareup.tape.QueueFile {
  public com.squareup.tape.QueueFile(java.io.File) throws java.io.IOException;
  public void add(byte[]) throws java.io.IOException;
  public synchronized void add(byte[], int, int) throws java.io.IOException;
  public synchronized boolean isEmpty();
  public synchronized byte[] peek() throws java.io.IOException;
  public synchronized void peek(com.squareup.tape.QueueFile$ElementReader) throws java.io.IOException;
  public synchronized void forEach(com.squareup.tape.QueueFile$ElementReader) throws java.io.IOException;
  public synchronized int size();
  public synchronized void remove() throws java.io.IOException;
  public synchronized void clear() throws java.io.IOException;
  public synchronized void close() throws java.io.IOException;
  public java.lang.String toString();
}
Compiled from "SerializedConverter.java"
public class com.squareup.tape.SerializedConverter<T extends java.io.Serializable> implements com.squareup.tape.FileObjectQueue$Converter<T> {
  public com.squareup.tape.SerializedConverter();
  public T from(byte[]) throws java.io.IOException;
  public void toStream(T, java.io.OutputStream) throws java.io.IOException;
  public void toStream(java.lang.Object, java.io.OutputStream) throws java.io.IOException;
  public java.lang.Object from(byte[]) throws java.io.IOException;
}
Compiled from "Task.java"
public interface com.squareup.tape.Task<T> extends java.io.Serializable {
  public abstract void execute(T);
}
Compiled from "TaskInjector.java"
public interface com.squareup.tape.TaskInjector<T extends com.squareup.tape.Task> {
  public abstract void injectMembers(T);
}
Compiled from "TaskQueue.java"
public class com.squareup.tape.TaskQueue<T extends com.squareup.tape.Task> implements com.squareup.tape.ObjectQueue<T> {
  public com.squareup.tape.TaskQueue(com.squareup.tape.ObjectQueue<T>, com.squareup.tape.TaskInjector<T>);
  public T peek();
  public int size();
  public void add(T);
  public void remove();
  public void setListener(com.squareup.tape.ObjectQueue$Listener<T>);
  public java.lang.Object peek();
  public void add(java.lang.Object);
}

And the original variable is: []

When using Windows cmd line:
python 3.3.3
javap 1.7.0_45

It only warns me with lot of:

Warning: Binary file com\squareup\tape\FileException contains com.squareup.tape.FileException
Warning: Binary file com\squareup\tape\FileObjectQueue$1 contains com.squareup.tape.FileObjectQueue$1
Warning: Binary file com\squareup\tape\FileObjectQueue$Converter contains com.squareup.tape.FileObjectQueue$Converter

The info variable is:

b'Compiled from "FileException.java"\r\npublic class com.squareup.tape.FileException extends java.lang.RuntimeException {\r\n  public com.squareup.tape.FileException(java.lang.String, java.io.IOException, java.io.File);\r\n  public java.io.File getFile();\r\n}\r\nCompiled from "FileObjectQueue.java"\r\nclass com.squareup.tape.FileObjectQueue$1 implements com.squareup.tape.QueueFile$ElementReader {\r\n  public void read(java.io.InputStream, int) throws java.io.IOException;\r\n}\r\nCompiled from "FileObjectQueue.java"\r\npublic interface com.squareup.tape.FileObjectQueue$Converter<T> {\r\n  public abstract T from(byte[]) throws java.io.IOException;\r\n  public abstract void toStream(T, java.io.OutputStream) throws java.io.IOException;\r\n}\r\nCompiled from "FileObjectQueue.java"\r\nclass com.squareup.tape.FileObjectQueue$DirectByteArrayOutputStream extends java.io.ByteArrayOutputStream {\r\n  public com.squareup.tape.FileObjectQueue$DirectByteArrayOutputStream();\r\n  public byte[] getArray();\r\n}\r\nCompiled from "FileObjectQueue.java"\r\npublic class com.squareup.tape.FileObjectQueue<T> implements com.squareup.tape.ObjectQueue<T> {\r\n  public com.squareup.tape.FileObjectQueue(java.io.File, com.squareup.tape.FileObjectQueue$Converter<T>) throwsjava.io.IOException;\r\n  public int size();\r\n  public final void add(T);\r\n  public T peek();\r\n  public final void remove();\r\n  public void setListener(com.squareup.tape.ObjectQueue$Listener<T>);\r\n}\r\nCompiled from "InMemoryObjectQueue.java"\r\npublic class com.squareup.tape.InMemoryObjectQueue<T> implements com.squareup.tape.ObjectQueue<T> {\r\n  public com.squareup.tape.InMemoryObjectQueue();\r\n  public voidadd(T);\r\n  public T peek();\r\n  public int size();\r\n  public void remove();\r\n  public void setListener(com.squareup.tape.ObjectQueue$Listener<T>);\r\n}\r\nCompiled from "ObjectQueue.java"\r\npublic interface com.squareup.tape.ObjectQueue$Listener<T> {\r\n  public abstract void onAdd(com.squareup.tape.ObjectQueue<T>, T);\r\n  public abstract void onRemove(com.squareup.tape.ObjectQueue<T>);\r\n}\r\nCompiled from "ObjectQueue.java"\r\npublic interface com.squareup.tape.ObjectQueue<T> {\r\n  public abstract int size();\r\n  public abstract void add(T);\r\n  public abstract T peek();\r\n  public abstract void remove();\r\n  public abstract void setListener(com.squareup.tape.ObjectQueue$Listener<T>);\r\n}\r\nCompiled from "QueueFile.java"\r\nclass com.squareup.tape.QueueFile$1 implements com.squareup.tape.QueueFile$ElementReader {\r\n  public void read(java.io.InputStream, int) throws java.io.IOException;\r\n}\r\nCompiled from "QueueFile.java"\r\nclass com.squareup.tape.QueueFile$Element {\r\n  public java.lang.String toString();\r\n}\r\nCompiled from "QueueFile.java"\r\nfinal class com.squareup.tape.QueueFile$ElementInputStream extends java.io.InputStream {\r\n  public int read(byte[], int, int) throws java.io.IOException;\r\n  public int read() throws java.io.IOException;\r\n}\r\nCompiled from "QueueFile.java"\r\npublic interface com.squareup.tape.QueueFile$ElementReader {\r\n  public abstractvoid read(java.io.InputStream, int) throws java.io.IOException;\r\n}\r\nCompiled from "QueueFile.java"\r\npublic class com.squareup.tape.QueueFile {\r\n  public com.squareup.tape.QueueFile(java.io.File) throws java.io.IOException;\r\n  public void add(byte[]) throws java.io.IOException;\r\n  public synchronized void add(byte[], int, int) throws java.io.IOException;\r\n  public synchronized boolean isEmpty();\r\n  public synchronized byte[] peek() throws java.io.IOException;\r\n  public synchronized void peek(com.squareup.tape.QueueFile$ElementReader) throws java.io.IOException;\r\n  public synchronized void forEach(com.squareup.tape.QueueFile$ElementReader) throws java.io.IOException;\r\n  public synchronized int size();\r\n  public synchronized void remove() throws java.io.IOException;\r\n  public synchronized void clear() throws java.io.IOException;\r\n  public synchronized void close() throws java.io.IOException;\r\n  public java.lang.String toString();\r\n}\r\nCompiled from "SerializedConverter.java"\r\npublic class com.squareup.tape.SerializedConverter<T extends java.io.Serializable> implements com.squareup.tape.FileObjectQueue$Converter<T> {\r\n  public com.squareup.tape.SerializedConverter();\r\n  public T from(byte[]) throws java.io.IOException;\r\n public void toStream(T, java.io.OutputStream) throws java.io.IOException;\r\n  public void toStream(java.lang.Object, java.io.OutputStream) throws java.io.IOException;\r\n  public java.lang.Object from(byte[]) throws java.io.IOException;\r\n}\r\nCompiled from "Task.java"\r\npublic interface com.squareup.tape.Task<T> extends java.io.Serializable {\r\n  public abstract void execute(T);\r\n}\r\nCompiled from "TaskInjector.java"\r\npublic interface com.squareup.tape.TaskInjector<T extends com.squareup.tape.Task> {\r\n  public abstract void injectMembers(T);\r\n}\r\nCompiled from "TaskQueue.java"\r\npublic class com.squareup.tape.TaskQueue<T extends com.squareup.tape.Task> implements com.squareup.tape.ObjectQueue<T> {\r\n  public com.squareup.tape.TaskQueue(com.squareup.tape.ObjectQueue<T>, com.squareup.tape.TaskInjector<T>);\r\n  public T peek();\r\n  public int size();\r\n  public void add(T);\r\n  public void remove();\r\n  public void setListener(com.squareup.tape.ObjectQueue$Listener<T>);\r\n  public java.lang.Object peek();\r\n  public void add(java.lang.Object);\r\n}\r\n'

And the original variable is: []

[ghost]

@SimonMarquis I'll investigate it 😄 Sorry for the delay

[SimonMarquis]

I think I finally found the issue with cygwin:

Temporary folders created by tempfile.mkdtemp() are under the /tmp of cygwin
And therefore, javap with relative path will simply fail.

The simplest workaround I found is to unzip under the same directory of the script:

def _unzip(name):
  destination = tempfile.mkdtemp(dir=os.path.dirname(os.path.realpath(__file__)))
  with zipfile.ZipFile(name) as f:
    f.extractall(destination)
  return destination

I've also discovered (runing the script on my own library) that jar must be construct with file compiled with the debug mode on. Otherwise, none of the files (excluding package-info.java) will have the Compiled from "MyClass.java" when runing the javap command.

Here is the result: https://github.com/SimonMarquis/jardiff/compare/JakeWharton:master...master