Processing Frozen (ParsingTask)

[RamiroMarinB]

Dear All-

For a few months I have been working with IPED 4.1.5 in my laboratory, I think it is a fantastic tool!

I'm working on a forensic image (E01), however the IPED indexing process has not finished, it still has a single task in "ParsingTask" mode for a file called "Program.part13.rar" from more than 5 hours ago.

I read discussion #787 and the documentation on "Frozen Processing", created the "script" and followed the instructions, however, no success:

First, I created the "script" and set it to the path: "/scripts/task/IgnoreFilesByPath", then, set the "script" to the file "/conf/TaskInstaller.xml/", and finally, I start IPED again with the command --continue without success.

Is there a way to ignore that file by running IPED again with the --continue command?  or, is there another method that can help me ignore that single task file in "ParsingTask" mode.

Thank you,

Ramiro

[lfcnassif]

Hello,

As per the documentation, could you take a thread dump jstack -l pid (from the biggest java process) after that file is being processed for a while? That would help us diagnose the situation. If you can provide that RAR file, that would help us even more.

About ignoring that file, please share your IgnoreFilesByPath script and also your TaskInstaller.xml file, so we can check if they were configured properly.

[RamiroMarinB]

IgnoreFilesByPathTask.txt
IPED_PID.txt

TaskInstaller.txt

Hi Nassif-

Thanks for your quick response,

I'm attaching the files that you asked me for, about the sample, I could not attach it because the size is too big, perhaps, you can share a link to a drive or similar so I can send it to you. After, reviewing the evidence (E01) with FTK Imager (attached image), It is evident that it is a design program or something similar.

Best,

Ramiro

[lfcnassif]

Hi @RamiroMarinB, thanks for the files.

I took a look at them and parsing is stuck into SevenZipJBinding:

"ParsingThread-2" #177 daemon prio=5 os_prio=0 cpu=537640.62ms elapsed=549.36s tid=0x00000221e8287800 nid=0x438c runnable  [0x00000081c5bfe000]
   java.lang.Thread.State: RUNNABLE
	at net.sf.sevenzipjbinding.impl.InArchiveImpl.nativeExtract(Native Method)
	at net.sf.sevenzipjbinding.impl.InArchiveImpl.extractSlow(InArchiveImpl.java:122)
	at iped.parsers.compress.SevenZipParser.parse(SevenZipParser.java:155)
	at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:298)
	at iped.parsers.standard.StandardParser.parse(StandardParser.java:245)
	at iped.engine.io.ParsingReader$BackgroundParsing.run(ParsingReader.java:247)
	at java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:515)
	at java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:264)
	at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run([email protected]/Thread.java:829)

Not sure why it didn't trigger a timeout... The file would help us, it is not that big (40MB), if you are not able to upload it here, please tell me your email so I can share a drive link with you.

IgnoreFilesByPathTask.txt

This script is not good, it has 4 process(...) functions with the same name, I think the last was used and it is not the right one. Try to keep just the first and delete the others. Of course, you also have to rename above script from txt to js.

Another approach would be to add this to ParsingConfig.xml:

<parser class="iped.parsers.compress.RARParser">
    <mime>application/x-rar-compressed</mime>
</parser>

It will use JUnrar instead of 7ZipJBiding to uncompress RAR files. Unfortunately, JUnrar doesn't support RAR 5 files, that's why we use 7ZipJBiding to uncompress RAR files by default.

[paulobreim]

@RamiroMarinB Do you have googledrive? You can upload there and send the link to @lfcnassif .

[RamiroMarinB]

Hi @lfcnassif

Sorry for the delay, early in the morning I was with a client and couldn't respond.

Thank you very much for taking the time to review my matter, I haven't worked on your suggestions yet, I think I understand the problem. For now I sent you a Pcloud link: https://e.pcloud.link/publink/show?code=kZa1znZYf3bckUmHYmSAyvrOaBSkbOwPYyV where you can download the files related to the "Altium" program, I hope they are useful to you.

Reggards,

Ramiro

[lfcnassif]

Thanks @RamiroMarinB, I just downloaded the files, you can remove them.

I reproduced the situation. Actually processing is not frozen, but it is extracting folders from the *.part13.rar file very very slowly, but it is going on. We currently use the 7ZipJBinding extractSlow() method. I will try to use an alternative method and see if it helps extracting files from this problematic rar file. Anyway, you should be able to ignore that file following the steps I've suggested.

[lfcnassif]

Hi @RamiroMarinB. It took 3 days to finish parsing your RAR. I implemented an optimization and it dropped to a few minutes. Please test this snapshot:
https://github.com/sepinf-inc/IPED/actions/runs/7999662913/artifacts/1265322522

[RamiroMarinB]

Hi @lfcnassif,

I downloaded the IPED version that you shared with me and the test results were perfect, the indexing process was faster. Maybe you can tell me, what changes were made? just to understand better.

Thank you.

Ramiro

[lfcnassif]

Hi @RamiroMarinB, the code changes are here: #2093. Basically a faster (and more complex) method to expand RAR files is being used now. I also fixed an issue with partial and corupted RAR files causing the internal tree to be displayed wrongly.