How to write a Loader that has to decode, decrypt, or decompress the data?

[hippietrail]

I'm sure this is a very basic question but it's hard to grep the repo to find an example and Googling for Loader dev docs finds a bunch of blog posts from 2020 of varying quality. Is there an official intro?

I've written a bunch of Loaders, all pretty simple. Most are like this:

• findSupportedLoadSpecs() - check for magic word if there is one, do sanity check if not, maybe store some offsets for load()
• load()
  • create a BinaryReader from the ByteProvider
  • startAddress = program.getAddressFactory().getDefaultAddressSpace().getAddress(start)
  • fileBytes = MemoryBlockUtils.createFileBytes(program, provider, monitor)
  • MemoryBlock block = program.getMemory().createInitializedBlock("memory", startAddress, filebytes, offset, length, false) (false means not an overlay)
  • entryAddress = program.getAddressFactory().getDefaultAddressSpace().getAddress(entry)
  • SymbolTable st = program.getSymbolTable()
  • st.createLabel(entryAddress, "entry", SourceType.IMPORTED)
  • st.addExternalEntryPoint(entryAddress)

The createInitializedBlock block part directly takes a flat part of the file.

I've got more complex loaders but none that don't take flat parts the file. Which ways do Loader writers use when they have to transform the bytes from the file to get them into a MemoryBlock? Which built-in ones to look at?

[ryanmkurtz]

You just use the version of MemoryBlockUtils.createInitializedBlock() that takes an InputStream rather than a FileBytes. The OmfLoader is a built-in example:

ghidra/Ghidra/Features/Base/src/main/java/ghidra/app/util/opinion/OmfLoader.java

Lines 131 to 146 in 39f388f

	// We don't use the file bytes to create block because the bytes are manipulated before
	// forming the block. Creating the FileBytes anyway in case later we want access to all
	// the original bytes.
	FileBytes fileBytes = MemoryBlockUtils.createFileBytes(program, provider, monitor);
	try {
	processSegmentHeaders(factory.getReader(), header, program, monitor, log);
	processPublicSymbols(header, program, monitor, log);
	processExternalSymbols(header, program, monitor, log);
	processRelocations(header, program, monitor, log);
	markupRecords(program, fileBytes, header, log, monitor);
	}
	catch (AddressOverflowException e) {
	throw new IOException(e);
	}
	}

ghidra/Ghidra/Features/Base/src/main/java/ghidra/app/util/opinion/OmfLoader.java

Lines 378 to 410 in 39f388f

	private void processSegmentHeaders(BinaryReader reader, OmfFileHeader header, Program program,
	TaskMonitor monitor, MessageLog log) throws AddressOverflowException, IOException {
	monitor.setMessage("Process segments...");
	final Language language = program.getLanguage();
	List<OmfSegmentHeader> segments = header.getSegments();
	for (OmfSegmentHeader segment : segments) {
	if (monitor.isCancelled()) {
	break;
	}
	Address segmentAddr = segment.getAddress(language);
	final long segmentSize = segment.getSegmentLength();
	if (segmentSize == 0) {
	continue;
	}
	if (segment.hasNonZeroData()) {
	MemoryBlockUtils.createInitializedBlock(program, false, segment.getName(),
	segmentAddr, segment.getRawDataStream(reader, log), segmentSize, "", "",
	segment.isReadable(), segment.isWritable(), segment.isExecutable(), log,
	monitor);
	}
	else {
	MemoryBlockUtils.createUninitializedBlock(program, false, segment.getName(),
	segmentAddr, segmentSize, "", "", segment.isReadable(), segment.isWritable(),
	segment.isExecutable(), log);
	}
	}
	}

[ryanmkurtz]

So if you just have a byte[], you can make a new ByteArrayInputStream from it and pass that in.

[ryanmkurtz]

FileBytes is a relatively new concept. All loaders used to use the InputStream variant.

[hippietrail]

Thanks. I can't seem to spot anywhere in OmfLoader where the uninitialized block is getting populated. My lack of familiarity with Java surely prevents me from seeing the difference between Ghidra-specific types and functions and ones part of Java. I don't see ByteArrayInputStream in OmfLoader but that helped me find this code in MotorolaHexLoader

ghidra/Ghidra/Features/Base/src/main/java/ghidra/app/util/opinion/MotorolaHexLoader.java

Lines 326 to 335 in 39f388f

	byte[] data = new byte[offset];
	System.arraycopy(dataBuffer, 0, data, 0, offset);
	Address start = baseAddr.add(startAddress);
	String name =
	blockName == null ? baseAddr.getAddressSpace().getName() : blockName;
	MemoryBlockUtils.createInitializedBlock(program, isOverlay, name, start,
	new ByteArrayInputStream(data), data.length, "", provider.getName(),
	true, isOverlay, isOverlay, log, monitor);

Which is pretty similar to what I'm working on. By the way, how are you quoting ranges of lines of code from the repo like that?

[ryanmkurtz]

I can't seem to spot anywhere in OmfLoader where the uninitialized block is getting populated.

Uninitialized blocks, by definition, don't get populated.

By the way, how are you quoting ranges of lines of code from the repo like that?

You can shift+click on a range of lines (in the margin) in github's presentation of the source, and then do "Copy permalink":