Ful Technical Documentation Support

[hackhaye]

Is there any detailed technical documentation? For example, architecture diagram.The detailed introduction of ghidra internal data structures is more user-friendly for developers new to ghidra development. The process of developing ghidra requires constant debugging of unfamiliar parts of the code, which makes the overall development rather slow.

[diommsantos]

I've been exploring the Ghidra source code and I can start to create such documentation. My knowlegde of Ghidra is quite limited compared to the devs, so contributions either from the Ghidra devs or the Ghidra community is basically a pre-requsite to make said documentation useful for the community. Is the creation of this documentation something the Ghidra developers would have interest on and would they be willing to contribute? @ryanmkurtz

[goatshriek]

As context for others, this has also been recently brought up on reddit.

I think a natural place for this would be in the overview page of the API javadocs, which is currently just a package list in 11.1.2. I think if this part of the javadoc also had a high-level architecture diagram and some pointers to the more detailed entry point classes (some OSGi info, some UI info, Plugin, etc) for specific parts it could go a long way towards making things easier to pick up for new folks. As I've gone through things I've found that specific classes have pretty thorough javadoc that describes things, but finding the right ones can be a chore. A high-level list of starting points might fill that gap in.

[dragonmacher]

Warning: philosophical ramblings follow; all thoughts are my own...

TL;DR: I do not believe the current Ghidra team has the energy, desire, resources or skill to have great / ideal documentation.

For me, I'd love to see us provide documentation not just for how the system works now (API javadocs), but also for the Ghidra Model, or the Ghidra Way. I think it would also be amazing if we provided documentation that would serve as training for those new to RE in general, using Ghidra as the training tool.

API Javadocs

Sadly, we have historically done a mediocre job at documenting our source code. We try to make sure our interfaces are documented, especially those that we expected others to use. We do use a list of packages to include when compiling the html javadocs. This is partly because our documentation is lacking, but it also helps to keep the size of the deliverable down. It would be technically possible for us to retroactively javadoc all of our source code, but I think the output would be poor if not downright incorrect for many classes. So much of the code we have now was created by others or by us so long ago that we have forgotten the most important design decisions made when writing the code. I don't have any stats, but my gut tells me that fully documenting all of our code would be a herculean task. Not only would it be a large effort, I doubt that any of the senior developers would enjoy such a task. I personally have arrived at a documentation philosophy that tries to balance utility vs effort. High-level interfaces meant for consumption or extension are given more attention. (Honestly, even if I wanted to write amazing documentation, I feel like I would struggle without community feedback to help me address conceptual shortcomings.) I think we could improve in this area, but I have no sense of how much end users are consuming our generated javadoc. As it stands now, I assume that if users need more API documentation than we provide, they will just download and read the source code, which is admittedly a lot of work.

Architectural Overview (The Ghidra Way)

This is probably the more important type of documentation we could provide. I would love to see us have multiple levels of documentation that address how Ghidra exists in the mind of the senior developers. The system is built and maintained using this model. An interesting thing to me is that since we do not have this model documented, working in the system feels very much like working in an apprenticeship style, requiring a great deal of collaboration. As pointed out by this ticket, working this way does not scale well for open source projects. Ideally, I'd like to see enough documentation to help users from various backgrounds, including: novice, RE professional, developer and even RE developer. Unlike the API javadocs, I think this type of documentation is more tractable. It may take a few years to get this in place and in a useful form, but I think I could make a strong justification for the value of such an endeavor. Another benefit of this type of documentation is that it helps preserve domain knowledge that will inevitably be lost as our developers age off the project.

In addition to conceptual overviews, I think it would be great to have a sort of FAQ style of documentation for common tasks, such as "How do I write a Plugin?", or "When should I write a Plugin instead of a script?".

Where We Are Now

Sadly, based on spending a professional lifetime working in this code, I'm not sure if 'great documentation' is something we will ever attain. I feel like the Ghidra team has a sea of work, but a limited window of opportunity and resources. We tend to work on the work with the highest perceived value. Documentation is not something that tends to make the top of the list. Also, as I mentioned, generally speaking, developers do not enjoy working on documentation. Additionally, the more documentation we have, the greater the upkeep and the more onerous the development process becomes. All of this makes it harder to get team buy-in when discussing how to create better documentation.

How to Proceed?

I personally place a high value on community interaction. I suggest that users upvote this ticket (and get their friends, family and anyone else they can to upvote as well).

[goatshriek]

All of your insight into where things are is super helpful, thanks for sharing your thoughts and time!

It's a LOT of work to maintain something like this, and I totally understand why things are where they are now. Even as I go through the source myself to find things, I rarely take the time to make and submit documentation improvements. If I had to choose between the Ghidra team writing docs or cool new features... I enjoy the features. 😄

On the topic of the small community tweaks, I think having a central/sanctioned place to keep community-generated stuff would close that gap. I understand that there would still be a review burden, but maybe it's more acceptable? Perhaps a spot in the repo that just doesn't end up in release builds, a way to submit Wiki pages, or something more wild west in the style of Gentoo's GURU repository. The way things are now with individual projects and blog posts is okay, but they are scattered all over, tend to age out, and can be misleading or wrong depending on the author.

I've run into this before while doing plugin dev. The training docs give a good starting point, but there's not a great place to look for more detailed questions like "how should I integrate this into a CI pipeline like github actions" and "how should I test this thing". That doesn't seem to fit into javadocs to me, so I don't know of a way to submit it today. I could document it for just my project, but then it's only really visible to people using my (niche) plugin. If there was a FAQ place like you suggested that we could submit to, that would be awesome!

tl;dr

All that to say that I get why this isn't a top priority, and do want to help when I have a clear ability to. Harnessing community support might make good progress with less effort for the Ghidra team - but I don't know if that "less effort" is "less" enough.