This is an actively developed library that parses GLSL using an ANTLR4 grammar and provides the facilities for transforming and re-printing the resulting parse tree.
glsl-transformer is developed and maintained by douira. This project was created as a hobby project initiated by the need for a more powerful GLSL shader patcher in the wonderful Iris shaders mod for Minecraft. This repo is now part of the IrisShaders org on GitHub.
glsl-transformer is a library for GLSL program transformation. It uses a parser generated with ANTLR based on a custom GLSL grammar to turn shader code into a parse tree. At this point there are two different ways of approaching transformation: The concrete syntax tree (CST) that is directly taken from the parser can be transformed. The other approach turns this CST into an abstract syntax tree (AST) and then does transformations on that. In both types of trees each syntactic piece of the code is represented as a node with children. However, the AST is a little closer to the semantic content of the code rather than how the parser understands it.
For transforming the CST, the API facilitates the creation of composable transformations that can be used to manipulate this parse tree. Transformations can iterate through the parse tree, insert or remove nodes, match patterns and extract information from it. An execution planner optimizes at what time each step of the whole transformation process is performed. After the parse tree has been changed, it is printed back into a string while preserving the original whitespace.
Transforming the AST requires the additional step of building the AST from the parse tree (the CST). Once the AST is available, transforming it is easier and more efficient since traversing the entire tree frequently (or even altogether) is avoided though the use of index structures that allow fast queries for specific identifiers and node types. After transformation the AST is directly printed into a string, optionally with some formatting applied. Since the AST only contains the semantically relevant data from the code, any non-code parts of the original string, such as comments and formatting whitespace, are not present in the re-printed string. AST transformations don't have an execution planner and dependency graph mechanism since they operate on the AST without traversal that needs synchronization and management.
- GLSL Lexing & Parsing
- AST and CST transformation
- CST: Composable parse tree transformations
- CST: Pattern and XPath matching
- Tree walking with visitors and listeners
- Parse tree manipulation and declaration injection
- New nodes are treated as part of the existing parse tree
- CST: Whitespace-preserving re-printing
- CST: The original input is preserved if no changes are made
- AST: Complex pattern matching
- AST: Templating for subtree generation
- AST: Printing with various formatting options
- AST: Index-Based queries
Further reading on Abstract vs Concrete (Parse) Syntax Trees
Currently, glsl-transformer mostly operates only on the syntactic level, even if an AST is constructed. This means it only knows how the code looks and what structure it has to have, not what it means and which structures are legal or not. It only has a limited semantic understanding of the code. Semantic processing of programs can be implemented by API users on a case-by-case basis for specific tasks. The implemented AST does not do type checking or check if the defined structures adhere to the GLSL spec. Implementing full semantics would require building what basically amounts to a GLSL compiler which is way out of scope for this project.
glsl-transformer does not do semantic validation of the code and will not error on code that is actually invalid for semantic reasons. It will only error on syntax errors. In particular, it will not error on type errors like assigning a boolean to an integer. It supports GLSL 4.6 with some extensions such as explicit arithmetic types and some others. It won't error on modern syntax features even if your driver doesn't support them. Do not rely on glsl-transformer for shader validation, only for syntax transformation.
It also doesn't validate that features aren't used which may not be available in older GLSL versions. The #version directive is not semantically interpreted. It may also fail to parse code that is technically legal in old GLSL version where certain tokens weren't declared as reserved words yet. Just don't use reserved words as tokens in this case. If it's really necessary, preprocess the code by simply replacing the reserved words with some placeholder before parsing.
This project uses semver for versioning. If there are frequent breaking API changes then the major version will change frequently. This is the way.
This library is written in Java 16 and using jabel compiled to Java 8 compatible classes. This means it doesn't use any newer Java language APIs. The tests are not affected by this and will only be run on the latest Java version (because it's annoying only use Java 8 in the tests). If nobody needs Java 8 support anymore in the future, it will be dropped with a major release. Currently, this is because Minecraft 1.16 uses Java 8.
Credit for the basics goes to https://github.com/gabriele-tomassetti/antlr-mega-tutorial which is part of the nice Java Setup section of the ANTLR Mega Tutorial.
The files in glslang-test are from glslang which seems to be appropriately licensed that they can be used here. HLSL and other files have been removed since they aren't relevant to this project.
Of course all of this wouldn't be possible without ANTLR4 and its contributors, in particular Terence Parr, the creator of ANTLR and author of the ANTLR book. Thanks!
If something breaks, please make an issue report with the details so I can fix the issue and add a test case. For more direct support, you can also ask in the glsl-transformer thread in the Iris discord server. I'm also interested in hearing from anyone using this library in their projects!
# generate grammar sources to enable IDE usage
gradle generateGrammarSource
# building also runs the tests
gradle build
# run the tests (also generates the jacoco coverage report)
gradle test
# generate javadoc
gradle javadocCST Transformation
// setup a transformer
var transformer = new CSTTransformer<>();
// register a transformation
manager.addConcurrent(transformation);
// after transformation
System.out.println(transformer.transform(input));AST Transformation
// setup a transformer
var transformer = new ASTTransformer<>();
// set the transformation
transformer.setTransformation(translationUnit -> {
// do things
});
// after transformation
System.out.println(transformer.transform(input));- Adding a new parsed node anywhere (as a new local root)
- Removing any node with its entire subtree
- Replacing any node (remove, then add)
- Moving a local root node anywhere
- Moving a node strictly inside its local root scope (not a subscope)
- Removing or moving a non-local-root node without touching its subtree, though this may break things that rely on grammar rules being followed
- Moving a node out of the scope of its local root (it would be missing a token stream)
- Entirely removing a local root node and replacing it with a new one, giving it the previous node's children
See the Javadocs for API documentation and some API-specific notes. See the documentation overview for detailed documentation. The tests also illustrate the usage of each individual feature. The core package uses glsl-transformer's features and extends them to provide structures for common tasks.
See the planned features and other work items in TODO.md.