awesome-graph-transformer

This repository contains a list of papers on the Graph Transformers; we categorize them based on their detailed techniques.

We will try to make this list updated. If you found any error or any missed paper, please don't hesitate to open an issue or pull request.

Structural Encoding / Postional Encoding for Graph Transformers

Spectral Positional Encoding

1. Rethinking Graph Transformers with Spectral Attention. NeurIPS 2021. [paper]
2. A Generalization of Transformer Networks to Graphs. AAAI workshop 2021. [paper]
3. Transformers Meet Directed Graphs. AAAI workshop 2023. [paper]

Other Structure-aware Encoding

1. Do Transformers Really Perform Bad for Graph Representation? NeurIPS 2021. [paper]
2. Graph Neural Networks with Learnable Structural and Positional Representations. ICLR 2022. [paper]
3. GRPE: Relative Positional Encoding for Graph Transformer. ICLR 2022 Workshop MLDD [paper]
4. Global Self-Attention as a Replacement for Graph Convolution. KDD 2022. [paper]
5. Pure Transformers are Powerful Graph Learners. NeurIPS 2022. [paper]
6. Are More Layers Beneficial to Graph Transformers? ICLR 2023. [paper]

Graph Neural Networks as Structural Encoder

1. GraphiT: Encoding Graph Structure in Transformers. arXiv 2021. [paper]
2. Representing Long-Range Context for Graph Neural Networks with Global Attention. NeurIPS 2021. [paper]
3. Structure-Aware Transformer for Graph Representation Learning. ICML 2022. [paper]
4. Recipe for a General, Powerful, Scalable Graph Transformer. NeurIPS 2022. [paper]

Scalability of Graph Transformers (Graph Transformers on Large Graphs)

Transformers with Sampling

1. A Self-Attention Network based Node Embedding Model. ECML-PKDD 2020. [paper]
2. Heterogeneous Graph Transformer. WWW 2020. [paper]
3. Gophormer: Ego-Graph Transformer for Node Classification. arXiv 2021. [paper]
4. Coarformer: Transformer for large graph via graph coarsening. Openreview 2021. [paper]
5. Hierarchical Graph Transformer with Adaptive Node Sampling. NeurIPS 2022. [paper]
6. NAGphormer: A Tokenized Graph Transformer for Node Classification in Large Graphs. ICLR 2023. [paper]

Transformers with Adapted Attention

1. From block-Toeplitz matrices to differential equations on graphs: towards a general theory for scalable masked Transformers. ICML 2022. [paper]
2. Recipe for a General, Powerful, Scalable Graph Transformer. NeurIPS 2022. [paper]
3. Deformable Graph Transformer. arXiv 2022. [paper]
4. NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classification. NeurIPS 2022. [paper]
5. DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion. ICLR 2023. [paper]

Applications of Graph Transformers (Molecules, Texts, Knowledge Graphs)

1. Modeling Graph Structure in Transformer for Better AMR-to-Text Generation. EMNLP 2019. [paper]
2. Heterogeneous Graph Transformer for Graph-to-Sequence Learning. ACL 2020. [paper]
3. Molecule Attention Transformer. arXiv 2020. [paper]
4. Interpretable Rumor Detection in Microblogs by Attending to User Interactions. AAAI 2020. [paper]
5. Graph Transformer for Graph-to-Sequence Learning. AAAI 2020. [paper]
6. Self-Supervised Graph Transformer on Large-Scale Molecular Data. NeurIPS 2020. [paper]
7. SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks. NeurIPS 2020. [paper]
8. Modeling Graph Structure via Relative Position for Text Generation from Knowledge Graphs. TextGraphs 2021. [paper]
9. GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph. NeurIPS 2021. [paper]
10. Systematic Generalization with Edge Transformers. NeurIPS 2021. [paper]
11. Mesh Graphormer. ICCV 2021. [paper]
12. Relative Molecule Self-Attention Transformer. arXiv 2021. [paper]
13. Neighbour Interaction based Click-Through Rate Prediction via Graph-masked Transformer. arXiv 2022. [paper]
14. Equivariant Transformers for Neural Network based Molecular Potentials. ICLR 2022. [paper]
15. Periodic Graph Transformers for Crystal Material Property Prediction. NeurIPS 2022. [pper]
16. Brain Network Transformer. NeurIPS 2022. [paper]
17. Multi-Relational Graph Transformer for Automatic Short Answer Grading. NAACL-HLT 2022. [paper]
18. Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queries. KDD 2022. [paper]
19. Relphormer: Relational Graph Transformer for Knowledge Graph Representations. arXiv 2022. [paper]
20. CoAtGIN: Marrying Convolution and Attention for Graph-based Molecule Property Prediction. BIBM 2022. [paper]
21. DiGress: Discrete Denoising diffusion for graph generation. ICLR 2023. [paper]
22. Relational Attention: Generalizing Transformers for Graph-Structured Tasks. ICLR 2023. [paper]
23. Heterformer: A Transformer Architecture for Node Representation Learning on Heterogeneous Text-Rich Networks. arXiv 2022. [paper]
24. Edgeformers: Graph-Empowered Transformers for Representation Learning on Textual-Edge Networks. ICLR 2023. [paper]

Pre-training with Graph Transformers

1. Selfsupervised graph transformer on large-scale molecular data. NeurIPS 2020. [paper]
2. Graph-Bert: Only Attention is Needed for Learning Graph Representations. arXiv 2020. [paper]
3. Graph Masked Autoencoders with Transformers. arXiv 2022. [paper]

Survey

1. Transformer for Graphs: An Overview from Architecture Perspective. arXiv 2022. [paper]
2. A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective. arXiv 2022. [paper]
3. Attending to Graph Transformers. arXiv 2022. [paper]

Neural Architecture Search (NAS) for Graph Transformers

1. AutoGT: Automated Graph Transformer Architecture Search. ICLR 2023. [paper]

Uncategorized

1. Transformers Generalize DeepSets and Can be Extended to Graphs & Hypergraphs. NeurIPS 2021. [paper]
2. Universal Graph Transformer Self-Attention Networks. WWW 2022. [paper]
3. Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification. IJCAI 2021. [paper]
4. Long Range Graph Benchmark. NeurIPS 2022. [paper]
5. Rethinking the Expressive Power of GNNs via Graph Biconnectivity. ICLR 2023. [paper]