Reading group for Fall 23

This reading group will go on from mid August till first week of december. We have approximately 14 weeks which means we can hold 28 presentations in total.

Relevant Links

1. (webpage link)
2. Sign up sheet (spreadsheet)
3. Unanswered questions doc (doc)

Contributors

Name	Role
Aditya Desai	TBD
Zhenghui Guo	R2
Gaurav Gupta	R3
Masa Maksimovic	R2
Atishay Jain	R1
Sanya Garg	R2
Benjamin Meisburger	R2
Apoorv Walia	R1
Jonah Yi	R1

Name	Title	Suggested to be read by	date	Unanswered questions updated?	slides
Aditya Desai	Moments, deviations, chernoff/Hoeffding	All	08/28/2023	Yes	N.A
Tony Zhang	Near neighbor graph, inner product transformation, inner product search	All	09/06/2023	Yes	Slides
Jonah Yi	Learning to Route	All	09/06/2023	Yes	Slides
Ben Meisburger	Deep Gradient Compression	All	09/11/2023	Yes	Slides
Masa Maksimovic	Convergence in ML	All	09/11/2023	Yes	❗
Zhenghui Guo(Kevin)	KV cache inference	All	09/18/2023	Yes	Slides
Atishay Jain	Knowledge neurons	All	09/18/2023	❗	❗
Gaurav	Vector Databases	All	09/25/2023	❗	❗
Apoorv	Randomized projection based compression	All	10/16/2023	❗	❗
Aditya Desai	Recent results on projection based compression	All	10/16/2023	❗	❗
Zhenghui Guo(Kevin)	You Only Sample (Almost) Once: Linear Cost Self-Attention Via Bernoulli Sampling	All	10/23/23	❗	[Slides]
Jonah Yi	Accelerating Large-Scale Inference with Anisotropic Vector Quantization	All	10/30/23	❗	Slides
Benjamin Meisburger	ROSE: Robust Caching for Amazon Search		11/13/23		Slides
Jonah Yi	ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms	All	11/20/23	❗	Slides

Announcements (Format - Title : Date of announcement) . Please put latest first.

Presentation by Jonah Yi 20th Nov.

1. ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms: https://arxiv.org/pdf/1807.05614.pdf
2. A Comprehensive Survey and Experimental Comparison of Graph-Based Approximate Nearest Neighbor Search: https://arxiv.org/pdf/2101.12631.pdf

Presentation by Benjamin Meisburger 13th Nov.

1. Robust Caching for Amazon Search: https://assets.amazon.science/dc/5e/b919974a4abdba7a3cd82c1bc86f/rose-robust-caches-for-amazon-product-search.pdf

Presentation by Jonah Yi 30th Oct.

1. Accelerating Large-Scale Inference with Anisotropic Vector Quantization: https://arxiv.org/pdf/1908.10396.pdf

Presentation by Zhenghui Guo(Kevin) 23rd Oct.

1. You Only Sample (Almost) Once: Linear Cost Self-Attention Via Bernoulli Sampling: https://proceedings.mlr.press/v139/zeng21a/zeng21a.pdf

Presentation by Sanya Garg: 235d Oct.

1. Learning without Forgetting for Vision-Language Models: https://arxiv.org/pdf/2305.19270.pdf

Presentation by Apoorv Walia: 16th Oct.

1. Introduction to Parameter Sharing Methods (Papers - Roast: Efficient Model Compression with Random Operation Access Specific Tile Hashing - https://arxiv.org/pdf/2207.10702.pdf)
2. Robe: Random Offset Block Embedding Array - https://arxiv.org/pdf/2108.02191.pdf
3. Compressing Neural Networks with the Hashing Trick - https://arxiv.org/pdf/1504.04788.pdf

Presentation by Aditya Desai: 16th Oct.

1. Results from "In defense of parameter sharing for model compression"

Presentation by Gaurav Gupta: 25th Sept.

1. Introduction to Vector databases (paper- Milvus: A purpose-built vector data management system, https://dl.acm.org/doi/abs/10.1145/3448016.3457550)

Presentation by Atishay Jain: 18th Sept.

1. Knowledge Neurons in Pretrained Transformers: https://arxiv.org/abs/2104.08696
2. Locating and Editing Factual Associations in GPT: https://arxiv.org/abs/2202.05262

Presentation by Zhenghui Guo(kevin): 18th Sept.

1. Efficient Memory Management for Large Language Model Serving with PagedAttention: https://arxiv.org/pdf/2309.06180.pdf

Presentation by Ben Meisburger: 11th Sept.

1. Deep Gradient Compression: https://openreview.net/pdf?id=SkhQHMW0W

Presentation by Masa Maksimovic: 11th Sept.

1. Convergence proofs for some simple settings: https://akyrillidis.github.io/comp414-514/schedule/?fbclid=IwAR1_ImKSLRQFgEGENLVBJD5stwFKf7fogHwse-w-NBrHxFVdMHl7iLRMUVA
2. A convergence theory for deep learning: https://arxiv.org/abs/1811.03962v1

Presentation by Tony Zhang : 1st Sept.

1. Hierarchical Navigable Small World (HNSW) data structure for approximate near neighbor search: https://arxiv.org/abs/1603.09320
2. Vector transformation for making them more amenable to inner product search: https://arxiv.org/abs/1405.5869
3. IP-NSW for maximum inner product search: https://proceedings.neurips.cc/paper/2018/hash/229754d7799160502a143a72f6789927-Abstract.html

Presentation by Jonah Yi : 1st Sept.

Learning to Route (LTR) in Similarity Graphs: https://arxiv.org/abs/1905.10987

Location Confirmed! DH1049 : 30th August

We have confirmed booking for DH1049 every monday (except Oct 16 where we will book a library room)

Presentation by Aditya Desai : 26th August

I will aim to cover parts of chapter 3 and chapter 4 from http://lib.ysu.am/open_books/413311.pdf

Roles in Fall 23 : 9th August 2023

We have the following roles in this edition.

R1:

This is focused research presentations. You will have to declare a particular problem that you are looking at and make three presentations:

1. The first presentation will be motivating the problem (signifiance, impact, urgency, etc) and some related work that people have done. (Example : Embedding table compression is important, why, how have people solved this in the past, what problems remain, etc)
2. Fundamentals of the approach. This is a "teaching presentation" where you explain the fundamental toolkit that you are using. (Example: I am going to use random projection based approach. what are random projections, how to analyse their usefulness, what are hash functions, why they are important, analysing hash functions)
3. Your proposed method and results . (Example, the recipe to apply random projection based compression in embedding tables, results - quality, efficiency, etc)

R2:

This is more general research presentations. You will pick an area of research (for example, training efficiency, model compression, ml on edge, etc). You ll be making 2 presentations. This is similar to how things ran in summer edition of reading group. You ll keep reading papers on a single topic and summarize your learnings in two presentations.

R3:

This is a role of "moderating presentations". You ll not be presenting yourself. But you ll shadow readings of others if possible and actively discuss while they are presenting. The goal of this role is to keep the presentations engaging and to get the best out of it.

R4:

This is a role of "general audience and shadow". You ll read papers from one of the broad topics shadowing other presenters, listen in to presentations and participate in discussions.

❗ Add your name to the table above about roles.

How to participate : 9th August 2023 ❗❗

1. If you are participating in the reading group, send me your github id at [email protected] . I will add you to the collaborators so that you can change the readme and thus update the webpage.
2. It will a collaborative effort to keep this webpage updated. So make sure to get yourself added to collaborators and update portions that are relevant to you over the period of entire edition.

Previous Editions

Summer 2023