Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AI Training - Max #59

Merged
merged 51 commits into from
Dec 7, 2023
Merged
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
3e8ced1
First pass at training
Nov 14, 2023
5c26613
Upd
Nov 14, 2023
5af7468
Added section place holders
profvjreddi Nov 15, 2023
624e6c1
Updated the introduction
profvjreddi Nov 15, 2023
bf96aca
Add overview
profvjreddi Nov 15, 2023
ac1f659
Updated introduction
profvjreddi Nov 15, 2023
aab7018
updated NN intro
profvjreddi Nov 15, 2023
fb27e38
Minor rework of the writing
profvjreddi Nov 15, 2023
5c0b895
Merge branch 'main' into pr/59
profvjreddi Nov 15, 2023
65b4812
Added training data content
profvjreddi Nov 15, 2023
3c67582
Focusing on the big items we cover in the chapter
profvjreddi Nov 15, 2023
d3902ee
Update backprop
Nov 20, 2023
307dafb
Markdown
Nov 20, 2023
cf3b67f
Upd training
Nov 20, 2023
0d0c613
Upd training
Nov 20, 2023
893b750
Upd training
Nov 20, 2023
fe2a424
Upd training
Nov 20, 2023
79757d1
Upd training
Nov 20, 2023
733ec36
Upd training
Nov 20, 2023
c58a6d0
Merge branch 'main' into pr/59
profvjreddi Nov 27, 2023
c7be42a
aitraining images
Nov 27, 2023
20fe3b7
Expanded the algorithms section and added references
profvjreddi Nov 30, 2023
752d3e4
Merge branch 'main' into training
profvjreddi Nov 30, 2023
e382921
Updated cover image
profvjreddi Nov 30, 2023
199779e
Update training.qmd
profvjreddi Nov 30, 2023
a38cddd
Remove grammar fix
profvjreddi Nov 30, 2023
7412fe1
Made a pass on the hyperparameter section
profvjreddi Nov 30, 2023
7beb911
Merge branch 'main' into pr/59
profvjreddi Dec 5, 2023
c2e48ec
References training
Dec 5, 2023
bdc4f3f
Adding in more details about regularization
profvjreddi Dec 5, 2023
55e4171
Minor updates to Regularization section
profvjreddi Dec 5, 2023
49794e9
Weight initialization update pass
profvjreddi Dec 5, 2023
076a599
Making weight initialization connections to weight init
profvjreddi Dec 5, 2023
4f7c6bc
Update training.qmd
profvjreddi Dec 5, 2023
3fe19f4
fix bibtex reference for he et al.
profvjreddi Dec 5, 2023
bfbdc4c
Added details to activation functions
profvjreddi Dec 5, 2023
7307e84
references
Dec 5, 2023
f5c468a
references bib
Dec 6, 2023
ea66794
references
Dec 6, 2023
cecadb0
Improving the MM section with details
profvjreddi Dec 6, 2023
14f6950
MD fixes
profvjreddi Dec 6, 2023
ad05af0
Merge branch 'training' of https://github.com/agnusmaximus/cs249r_boo…
profvjreddi Dec 6, 2023
a60904a
sorting references
profvjreddi Dec 6, 2023
1bc8513
Improving the optimizations section
profvjreddi Dec 6, 2023
a42d79d
Roofline / big batch-size training
Dec 7, 2023
d92a50b
training
Dec 7, 2023
a16a060
Updated the training parallelization section
profvjreddi Dec 7, 2023
bff3e9e
Merge branch 'training' of https://github.com/agnusmaximus/cs249r_boo…
profvjreddi Dec 7, 2023
606cee9
Updated the learning objectives.
profvjreddi Dec 7, 2023
8c0c3bf
Updated the conclusion section.
profvjreddi Dec 7, 2023
5817210
MD fixes
profvjreddi Dec 7, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added images/ai_training.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/aitrainingfit.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/aitrainingnn.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/aitrainingpara.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/aitrainingroof.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/aitrainingsgd.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
46 changes: 46 additions & 0 deletions references.bib
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,20 @@ @article{oecd22
doi = "https://doi.org/https://doi.org/10.1787/7babf571-en"
}

@article{ruder2016overview,
title={An overview of gradient descent optimization algorithms},
author={Ruder, Sebastian},
journal={arXiv preprint arXiv:1609.04747},
year={2016}
}

@inproceedings{abadi2016tensorflow,
title = {$\{$TensorFlow\$\}\$: a system for \$\{\$Large-Scale\$\}\$ machine learning},
author = {Abadi, Mart{\'\i}n and Barham, Paul and Chen, Jianmin and Chen, Zhifeng and Davis, Andy and Dean, Jeffrey and Devin, Matthieu and Ghemawat, Sanjay and Irving, Geoffrey and Isard, Michael and others},
year = 2016,
booktitle = {12th USENIX symposium on operating systems design and implementation (OSDI 16)},
pages = {265--283},
}

@inproceedings{sculley2015hidden,
title = {"Everyone wants to do the model work, not the data work": Data Cascades in High-Stakes AI},
Expand All @@ -31,6 +44,39 @@ @inproceedings{kocher1996timing
organization={Springer}
}

@inproceedings{DBLP:journals/corr/KingmaB14,
author = {Diederik P. Kingma and
Jimmy Ba},
editor = {Yoshua Bengio and
Yann LeCun},
title = {Adam: {A} Method for Stochastic Optimization},
booktitle = {3rd International Conference on Learning Representations, {ICLR} 2015,
San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings},
year = {2015},
url = {http://arxiv.org/abs/1412.6980},
timestamp = {Thu, 25 Jul 2019 14:25:37 +0200},
biburl = {https://dblp.org/rec/journals/corr/KingmaB14.bib},
bibsource = {dblp computer science bibliography, https://dblp.org}
}

@article{dahl2023benchmarking,
title={Benchmarking Neural Network Training Algorithms},
author={Dahl, George E and Schneider, Frank and Nado, Zachary and Agarwal, Naman and Sastry, Chandramouli Shama and Hennig, Philipp and Medapati, Sourabh and Eschenhagen, Runa and Kasimbeg, Priya and Suo, Daniel and others},
journal={arXiv preprint arXiv:2306.07179},
year={2023}
}


@article{afib,
title = {Mobile Photoplethysmographic Technology to Detect Atrial Fibrillation},
author = {Yutao Guo and Hao Wang and Hui Zhang and Tong Liu and Zhaoguang Liang and Yunlong Xia and Li Yan and Yunli Xing and Haili Shi and Shuyan Li and Yanxia Liu and Fan Liu and Mei Feng and Yundai Chen and Gregory Y.H. Lip and null null},
year = 2019,
journal = {Journal of the American College of Cardiology},
volume = 74,
number = 19,
pages = {2365--2375},
}

@inproceedings{agrawal2003side,
title={The EM side—channel (s)},
author={Agrawal, Dakshi and Archambeault, Bruce and Rao, Josyula R and Rohatgi, Pankaj},
Expand Down
674 changes: 621 additions & 53 deletions training.qmd

Large diffs are not rendered by default.