updates

Signed-off-by: (Bit-Mage) <[email protected]>

Content/20230712223044-lisp.org

@@ -34,6 +34,10 @@ Maxwell's equations of Software (~Alan Kay in some Interview~)
 	(#t       (apply (car e) (evlis (cdr e) a) a))))
 #+end_src
 * Comment-Stream
+** 0x2303
+ - read up on the sharp dot (#.) read macro
+ - https://www.lispworks.com/documentation/HyperSpec/Body/02_dhf.htm
+
 ** 0x22DA
  - what goes into [[id:2dfa41e9-07ba-4deb-a466-af409fa8b465][writing a lisp]] interpreter?
 ** 0x227B

Content/20230715180212-mutability.org

@@ -2,11 +2,4 @@
 :ID:       20230715T180212.260696
 :END:
 #+title: Mutability
-#+filetags: :trait:
-
-* 0x211A
-
-The ability to be changed. If denied, we refer the said entity to be immutable.
-Different styles of programming arise when one chooses a mutable/immutable representation to work with.
-
-This in turn gives rise to the notion of a State.
+#+filetags: :programming:

Content/20230717122922-vector_space_model.org

@@ -1,5 +1,6 @@
 :PROPERTIES:
 :ID:       9bb733a2-8540-4f7e-acd8-63547efa9b7e
+:ROAM_ALIASES: "Spatial Data"
 :END:
 #+title: vector space model
 #+filetags: :math:

Content/20230718222456-input_queue.org

@@ -28,6 +28,7 @@ These are meant to be navigated with org-roam emacs and accessing them is conven
 ** https://news.ycombinator.com/
 ** http://wiki.c2.com/
 ** https://www.cliki.net/
+** https://explodinginsights.com/
 * Blogs
 ** The CTO Club : https://thectoclub.com/
  - good for the bigger picture on emerging tech

Content/20230912141403-functions.org

@@ -2,4 +2,4 @@
 :ID:       a31671c6-12ea-4fc9-93cb-73d29fd508a6
 :END:
 #+title: Functions
-#+filetags: :math:
+#+filetags: :programming:math:

Content/20231003160003-attention.org

@@ -71,37 +71,6 @@
 - *[[id:799650ff-fea0-45b0-aabf-3f4d80540439][Softmax]] Scaling:* The division by \(\sqrt{d_k}\) stabilizes the gradients by preventing the dot-product values from becoming too large.
 - *Attention Mechanism:* It enables the transformer to focus differently on different input parts, capturing dependencies irrespective of their position.
 
-
-* Multihead Attention Detailed Elaboration
-
-Multihead Attention is a fundamental component of the Transformer architecture, which has been pivotal in advancing natural language processing and other fields.
-
-- *Core Concept*:
-  - Involves multiple attention mechanisms running in parallel, allowing the model to focus on different parts of the input sequence simultaneously.
-
-- *Multihead Attention*:
-  - Instead of having a single set of queries, keys, and values, Multihead Attention uses multiple sets, each called a 'head'.
-  - Each head performs its own scaled dot-product attention operation in parallel, and the results are concatenated and projected to obtain the final output.
-  - This allows the model to jointly attend to information from different representation subspaces at different positions.
-
-- *Benefits*:
-  - Enhanced modeling capacity: The use of multiple attention heads allows capturing diverse types of dependencies in the input.
-  - Parallelizability: Multihead attention enables parallel processing and efficient use of computational resources.
-
-- *Implementation Details*:
-  - In practice, the input is often split into a fixed number of heads, where each head performs attention on a portion of the embedding dimensions.
-  - The dimensions of Q, K, and V are typically chosen such that the dimensions after concatenation match the input dimension, facilitating residual connections, a key feature in Transformer architectures.
-
-- *Applications and Extensions*:
-  - Widely used in language models, such as BERT and GPT.
-  - Variants include multihead attention with relative positional encodings, multihead attention with local constraints in [[id:6145692e-b916-42d6-83a5-df1bc8eb2a6b][Vision Transformers]] (ViTs), etc.
-
-** Connections between these entities:
-
-- *Operational Process*: The attention mechanism's ability to focus comes from the interaction between Q, K, V, which is optimized through the training process, allowing for flexible modeling of input dependencies.
-- *Architectural Design*: Multihead mechanism builds upon the basic attention to leverage positional and contextual variations that a single head may miss.
-- *Use Cases*: This mechanism's applicability extends beyond NLP, making it versatile for problems involving sequential or spatial data.
-
 * Relevant Nodes
  - [[id:4f9006cf-6e6f-4019-bb8d-e7d5d85e191e][Transformers]]
  - [[id:affff439-329d-4962-bf5f-def85d75042e][Large Language Models]]

Content/20231012140139-large_language_models.org

@@ -96,6 +96,70 @@ The scaling laws for Large Language Models (LLMs) describe how changes in differ
 *** Truthful QA
 *** Massive Multitask Language Understanding
 *** Word in Context
-* Resources
+** Hyperparameters of an LLM
+*** During Training:
+- *Learning Rate*:
+  - Controls the step size for updating model weights.
+  - A crucial hyperparameter as it affects convergence and stability.
+
+- *Batch Size*:
+  - Number of training examples used in one iteration.
+  - Larger batch sizes can stabilize gradient updates but require more memory.
+
+- *Number of Epochs*:
+  - Defines how many times the entire training dataset is passed through the model.
+  - Needed to ensure adequate learning without overfitting.
+
+- *Optimizer Type*:
+  - Algorithms like Adam, SGD, or RMSProp used to adjust weights.
+  - Different optimizers can result in varying convergence speeds and outcomes.
+
+- *Dropout Rate*:
+  - Probability of dropping units in neural networks to prevent overfitting.
+  - Applied to the network layers during training.
+
+- *Weight Initialization*:
+  - Strategy for initializing model weights.
+  - Influences how quickly and effectively the model converges.
+
+- *Gradient Clipping*:
+  - Limits the maximum value of gradients to prevent exploding gradient issues.
+  - Especially useful in training large networks.
+
+- *Warmup Steps*:
+  - Number of initial training steps with a gradually increasing learning rate.
+  - Helps avoid large sudden updates in early training.
+
+*** During Inference:
+- *Beam Size (in beam search)*:
+  - Number of beams (alternate sequences) considered for output generation.
+  - Balances between computational resources and quality of output.
+
+- *Temperature*:
+  - Controls randomness during sampling; higher values increase randomness.
+  - Influences creativity versus coherence of generated text.
+
+- *Top-k Sampling*:
+  - Limits the next word selection to the top k probable entries.
+  - Reduces unpredictability by narrowing down the choice of words.
+
+- *Top-p Sampling (Nucleus Sampling)*:
+  - Extends top-k by choosing from a dynamically determined set of most probable outputs.
+  - Balances diversity and coherence more effectively than fixed k.
+
+- *Max Token Length*:
+  - Maximum number of tokens to generate in the output.
+  - Used to allocate computational resources appropriately.
+
+*** Connections:
+- *Learning Rate and Warmup Steps*:
+  - Both influence how learning is paced and stabilized during the early training stages.
+
+- *Batch Size and Gradient Clipping*:
+  - Larger batch sizes might affect the stability of gradients, where clipping can help to prevent instabilities.
+
+- *Temperature, Top-k, and Top-p Sampling*:
+  - These hyperparameters work together to modulate the randomness and quality of the generated text during inference.
+
 ** Book: Building LLMs for production
  - https://towardsai.net/book

Content/20231116192349-the_evolution_of_knowledge.org

@@ -38,3 +38,7 @@ IOU(objectivist epistemology, ontology) > IOU(epistemology, ontology)
 :ID:       d0d75b64-13e5-4b50-8833-a2b6a47cdf19
 :END:
 One can factorize a complex domain into its constituent abstractions by identifying the separate degrees of freedom in its description. Varying the extent of a particular degree at a time allows one to meaningfully study variants rather than simply diffing domains.
+
+* Generic Tools
+** [[id:cbd8cb36-6cba-46c3-84bf-077756dc3b37][Etymology]]
+** [[id:c30a1e1b-f615-4b02-884f-656fee743e1d][Calculi]]

Content/20240429115034-immutability.org

@@ -2,5 +2,4 @@
 :ID:       efba8f9b-a5df-4212-94c9-230bef916b5c
 :END:
 #+title: Immutability
-#+filetags: :trait:
-
+#+filetags: :programming:

Content/20240630180307-video_processing.org

@@ -1,8 +1,8 @@
 :PROPERTIES:
 :ID:       c3228445-00c4-4451-a0cc-b8742673753d
 :END:
-#+title: video processing
-#+filetags: :video:
+#+title: Video Processing
+#+filetags: :programming:bs:
 
 * Abstract
 Phases of producing a video

Content/20240630182300-ffmpeg.org

@@ -2,7 +2,7 @@
 :ID:       88f03140-7c60-41aa-b191-63e1460e76bd
 :END:
 #+title: ffmpeg
-#+filetags: :video:cli:tool:
+#+filetags: :cli:tool:
 
 * Abstract
 ** FFmpeg Overview

Content/20240707170507-handbrake.org

@@ -2,4 +2,4 @@
 :ID:       02abc3f4-0caf-481d-a439-80bf86f1740a
 :END:
 #+title: HandBrake
-#+filetags: :video:gui:tool:
+#+filetags: :gui:tool:

Content/20240707170716-non_linear_editing.org

@@ -2,4 +2,4 @@
 :ID:       4f1aa1e0-5409-4290-8564-be2e908c5c76
 :END:
 #+title: Non-Linear Editing
-#+filetags: :video:
+#+filetags: :tools:

Content/20240818165817-langchain.org

@@ -32,7 +32,6 @@
 ** [[id:46f09529-c273-49ed-9bf7-7e0a6d97d65c][Callbacks]]
  LangChain provides a callbacks system that allows you to hook into the various stages of your LLM application. This is useful for logging, monitoring, and streaming.
 
-
 * LangChain Agents & Tools Overview
 ** What?
  - complete tasks using chains, prompts, memory, and tools

Content/20240818170034-llammaindex.org

@@ -4,5 +4,7 @@
 #+title: Llammaindex
 #+filetags: :ai:tool:
 
+* Relevant Nodes
+** [[id:d9d30a75-f1aa-4ca0-8480-cb617afe29ab][Deep Lake Vector Stores]]
 * Resources
 - https://www.llamaindex.ai/

Content/20241115120450-variable_scope.org

@@ -0,0 +1,5 @@
+:PROPERTIES:
+:ID:       912ba440-5083-41a9-b2ab-553cd9a7f8c7
+:END:
+#+title: Variable Scope
+#+filetags: :programming:

Content/20241115120547-referential_transparency.org

@@ -0,0 +1,19 @@
+:PROPERTIES:
+:ID:       c6ef7bd2-dd62-44bf-90d5-a7520b873fcd
+:END:
+#+title: Referential Transparency
+#+filetags: :programming:
+
+* Overview
+- *Definition*:
+  - Referential transparency is a property of parts of a computer program where an expression can be replaced with its value without changing the program's behavior.
+  - often associated with [[id:20231212T081907.908301][functional programming]].
+
+- *Benefits*:
+  - *Improved Readability and Understandability*: Because expressions can be replaced consistently with their values, it simplifies reasoning about code.
+  - *Easier Debugging and Testing*: With referentially transparent code, testing becomes easier as expressions yield the same result every time they are evaluated with the same inputs.
+  - *Facilitates Optimization*: Compilers and interpreters can more easily optimize code through techniques like [[id:1bdc93aa-b564-4520-8590-c1ffcb026f55][memoization]] or [[id:eba75a47-6e35-4146-a152-b05089bbd8ec][lazy evaluation]].
+
+- *Key Concepts*:
+  - *Pure Functions*: Functions that are deterministic (same inputs lead to the same outputs) and do not produce [[id:ca98597a-027f-4eda-bbb2-801b8f74d0a5][side effects]], a critical component for achieving referential transparency.
+  - *[[id:efba8f9b-a5df-4212-94c9-230bef916b5c][Immutable]] Data*: Often accompanies referential transparency by ensuring that data is not altered after creation which supports consistency in function results.

Content/20241115120657-memoization.org

@@ -0,0 +1,5 @@
+:PROPERTIES:
+:ID:       1bdc93aa-b564-4520-8590-c1ffcb026f55
+:END:
+#+title: Memoization
+#+filetags: :programming:

Content/20241115120712-lazy_evaluation.org

@@ -0,0 +1,5 @@
+:PROPERTIES:
+:ID:       eba75a47-6e35-4146-a152-b05089bbd8ec
+:END:
+#+title: Lazy Evaluation
+#+filetags: :programming:

Content/20241115121014-side_effects.org

@@ -0,0 +1,5 @@
+:PROPERTIES:
+:ID:       ca98597a-027f-4eda-bbb2-801b8f74d0a5
+:END:
+#+title: Side Effects
+#+filetags: :programming:

Content/20241115121718-calculus.org

@@ -0,0 +1,32 @@
+:PROPERTIES:
+:ID:       c30a1e1b-f615-4b02-884f-656fee743e1d
+:ROAM_ALIASES: Calculi
+:END:
+#+title: Calculus
+#+filetags: :meta:
+
+* Etymological roots
+
+- *Origin Language*: The word "calculus" comes from Latin.
+- *Latin Meaning*: In Latin, "calculus" means "small stone" or "pebble."
+- *Historical Context*:
+  - /Counting Stones/: This term was historically used because small stones were used for counting and calculations, particularly in the ancient abacus.
+  - /Mathematical Practice/: As methods for systematic calculation evolved, the name stuck to represent the burgeoning mathematical field.
+- *Development of Mathematical Concept*:
+  - /17th Century and Beyond/: The development of calculus as we understand it today is attributed to the work of mathematicians Neumann, Leibniz, and Newton in the 17th century.
+  - /Use as a Mathematical Term/: Over time, "calculus" began to be used more broadly in the context of mathematics to specifically signify the advanced study of rates of change and quantities found under curves.
+
+Connections and Further Exploration:
+
+- The term’s roots in counting and measurement reflect the foundational aspects of calculus as a system for analyzing changes quantitatively.
+- The use of a metaphorical "small stones" indicates the fundamental building blocks nature of calculus in mathematics, similar to atoms in matter.
+- The transition from physical representations (stones) to abstract mathematical formulations mirrors the broader evolution of scientific thought from tangible observations to abstract theories.
+
+Questions for Further Exploration:
+
+- How did the transition from physical methods of calculation to theoretical calculus influence other scientific domains?
+- What impact did the work of Newton and Leibniz have on the understanding and teaching of calculus in modern education systems?
+- How has the notion of 'building blocks' evolved in other areas of mathematics and science from the etymological roots of calculus?
+
+* Branches
+** [[id:b2efd76b-740b-486d-b307-7ea4bed72b5d][Pi Calculus]]

Content/20241115122742-etymology.org

@@ -0,0 +1,5 @@
+:PROPERTIES:
+:ID:       cbd8cb36-6cba-46c3-84bf-077756dc3b37
+:END:
+#+title: Etymology
+#+filetags: :linguistics:meta:

Content/20241115141751-recursion.org

@@ -0,0 +1,63 @@
+:PROPERTIES:
+:ID:       95edc4bc-c364-4b18-833a-ba476b3283e8
+:END:
+#+title: Recursion
+#+filetags: :programming:
+
+* Overview
+
+- *Definition and Concept*:
+  - Recursion is a programming technique where a [[id:a31671c6-12ea-4fc9-93cb-73d29fd508a6][function]] calls itself, either directly or indirectly, to solve a problem.
+  - It typically involves a base case to stop the recursion and a recursive case to divide the problem into smaller instances of the same problem.
+
+- *Base Case*:
+  - The base case is the termination condition that stops the recursion.
+  - Without a base case, the recursive calls would continue indefinitely, leading to a [[id:ea557127-190f-4861-aecb-f727fe6e676b][stack overflow]].
+
+- *Recursive Case*:
+  - This is the part of the function that includes a call to itself.
+  - It reduces the problem at each step, bringing it closer to the base case.
+
+- *Applications*:
+  - Used in algorithms for traversing data structures like trees and graphs (e.g., depth-first search).
+  - Useful in problems that can be divided into similar sub-problems, like divide and conquer algorithms (e.g., mergesort, quicksort).
+
+*Connections*:
+- Recursion is a fundamental concept in computer science closely related to [[id:120cade5-1cf0-4afe-b541-e2b607ae77da][mathematical induction]], as both involve solving problems by breaking them down into base cases and smaller sub-problems.
+- Recursion is frequently compared to [[id:40722d92-1d10-445e-bcd9-f41999ccdf52][iteration]], as both can be used to repeat tasks but have different trade-offs in terms of complexity and efficiency.
+
+* Interconverting Recursion and [[id:40722d92-1d10-445e-bcd9-f41999ccdf52][Iteration]]
+** Steps for Interconverting Recursion to Iteration:
+
+1. *Identify Base Case*:
+   - Determine the base case(s) in recursion which serves as the termination condition.
+
+2. *Set Initial Conditions*:
+   - Translate the base case of recursion into initial conditions for the iterative loop.
+
+3. *Transform Recursive Logic*:
+   - Convert the recursive calls into loop updates. This often involves using auxiliary data structures like stacks to simulate the recursive call stack.
+
+4. *Loop Until Condition Met*:
+   - Replace the recursive function call mechanism with a loop structure to handle repetitive execution until the condition is met.
+
+5. *End Condition*:
+   - Ensure the loop has a termination condition similar to how a recursive function has a base case.
+
+** Steps for Interconverting Iteration to Recursion:
+
+1. *Define Base Case*:
+   - Translate the loop’s termination condition into a base case for the recursive function.
+
+2. *Create Recursive Function*:
+   - Define a recursive function that includes parameters reflecting loop variables or state.
+
+3. *Inject Recursive Call*:
+   - Replace loop iteration with a self-referential function call, where each call updates the state similar to loop iteration.
+
+4. *Ensure Termination*:
+   - Implement a termination check within the recursive function to ensure the function exits appropriately.
+
+* Relevant Nodes
+- [[id:3a717d24-64ef-4d38-936a-6814baaa1e6a][Tail Recursion]]
+- [[id:1bdc93aa-b564-4520-8590-c1ffcb026f55][Memoization]]

Content/20241115141836-stack_overflow.org

@@ -0,0 +1,5 @@
+:PROPERTIES:
+:ID:       ea557127-190f-4861-aecb-f727fe6e676b
+:END:
+#+title: stack overflow
+#+filetags: :programming:

Content/20241115141952-mathematical_induction.org

@@ -0,0 +1,5 @@
+:PROPERTIES:
+:ID:       120cade5-1cf0-4afe-b541-e2b607ae77da
+:END:
+#+title: mathematical induction
+#+filetags: :math:

Content/20241115142008-iteration.org

@@ -0,0 +1,5 @@
+:PROPERTIES:
+:ID:       40722d92-1d10-445e-bcd9-f41999ccdf52
+:END:
+#+title: Iteration
+#+filetags: :programming:

Content/20241115142039-tail_recursion.org

@@ -0,0 +1,5 @@
+:PROPERTIES:
+:ID:       3a717d24-64ef-4d38-936a-6814baaa1e6a
+:END:
+#+title: Tail Recursion
+#+filetags: :programming:

Content/20241115142152-computational_complexity.org

@@ -0,0 +1,5 @@
+:PROPERTIES:
+:ID:       bb91d3d1-c6f9-4864-9291-35c10ce79c42
+:END:
+#+title: Computational Complexity
+#+filetags: :programming: