FiveTech Software tech support forums

graph TD subgraph Input Tokens --> Embedding Embedding --> Positional_Encoding end subgraph Encoder Positional_Encoding --> MultiHead_Attention1 MultiHead_Attention1 --> AddNorm1 AddNorm1 --> FeedForward1 FeedForward1 --> AddNorm2 AddNorm2 --> Encoder_Output end subgraph Decoder Encoder_Output --> Masked_MultiHead_Attention Masked_MultiHead_Attention --> AddNorm3 AddNorm3 --> MultiHead_Attention2 MultiHead_Attention2 --> AddNorm4 AddNorm4 --> FeedForward2 FeedForward2 --> AddNorm5 AddNorm5 --> Linear Linear --> Softmax Softmax --> Output end subgraph Attention Query --> Scaled_Dot_Product Key --> Scaled_Dot_Product Value --> Scaled_Dot_Product Scaled_Dot_Product --> Mask[Optional Mask] Mask --> Softmax_Attention Softmax_Attention --> Weighted_Sum end subgraph MultiHead Weighted_Sum --> Concat Concat --> Linear_Projection end %% Backpropagation (punteado e inverso) %% Desde la pérdida en la salida Output -. grad .-> Softmax Softmax -. dL/dlogits .-> Linear %% Decoder backward Linear -. grad .-> AddNorm5 AddNorm5 -. grad .-> FeedForward2 FeedForward2 -. grad .-> AddNorm4 AddNorm4 -. grad .-> MultiHead_Attention2 MultiHead_Attention2 -. grad to Q,K,V,W .-> AddNorm3 AddNorm3 -. grad .-> Masked_MultiHead_Attention Masked_MultiHead_Attention -. grad .-> Encoder_Output %% Encoder backward (vía cross-attention) Encoder_Output -. grad .-> AddNorm2 AddNorm2 -. grad .-> FeedForward1 FeedForward1 -. grad .-> AddNorm1 AddNorm1 -. grad .-> MultiHead_Attention1 MultiHead_Attention1 -. grad .-> Positional_Encoding Positional_Encoding -. dL/dEmb .-> Embedding Embedding -. grad .-> Tokens %% Atención interna (backprop por atención) %% Nota: se muestran gradientes típicos dentro de la subcapa de atención Softmax_Attention -. dL/dAttn .-> Mask Mask -. grad .-> Scaled_Dot_Product Scaled_Dot_Product -. grad Q,K,V .-> Query Scaled_Dot_Product -. grad Q,K,V .-> Key Scaled_Dot_Product -. grad Q,K,V .-> Value Weighted_Sum -. grad .-> Softmax_Attention Concat -. grad .-> Weighted_Sum Linear_Projection -. grad .-> Concat

flowchart TD A[Tech Tree] --> B[Python] %% --- Dataset --- subgraph DS[Dataset] C[Basic Dataset] --> R[Data Collection] R --> S[Data Cleaning] S --> T[Tokenisation] T --> U[Byte Pair Encoding] U --> V[Causal Language Modelling] V --> W[Positional Encoding] W --> X[Rotary Positional Encoding] end %% --- Operaciones Básicas --- subgraph OP[Operaciones Básicas] D[Tensor] E[Basic Operations] Y[Linear Layer] Z[ReLU] AA[Swish] AB[GeLU] AC[SiLU/Swish] AD[Attention] AE[Multi-Head Attention] end %% --- Entrenamiento --- subgraph TR[Entrenamiento] F[Autograd] G[Mean Squared Error] H[Cross Entropy] I[Backpropagation] J[Stochastic Gradient Descent] K[Weight Decay] L[Momentum] M[Learning rate] N[Gradient Clipping] O[AdamW] AN[Hyperparameter Tuning] AO[μ-Scaling Laws] end %% --- Arquitectura --- subgraph ARQ[Arquitectura] AF[Multi-Layer Perceptron] Q[Transformer Block] AG[Transformer] AH[Small Language Model] AI[Modern Language Model] AJ[Quantisation] AK[Batch Norm] AL[Layer Norm] AM[RMSNorm] end %% Relaciones principales B --> C B --> D B --> E D --> F F --> G F --> H F --> I I --> J J --> K J --> L J --> M J --> N J --> O O --> Q E --> Y Y --> Z Y --> AA Y --> AB AA --> AC Y --> AD AD --> AE Z --> AF AA --> AF AB --> AF AC --> AF AF --> Q AE --> Q I --> Y Q --> AG AG --> AH AH --> AI AI --> AJ AH --> AK AH --> AL AH --> AM K --> Q L --> Q M --> Q N --> Q AN --> AO --> Q