Explain penultimate layer choice for trigger extraction
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -63,7 +63,9 @@ input tokens. Weights never modified.
|
||||
\textbf{Reflex bank}: stores (trigger, response) pairs:
|
||||
\begin{itemize}
|
||||
\item \textbf{Trigger}: hidden-state vector $\mathbf{h}$ at the final token
|
||||
position---the model's activation pattern for a given input.
|
||||
position, extracted from the penultimate layer ($N{-}1$). The final layer is
|
||||
optimized for next-token prediction via the lm\_head projection; earlier layers
|
||||
retain richer semantic structure for similarity matching.
|
||||
\item \textbf{Response}: per-position logit biases
|
||||
$\{(t_i, b_i)\}$---one pair per answer token.
|
||||
\end{itemize}
|
||||
|
||||
Reference in New Issue
Block a user