tommi/cri
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Explain penultimate layer choice for trigger extraction

Browse Source 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Tommi Niemi
2026-04-06 19:46:32 +07:00
parent eeada7810a
commit 2e61c2353a
4 changed files with 11 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
paper.log
View File

@@ -1,4 +1,4 @@
This is pdfTeX, Version 3.141592653-2.6-1.40.29 (TeX Live 2026/Arch Linux) (preloaded format=pdflatex 2026.3.11) 6 APR 2026 19:44 This is pdfTeX, Version 3.141592653-2.6-1.40.29 (TeX Live 2026/Arch Linux) (preloaded format=pdflatex 2026.3.11) 6 APR 2026 19:46
entering extended mode entering extended mode
restricted \write18 enabled. restricted \write18 enabled.
%&-line parsing enabled. %&-line parsing enabled.
@@ -429,7 +429,7 @@ File: umsb.fd 2013/01/14 v3.01 AMS symbols B
) [1 ) [1
{/var/lib/texmf/fonts/map/pdftex/updmap/pdftex.map}] {/var/lib/texmf/fonts/map/pdftex/updmap/pdftex.map}]
Overfull \hbox (41.58173pt too wide) in paragraph at lines 118--122 Overfull \hbox (41.58173pt too wide) in paragraph at lines 120--124
\OT1/cmr/m/n/10.95 it (1536-dim, 35 lay-ers), E4B base (2560-dim, 42 lay-ers). \OT1/cmr/m/n/10.95 it (1536-dim, 35 lay-ers), E4B base (2560-dim, 42 lay-ers).
Quan-ti-za-tion tested at f32/f16/bf16/int8/int4 Quan-ti-za-tion tested at f32/f16/bf16/int8/int4
[] []
@@ -440,18 +440,14 @@ Quan-ti-za-tion tested at f32/f16/bf16/int8/int4
LaTeX2e <2025-11-01> LaTeX2e <2025-11-01>
L3 programming layer <2026-01-19> L3 programming layer <2026-01-19>
*********** ***********
LaTeX Warning: Label(s) may have changed. Rerun to get cross-references right.
Package rerunfilecheck Info: File `paper.out' has not changed. Package rerunfilecheck Info: File `paper.out' has not changed.
(rerunfilecheck) Checksum: 4198C661EF4EF302CF6CA8FE934684BD;2020. (rerunfilecheck) Checksum: 4198C661EF4EF302CF6CA8FE934684BD;2020.
) )
Here is how much of TeX's memory you used: Here is how much of TeX's memory you used:
12070 strings out of 467525 12071 strings out of 467525
177761 string characters out of 5425861 177779 string characters out of 5425861
607691 words of memory out of 5000000 607805 words of memory out of 5000000
40819 multiletter control sequences out of 15000+600000 40820 multiletter control sequences out of 15000+600000
639085 words of font info for 84 fonts, out of 8000000 for 9000 639085 words of font info for 84 fonts, out of 8000000 for 9000
1141 hyphenation exceptions out of 8191 1141 hyphenation exceptions out of 8191
75i,8n,79p,324b,557s stack positions out of 10000i,1000n,20000p,200000b,200000s 75i,8n,79p,324b,557s stack positions out of 10000i,1000n,20000p,200000b,200000s
@@ -468,7 +464,7 @@ public/amsfonts/cm/cmr17.pfb></usr/share/texmf-dist/fonts/type1/public/amsfonts
mf-dist/fonts/type1/public/amsfonts/cm/cmtt10.pfb></usr/share/texmf-dist/fonts/ mf-dist/fonts/type1/public/amsfonts/cm/cmtt10.pfb></usr/share/texmf-dist/fonts/
type1/public/amsfonts/cm/cmtt12.pfb></usr/share/texmf-dist/fonts/type1/public/c type1/public/amsfonts/cm/cmtt12.pfb></usr/share/texmf-dist/fonts/type1/public/c
m-super/sfrm1095.pfb> m-super/sfrm1095.pfb>
Output written on paper.pdf (5 pages, 192843 bytes). Output written on paper.pdf (5 pages, 193342 bytes).
PDF statistics: PDF statistics:
207 PDF objects out of 1000 (max. 8388607) 207 PDF objects out of 1000 (max. 8388607)
168 compressed objects within 2 object streams 168 compressed objects within 2 object streams

2
paper.md
View File

@@ -23,7 +23,7 @@ Fine-tuning modifies weights. RAG re-encodes text each time. LoRA requires gradi
**Frozen backbone**: any transformer. Produces hidden-state vectors from input tokens. Weights never modified. **Frozen backbone**: any transformer. Produces hidden-state vectors from input tokens. Weights never modified.
**Reflex bank**: stores (trigger, response) pairs: **Reflex bank**: stores (trigger, response) pairs:
- **Trigger**: hidden-state vector h at the final token position — the model's activation pattern for a given input. - **Trigger**: hidden-state vector h at the final token position, extracted from the penultimate layer (N-1). The final layer is optimized for next-token prediction via the lm_head projection; earlier layers retain richer semantic structure for similarity matching.
- **Response**: per-position logit biases [(token_id, boost)] — one pair per answer token. - **Response**: per-position logit biases [(token_id, boost)] — one pair per answer token.
Both are sub-symbolic. The trigger is an opaque high-dimensional vector; the response is a list of (integer, float) pairs. The reflex bank resists inspection without the backbone that produced it. Both are sub-symbolic. The trigger is an opaque high-dimensional vector; the response is a list of (integer, float) pairs. The reflex bank resists inspection without the backbone that produced it.

BIN
paper.pdf
View File

Binary file not shown.

4
paper.tex
View File

@@ -63,7 +63,9 @@ input tokens. Weights never modified.
\textbf{Reflex bank}: stores (trigger, response) pairs: \textbf{Reflex bank}: stores (trigger, response) pairs:
\begin{itemize} \begin{itemize}
\item \textbf{Trigger}: hidden-state vector $\mathbf{h}$ at the final token \item \textbf{Trigger}: hidden-state vector $\mathbf{h}$ at the final token
position---the model's activation pattern for a given input. position, extracted from the penultimate layer ($N{-}1$). The final layer is
optimized for next-token prediction via the lm\_head projection; earlier layers
retain richer semantic structure for similarity matching.
\item \textbf{Response}: per-position logit biases \item \textbf{Response}: per-position logit biases
$\{(t_i, b_i)\}$---one pair per answer token. $\{(t_i, b_i)\}$---one pair per answer token.
\end{itemize} \end{itemize}