Explain penultimate layer choice for trigger extraction
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2
paper.md
2
paper.md
@@ -23,7 +23,7 @@ Fine-tuning modifies weights. RAG re-encodes text each time. LoRA requires gradi
|
||||
**Frozen backbone**: any transformer. Produces hidden-state vectors from input tokens. Weights never modified.
|
||||
|
||||
**Reflex bank**: stores (trigger, response) pairs:
|
||||
- **Trigger**: hidden-state vector h at the final token position — the model's activation pattern for a given input.
|
||||
- **Trigger**: hidden-state vector h at the final token position, extracted from the penultimate layer (N-1). The final layer is optimized for next-token prediction via the lm_head projection; earlier layers retain richer semantic structure for similarity matching.
|
||||
- **Response**: per-position logit biases [(token_id, boost)] — one pair per answer token.
|
||||
|
||||
Both are sub-symbolic. The trigger is an opaque high-dimensional vector; the response is a list of (integer, float) pairs. The reflex bank resists inspection without the backbone that produced it.
|
||||
|
||||
Reference in New Issue
Block a user