Explain penultimate layer choice for trigger extraction

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-04-06 19:46:32 +07:00
parent eeada7810a
commit 2e61c2353a
4 changed files with 11 additions and 13 deletions

View File

@@ -23,7 +23,7 @@ Fine-tuning modifies weights. RAG re-encodes text each time. LoRA requires gradi
**Frozen backbone**: any transformer. Produces hidden-state vectors from input tokens. Weights never modified.
**Reflex bank**: stores (trigger, response) pairs:
- **Trigger**: hidden-state vector h at the final token position — the model's activation pattern for a given input.
- **Trigger**: hidden-state vector h at the final token position, extracted from the penultimate layer (N-1). The final layer is optimized for next-token prediction via the lm_head projection; earlier layers retain richer semantic structure for similarity matching.
- **Response**: per-position logit biases [(token_id, boost)] — one pair per answer token.
Both are sub-symbolic. The trigger is an opaque high-dimensional vector; the response is a list of (integer, float) pairs. The reflex bank resists inspection without the backbone that produced it.