Intelligent Extraction
Pensyve's Tier 2 extraction pipeline uses LLM inference to automatically extract structured knowledge from unstructured text — facts, entities, relationships, and contradictions.
Two Tiers of Extraction
Tier 1: Pattern Extraction (always on)
The Rust-native pattern extractor runs on every remember() call at zero additional cost:
- Emails —
user@example.com - Dates —
2026-03-27 - URLs —
https://...
Extracted patterns are stored as metadata alongside the memory, making them searchable without embedding overhead.
Tier 2: LLM Extraction (opt-in)
When enabled, Tier 2 uses an LLM to perform deep extraction:
| Extracted | Description | Example |
|---|---|---|
| Facts | Subject-predicate-object triples | (Alice, prefers, dark mode) |
| Entities | Named people, tools, projects, concepts | Alice, VS Code, Project Mercury |
| Relations | Typed connections between entities | Alice → works-on → Project Mercury |
| Contradictions | Conflicts with existing knowledge | "Alice now prefers light mode" vs. stored "Alice prefers dark mode" |
Extracted facts become semantic memories automatically. Relations are added as edges in the knowledge graph, enabling graph-based retrieval.
Enabling Tier 2
Self-hosted
Set these environment variables:
export PENSYVE_TIER2_ENABLED=true
export PENSYVE_TIER2_MODEL_PATH=/path/to/model.ggufTier 2 uses a local GGUF model for inference — no external API calls.
Pensyve Cloud
Tier 2 extraction is available as a metered operation. Every LLM extraction call is billed at $0.01/op — reflecting the significant compute cost of running LLM inference per memory.
Your first 25 extraction operations each month are free, so you can experiment before committing.
Contradiction Detection
One of the most powerful features of Tier 2 is contradiction detection. When a new fact contradicts an existing memory, Pensyve:
- Flags the contradiction in the
recall()response - Creates a
Supersedesedge in the knowledge graph - Reduces the confidence of the older memory
This means your agents' knowledge stays current without manual intervention. When Alice says "I switched to VS Code" and you have a stored memory "Alice uses Vim", the system detects the conflict and updates accordingly.
How It Integrates
Extraction runs inline during remember() and at episode close (episode.end()). The flow:
- Input text → Tier 1 pattern extraction (always)
- If Tier 2 enabled → LLM extracts facts, entities, relations
- Deduplication — extracted facts are compared against existing memories
- Storage — new facts become semantic memories; relations become graph edges
- Contradiction check — conflicts with existing facts are flagged
Tier 2 extraction is the most expensive operation in Pensyve because it runs LLM inference for each call. Use it strategically — on high-value conversations, onboarding flows, or summarization steps — rather than on every message.