benmaster82-Kwipu

repository

loading code, commits, and activity

repositories

loading repo index

#1	# Kwipu
#2
#3	[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
#4	[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
#5	[![Ollama](https://img.shields.io/badge/LLM-Ollama-orange.svg)](https://ollama.ai/)
#6	[![LlamaIndex](https://img.shields.io/badge/framework-LlamaIndex-purple.svg)](https://www.llamaindex.ai/)
#7	[![Obsidian Compatible](https://img.shields.io/badge/Obsidian-compatible-7C3AED.svg)](https://obsidian.md/)
#8	[![MCP Server](https://img.shields.io/badge/MCP-compatible-blue.svg)](https://modelcontextprotocol.io/)
#9
#10	[中文文档](README.zh-CN.md)
#11
#12	A local Graph RAG system that turns your markdown notes into a queryable knowledge graph. Ask questions in natural language and get answers that connect information across multiple files.
#13
#14	Built for [Obsidian](https://obsidian.md/) vaults but works with any folder of markdown files.
#15
#16	![Kwipu in action](img/screen.png)
#17
#18	![Query response example](img/screen_2.png)
#19
#20	## What's New
#21
#22	- MCP Server - use Kwipu as a tool inside Claude Desktop, Cursor, Windsurf, or any MCP-compatible agent. All processing runs locally via Ollama.
#23	- Incremental updates - editing a note no longer rebuilds the entire graph. Modified files are updated in-place in seconds.
#24	- CLI model override - switch LLM or embedding model without editing code: `--llm-model`, `--embed-model`
#25	- Startup validation - checks that Ollama is running and models are available before starting. Clear error messages with suggested commands.
#26	- Storage integrity - detects embedding model mismatches to prevent silent corruption of the graph.
#27	- Refined prompt - rewritten anti-hallucination rules reduce false information while keeping answers complete.
#28
#29	## Features
#30
#31	- Property Graph Index - builds a knowledge graph from your notes using LLM-extracted relationships
#32	- Obsidian-native - automatically parses `[[wikilinks]]` and YAML frontmatter into structured graph triples
#33	- Multilingual - supports Italian, English, French, German, Spanish, Portuguese (auto-detected)
#34	- Hybrid retrieval - combines 4 retrieval strategies:
#35	- LLM synonym expansion (optional, disable with `--fast`)
#36	- Vector similarity search
#37	- BM25 keyword scoring
#38	- Temporal/metadata matching
#39	- Real-time sync - watches your folder for changes and updates the graph incrementally
#40	- Incremental updates - modified files are updated in-place (delete + re-insert) without rebuilding the entire graph
#41	- Anti-hallucination prompt - strict instructions to cite sources and avoid inventing facts
#42	- Fully local - runs on Ollama, no data leaves your machine
#43	- Startup checks - verifies Ollama is running and models are available before starting
#44	- Storage validation - detects embedding model mismatches to prevent silent corruption
#45	- CLI model override - switch models without editing code via `--llm-model` and `--embed-model`
#46
#47	## Requirements
#48
#49	- Python 3.11+
#50	- [Ollama](https://ollama.ai/) running locally
#51	- An LLM model (e.g. `llama3.1:8b`, `qwen2.5:7b`, `mistral:7b`)
#52	- An embedding model (default: `nomic-embed-text`)
#53
#54	## Setup
#55
#56	```bash
#57	# Install dependencies
#58	pip install -r requirements.txt
#59
#60	# Pull models in Ollama
#61	ollama pull llama3.1:8b
#62	ollama pull nomic-embed-text
#63	```
#64
#65	## MCP Server (Claude Desktop, Cursor, Windsurf)
#66
#67	Kwipu can run as an MCP server, allowing AI agents to query your knowledge graph directly. All processing happens locally via Ollama - the agent only sends the question and receives the answer.
#68
#69	Add to your `claude_desktop_config.json` (or equivalent MCP config):
#70
#71	```json
#72	{
#73	"mcpServers": {
#74	"kwipu": {
#75	"command": "C:/path/to/python.exe",
#76	"args": ["C:/path/to/kwipu_mcp_server.py"]
#77	}
#78	}
#79	}
#80	```
#81
#82	Replace paths with your actual Python and project locations. Requires Ollama running with the configured model.
#83
#84	## Usage
#85
#86	```bash
#87	# Full mode (default, all retrievers)
#88	python geode_graph.py
#89
#90	# Fast mode (skips LLM synonym retriever, faster queries)
#91	python geode_graph.py --fast
#92
#93	# Override models from CLI (no need to edit the file)
#94	python geode_graph.py --llm-model qwen2.5:7b --embed-model nomic-embed-text
#95
#96	# Build with cloud model, then query with local model
#97	python geode_graph.py --llm-model gpt-oss:20b-cloud
#98	# After build completes, restart with:
#99	python geode_graph.py --llm-model qwen2.5:3b --fast
#100	```
#101
#102	Place your markdown files in `./knowledge_base/` (or change `KNOWLEDGE_DIR` in the config). The system builds the graph on first run and watches for changes.
#103
#104	## How It Works
#105
#106	```
#107	Your Notes (.md)
#108	│
#109	▼
#110	┌─────────────────────┐
#111	│ Pre-processing │ ← Extracts [[wikilinks]], YAML frontmatter
#112	│ (lang_config.py) │ ← Infers relations from context (multilingual)
#113	└─────────┬───────────┘
#114	│
#115	▼
#116	┌─────────────────────┐
#117	│ LLM Extraction │ ← Extracts additional entity-relation triples
#118	│ (SimpleLLMPath) │
#119	└─────────┬───────────┘
#120	│
#121	▼
#122	┌─────────────────────┐
#123	│ Property Graph │ ← Merges structural + LLM triples
#124	│ Index │ ← Persisted to disk (storage_graph/)
#125	└─────────┬───────────┘
#126	│
#127	▼
#128	┌─────────────────────┐
#129	│ Hybrid Retrieval │ ← Synonym + Vector + BM25 + Temporal
#130	└─────────┬───────────┘
#131	│
#132	▼
#133	┌─────────────────────┐
#134	│ LLM Response │ ← Generates answer from retrieved context
#135	└─────────────────────┘
#136	```
#137
#138	## Project Structure
#139
#140	```
#141	├── geode_graph.py # Main application (terminal interface)
#142	├── kwipu_mcp_server.py # MCP server for AI agent integration
#143	├── lang_config.py # Multilingual configuration (stopwords, patterns, relations)
#144	├── requirements.txt # Python dependencies
#145	├── knowledge_base/ # Your notes go here
#146	│ └── examples/ # Example notes to get started
#147	└── storage_graph/ # Generated graph index (auto-created, gitignored)
#148	```
#149
#150	## Pointing to an Obsidian Vault
#151
#152	Change `KNOWLEDGE_DIR` to your vault path:
#153
#154	```python
#155	KNOWLEDGE_DIR = "C:/Users/YourName/Documents/MyVault"
#156	```
#157
#158	The system reads files without modifying them. It ignores `.obsidian/` configuration files automatically.
#159
#160	## Model Recommendations
#161
#162	\| Model \| RAM (Q4) \| Quality \| Speed per query (CPU) \| Speed per query (GPU) \|
#163	\|-------\|----------\|---------\|----------------------\|----------------------\|
#164	\| 1B \| ~2 GB \| Basic \| ~8s \| ~2s \|
#165	\| 3B \| ~3 GB \| Good \| ~30-60s \| ~5-8s \|
#166	\| 7-8B \| ~5-6 GB \| Great \| ~2-5 min \| ~15-25s \|
#167	\| 20B \| ~12 GB \| Best \| Not practical \| ~15s \|
#168
#169	For serious use, 7B+ with a GPU is the sweet spot. The 3B is a good compromise for CPU-only setups.
#170
#171	## Build Time Estimates
#172
#173	First-time graph construction requires an LLM call for each document chunk. Subsequent runs load the graph from disk instantly. Times can vary ±2x depending on note length and model.
#174
#175	\| Notes \| GPU (7B) \| CPU (7B) \| CPU (3B) \|
#176	\|-------\|----------\|----------\|----------\|
#177	\| 5 \| ~2 min \| ~8 min \| ~4 min \|
#178	\| 20 \| ~8 min \| ~30 min \| ~15 min \|
#179	\| 50 \| ~20 min \| ~1.5 hrs \| ~40 min \|
#180	\| 100 \| ~40 min \| ~3 hrs \| ~1.5 hrs \|
#181	\| 500+ \| ~3 hrs \| Not recommended \| Not recommended \|
#182
#183	Adding a single new file is incremental (~20-60s) and does not rebuild the full graph. Modifying an existing file also uses incremental update (delete + re-insert). Only file deletion triggers a full rebuild.
#184
#185	## Resource Usage
#186
#187	\| Component \| RAM \| Notes \|
#188	\|-----------\|-----\|-------\|
#189	\| Ollama (LLM) \| 2-14 GB \| Depends on model size and quantization \|
#190	\| Ollama (embeddings) \| ~300 MB \| nomic-embed-text \|
#191	\| Kwipu (indexing) \| 0.5-4 GB \| Depends on number of notes \|
#192	\| Kwipu (queries) \| 200-500 MB \| After graph is built \|
#193	\| Total (7B Q4) \| ~8-12 GB \| Recommended minimum: 16 GB system RAM \|
#194
#195	## Tip: Use Cloud Models for Graph Building
#196
#197	If your hardware is limited, you can use a powerful cloud model via Ollama to build the graph once, then switch to a smaller local model for daily queries. The graph is persisted to disk, so you only need the large model during construction.
#198
#199	```bash
#200	# Step 1: Build the graph with a cloud model (one-time, high quality extraction)
#201	python geode_graph.py --llm-model gpt-oss:20b-cloud
#202	# Wait for "Graph built and saved successfully", then exit.
#203
#204	# Step 2: Switch to a small local model for queries (fast, low resource)
#205	python geode_graph.py --llm-model qwen2.5:3b --fast
#206	```
#207
#208	This gives you the best of both worlds: a high-quality graph built by a 20B+ model, with fast and lightweight queries on a 3B model. The graph structure (entities, relations, triples) doesn't change when you switch models - only the response generation uses the smaller model.
#209
#210	> Note: If you change the embedding model (`--embed-model`), you must delete `storage_graph/` and rebuild. Kwipu will detect the mismatch and warn you.
#211
#212	## Contributing
#213
#214	Contributions are welcome. Here's how to get started:
#215
#216	```bash
#217	# Clone and setup
#218	git clone https://github.com/benmaster82/Kwipu.git
#219	cd Kwipu
#220	pip install -r requirements.txt
#221	```
#222
#223	Areas where help is needed:
#224
#225	- CJK language support - Looking for help adding Chinese, Japanese, and Korean. Requires word segmentation and language-specific patterns. See [open issues](https://github.com/benmaster82/Kwipu/issues).
#226	- Retriever attribution logging - Log which retriever (vector, BM25, temporal, synonym) contributed context for each answer
#227	- Evaluation set - Build a categorized test suite (exact-source, multi-hop, temporal, negative questions)
#228	- Provenance inspector - Surface the chain: answer claim -> cited note -> extracted entity/edge
#229	- Telegram bot integration - Query the knowledge base remotely via Telegram
#230	- Performance - Incremental update on file modification (currently triggers full rebuild)
#231
#232	Guidelines:
#233
#234	- Keep it simple. This is a local-first tool, not an enterprise platform.
#235	- Test with real Obsidian vaults when possible.
#236	- One feature per PR. Small PRs get reviewed faster.
#237	- English for code, comments, and commit messages.
#238
#239	Open an issue first if you want to discuss an approach before coding.
#240
#241	## Roadmap
#242
#243	- Telegram Bot - Query your Obsidian vault or knowledge base from anywhere via Telegram
#244
#245	## License
#246
#247	MIT
#248

z6MkqRzA/benmaster82-Kwipu