Commit Graph

3 Commits

Author SHA1 Message Date
woikos
0dac41e35e Add documentation and improve BBolt import memory efficiency (v0.48.8)
Some checks failed
Go / build-and-release (push) Has been cancelled
- Add README.md table of contents for easier navigation
- Add Curation ACL documentation section to README.md
- Create detailed Curation Mode Guide (docs/CURATION_MODE_GUIDE.md)
- Fix OOM during BBolt index building by closing temp file before build
- Add GC calls before index building to reclaim batch buffer memory
- Improve import-export.go with processJSONLEventsReturningCount
- Add policy-aware import path for sync operations

Files modified:
- README.md: Added TOC and curation ACL documentation
- docs/CURATION_MODE_GUIDE.md: New comprehensive curation mode guide
- pkg/bbolt/import-export.go: Memory-safe import with deferred cleanup
- pkg/bbolt/import-minimal.go: Added GC before index build
- pkg/version/version: Bump to v0.48.8

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 15:37:06 +01:00
woikos
2480be3a73 Fix OOM in BuildIndexes by processing in chunks (v0.48.6)
- Process events in 200k chunks instead of loading all at once
- Write indexes to disk after each chunk, then free memory
- Call debug.FreeOSMemory() between chunks to release memory to OS
- Memory usage now ~150-200MB per chunk instead of 5GB+

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 09:10:50 +01:00
woikos
d363f5da04 Implement BBolt ImportEventsFromReader for migration (v0.48.1)
Some checks failed
Go / build-and-release (push) Has been cancelled
- Add import-export.go with full JSONL import support for bbolt
- Remove Import/Export/ImportEventsFromReader stubs from stubs.go
- Includes batched write flush after import completion
- Progress logging every 5 seconds during import

Files modified:
- pkg/bbolt/import-export.go: New file with import functionality
- pkg/bbolt/stubs.go: Remove implemented stubs
- pkg/version/version: v0.48.0 -> v0.48.1

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 06:57:06 +01:00