Add documentation and improve BBolt import memory efficiency (v0.48.8)
Some checks failed
Go / build-and-release (push) Has been cancelled
Some checks failed
Go / build-and-release (push) Has been cancelled
- Add README.md table of contents for easier navigation - Add Curation ACL documentation section to README.md - Create detailed Curation Mode Guide (docs/CURATION_MODE_GUIDE.md) - Fix OOM during BBolt index building by closing temp file before build - Add GC calls before index building to reclaim batch buffer memory - Improve import-export.go with processJSONLEventsReturningCount - Add policy-aware import path for sync operations Files modified: - README.md: Added TOC and curation ACL documentation - docs/CURATION_MODE_GUIDE.md: New comprehensive curation mode guide - pkg/bbolt/import-export.go: Memory-safe import with deferred cleanup - pkg/bbolt/import-minimal.go: Added GC before index build - pkg/version/version: Bump to v0.48.8 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -62,9 +62,12 @@ func (b *B) BuildIndexes(ctx context.Context) error {
|
||||
log.I.F("bbolt: starting index build...")
|
||||
startTime := time.Now()
|
||||
|
||||
// Process in chunks to avoid OOM
|
||||
// With ~15 indexes per event and ~50 bytes per key, 200k events = ~150MB
|
||||
const chunkSize = 200000
|
||||
// Force GC before starting to reclaim batch buffer memory
|
||||
debug.FreeOSMemory()
|
||||
|
||||
// Process in small chunks to avoid OOM on memory-constrained systems
|
||||
// With ~15 indexes per event and ~50 bytes per key, 50k events = ~37.5MB per chunk
|
||||
const chunkSize = 50000
|
||||
|
||||
var totalEvents int
|
||||
var lastSerial uint64 = 0
|
||||
|
||||
Reference in New Issue
Block a user