Add memory optimization improvements for reduced GC pressure (v0.36.16)
Some checks failed
Go / build-and-release (push) Has been cancelled

- Add buffer pool (pkg/database/bufpool) with SmallPool (64B) and MediumPool (1KB)
  for reusing bytes.Buffer instances on hot paths
- Fix escape analysis in index types (uint40, letter, word) by using fixed-size
  arrays instead of make() calls that escape to heap
- Add handler concurrency limiter (ORLY_MAX_HANDLERS_PER_CONN, default 100) to
  prevent unbounded goroutine growth under WebSocket load
- Add pre-allocation hints to Uint40s.Union/Intersection/Difference methods
- Update compact_event.go, save-event.go, serial_cache.go, and
  get-indexes-for-event.go to use pooled buffers

Files modified:
- app/config/config.go: Add MaxHandlersPerConnection config
- app/handle-websocket.go: Initialize handler semaphore
- app/listener.go: Add semaphore acquire/release in messageProcessor
- pkg/database/bufpool/pool.go: New buffer pool package
- pkg/database/compact_event.go: Use buffer pool, fix escape analysis
- pkg/database/get-indexes-for-event.go: Reuse single buffer for all indexes
- pkg/database/indexes/types/letter.go: Fixed array in UnmarshalRead
- pkg/database/indexes/types/uint40.go: Fixed arrays, pre-allocation hints
- pkg/database/indexes/types/word.go: Fixed array in UnmarshalRead
- pkg/database/save-event.go: Use buffer pool for key encoding
- pkg/database/serial_cache.go: Use buffer pool for lookups

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
2025-12-25 06:03:53 +01:00
parent 24383ef1f4
commit eddd05eabf
13 changed files with 204 additions and 70 deletions

View File

@@ -86,6 +86,12 @@ whitelist:
})
defer conn.Close()
// Determine handler semaphore size from config
handlerSemSize := s.Config.MaxHandlersPerConnection
if handlerSemSize <= 0 {
handlerSemSize = 100 // Default if not configured
}
listener := &Listener{
ctx: ctx,
cancel: cancel,
@@ -98,6 +104,7 @@ whitelist:
writeDone: make(chan struct{}),
messageQueue: make(chan messageRequest, 100), // Buffered channel for message processing
processingDone: make(chan struct{}),
handlerSem: make(chan struct{}, handlerSemSize), // Limits concurrent handlers
subscriptions: make(map[string]context.CancelFunc),
}