Fix directory spider tag loss: size limits and validation
Some checks failed
Go / build-and-release (push) Has been cancelled

- Increase WebSocket message size limit from 500KB to 10MB to prevent
  truncation of large kind 3 follow list events (8000+ follows)
- Add validation in SaveEvent to reject kind 3 events without p tags
  before storage, preventing malformed events from buggy relays
- Implement CleanupKind3WithoutPTags() to remove existing malformed
  kind 3 events at startup
- Add enhanced logging showing tag count and event ID when rejecting
  invalid kind 3 events for better observability
- Create round-trip test proving binary tag encoding preserves p tags
  correctly through JSON→binary→JSON cycle
- Root cause: 500KB limit was truncating large follow lists during
  WebSocket receive, causing tags to be lost or incomplete
- Three-layer defense: prevent at gate (size), validate (save time),
  and cleanup (startup)

Files modified:
- app/handle-websocket.go: Increase DefaultMaxMessageSize to 10MB
- pkg/database/save-event.go: Add kind 3 validation with logging
- pkg/database/cleanup-kind3.go: New cleanup function
- pkg/database/cleanup-kind3_test.go: Round-trip test
- app/main.go: Invoke cleanup at startup
This commit is contained in:
2025-11-27 13:49:33 +00:00
parent 7fedcd24d3
commit 48b0b6984c
7 changed files with 236 additions and 2 deletions

View File

@@ -21,7 +21,10 @@ const (
DefaultPongWait = 60 * time.Second
DefaultPingWait = DefaultPongWait / 2
DefaultWriteTimeout = 3 * time.Second
DefaultMaxMessageSize = 512000 // Match khatru's MaxMessageSize
// DefaultMaxMessageSize is the maximum message size for WebSocket connections
// Increased from 512KB to 10MB to support large kind 3 follow lists (10k+ follows)
// and other large events without truncation
DefaultMaxMessageSize = 10 * 1024 * 1024 // 10MB
// ClientMessageSizeLimit is the maximum message size that clients can handle
// This is set to 100MB to allow large messages
ClientMessageSizeLimit = 100 * 1024 * 1024 // 100MB

View File

@@ -92,6 +92,13 @@ func Run(
// Continue anyway - follows can be loaded when admins update their follow lists
}
// Cleanup any kind 3 events that lost their p tags (only for Badger backend)
if badgerDB, ok := db.(*database.D); ok {
if err := badgerDB.CleanupKind3WithoutPTags(ctx); chk.E(err) {
log.E.F("failed to cleanup kind 3 events: %v", err)
}
}
// Initialize spider manager based on mode (only for Badger backend)
if badgerDB, ok := db.(*database.D); ok && cfg.SpiderMode != "none" {
if l.spiderManager, err = spider.New(ctx, badgerDB, l.publishers, cfg.SpiderMode); chk.E(err) {