Fix directory spider tag loss: size limits and validation
Some checks failed
Go / build-and-release (push) Has been cancelled
Some checks failed
Go / build-and-release (push) Has been cancelled
- Increase WebSocket message size limit from 500KB to 10MB to prevent truncation of large kind 3 follow list events (8000+ follows) - Add validation in SaveEvent to reject kind 3 events without p tags before storage, preventing malformed events from buggy relays - Implement CleanupKind3WithoutPTags() to remove existing malformed kind 3 events at startup - Add enhanced logging showing tag count and event ID when rejecting invalid kind 3 events for better observability - Create round-trip test proving binary tag encoding preserves p tags correctly through JSON→binary→JSON cycle - Root cause: 500KB limit was truncating large follow lists during WebSocket receive, causing tags to be lost or incomplete - Three-layer defense: prevent at gate (size), validate (save time), and cleanup (startup) Files modified: - app/handle-websocket.go: Increase DefaultMaxMessageSize to 10MB - pkg/database/save-event.go: Add kind 3 validation with logging - pkg/database/cleanup-kind3.go: New cleanup function - pkg/database/cleanup-kind3_test.go: Round-trip test - app/main.go: Invoke cleanup at startup
This commit is contained in:
@@ -21,7 +21,10 @@ const (
|
||||
DefaultPongWait = 60 * time.Second
|
||||
DefaultPingWait = DefaultPongWait / 2
|
||||
DefaultWriteTimeout = 3 * time.Second
|
||||
DefaultMaxMessageSize = 512000 // Match khatru's MaxMessageSize
|
||||
// DefaultMaxMessageSize is the maximum message size for WebSocket connections
|
||||
// Increased from 512KB to 10MB to support large kind 3 follow lists (10k+ follows)
|
||||
// and other large events without truncation
|
||||
DefaultMaxMessageSize = 10 * 1024 * 1024 // 10MB
|
||||
// ClientMessageSizeLimit is the maximum message size that clients can handle
|
||||
// This is set to 100MB to allow large messages
|
||||
ClientMessageSizeLimit = 100 * 1024 * 1024 // 100MB
|
||||
|
||||
@@ -92,6 +92,13 @@ func Run(
|
||||
// Continue anyway - follows can be loaded when admins update their follow lists
|
||||
}
|
||||
|
||||
// Cleanup any kind 3 events that lost their p tags (only for Badger backend)
|
||||
if badgerDB, ok := db.(*database.D); ok {
|
||||
if err := badgerDB.CleanupKind3WithoutPTags(ctx); chk.E(err) {
|
||||
log.E.F("failed to cleanup kind 3 events: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Initialize spider manager based on mode (only for Badger backend)
|
||||
if badgerDB, ok := db.(*database.D); ok && cfg.SpiderMode != "none" {
|
||||
if l.spiderManager, err = spider.New(ctx, badgerDB, l.publishers, cfg.SpiderMode); chk.E(err) {
|
||||
|
||||
Reference in New Issue
Block a user