next.orly.dev/pkg/database/PUBKEY_GRAPH.md

# Pubkey Graph System

## Overview

The pubkey graph system provides efficient social graph queries by creating bidirectional, direction-aware edges between events and pubkeys in the ORLY relay.

## Architecture

### 1. Pubkey Serial Assignment

**Purpose**: Compress 32-byte pubkeys to 5-byte serials for space efficiency.

**Tables**:
- `pks|pubkey_hash(8)|serial(5)` - Hash-to-serial lookup (16 bytes)
- `spk|serial(5)` → 32-byte pubkey (value) - Serial-to-pubkey reverse lookup

**Space Savings**: Each graph edge saves 27 bytes per pubkey reference (32 → 5 bytes).

### 2. Graph Edge Storage

**Bidirectional edges with metadata**:

#### EventPubkeyGraph (Forward)
```
epg|event_serial(5)|pubkey_serial(5)|kind(2)|direction(1) = 16 bytes
```

#### PubkeyEventGraph (Reverse)
```
peg|pubkey_serial(5)|kind(2)|direction(1)|event_serial(5) = 16 bytes
```

### 3. Direction Byte

The direction byte distinguishes relationship types:

| Value | Direction | From Event Perspective | From Pubkey Perspective |
|-------|-----------|------------------------|-------------------------|
| `0` | Author | This pubkey is the event author | I am the author of this event |
| `1` | P-Tag Out | Event references this pubkey | *(not used in reverse)* |
| `2` | P-Tag In | *(not used in forward)* | I am referenced by this event |

**Location in keys**:
- **EventPubkeyGraph**: Byte 13 (after 3+5+5)
- **PubkeyEventGraph**: Byte 10 (after 3+5+2)

## Graph Edge Creation

When an event is saved:

1. **Extract pubkeys**:
   - Event author: `ev.Pubkey`
   - P-tags: All `["p", "<hex-pubkey>", ...]` tags

2. **Get or create serials**: Each unique pubkey gets a monotonic 5-byte serial

3. **Create bidirectional edges**:

   For **author** (pubkey = event author):
   ```
   epg|event_serial|author_serial|kind|0  (author edge)
   peg|author_serial|kind|0|event_serial  (is-author edge)
   ```

   For each **p-tag** (referenced pubkey):
   ```
   epg|event_serial|ptag_serial|kind|1    (outbound reference)
   peg|ptag_serial|kind|2|event_serial    (inbound reference)
   ```

## Query Patterns

### Find all events authored by a pubkey
```
Prefix scan: peg|pubkey_serial|*|0|*
Filter: direction == 0 (author)
```

### Find all events mentioning a pubkey (inbound p-tags)
```
Prefix scan: peg|pubkey_serial|*|2|*
Filter: direction == 2 (p-tag inbound)
```

### Find all kind-1 events mentioning a pubkey
```
Prefix scan: peg|pubkey_serial|0x0001|2|*
Exact match: kind == 1, direction == 2
```

### Find all pubkeys referenced by an event (outbound p-tags)
```
Prefix scan: epg|event_serial|*|*|1
Filter: direction == 1 (p-tag outbound)
```

### Find the author of an event
```
Prefix scan: epg|event_serial|*|*|0
Filter: direction == 0 (author)
```

## Implementation Details

### Thread Safety

The `GetOrCreatePubkeySerial` function uses:
1. Read transaction to check for existing serial
2. If not found, get next sequence number
3. Write transaction with double-check to handle race conditions
4. Returns existing serial if another goroutine created it concurrently

### Deduplication

The save-event function deduplicates pubkeys before creating serials:
- Map keyed by hex-encoded pubkey
- Prevents duplicate edges when author is also in p-tags

### Edge Cases

1. **Author in p-tags**: Only creates author edge (direction=0), skips duplicate p-tag edge
2. **Invalid p-tags**: Silently skipped if hex decode fails or length != 32 bytes
3. **No p-tags**: Only author edge is created

## Performance Characteristics

### Space Efficiency

Per event with N unique pubkeys:
- **Old approach** (storing full pubkeys): N × 32 bytes = 32N bytes
- **New approach** (using serials): N × 5 bytes = 5N bytes
- **Savings**: 27N bytes per event (84% reduction)

Example: Event with author + 10 p-tags:
- Old: 11 × 32 = 352 bytes
- New: 11 × 5 = 55 bytes
- **Saved: 297 bytes (84%)**

### Query Performance

1. **Pubkey lookup**: O(1) hash lookup via 8-byte truncated hash
2. **Serial generation**: O(1) atomic increment
3. **Graph queries**: Sequential scan with prefix optimization
4. **Kind filtering**: Built into key ordering, no event decoding needed

## Testing

Comprehensive tests verify:
- ✅ Serial assignment and deduplication
- ✅ Bidirectional graph edge creation
- ✅ Multiple events sharing pubkeys
- ✅ Direction byte correctness
- ✅ Edge cases (invalid pubkeys, non-existent keys)

## Future Query APIs

The graph structure supports efficient queries for:

1. **Social Graph Queries**:
   - Who does Alice follow? (p-tags authored by Alice)
   - Who follows Bob? (p-tags referencing Bob)
   - Common connections between Alice and Bob

2. **Event Discovery**:
   - All replies to Alice's events (kind-1 events with p-tag to Alice)
   - All events Alice has replied to (kind-1 events by Alice with p-tags)
   - Quote reposts, mentions, reactions by event kind

3. **Analytics**:
   - Most-mentioned pubkeys (count p-tag-in edges)
   - Most active authors (count author edges)
   - Interaction patterns by kind

## Migration Notes

This is a **new index** that:
- Runs alongside existing event indexes
- Populated automatically for all new events
- Does NOT require reindexing existing events (yet)
- Can be backfilled via a migration if needed

To backfill existing events, run a migration that:
1. Iterates all events
2. Extracts pubkeys and creates serials
3. Creates graph edges for each event