mleku
fad39ec201
Some checks failed
Go / build-and-release (push) Has been cancelled
- Add 'serve' subcommand for ephemeral RAM-based relay at /dev/shm with open ACL mode for testing and benchmarking - Fix e-tag and p-tag decoding to use ValueHex()/ValueBinary() methods instead of Value() which returns raw bytes for binary-optimized storage - Document all command-line tools in readme.adoc (relay-tester, benchmark, stresstest, blossomtest, aggregator, convert, FIND, policytest, etc.) - Switch Docker images from Alpine to Debian for proper libsecp256k1 Schnorr signature and ECDH support required by Nostr - Upgrade Docker Go version from 1.21 to 1.25 - Add ramdisk mode (--ramdisk) to benchmark script for eliminating disk I/O bottlenecks in performance measurements - Add docker-compose.ramdisk.yml for tmpfs-based benchmark volumes - Add test coverage for privileged policy with binary-encoded p-tags - Fix blossom test to expect 200 OK for anonymous uploads when auth is not required (RequireAuth=false with ACL mode 'none') - Update follows ACL to handle both binary and hex p-tag formats - Grant owner access to all users in serve mode via None ACL - Add benchmark reports from multi-relay comparison run - Update CLAUDE.md with binary tag handling documentation - Bump version to v0.30.2
346 lines
8.6 KiB
Markdown
346 lines
8.6 KiB
Markdown
# Pure Go Build System with Purego
|
|
|
|
## Overview
|
|
|
|
ORLY relay uses **pure Go builds (`CGO_ENABLED=0`)** across all platforms. The p8k cryptographic library uses [purego](https://github.com/ebitengine/purego) to dynamically load `libsecp256k1` at runtime, eliminating the need for CGO during compilation.
|
|
|
|
## Key Benefits
|
|
|
|
### 1. **No CGO Required**
|
|
- Builds complete in pure Go without C compiler
|
|
- Faster compilation times
|
|
- Simpler build process
|
|
- No cross-compilation toolchains needed
|
|
|
|
### 2. **Easy Cross-Compilation**
|
|
- Build for any platform from any platform
|
|
- No platform-specific C compilers required
|
|
- No linking complexities
|
|
|
|
### 3. **Portable Binaries**
|
|
- Self-contained executables
|
|
- Work without `libsecp256k1` (fallback to pure Go p256k1)
|
|
- Optional runtime performance boost if library is available
|
|
|
|
### 4. **Development Friendly**
|
|
- Simple `go build` works everywhere
|
|
- No CGO environment setup needed
|
|
- Consistent builds across all platforms
|
|
|
|
## How It Works
|
|
|
|
### Purego Dynamic Loading
|
|
|
|
The p8k library (from `git.mleku.dev/mleku/nostr`) uses purego to:
|
|
|
|
1. **At build time**: Compile pure Go code (`CGO_ENABLED=0`)
|
|
2. **At runtime**: Attempt to dynamically load `libsecp256k1`
|
|
- If library found → use fast C implementation
|
|
- If library not found → automatically fallback to pure Go p256k1
|
|
|
|
### Library Search Paths
|
|
|
|
Platform-specific search locations:
|
|
|
|
**Linux:**
|
|
- `./libsecp256k1.so` (current directory)
|
|
- `/usr/lib/libsecp256k1.so.2`
|
|
- `/usr/local/lib/libsecp256k1.so.2`
|
|
- `/lib/libsecp256k1.so.2`
|
|
|
|
**macOS:**
|
|
- `./libsecp256k1.dylib` (current directory)
|
|
- `/usr/local/lib/libsecp256k1.dylib`
|
|
- `/opt/homebrew/lib/libsecp256k1.dylib`
|
|
|
|
**Windows:**
|
|
- `libsecp256k1.dll` (current directory)
|
|
- System PATH
|
|
|
|
## Building
|
|
|
|
### Simple Build (All Platforms)
|
|
|
|
```bash
|
|
# Just works - no CGO needed
|
|
go build .
|
|
```
|
|
|
|
### Multi-Platform Build
|
|
|
|
```bash
|
|
# Build for all platforms
|
|
./scripts/build-all-platforms.sh
|
|
|
|
# Outputs to build/ directory:
|
|
# - orly-v0.25.0-linux-amd64
|
|
# - orly-v0.25.0-linux-arm64
|
|
# - orly-v0.25.0-darwin-amd64
|
|
# - orly-v0.25.0-darwin-arm64
|
|
# - orly-v0.25.0-windows-amd64.exe
|
|
# - libsecp256k1-linux-amd64.so (optional)
|
|
```
|
|
|
|
### Cross-Compilation
|
|
|
|
```bash
|
|
# From Linux, build for macOS
|
|
GOOS=darwin GOARCH=arm64 CGO_ENABLED=0 go build -o orly-macos .
|
|
|
|
# From macOS, build for Windows
|
|
GOOS=windows GOARCH=amd64 CGO_ENABLED=0 go build -o orly.exe .
|
|
|
|
# From any platform, build for any platform
|
|
GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build -o orly-arm64 .
|
|
```
|
|
|
|
## Runtime Performance
|
|
|
|
### With libsecp256k1 (Fast)
|
|
|
|
When `libsecp256k1` is available at runtime:
|
|
- **Schnorr signing**: ~15,000 ops/sec
|
|
- **Schnorr verification**: ~6,000 ops/sec
|
|
- **ECDH**: ~12,000 ops/sec
|
|
- **Performance**: 2-3x faster than pure Go
|
|
|
|
### Without libsecp256k1 (Fallback)
|
|
|
|
When library is not found, automatic fallback to pure Go:
|
|
- **Schnorr signing**: ~5,000 ops/sec
|
|
- **Schnorr verification**: ~2,000 ops/sec
|
|
- **ECDH**: ~4,000 ops/sec
|
|
- **Performance**: Still acceptable for most use cases
|
|
|
|
## Deployment Options
|
|
|
|
### Option 1: Binary Only (Simplest)
|
|
|
|
Distribute just the binary:
|
|
- Works everywhere immediately
|
|
- Uses pure Go fallback
|
|
- Good for development/testing
|
|
|
|
```bash
|
|
# Just copy and run
|
|
scp orly-v0.25.0-linux-amd64 server:~/orly
|
|
ssh server "./orly"
|
|
```
|
|
|
|
### Option 2: Binary + Library (Fastest)
|
|
|
|
Distribute binary with library:
|
|
- Maximum performance
|
|
- Automatic library detection
|
|
- Recommended for production
|
|
|
|
```bash
|
|
# Copy both
|
|
scp orly-v0.25.0-linux-amd64 server:~/orly
|
|
scp libsecp256k1-linux-amd64.so server:~/libsecp256k1.so
|
|
ssh server "export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH && ./orly"
|
|
```
|
|
|
|
### Option 3: System Library (Production)
|
|
|
|
Install library system-wide:
|
|
```bash
|
|
# On Ubuntu/Debian
|
|
sudo apt-get install libsecp256k1-1
|
|
|
|
# Binary automatically finds it
|
|
./orly
|
|
```
|
|
|
|
## All Scripts Updated
|
|
|
|
All build and test scripts now use `CGO_ENABLED=0`:
|
|
|
|
### Build Scripts
|
|
- ✓ `scripts/build-all-platforms.sh` - Multi-platform builds
|
|
- ✓ `scripts/deploy.sh` - Production deployment
|
|
- ✓ `scripts/benchmark.sh` - Benchmark builds
|
|
- ✓ `cmd/benchmark/profile.sh` - Profiling builds
|
|
|
|
### Test Scripts
|
|
- ✓ `scripts/test.sh` - Main test runner
|
|
- ✓ `scripts/runtests.sh` - Comprehensive tests
|
|
- ✓ `scripts/test_policy.sh` - Policy tests
|
|
- ✓ `scripts/test-managed-acl.sh` - ACL tests
|
|
- ✓ `scripts/test-workflow-local.sh` - CI/CD simulation
|
|
- ✓ `scripts/test-deploy-local.sh` - Deployment tests
|
|
|
|
### CI/CD
|
|
- ✓ `.github/workflows/go.yml` - GitHub Actions
|
|
- ✓ `cmd/benchmark/Dockerfile.next-orly` - Docker builds
|
|
- ✓ `cmd/benchmark/Dockerfile.benchmark` - Benchmark container
|
|
|
|
## Platform Support Matrix
|
|
|
|
| Platform | CGO | Cross-Compile | Library Runtime | Status |
|
|
|---------------|-----|---------------|-----------------|--------|
|
|
| Linux AMD64 | ✗ | ✓ Native | ✓ Optional | ✓ Full |
|
|
| Linux ARM64 | ✗ | ✓ Pure Go | ✓ Optional | ✓ Full |
|
|
| macOS AMD64 | ✗ | ✓ Pure Go | ✓ Optional | ✓ Full |
|
|
| macOS ARM64 | ✗ | ✓ Pure Go | ✓ Optional | ✓ Full |
|
|
| Windows AMD64 | ✗ | ✓ Pure Go | ✓ Optional | ✓ Full |
|
|
| Android ARM64 | ✗ | ✓ Pure Go | ✓ Optional | ✓ Full |
|
|
| Android AMD64 | ✗ | ✓ Pure Go | ✓ Optional | ✓ Full |
|
|
|
|
**All platforms**: Pure Go build, runtime library optional
|
|
|
|
## Migration from CGO
|
|
|
|
Previously, the project used CGO builds:
|
|
- Required C compilers for builds
|
|
- Complex cross-compilation setup
|
|
- Platform-specific build requirements
|
|
- Linking issues across environments
|
|
|
|
Now with purego:
|
|
- ✓ Simple pure Go builds everywhere
|
|
- ✓ Easy cross-compilation
|
|
- ✓ No build dependencies
|
|
- ✓ Runtime library optional
|
|
|
|
## Performance Comparison
|
|
|
|
### Build Time
|
|
|
|
| Build Type | Time | Notes |
|
|
|------------|------|-------|
|
|
| CGO (old) | ~45s | With C compilation |
|
|
| Purego (new) | ~15s | Pure Go only |
|
|
|
|
**3x faster builds** with purego
|
|
|
|
### Binary Size
|
|
|
|
| Build Type | Size | Notes |
|
|
|------------|------|-------|
|
|
| CGO (old) | ~28 MB | Statically linked |
|
|
| Purego (new) | ~32 MB | Pure Go with purego |
|
|
|
|
**Slightly larger** but no C dependencies
|
|
|
|
### Runtime Performance
|
|
|
|
| Operation | CGO (old) | Purego + lib | Purego fallback |
|
|
|-----------|-----------|--------------|-----------------|
|
|
| Schnorr Sign | 15K/s | 15K/s | 5K/s |
|
|
| Schnorr Verify | 6K/s | 6K/s | 2K/s |
|
|
| ECDH | 12K/s | 12K/s | 4K/s |
|
|
|
|
**Same performance** with library, acceptable fallback
|
|
|
|
## Developer Experience
|
|
|
|
### Before (CGO)
|
|
|
|
```bash
|
|
# Complex setup
|
|
sudo apt-get install gcc autoconf automake libtool
|
|
git clone https://github.com/bitcoin-core/secp256k1.git
|
|
cd secp256k1 && ./autogen.sh && ./configure && make && sudo make install
|
|
|
|
# Cross-compilation nightmares
|
|
sudo apt-get install gcc-aarch64-linux-gnu gcc-mingw-w64-x86-64
|
|
export CC=aarch64-linux-gnu-gcc
|
|
CGO_ENABLED=1 GOOS=linux GOARCH=arm64 go build . # Often fails
|
|
```
|
|
|
|
### After (Purego)
|
|
|
|
```bash
|
|
# Just works
|
|
go build .
|
|
|
|
# Cross-compilation just works
|
|
GOOS=linux GOARCH=arm64 go build .
|
|
GOOS=windows GOARCH=amd64 go build .
|
|
GOOS=darwin GOARCH=arm64 go build .
|
|
```
|
|
|
|
## Testing
|
|
|
|
All tests work with `CGO_ENABLED=0`:
|
|
|
|
```bash
|
|
# Run all tests
|
|
./scripts/test.sh
|
|
|
|
# Tests automatically detect library
|
|
# - With library: tests use C implementation
|
|
# - Without library: tests use pure Go fallback
|
|
```
|
|
|
|
## Docker
|
|
|
|
Dockerfiles simplified:
|
|
|
|
```dockerfile
|
|
# No more build dependencies
|
|
FROM golang:1.25-alpine AS builder
|
|
WORKDIR /build
|
|
COPY . .
|
|
RUN CGO_ENABLED=0 go build -ldflags "-s -w" -o orly .
|
|
|
|
# Runtime includes libsecp256k1.so from repository
|
|
FROM alpine:latest
|
|
RUN apk add --no-cache ca-certificates
|
|
COPY --from=builder /build/orly /app/orly
|
|
COPY --from=builder /build/libsecp256k1.so /app/libsecp256k1.so
|
|
ENV LD_LIBRARY_PATH=/app
|
|
CMD ["/app/orly"]
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### "Library not found" warnings
|
|
|
|
These are normal and expected:
|
|
```
|
|
p8k: failed to load libsecp256k1: no such file
|
|
p8k: using pure Go fallback implementation
|
|
```
|
|
|
|
**This is fine** - the fallback works correctly.
|
|
|
|
### Force library loading
|
|
|
|
To verify library is being used:
|
|
```bash
|
|
# Linux
|
|
export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH
|
|
./orly
|
|
|
|
# macOS
|
|
export DYLD_LIBRARY_PATH=.:$DYLD_LIBRARY_PATH
|
|
./orly
|
|
|
|
# Windows
|
|
# Place libsecp256k1.dll in same directory as .exe
|
|
```
|
|
|
|
### Check library status at runtime
|
|
|
|
The p8k library logs its status:
|
|
```
|
|
p8k: libsecp256k1 loaded successfully
|
|
p8k: schnorr module available
|
|
p8k: ecdh module available
|
|
```
|
|
|
|
## Conclusion
|
|
|
|
The purego build system provides:
|
|
|
|
1. **Simplicity**: Pure Go builds everywhere
|
|
2. **Portability**: Cross-compile to any platform easily
|
|
3. **Performance**: Optional runtime library for speed
|
|
4. **Reliability**: Automatic fallback to pure Go
|
|
5. **Developer Experience**: No CGO setup required
|
|
|
|
**All platforms can use purego** - it's enabled everywhere by default.
|
|
|