17 Commits

Author SHA1 Message Date
1c4c4e1d27 go.mod stuff 2025-02-02 14:32:45 -01:06
5cde988f66 add timestamp 2025-02-02 14:30:26 -01:06
d29ea74783 remove all the test logging 2025-02-02 11:31:25 -01:06
76297bf73e add request URL and notes about always signed requests 2025-02-02 11:22:58 -01:06
786cc0108c add http authentication protocol 2025-02-02 11:12:50 -01:06
7eba0a1ddb notes about chanserv/nickserv cache/auth relay types 2025-02-02 09:39:05 -01:06
a3b07cf68a add full subscription option 2025-02-02 09:31:56 -01:06
4983d6095f fixed main readme links to docs 2025-02-02 09:15:37 -01:06
eaefa6c0bc refine documentation to be more clear 2025-02-02 09:11:12 -01:06
0fd7151094 completed tags codec 2025-02-02 09:10:25 -01:06
4c6e7b08ac add content codec 2025-01-31 03:45:06 -01:06
d36bcdb243 switch base64 to raw 2025-01-31 03:01:15 -01:06
ab032cd296 switch to asciidoc and explain relay specialization 2025-01-31 02:20:48 -01:06
0b6772ca83 switch to asciidoc and explain relay specialization 2025-01-31 02:15:54 -01:06
bbf79bb91f describe event delivery format 2025-01-30 19:14:36 -01:06
d7b9415037 small additional notes in spec 2025-01-30 19:06:35 -01:06
80e4c54c08 expand events info and submit/query messages 2025-01-30 18:59:26 -01:06
33 changed files with 1017 additions and 116 deletions

1
.gitignore vendored
View File

@@ -75,6 +75,7 @@ key
!.openapi-generator-ignore
!.gitignore
!*.jsonl
!*.adoc
# ...even if they are in subdirectories
!*/

158
doc/events_queries.adoc Normal file
View File

@@ -0,0 +1,158 @@
= REALY protocol event/query specification
JSON is awful, and space inefficient, and complex to parse due to its intolerance of terminal commas and annoying to work with because of its retarded, multi-standards of string escaping.
Line structured documents are much more readily amenable to human reading and editing, and `\n`/`;`/`:` is more efficient than `","` as an item separator. Data structures can be much more simply expressed in a similar way as how they are in programming languages.
It is one of the guiding principles of the Unix philosophy to keep data in plain text, human readable format wherever possible, forcing the interposition of a parser just for humans to read the data adds extra brittleness to a protocol.
REALY protocol format is extremely simple and should be trivial to parse in any programming language with basic string slicing operators.
---
== Base64 Encoding
To save space and eliminate the need for ugly `=` padding characters, we invoke link:https://datatracker.ietf.org/doc/html/rfc4648#section-3.2[RFC 4648 section 3.2] for the case of using base64 URL encoding without padding because we know the data length. In this case, it is used for IDs and pubkeys (32 bytes payload each, 43 characters base64 raw URL encoded) and signatures (64 bytes payload, 86 characters base64 raw URL encoded) - the further benefit here is the exact same string can be used in HTTP GET parameters `?key=value&...` context. The standard `=` padding would break this usage as well.
For ease of human usage, also, it is recommended when the value is printed in plain text that it be on its own line so triple click catches all of it including the normally word-wise separated `-` hyphen/minus character, as follows:
CF4I5dXYPZ_lu2pYRjey1QMDmgNJEyT-MM8Vvj6EnZM
For those who can't find a "raw" codec for base64, the 32 byte length has 1`=` pad suffix and the 64 byte length has 2: `==` and this can be trimmed off and added back to conform to this requirement. Due to the fact that potentially there can be hundreds if not thousands of these in event content and tag fields the benefit can be quite great, as well as the benefit of being able to use these codes also in URL parameter values.
== Sockets and HTTP
Only subscriptions require server push messaging pattern, thus all other queries in REALY can be done with simple HTTP POST requests.
A relay should respond to a `subscribe` request by upgrading from http to a websocket.
It is unnecessary messages and work to use websockets for queries that match the HTTP request/response pattern, and by only requiring sockets for APIs that actually need server initiated messaging, the complexity of the relay is greatly reduced.
There can be a separate subscription type also, where there is delivering the IDs only, or forwarding the whole event.
=== HTTP Authentication
For the most part, all queries and submissions must be authenticated in order to enable a REALY relay to allow access.
To enable this, a suffix is added to messages with the following format:
`<message payload>\n` // all messages must be terminated with a newline
`<request URL>\n`
`<unix timestamp in decimal ascii>\n`
`<public key of signer>\n`
`<signature>\n`
For simplicity, the signature is on a separate line, just as it is in the event format, this avoids needing to have a separate codec, and for the same reason the timestamp and public key.
For reasons of security, a relay should not allow a time skew in the timestamp of more than 15 seconds.
The signature is upon the Blake 2b message hash of everything up to the semicolon preceding it, and only relates to the HTTP POST payload, not including the header.
Even subscription messages should be signed the same way, to avoid needing a secondary protocol. "open" relays that have no access control (which is retarded, but just to be complete) must still require this authentication message, but simply the client can use one-shot keys to sign with, as it also serves as a HMAC to validate the consistency of the request data, since it is based on the hash.
== Events
The format of events is as follows - the monospace segments are the exact text, including the necessary linebreak characters, the rest is descriptive.
---
`<type name>\n` // can be anything, hierarchic names like note/html note/md are possible, or type.subtype or whatever
`<pubkey>\n` // encoded in URL-base64 with the padding `=` elided
`<unix second precision timestamp in decimal ascii>\n`
`tags:\n`
`key:value;extra;...\n` // zero or more line separated, fields cannot contain a semicolon, end with newline instead of semicolon, key lowercase alphanumeric, first alpha, no whitespace or symbols, only key and following `:` are mandatory
`\n` // tags end with a double linebreak
`content:\n` // literally this word on one line *directly* after the newline of the previous
`<content>\n` // any number of further line breaks, last line is signature, everything before signature line is part of the canonical hash
-> The canonical form is the above, creating the message hash that is generated with Blake 2b <-
---
`<ed25519 signature encoded in URL-base64>\n` // this field would have two padding chars `==`, these should be elided
---
The binary data - Event Ids, Pubkeys and Signatures are encoded in raw base64 URL encoding (without padding), Signatures are 86 characters long, with the two padding characters elided `==`, Ids and Pubkeys are 43 characters long, with a single padding character elided `=`.
The database stored form of this event should make use of an event ID hash to monotonic serial ID number as the key to associating the filter indexes of an event store.
Event ID hashes will be encoded in URL-base64 where used in tags or mentioned in content with the prefix `e:`. Public keys must be prefixed with `p:` Tag keys should be intelligible words and a specification for their structure should be defined by users of them and shared with other REALY devs.
Indexing tag keys should be done with a truncated Blake2b hash cut at 8 bytes in the event store, keys should be short and thus the chances of collisions are practically zero.
== Publishing
Submitting an event to be stored is the same as a result sent from an Event Id query except with the type of operation inteded: `store\n` to store an event, `replace:<Event Id>\n` to replace an existing event and `relay\n` to not store but send to subscribers with open matching filters. Replace will not be accepted if the message type and pubkey are different to the original that is specified.
The use of specific different types of store requests eliminates the complexity of defining event types as replaceable, by making this intent explicit. A relay can also only allow one kind, such as a pure relay, which only accepts `relay` requests but neither `store` nor `replace`.
An event is then acknowledged to be stored or rejected with a message `ok:<true/false>;<Event Id>;<reason type>:human readable part` where the reason type is one of a set of common types to indicate the reason for the false
Events that are returned have the `<subscription Id>:<Event Id>\n` as the first line, and then the event in the format described above afterwards.
== Queries
There is three types of queries in REALY:
=== Filter
A filter has one or more of the fields listed below, and headed with `filter`:
----
filter:<subscription Id>\n
pubkeys:<one>;<two>;...\n // these match as OR
timestamp:<since>;<until\n // either can be empty but not both, omit line for this, both are inclusive
tags:
<key>:<value>\n // indexes are not required or used for more than the key and value
... // several matches can be present, they will act as OR
----
The result returned from this is a newline separated list of event ID hashes encoded in base64, a following Event Id search is required to retrieve them. This obviates the need for pagination as the 45 bytes per event per result is far less than sending the whole event and the client is then free to paginate how they like without making for an onerous implementation requirement or nebulous result limit specification.
The results must be in reverse chronological order so the client knows it can paginate them from newest to oldest as required by the user interface.
If instead of `filter\n` at the top there is `subscribe:<subscription Id>\n` the relay should return any events it finds the Id for and then subsequently will forward the Event Id of any new matching event that comes in until the client sends a `close:<subscription Id>\n` message.
Once all stored events are returned, the relay will send `end:<subscription Id>\n` to notify the client that here after will only be events that just arrived.
`subscribe_full:<subscription Id>` should be used to request the events be directly delivered instead of just the event IDs associated with the subscription filter.
In the case of events that are published via the `relay` command, it is necessary that therefore there must be one or more "chanserv" style relays also connected to the relay to whom the clients know they can request such events, and a "nickserv" type specialized relay would need to exist also for creating access whitelists - by compiling singular edits to these lists and using a subscription mechanism to notify such clients of the need to update their ACL.
=== Text
A text search is just `search:<subscription Id>:` followed by a series of space separated tokens if the event store has a full text index, terminated with a newline.
=== Event Id
Event requests are as follows:
----
events:<subscription Id>\n
<event ID one>\n
...
----
Unlike in event tags and content, the `e:` prefix is unnecessary. The previous two query types only have lists of events in return, and to fetch the event a client then must send an `events` request.
Normally clients will gather a potentially longer list of events and then send Event Id queries in segments according to the requirements of the user interface.
The results are returned as a series as follows, for each item returned:
----
event:<subscription Id>:<Event Id>\n
<event>\n
...
----

26
doc/relays.adoc Normal file
View File

@@ -0,0 +1,26 @@
= relays
A key design principle employed in REALY is that of relay specialization.
Instead of making a relay a hybrid event store and router, in REALY a relay does only one thing. Thus there can be
- a simple event repository that only understands queries to fetch a list of events by ID,
- a relay that only indexes and keeps a space/time limited cache of events to process filters
- a relay that only keeps a full text search index and a query results cache
- a relay that only accepts list change CRDT events such as follow, join/create/delete/leave group, block, delete, report and compiles these events into single lists that are accessible to another relay that can use these compiled lists to control access either via explicit lists or by matching filters
- a relay that stores and fetches media, including being able to convert and cache such as image size and formats
- ...and many others are possible
By constraining the protocol interoperability compliance down to small simple sub-protocols the ability for clients to maintain currency with other clients and with relays is greatly simplified, without gatekeepers.
In addition, it should be normalized that relays can include clients that query other specialist relays, especially for such things as caching events. Thus one relay can be queried for a filter index, and the list of Event Ids returned can then be fetched from another relay that specialises in storing events and returning them on request by lists of Event Ids, and still other relays could store media files and be able to convert them on demand.
For this reason, instead of a single centralised mechanism, aside from the basic specifications as you can see in link:./events_queries.adoc[REALY protocol event/query specification] it is possible to add more to this list without needing to negotiate to have this specification added to this repository, though once it comes into use it can be done.
Along with the use of human-readable type identifiers for documents and the almost completely human-composable event encoding, the specification of REALY is not dependent on any kind of authoritative gatekeeping organisation, but instead organisations can add these to their own specifications lists as they see fit, eliminating a key problem with the operation of the nostr protocol.
There need not be bureaucratic RFC style specifications, but instead use human-readable names and be less formally described, the formality improving as others adopt it and expand or refine it.
Thus also it is recommended that implementations of any or all REALY servers and clients should keep a copy of the specification documents found in other implementations and converge them to each other as required when their repositories update support to changes and new sub-protocols.
Lastly, as part of making this ecosystem as heterogeneous and decentralized as possible, the notion of relay operators subscribing to other relay services such as media storage/conversion specialists or event archivists and focusing each relay service on simple, single purposes and protocols enables a more robust and failure resistant ecosystem where multiple providers can compete for clients and to be suppliers for other providers and replicate data and potentially enable specialisations like archival data access for providers that aggregate data from multiple other providers.

View File

@@ -1,25 +0,0 @@
# realy protocol event specification
JSON is awful, and space inefficient, and complex to parse due to its intolerance of terminal commas and annoying to work with because of its retarded, multi-standards of string escaping.
Line structured documents are much more readily amenable to human reading and editing, and `\n`/`;`/`:` is more efficient than `","` as an item separator. Data structures can be much more simply expressed in a similar way as how they are in programming languages.
It is one of the guiding principles of the Unix philosophy to keep data in plain text, human readable format wherever possible, forcing the interposition of a parser just for humans to read the data adds extra brittleness to a protocol.
So, this is how realy events look:
```
<type name>\n
<pubkey>\n // encoded in URL-base64
<unix second precision timestamp in decimal ascii>\n
key:value;extra;...\n // zero or more line separated, fields cannot contain a semicolon, end with newline instead of semicolon, key lowercase alphanumeric, first alpha, only key is mandatory, only reserved is `content`
content: // literally this word on one line
<content>\n // any number of further line breaks, last line is signature
<bip-340 schnorr signature encoded in URL-base64>\n
```
The canonical form is exactly this, except for the signature and following linebreak, hashed with Blake2b
The database stored form of this event should make use of an event ID hash to monotonic collision free serial table and an event table.
Event ID hashes will be encoded in URL-base64 where used in tags or mentioned in content with the prefix `event:`. Public keys must be prefixed with `pubkey:` Tag keys should be intelligible words and a specification for their structure should be defined by users of them and shared with other REALY devs.

View File

@@ -1,4 +1,4 @@
# why realy?
= why REALY?
Since the introduction of the idea of a general "public square" style social network as seen with Facebook and Twitter, the whole world has been overcome by something of a plague of mind control brainwashing cults.
@@ -6,16 +6,23 @@ Worse than "Beatlemania" people are being lured into the control of various kind
Nostr protocol is a super simple event bus architecture, blended with a post office protocol, and due to various reasons related to the recent buyout of Twitter by Elon Musk, who plainly wants to turn it into the Western version of Wechat, it has become plagued with bad subprotocol designs that negate the benefits of self sovereign identity (elliptic curve asymmetric cryptography) and a dominant form of client that is essentially a travesty of Twitter itself.
Realy is being designed with the lessons learned from Nostr and the last 30 years of experience of internet communications protocols to aim to resist this kind of Embrace/Extend/Extinguish protocol that has repeatedly been performed on everything from email, to RSS, to threaded forums and instant messaging, by starting with the distilled essence of how these protocols should work so as to not be so easily vulnerable to being coopted by what is essentially in all but name the same centralised event bus architecture of social networks like Facebook and Twitter.
REALY is being designed with the lessons learned from Nostr and the last 30 years of experience of internet communications protocols to aim to resist this kind of Embrace/Extend/Extinguish protocol that has repeatedly been performed on everything from email, to RSS, to threaded forums and instant messaging, by starting with the distilled essence of how these protocols should work so as to not be so easily vulnerable to being coopted by what is essentially in all but name the same centralised event bus architecture of social networks like Facebook and Twitter.
The main purposes that Realy will target are:
The main purposes that REALY will target are:
- synchronous instant messaging protocols with IRC style nickserv and chanserv permissions and persistence, built from the ground up to take advantage of the cryptographic identities created by BIP-340 signatures, with an intuitive threaded structure that allows users to peruse a larger discussion without the problem of threads of discussion breaking the top level structure
- structured document repositories primarily for text media, as a basis for collaborative documentation and literature collections, and software source code (breaking out of the filesystem tree structure to permit much more flexible ways of organising code)
- persistent threaded discussion forums for longer form messages than the typical single sentence/paragraph of instant messaging
- simple cross-relay data query protocol that enables minimising the data cost of traffic to clients
- push style notification systems that can be programmed by the users' clients to respond to any kind of event breadcast to a relay
* synchronous instant messaging protocols with IRC style nickserv and chanserv permissions and persistence, built from the ground up to take advantage of the cryptographic identities created by BIP-340 signatures, with an intuitive threaded structure that allows users to peruse a larger discussion without the problem of threads of discussion breaking the top level structure
* structured document repositories primarily for text media, as a basis for collaborative documentation and literature collections, and software source code (breaking out of the filesystem tree structure to permit much more flexible ways of organising code)
* persistent threaded discussion forums for longer form messages than the typical single sentence/paragraph of instant messaging
* simple cross-relay data query protocol that enables minimising the data cost of traffic to clients
* push style notification systems that can be programmed by the users' clients to respond to any kind of event breadcast to a relay
A key concept in the R.E.A.L.Y. architecture is that of relays being a heteregenous group of data repositories and relaying systems that are built specific to purpose, such as a chat relay, which does not store any messages but merely bounces messages around ot subscribers, a document repository, which provides read access to data with full text search capability, that can ne specialised for a singular data format (eg markdown, eg mediawiki, eg code), a threaded, moderated forum, and others.
A key concept in the REALY architecture is that of relays being a heteregenous group of data repositories and relaying systems that are built specific to purpose, such as
A second key concept in R.E.A.L.Y. is the integration of Lightning Network payments - again mostly copying what is done with Nostr but enabling both per-use, micro-accounts and long term subscription styles of access, and the promotion of a notion of user-pays - where all data writing must be charged for, and most reading must be paid for. Lightning is perfect for this because it can currently cope with enormous volumes of payments with mere seconds of delay for settlement and a granularity of denomination that lends itself to the very low cost of delivering a one-time service, or maintaining a micro-account.
- a chat relay, which does not store any messages but merely bounces messages around ot subscribers,
- a document repository, which provides read access to data with full text search capability, that can ne specialised for a singular data format (eg markdown, eg mediawiki, eg code), a threaded, moderated forum, and others,
- a directory relay which stores and distributes user metadata such as profiles, relay lists, follows, mutes, deletes and reports
- an authentication relay, which can be sent messages to add or remove users from access whitelists and blacklists, that provides this state data to relays it is used by
A second key concept in REALY is the integration of Lightning Network payments - again mostly copying what is done with Nostr but enabling both pseudonymous micro-accounts and long term subscription styles of access payment, and the promotion of a notion of user-pays - where all data writing must be charged for, and most reading must be paid for.
Lightning is perfect for this because it can currently cope with enormous volumes of payments with mere seconds of delay for settlement and a granularity of denomination that lends itself to the very low cost of delivering a one-time service, or maintaining a micro-account.

2
go.mod
View File

@@ -6,6 +6,8 @@ require (
github.com/davecgh/go-spew v1.1.1
github.com/fatih/color v1.18.0
go.uber.org/atomic v1.11.0
golang.org/x/exp v0.0.0-20250128182459-e0ece0dbea4c
lukechampine.com/frand v1.5.1
)
require (

4
go.sum
View File

@@ -12,8 +12,12 @@ github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOf
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
golang.org/x/exp v0.0.0-20250128182459-e0ece0dbea4c h1:KL/ZBHXgKGVmuZBZ01Lt57yE5ws8ZPSkkihmEyq7FXc=
golang.org/x/exp v0.0.0-20250128182459-e0ece0dbea4c/go.mod h1:tujkw807nyEEAamNbDrEGzRav+ilXA7PCRAd6xsmwiU=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.29.0 h1:TPYlXGxvx1MGTn2GiZDhnjPA9wZzZeGKHHmKhHYvgaU=
golang.org/x/sys v0.29.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
lukechampine.com/frand v1.5.1 h1:fg0eRtdmGFIxhP5zQJzM1lFDbD6CUfu/f+7WgAZd5/w=
lukechampine.com/frand v1.5.1/go.mod h1:4VstaWc2plN4Mjr10chUD46RAVGWhpkZ5Nja8+Azp0Q=

54
pkg/content/content.go Normal file
View File

@@ -0,0 +1,54 @@
package content
import (
"bytes"
)
// C is raw content bytes of a message. This can contain anything but when it is
// unmarshalled it is assumed that the last line (content between the second
// last and last line break) is not part of the content, as this is where the
// signature is placed.
//
// The only guaranteed property of an encoded content.C is that it has two
// newline characters, one at the very end, and a second one before it that
// demarcates the end of the actual content. It can be entirely binary and mess
// up a terminal to render the unsanitized possible control characters.
type C struct{ Content []byte }
// Marshal just writes the provided data with a `content:\n` prefix and adds a
// terminal newline.
func (c *C) Marshal(dst []byte) (result []byte, err error) {
result = append(append(append(dst, "content:\n"...), c.Content...), '\n')
return
}
var Prefix = "content:\n"
// Unmarshal expects the `content:\n` prefix and stops at the second last
// newline. The data between the second last and last newline in the data is
// assumed to be a signature but it could be anything in another use case.
func (c *C) Unmarshal(data []byte) (rem []byte, err error) {
if !bytes.HasPrefix(data, []byte("content:\n")) {
err = errorf.E("content prefix `content:\\n' not found: '%s'", data[:len(Prefix)+1])
return
}
// trim off the prefix.
data = data[len(Prefix):]
// check that there is a last newline.
if data[len(data)-1] != '\n' {
err = errorf.E("input data does not end with newline")
return
}
// we start at the second last, previous to the terminal newline byte.
lastPos := len(data) - 2
for ; lastPos >= len(Prefix); lastPos-- {
// the content ends at the byte before the second last newline byte.
if data[lastPos] == '\n' {
break
}
}
c.Content = data[:lastPos]
// return the remainder after the content-terminal newline byte.
rem = data[lastPos+1:]
return
}

View File

@@ -0,0 +1,37 @@
package content
import (
"bytes"
"crypto/rand"
mrand "math/rand"
"testing"
)
func TestC_Marshal_Unmarshal(t *testing.T) {
c := make([]byte, mrand.Intn(100)+25)
_, err := rand.Read(c)
if err != nil {
t.Fatal(err)
}
c1 := new(C)
c1.Content = c
var res []byte
if res, err = c1.Marshal(nil); chk.E(err) {
t.Fatal(err)
}
// append a fake zero length signature
res = append(res, '\n')
c2 := new(C)
var rem []byte
if rem, err = c2.Unmarshal(res); chk.E(err) {
t.Fatal(err)
}
if !bytes.Equal(c1.Content, c2.Content) {
log.I.S(c1, c2)
t.Fatal("content not equal")
}
if !bytes.Equal(rem, []byte{'\n'}) {
log.I.S(rem)
t.Fatalf("remainder not found")
}
}

9
pkg/content/log.go Normal file
View File

@@ -0,0 +1,9 @@
package content
import (
"protocol.realy.lol/pkg/lol"
)
var (
log, chk, errorf = lol.Main.Log, lol.Main.Check, lol.Main.Errorf
)

View File

@@ -1,14 +1,19 @@
package event
import (
"protocol.realy.lol/pkg/content"
"protocol.realy.lol/pkg/event/types"
"protocol.realy.lol/pkg/pubkey"
"protocol.realy.lol/pkg/signature"
"protocol.realy.lol/pkg/tags"
"protocol.realy.lol/pkg/timestamp"
)
type Event struct {
Type types.T
Pubkey []byte
Timestamp int64
Tags [][]byte
Content []byte
Signature []byte
Type *types.T
Pubkey *pubkey.P
Timestamp *timestamp.T
Tags *tags.T
Content *content.C
Signature *signature.S
}

View File

@@ -12,7 +12,6 @@ func TestT_Marshal_Unmarshal(t *testing.T) {
if res, err = typ.Marshal(nil); chk.E(err) {
t.Fatal(err)
}
log.I.S(res)
t2 := new(T)
var rem []byte
if rem, err = t2.Unmarshal(res); chk.E(err) {
@@ -21,7 +20,6 @@ func TestT_Marshal_Unmarshal(t *testing.T) {
if len(rem) > 0 {
log.I.S(rem)
}
log.I.S(t2)
if !bytes.Equal(typ, *t2) {
t.Fatal("types.T did not encode/decode faithfully")
}

74
pkg/id/id.go Normal file
View File

@@ -0,0 +1,74 @@
package id
import (
"bytes"
"crypto/ed25519"
"encoding/base64"
"io"
)
const Len = 43
type P struct{ b []byte }
func New(id []byte) (p *P, err error) {
if len(id) != ed25519.PublicKeySize {
err = errorf.E("invalid public key size: %d; require %d",
len(id), ed25519.PublicKeySize)
return
}
p = &P{id}
return
}
func (p *P) Marshal(dst []byte) (result []byte, err error) {
result = dst
if p == nil || p.b == nil || len(p.b) == 0 {
err = errorf.E("nil/zero length pubkey")
return
}
if len(p.b) != ed25519.PublicKeySize {
err = errorf.E("invalid public key length %d; require %d '%0x'",
len(p.b), ed25519.PublicKeySize, p.b)
return
}
buf := bytes.NewBuffer(result)
w := base64.NewEncoder(base64.RawURLEncoding, buf)
if _, err = w.Write(p.b); chk.E(err) {
return
}
if err = w.Close(); chk.E(err) {
return
}
result = append(buf.Bytes(), '\n')
return
}
func (p *P) Unmarshal(data []byte) (rem []byte, err error) {
rem = data
if p == nil {
err = errorf.E("can't unmarshal into nil types.T")
return
}
if len(rem) < 2 {
err = errorf.E("can't unmarshal nothing")
return
}
for i := range rem {
if rem[i] == '\n' {
if i != Len {
err = errorf.E("invalid encoded pubkey length %d; require %d '%0x'",
i, Len, rem[:i])
return
}
p.b = make([]byte, ed25519.PublicKeySize)
if _, err = base64.RawURLEncoding.Decode(p.b, rem[:i]); chk.E(err) {
return
}
rem = rem[i+1:]
return
}
}
err = io.EOF
return
}

37
pkg/id/id_test.go Normal file
View File

@@ -0,0 +1,37 @@
package id
import (
"bytes"
"crypto/ed25519"
"crypto/rand"
"testing"
)
func TestT_Marshal_Unmarshal(t *testing.T) {
var err error
for range 10 {
pk := make([]byte, ed25519.PublicKeySize)
if _, err = rand.Read(pk); chk.E(err) {
t.Fatal(err)
}
var p *P
if p, err = New(pk); chk.E(err) {
t.Fatal(err)
}
var o []byte
if o, err = p.Marshal(nil); chk.E(err) {
t.Fatal(err)
}
p2 := &P{}
var rem []byte
if rem, err = p2.Unmarshal(o); chk.E(err) {
t.Fatal(err)
}
if len(rem) > 0 {
log.I.F("%d %s", len(rem), rem)
}
if !bytes.Equal(pk, p2.b) {
t.Fatal("public key did not encode/decode faithfully")
}
}
}

9
pkg/id/log.go Normal file
View File

@@ -0,0 +1,9 @@
package id
import (
"protocol.realy.lol/pkg/lol"
)
var (
log, chk, errorf = lol.Main.Log, lol.Main.Check, lol.Main.Errorf
)

View File

@@ -7,7 +7,7 @@ import (
"io"
)
const Len = 44
const Len = 43
type P struct{ ed25519.PublicKey }
@@ -33,7 +33,7 @@ func (p *P) Marshal(dst []byte) (result []byte, err error) {
return
}
buf := bytes.NewBuffer(result)
w := base64.NewEncoder(base64.URLEncoding, buf)
w := base64.NewEncoder(base64.RawURLEncoding, buf)
if _, err = w.Write(p.PublicKey); chk.E(err) {
return
}
@@ -62,7 +62,7 @@ func (p *P) Unmarshal(data []byte) (rem []byte, err error) {
return
}
p.PublicKey = make([]byte, ed25519.PublicKeySize)
if _, err = base64.URLEncoding.Decode(p.PublicKey, rem[:i]); chk.E(err) {
if _, err = base64.RawURLEncoding.Decode(p.PublicKey, rem[:i]); chk.E(err) {
return
}
rem = rem[i+1:]

View File

@@ -8,31 +8,30 @@ import (
)
func TestP_Marshal_Unmarshal(t *testing.T) {
pk := make([]byte, ed25519.PublicKeySize)
var err error
if _, err = rand.Read(pk); chk.E(err) {
t.Fatal(err)
}
log.I.S(pk)
var p *P
if p, err = New(pk); chk.E(err) {
t.Fatal(err)
}
var o []byte
if o, err = p.Marshal(nil); chk.E(err) {
t.Fatal(err)
}
log.I.F("%d %s", len(o), o)
p2 := &P{}
var rem []byte
if rem, err = p2.Unmarshal(o); chk.E(err) {
t.Fatal(err)
}
if len(rem) > 0 {
log.I.F("%d %s", len(rem), rem)
}
log.I.S(p2.PublicKey)
if !bytes.Equal(pk, p2.PublicKey) {
t.Fatal("public key did not encode/decode faithfully")
for range 10 {
pk := make([]byte, ed25519.PublicKeySize)
if _, err = rand.Read(pk); chk.E(err) {
t.Fatal(err)
}
var p *P
if p, err = New(pk); chk.E(err) {
t.Fatal(err)
}
var o []byte
if o, err = p.Marshal(nil); chk.E(err) {
t.Fatal(err)
}
p2 := &P{}
var rem []byte
if rem, err = p2.Unmarshal(o); chk.E(err) {
t.Fatal(err)
}
if len(rem) > 0 {
log.I.F("%d %s", len(rem), rem)
}
if !bytes.Equal(pk, p2.PublicKey) {
t.Fatal("public key did not encode/decode faithfully")
}
}
}

View File

@@ -7,7 +7,7 @@ import (
"io"
)
const Len = 88
const Len = 86
type S struct{ Signature []byte }
@@ -33,7 +33,7 @@ func (p *S) Marshal(dst []byte) (result []byte, err error) {
return
}
buf := bytes.NewBuffer(result)
w := base64.NewEncoder(base64.URLEncoding, buf)
w := base64.NewEncoder(base64.RawURLEncoding, buf)
if _, err = w.Write(p.Signature); chk.E(err) {
return
}
@@ -62,7 +62,7 @@ func (p *S) Unmarshal(data []byte) (rem []byte, err error) {
return
}
p.Signature = make([]byte, ed25519.SignatureSize)
if _, err = base64.URLEncoding.Decode(p.Signature, rem[:i]); chk.E(err) {
if _, err = base64.RawURLEncoding.Decode(p.Signature, rem[:i]); chk.E(err) {
return
}
rem = rem[i+1:]

View File

@@ -8,31 +8,31 @@ import (
)
func TestS_Marshal_Unmarshal(t *testing.T) {
sig := make([]byte, ed25519.SignatureSize)
var err error
if _, err = rand.Read(sig); chk.E(err) {
t.Fatal(err)
}
log.I.S(sig)
var s *S
if s, err = New(sig); chk.E(err) {
t.Fatal(err)
}
var o []byte
if o, err = s.Marshal(nil); chk.E(err) {
t.Fatal(err)
}
log.I.F("%d %s", len(o), o)
p2 := &S{}
var rem []byte
if rem, err = p2.Unmarshal(o); chk.E(err) {
t.Fatal(err)
}
if len(rem) > 0 {
log.I.F("%d %s", len(rem), rem)
}
log.I.S(p2.Signature)
if !bytes.Equal(sig, p2.Signature) {
t.Fatal("signature did not encode/decode faithfully")
for range 10 {
sig := make([]byte, ed25519.SignatureSize)
if _, err = rand.Read(sig); chk.E(err) {
t.Fatal(err)
}
var s *S
if s, err = New(sig); chk.E(err) {
t.Fatal(err)
}
var o []byte
if o, err = s.Marshal(nil); chk.E(err) {
t.Fatal(err)
}
p2 := &S{}
var rem []byte
if rem, err = p2.Unmarshal(o); chk.E(err) {
t.Fatal(err)
}
if len(rem) > 0 {
log.I.F("%d %s", len(rem), rem)
}
if !bytes.Equal(sig, p2.Signature) {
t.Fatal("signature did not encode/decode faithfully")
}
}
}

9
pkg/tag/log.go Normal file
View File

@@ -0,0 +1,9 @@
package tag
import (
"protocol.realy.lol/pkg/lol"
)
var (
log, chk, errorf = lol.Main.Log, lol.Main.Check, lol.Main.Errorf
)

135
pkg/tag/tag.go Normal file
View File

@@ -0,0 +1,135 @@
// Package tag defines a format for event tags that follows the following rules:
//
// First field is the key, this is to be hashed using Blake2b and truncated to 8 bytes for indexing. These keys should
// not be long, and thus will not have any collisions as a truncated hash. The terminal byte of a key is the colon `:`
//
// Subsequent fields are separated by semicolon ';' and they can contain any data except a semicolon or newline.
//
// The tag is terminated by a newline.
package tag
import (
"bytes"
)
type fields [][]byte
type T struct{ fields }
func New[V ~[]byte | ~string](v ...V) (t *T, err error) {
t = new(T)
var k []byte
if k, err = ValidateKey([]byte(v[0])); err != nil {
err = errorf.E("")
return
}
v = v[1:]
t.fields = append(t.fields, k)
for i, val := range v {
var b []byte
if b, err = ValidateField(val, i); chk.E(err) {
return
}
t.fields = append(t.fields, b)
}
return
}
// ValidateKey checks that the key is valid. Keys must be the same most language symbols:
//
// - first character is alphabetic [a-zA-Z]
// - subsequent characters can be alphanumeric and underscore [a-zA-Z0-9_]
//
// If the key is not valid this function returns a nil value.
func ValidateKey[V ~[]byte | ~string](key V) (k []byte, err error) {
if len(key) < 1 {
return
}
kb := []byte(key)
switch {
case kb[0] < 'a' && k[0] > 'z' || kb[0] < 'A' && kb[0] > 'Z':
for i, b := range kb[1:] {
switch {
case (b > 'a' && b < 'z') || b > 'A' && b < 'Z' || b == '_' || b > '0' && b < '9':
default:
err = errorf.E("invalid character in tag key at index %d '%c': \"%s\"", i, b, kb)
return
}
}
}
// if we got to here, the whole string is compliant
k = kb
return
}
func ValidateField[V ~[]byte | ~string](f V, i int) (k []byte, err error) {
b := []byte(f)
if bytes.Contains(b, []byte(";")) {
err = errorf.E("key %d cannot contain ';': '%s'", i, b)
return
}
if bytes.Contains(b, []byte("\n")) {
err = errorf.E("key %d cannot contain '\\n': '%s'", i, b)
return
}
// if we got to here, the whole string is compliant
k = b
return
}
func (t *T) Marshal(dst []byte) (result []byte, err error) {
result = dst
if len(t.fields) == 0 {
return
}
for i, field := range t.fields {
result = append(result, field...)
if i == 0 {
result = append(result, ':')
} else if i == len(t.fields)-1 {
result = append(result, '\n')
} else {
result = append(result, ';')
}
}
return
}
func (t *T) Unmarshal(data []byte) (rem []byte, err error) {
var i int
var v byte
var dat []byte
// first find the end
for i, v = range data {
if v == '\n' {
dat, rem = data[:i], data[i+1:]
break
}
}
if len(dat) == 0 {
err = errorf.E("invalid empty tag")
return
}
for i, v = range dat {
if v == ':' {
f := dat[:i]
dat = dat[i+1:]
t.fields = append(t.fields, f)
break
}
}
for len(dat) > 0 {
for i, v = range dat {
if v == ';' {
t.fields = append(t.fields, dat[:i])
dat = dat[i+1:]
break
}
if i == len(dat)-1 {
t.fields = append(t.fields, dat)
return
}
}
}
return
}

27
pkg/tag/tag_test.go Normal file
View File

@@ -0,0 +1,27 @@
package tag
import (
"testing"
)
func TestT_Marshal_Unmarshal(t *testing.T) {
var err error
var t1 *T
if t1, err = New("reply", "e:l_T9Of4ru-PLGUxxvw3SfZH0e6XW11VYy8ZSgbcsD9Y",
"realy.example.com/repo"); chk.E(err) {
t.Fatal(err)
}
var tb []byte
if tb, err = t1.Marshal(nil); chk.E(err) {
t.Fatal(err)
}
t2 := new(T)
var rem []byte
if rem, err = t2.Unmarshal(tb); chk.E(err) {
t.Fatal(err)
}
if len(rem) > 0 {
log.I.F("%s", rem)
t.Fatal("remainder after tag should have been nothing")
}
}

9
pkg/tags/log.go Normal file
View File

@@ -0,0 +1,9 @@
package tags
import (
"protocol.realy.lol/pkg/lol"
)
var (
log, chk, errorf = lol.Main.Log, lol.Main.Check, lol.Main.Errorf
)

56
pkg/tags/tags.go Normal file
View File

@@ -0,0 +1,56 @@
package tags
import (
"bytes"
"fmt"
"protocol.realy.lol/pkg/tag"
)
const Sentinel = "tags:\n"
var SentinelBytes = []byte(Sentinel)
type tags []*tag.T
type T struct{ tags }
func New(v ...*tag.T) *T { return &T{tags: v} }
func (t *T) Marshal(dst []byte) (result []byte, err error) {
result = dst
result = append(result, Sentinel...)
for _, tt := range t.tags {
if result, err = tt.Marshal(result); chk.E(err) {
return
}
}
result = append(result, '\n')
return
}
func (t *T) Unmarshal(data []byte) (rem []byte, err error) {
if len(data) < len(Sentinel) {
err = fmt.Errorf("bytes too short to contain tags")
return
}
var dat []byte
if bytes.Equal(data[:len(Sentinel)], SentinelBytes) {
dat = data[len(Sentinel):]
}
if len(dat) < 1 {
return
}
for len(dat) > 0 {
if len(dat) == 1 && dat[0] == '\n' {
break
}
// log.I.S(dat)
tt := new(tag.T)
if dat, err = tt.Unmarshal(dat); chk.E(err) {
return
}
t.tags = append(t.tags, tt)
}
return
}

45
pkg/tags/tags_test.go Normal file
View File

@@ -0,0 +1,45 @@
package tags
import (
"bytes"
"testing"
"protocol.realy.lol/pkg/tag"
)
func TestT_Marshal_Unmarshal(t *testing.T) {
var tegs = [][]string{
{"reply", "e:l_T9Of4ru-PLGUxxvw3SfZH0e6XW11VYy8ZSgbcsD9Y", "realy.example.com/repo1"},
{"root", "e:l_T9Of4ru-PLGUxxvw3SfZH0e6XW11VYy8ZSgbcsD9Y", "realy.example.com/repo2"},
{"mention", "p:JMkZVnu9QFplR4F_KrWX-3chQsklXZq_5I6eYcXfz1Q", "realy.example.com/repo3"},
}
var err error
var tgs []*tag.T
for _, teg := range tegs {
var tg *tag.T
if tg, err = tag.New(teg...); chk.E(err) {
t.Fatal(err)
}
tgs = append(tgs, tg)
}
t1 := New(tgs...)
var m1 []byte
if m1, err = t1.Marshal(nil); chk.E(err) {
t.Fatal(err)
}
t2 := new(T)
var rem []byte
if rem, err = t2.Unmarshal(m1); chk.E(err) {
t.Fatal(err)
}
if len(rem) > 0 {
t.Fatalf("%s", rem)
}
var m2 []byte
if m2, err = t2.Marshal(nil); chk.E(err) {
t.Fatal(err)
}
if !bytes.Equal(m1, m2) {
t.Fatalf("not equal:\n%s\n%s", m1, m2)
}
}

File diff suppressed because one or more lines are too long

9
pkg/timestamp/gen/log.go Normal file
View File

@@ -0,0 +1,9 @@
package main
import (
"protocol.realy.lol/pkg/lol"
)
var (
log, chk, errorf = lol.Main.Log, lol.Main.Check, lol.Main.Errorf
)

View File

@@ -0,0 +1,17 @@
package main
import (
"fmt"
"os"
)
func main() {
var err error
var fh *os.File
if fh, err = os.Create("base10k.txt"); chk.E(err) {
panic(err)
}
for i := range 10000 {
fmt.Fprintf(fh, "%04d", i)
}
}

9
pkg/timestamp/log.go Normal file
View File

@@ -0,0 +1,9 @@
package timestamp
import (
"protocol.realy.lol/pkg/lol"
)
var (
log, chk, errorf = lol.Main.Log, lol.Main.Check, lol.Main.Errorf
)

109
pkg/timestamp/timestamp.go Normal file
View File

@@ -0,0 +1,109 @@
package timestamp
import (
_ "embed"
"golang.org/x/exp/constraints"
)
// run this to regenerate (pointlessly) the base 10 array of 4 places per entry
//go:generate go run ./gen/.
//go:embed base10k.txt
var base10k []byte
const base = 10000
type T struct {
N uint64
}
func New[V constraints.Integer](n V) *T { return &T{uint64(n)} }
func (n *T) Uint64() uint64 { return n.N }
func (n *T) Int64() int64 { return int64(n.N) }
func (n *T) Uint16() uint16 { return uint16(n.N) }
var powers = []*T{
{1},
{1_0000},
{1_0000_0000},
{1_0000_0000_0000},
{1_0000_0000_0000_0000},
}
const zero = '0'
const nine = '9'
func (n *T) Marshal(dst []byte) (b []byte) {
nn := n.N
b = dst
if n.N == 0 {
b = append(b, '0')
return
}
var i int
var trimmed bool
k := len(powers)
for k > 0 {
k--
q := n.N / powers[k].N
if !trimmed && q == 0 {
continue
}
offset := q * 4
bb := base10k[offset : offset+4]
if !trimmed {
for i = range bb {
if bb[i] != '0' {
bb = bb[i:]
trimmed = true
break
}
}
}
b = append(b, bb...)
n.N = n.N - q*powers[k].N
}
n.N = nn
return
}
// Unmarshal reads a string, which must be a positive integer int larger than math.MaxUint64,
// skipping any non-numeric content before it.
//
// Note that leading zeros are not considered valid, but basically int such thing as machine
// generated JSON integers with leading zeroes. Until this is disproven, this is the fastest way
// to read a positive json integer, and a leading zero is decoded as a zero, and the remainder
// returned.
func (n *T) Unmarshal(b []byte) (r []byte, err error) {
if len(b) < 1 {
err = errorf.E("zero length number")
return
}
var sLen int
if b[0] == zero {
r = b[1:]
n.N = 0
return
}
// count the digits
for ; sLen < len(b) && b[sLen] >= zero && b[sLen] <= nine && b[sLen] != ','; sLen++ {
}
if sLen == 0 {
err = errorf.E("zero length number")
return
}
if sLen > 20 {
err = errorf.E("too big number for uint64")
return
}
// the length of the string found
r = b[sLen:]
b = b[:sLen]
for _, ch := range b {
ch -= zero
n.N = n.N*10 + uint64(ch)
}
return
}

View File

@@ -0,0 +1,77 @@
package timestamp
import (
"math"
"strconv"
"testing"
"lukechampine.com/frand"
)
func TestMarshalUnmarshal(t *testing.T) {
b := make([]byte, 0, 8)
var rem []byte
var n *T
var err error
for _ = range 10000000 {
n = New(uint64(frand.Intn(math.MaxInt64)))
b = n.Marshal(b)
m := New(0)
if rem, err = m.Unmarshal(b); chk.E(err) {
t.Fatal(err)
}
if n.N != m.N {
t.Fatalf("failed to convert to int64 at %d %s %d", n.N, b, m.N)
}
if len(rem) > 0 {
t.Fatalf("leftover bytes after converting back: '%s'", rem)
}
b = b[:0]
}
}
func BenchmarkByteStringToInt64(bb *testing.B) {
b := make([]byte, 0, 19)
var i int
const nTests = 10000000
testInts := make([]*T, nTests)
for i = range nTests {
testInts[i] = New(frand.Intn(math.MaxInt64))
}
bb.Run("Marshal", func(bb *testing.B) {
bb.ReportAllocs()
for i = 0; i < bb.N; i++ {
n := testInts[i%10000]
b = n.Marshal(b)
b = b[:0]
}
})
bb.Run("Itoa", func(bb *testing.B) {
bb.ReportAllocs()
var s string
for i = 0; i < bb.N; i++ {
n := testInts[i%10000]
s = strconv.Itoa(int(n.N))
_ = s
}
})
bb.Run("MarshalUnmarshal", func(bb *testing.B) {
bb.ReportAllocs()
m := New(0)
for i = 0; i < bb.N; i++ {
n := testInts[i%10000]
b = m.Marshal(b)
_, _ = n.Unmarshal(b)
b = b[:0]
}
})
bb.Run("ItoaAtoi", func(bb *testing.B) {
bb.ReportAllocs()
var s string
for i = 0; i < bb.N; i++ {
n := testInts[i%10000]
s = strconv.Itoa(int(n.N))
_, _ = strconv.Atoi(s)
}
})
}

20
readme.adoc Normal file
View File

@@ -0,0 +1,20 @@
= REALY Protocol
____
relay, events and like… yeah
____
image:https://img.shields.io/badge/godoc-documentation-blue.svg[Documentation,link=https://pkg.go.dev/protocol.realy.lol]
image:https://img.shields.io/badge/matrix-chat-green.svg[matrix chat,link=https://matrix.to/#/#realy-general:matrix.org]
zap mleku: ⚡mleku@getalby.com
Inspired by the event bus architecture of https://github.com/nostr-protocol[nostr] but redesigned to avoid the
serious deficiencies of that protocol for both developers and users.
* link:./doc/why.adoc[why]
* link:./doc/events_queries.adoc[events and queries]
* link:./doc/relays.adoc[relays]
* link:./relays/readme.md[reference relays]
* link:./clients/readme.md[reference clients]
* link:./pkg/readme.md[_GO⌯_ libraries]

View File

@@ -1,17 +0,0 @@
# R.E.A.L.Y. Protocol
> relay events and like... yeah
[![Documentation](https://img.shields.io/badge/godoc-documentation-blue.svg)](https://pkg.go.dev/protocol.realy.lol)
[![matrix chat](https://img.shields.io/badge/matrix-chat-green.svg)](https://matrix.to/#/#realy-general:matrix.org)
zap mleku: ⚡mleku@getalby.com
Inspired by the event bus architecture of [nostr](https://github.com/nostr-protocol) but redesigned to avoid the
serious deficiencies of that protocol for both developers and users.
- [why](./doc/why.md)
- [event spec](./doc/spec.md)
- [reference relays](./relays/readme.md)
- [reference clients](./clients/readme.md)
- [GO libraries](./pkg/readme.md)