switch to asciidoc and explain relay specialization

.gitignore

@@ -75,6 +75,7 @@ key
 !.openapi-generator-ignore
 !.gitignore
 !*.jsonl
+!*.adoc
 
 # ...even if they are in subdirectories
 !*/

doc/events_queries.adoc

@@ -1,4 +1,4 @@
-# realy protocol event specification
+= REALY protocol event/query specification
 
 JSON is awful, and space inefficient, and complex to parse due to its intolerance of terminal commas and annoying to work with because of its retarded, multi-standards of string escaping.
 
@@ -8,19 +8,19 @@ It is one of the guiding principles of the Unix philosophy to keep data in plain
 
 REALY protocol format is extremely simple and should be trivial to parse in any programming language with basic string slicing operators.
 
-## Events
+== Events
 
 So, this is how realy events look:
 
-```
-<type name>\n
+----
+<type name>\n // can be anything, hierarchic names like note/html note/md are possible
 <pubkey>\n // encoded in URL-base64
 <unix second precision timestamp in decimal ascii>\n
 key:value;extra;...\n // zero or more line separated, fields cannot contain a semicolon, end with newline instead of semicolon, key lowercase alphanumeric, first alpha, no whitespace or symbols, only key is mandatory, only reserved is `content`
-content:\n // literally this word on one line
+content:\n // literally this word on one line *directly* after the newline of the previous
 <content>\n // any number of further line breaks, last line is signature, everything before signature line is part of the canonical hash
 <ed25519 signature encoded in URL-base64>\n
-```
+----
 
 The canonical form is exactly this, except for the signature and following linebreak, hashed with Blake2b.
 
@@ -36,22 +36,23 @@ An event is then acknowledged to be stored or rejected with a message `ok:<true/
 
 Events that are returned have the `<subscription Id>:<Event Id>\n` as the first line.
 
-## Queries
+== Queries
 
 There is three types of queries in REALY:
 
-### Filter
+=== Filter
 
 A filter has one or more of the fields listed below, and headed with `filter`:
 
-```
+----
 filter:<subscription Id>\n
 pubkeys:<one>;<two>;...\n // these match as OR
 timestamp:<since>;<until\n // either can be empty but not both, omit line for this, both are inclusive
 tags:
 <key>:<value>\n // indexes are not required or used for more than the key and value
 ... // several matches can be present, they will act as OR
-```
+----
+
 The result returned from this is a newline separated list of event ID hashes encoded in base64, a following Event Id search is required to retrieve them. This obviates the need for pagination as the 45 bytes per event per result is far less than sending the whole event and the client is then free to paginate how they like without making for an onerous implementation requirement or nebulous result limit specification.
 
 The results must be in reverse chronological order so the client knows it can paginate them from newest to oldest as required by the user interface.
@@ -60,24 +61,25 @@ If instead of `filter\n` at the top there is `subscribe:<subscription Id>\n` the
 
 Once all stored events are returned, the relay will send `end:<subscription Id>\n` to notify the client the query is finished. If the client wants a subscription it must use `subscribe`. The client should end subscriptions with `close:<subscription Id>\n` or if the socket is closed.
 
-### Text
+=== Text
 
 A text search is just `search:<subscription Id>:` followed by a series of space separated tokens if the event store has a full text index, terminated with a newline.
 
-### Event Id
+=== Event Id
 
 Event requests are as follows:
 
-```
+----
 events:<subscription Id>\n
 <one>\n
 ...
-```
+----
+
 Normally clients will gather a potentially longer list of events and then send Event Id queries in segments according to the requirements of the user interface.
 
 The results are returned as a series as follows, for each item returned:
 
-```
+----
 event:<subscription Id>:<Event Id>\n
 <event>
-```
+----

doc/relays.adoc

@@ -0,0 +1,24 @@
+= relays
+
+A key design principle employed in REALY is that of relay specialization.
+
+Instead of making a relay a hybrid event store and router, in REALY a relay does only one thing. Thus there can be
+
+- a simple event repository that only understands queries to fetch a list of events by ID,
+- a relay that only indexes and keeps a space/time limited cache of events to process filters
+- a relay that only keeps a full text search index and a query results cache
+- a relay that only accepts list change CRDT events such as follow, join/create/delete/leave group, block, delete, report and compiles these events into single lists that are accessible to another relay that can use these compiled lists to control access either via explicit lists or by matching filters
+- a relay that stores and fetches media, including being able to convert and cache such as image size and formats
+- ...and many others are possible
+
+In addition, it should be normalized that relays can include clients that query other specialist relays, especially for such things as caching events. Thus one relay can be queried for a filter index, and the list of Event Ids returned can then be fetched from another relay that specialises in storing events and returning them on request by lists of Event Ids, and still other relays could store media files and be able to convert them on demand.
+
+For this reason, instead of a single centralised mechanism, aside from the basic specifications as you can see in link:./events_queries.adoc[REALY protocol event/query specification] it is possible to add more to this list without needing to negotiate to have this specification added to this repository, though once it comes into use it can be done.
+
+Along with the use of human-readable type identifiers for documents and the almost completely human-composable event encoding, the specification of REALY is not dependent on any kind of authoritative gatekeeping organisation, but instead organisations can add these to their own specifications lists as they see fit, eliminating a key problem with the operation of the nostr protocol.
+
+There need not be bureaucratic RFC style specifications, but instead use human-readable names and be less formally described, the formality improving as others adopt it and expand or refine it.
+
+Thus also it is recommended that implementations of any or all REALY servers and clients should keep a copy of the specification documents found in other implementations and converge them to each other as required when their repositories update support to changes and new sub-protocols.
+
+Lastly, as part of making this ecosystem as heterogeneous and decentralized as possible, the notion of relay operators subscribing to other relay services such as media storage/conversion specialists or event archivists and focusing each relay service on simple, single purposes and protocols enables a more robust and failure resistant ecosystem where multiple providers can compete for clients and to be suppliers for other providers and replicate data and potentially enable specialisations like archival data access for providers that aggregate data from multiple other providers.

doc/why.adoc

@@ -1,4 +1,4 @@
-# why realy?
+= why REALY?
 
 Since the introduction of the idea of a general "public square" style social network as seen with Facebook and Twitter, the whole world has been overcome by something of a plague of mind control brainwashing cults.
 
@@ -6,16 +6,23 @@ Worse than "Beatlemania" people are being lured into the control of various kind
 
 Nostr protocol is a super simple event bus architecture, blended with a post office protocol, and due to various reasons related to the recent buyout of Twitter by Elon Musk, who plainly wants to turn it into the Western version of Wechat, it has become plagued with bad subprotocol designs that negate the benefits of self sovereign identity (elliptic curve asymmetric cryptography) and a dominant form of client that is essentially a travesty of Twitter itself.
 
-Realy is being designed with the lessons learned from Nostr and the last 30 years of experience of internet communications protocols to aim to resist this kind of Embrace/Extend/Extinguish protocol that has repeatedly been performed on everything from email, to RSS, to threaded forums and instant messaging, by starting with the distilled essence of how these protocols should work so as to not be so easily vulnerable to being coopted by what is essentially in all but name the same centralised event bus architecture of social networks like Facebook and Twitter.
+REALY is being designed with the lessons learned from Nostr and the last 30 years of experience of internet communications protocols to aim to resist this kind of Embrace/Extend/Extinguish protocol that has repeatedly been performed on everything from email, to RSS, to threaded forums and instant messaging, by starting with the distilled essence of how these protocols should work so as to not be so easily vulnerable to being coopted by what is essentially in all but name the same centralised event bus architecture of social networks like Facebook and Twitter.
 
-The main purposes that Realy will target are:
+The main purposes that REALY will target are:
 
-- synchronous instant messaging protocols with IRC style nickserv and chanserv permissions and persistence, built from the ground up to take advantage of the cryptographic identities created by BIP-340 signatures, with an intuitive threaded structure that allows users to peruse a larger discussion without the problem of threads of discussion breaking the top level structure
-- structured document repositories primarily for text media, as a basis for collaborative documentation and literature collections, and software source code (breaking out of the filesystem tree structure to permit much more flexible ways of organising code)
-- persistent threaded discussion forums for longer form messages than the typical single sentence/paragraph of instant messaging
-- simple cross-relay data query protocol that enables minimising the data cost of traffic to clients
-- push style notification systems that can be programmed by the users' clients to respond to any kind of event breadcast to a relay
+* synchronous instant messaging protocols with IRC style nickserv and chanserv permissions and persistence, built from the ground up to take advantage of the cryptographic identities created by BIP-340 signatures, with an intuitive threaded structure that allows users to peruse a larger discussion without the problem of threads of discussion breaking the top level structure
+* structured document repositories primarily for text media, as a basis for collaborative documentation and literature collections, and software source code (breaking out of the filesystem tree structure to permit much more flexible ways of organising code)
+* persistent threaded discussion forums for longer form messages than the typical single sentence/paragraph of instant messaging
+* simple cross-relay data query protocol that enables minimising the data cost of traffic to clients
+* push style notification systems that can be programmed by the users' clients to respond to any kind of event breadcast to a relay
 
-A key concept in the R.E.A.L.Y. architecture is that of relays being a heteregenous group of data repositories and relaying systems that are built specific to purpose, such as a chat relay, which does not store any messages but merely bounces messages around ot subscribers, a document repository, which provides read access to data with full text search capability, that can ne specialised for a singular data format (eg markdown, eg mediawiki, eg code), a threaded, moderated forum, and others.
+A key concept in the REALY architecture is that of relays being a heteregenous group of data repositories and relaying systems that are built specific to purpose, such as
 
-A second key concept in R.E.A.L.Y. is the integration of Lightning Network payments - again mostly copying what is done with Nostr but enabling both per-use, micro-accounts and long term subscription styles of access, and the promotion of a notion of user-pays - where all data writing must be charged for, and most reading must be paid for. Lightning is perfect for this because it can currently cope with enormous volumes of payments with mere seconds of delay for settlement and a granularity of denomination that lends itself to  the very low cost of delivering a one-time service, or maintaining a micro-account.
+- a chat relay, which does not store any messages but merely bounces messages around ot subscribers,
+- a document repository, which provides read access to data with full text search capability, that can ne specialised for a singular data format (eg markdown, eg mediawiki, eg code), a threaded, moderated forum, and others,
+- a directory relay which stores and distributes user metadata such as profiles, relay lists, follows, mutes, deletes and reports
+- an authentication relay, which can be sent messages to add or remove users from access whitelists and blacklists, that provides this state data to relays it is used by
+
+A second key concept in REALY is the integration of Lightning Network payments - again mostly copying what is done with Nostr but enabling both pseudonymous micro-accounts and long term subscription styles of access payment, and the promotion of a notion of user-pays - where all data writing must be charged for, and most reading must be paid for.
+
+Lightning is perfect for this because it can currently cope with enormous volumes of payments with mere seconds of delay for settlement and a granularity of denomination that lends itself to the very low cost of delivering a one-time service, or maintaining a micro-account.

readme.adoc

@@ -0,0 +1,22 @@
+= REALY Protocol
+
+____
+
+relay events and like… yeah
+
+____
+
+image:https://img.shields.io/badge/godoc-documentation-blue.svg[Documentation,link=https://pkg.go.dev/protocol.realy.lol]
+image:https://img.shields.io/badge/matrix-chat-green.svg[matrix chat,link=https://matrix.to/#/#realy-general:matrix.org]
+
+zap mleku: ⚡️mleku@getalby.com
+
+Inspired by the event bus architecture of https://github.com/nostr-protocol[nostr] but redesigned to avoid the
+serious deficiencies of that protocol for both developers and users.
+
+* link:./doc/why.md[why]
+* link:./doc/events_queries.md[events and queries]
+* link:./doc/relays.md[relays]
+* link:./relays/readme.md[reference relays]
+* link:./clients/readme.md[reference clients]
+* link:./pkg/readme.md[GO libraries]

readme.md

@@ -1,17 +0,0 @@
-# R.E.A.L.Y. Protocol
-
-> relay events and like... yeah
-
-[![Documentation](https://img.shields.io/badge/godoc-documentation-blue.svg)](https://pkg.go.dev/protocol.realy.lol)
-[![matrix chat](https://img.shields.io/badge/matrix-chat-green.svg)](https://matrix.to/#/#realy-general:matrix.org)
-
-zap mleku: ⚡️mleku@getalby.com
-
-Inspired by the event bus architecture of [nostr](https://github.com/nostr-protocol) but redesigned to avoid the 
-serious deficiencies of that protocol for both developers and users.
-
-- [why](./doc/why.md)
-- [event spec](./doc/spec.md)
-- [reference relays](./relays/readme.md)
-- [reference clients](./clients/readme.md)
-- [GO libraries](./pkg/readme.md)