initial replication from docker-gentoo-steemd

.gitignore

@@ -0,0 +1,10 @@
+block_log*
+shared_memory*
+*.log
+config.ini
+*.json
+config
+config.ini
+config.py
+*.pyc
+__pycache__

LICENSE

@@ -0,0 +1,24 @@
+This is free and unencumbered software released into the public domain.
+
+Anyone is free to copy, modify, publish, use, compile, sell, or
+distribute this software, either in source code form or as a compiled
+binary, for any purpose, commercial or non-commercial, and by any
+means.
+
+In jurisdictions that recognize copyright laws, the author or authors
+of this software dedicate any and all copyright interest in the
+software to the public domain. We make this dedication for the benefit
+of the public at large and to the detriment of our heirs and
+successors. We intend this dedication to be an overt act of
+relinquishment in perpetuity of all present and future rights to this
+software under copyright law.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
+
+For more information, please refer to <http://unlicense.org>

README.md

@@ -0,0 +1,107 @@
+# docker-gentoo-steemd
+`steemd` built inside a Docker container using a Gentoo base with size optimization
+
+In the `dkr` directory is a short shell script that sets some handy aliases, which you
+will need to run this docker container:
+
+    source init.sh
+
+I recommend on the servers you use this script that you put this on your shell startup
+script, it will make it a lot simpler for you. While you are still in the `dkr` directory,
+run this command:
+
+    echo "source `pwd`/init.sh" >> ~/.bashrc
+
+Substitute `.bashrc` with `.zshrc` if you, like me, use zsh.
+
+Note that you can 'source' this file from any other part of the filesystem and it will
+still know where its' parent directory and all the bits are.
+
+Then you can type `halp` and it will show you the short commands to perform all the 
+functions. The script stores its own location automatically so all commands will work 
+no matter where you move in your filesystem. The script can be edited with the command:
+`.editsh`. Note that the halp command simply processes the script itself to produce
+the output, the code is the documentation, is the code.
+
+## Basic info about managing a Docker Witness instance
+
+Because it can be a bit confusing (it confused me for a long time) I will explain the
+basic procedure to using this, beyond what the help in the `init.sh` provides:
+
+To build the image, `.build` will run the procedure defined in the `Dockerfile`. Then
+there will be an image you can see with `dkr ps -a` (probably at the top of the output).
+
+To get the image running, type `.run` (this will also start the `steemd` which, with an empty `data`
+folder will begin syncing from scratch, and use an empty configuration). There is
+a file in `data/witness_node_data_dir/` called config.ini.example which will help
+you create a basic witness configuration, I have annotated it with comments to
+explain what you need to put in.
+
+It is confusing, but `.run` is very different to `.start`. The former initialises 
+a newly built container and starts it up, the latter starts an already initialised
+container again after you have stopped it. It will complain if you try to `.run`
+after `.run` has already been issued, and will only work if you `.stop` and `.rm`
+to remove the container. Then you will have to `.build` again. This is what you
+want to do if you edit the `Dockerfile`.
+
+`.stop` will stop a running container, `.start` will start it again. `.log` will
+show you the log file output, which is created in `data/steemd.sh` and is there
+so that other scripts can monitor and parse the log file. If you use the 
+`sudo docker <image name> logs` file instead, you will see a notice that the
+logs are going to a log file and how to view them as they come down.
+
+`.enter` will let you run a shell inside the container and poke around if you want
+to do that (any changes will be lost on a `.stop;.rm;` though the contents of 
+`/work` are persistent outside of the container.
+
+The advantage of this arrangement is that even if you delete the Docker
+container (not this folder), you won't lose the block_log and configuration files.
+
+The data directory contains a volume that is mounted inside the container as `/work`
+which I have done so you can easily import `block_log` files or even the whole
+`blockchain` folder inside `witness_node_data_dir`. However, note that this by default
+will build v0.17.1 so an older `shared_memory.*` file will not work.
+
+Lastly, in my `init.sh` script you also have two commands, one, `.editsh` lets you
+edit the `init.sh` file (after you first run it from the `dkr/` directory it will
+remember even if you edit it where it is) and the other is `.editdkr` which will
+let you edit the `Dockerfile`.
+
+There is more commands now available in there, but the 'halp' command will provide enough
+information to use most of them.
+
+There is now a monitor script that can be used, and there is a command in the `init.sh`
+that launches it inside a screen session detached. Copy or rename the `config.example`
+and `config.py.example` to the same names without the `.example` ending and the monitor
+script will have all the details. Some of the parameters are unneccessary in the current
+configuration but I intend to add more functions that will use them (such as a manual
+toggle between your primary and secondary witness).
+
+There is also a price feed setter script that draws its numbers from coinmarketcap,
+it also draws on the the same configurations, and note that the `init.sh` brings the
+environment variables set in `config` into your environment along with all the aliases
+to make managing a witness is nice and easy.
+
+This is a super simple, minimalistic docker container, designed in the way that *I* do
+things, which is usually different to other people. I think it is simpler and more
+efficient.
+
+The configuration of the Gentoo system inside the container is built with `-Os` 
+compiler optimisation flag enabled, so that most of the support libraries are probably
+smaller and faster than the ones that the host system would use. Neither Boost nor 
+steemd is optimised this way, however. I may amend this in the future, but for
+now, this container is fully working and has a side benefit that following the 
+commands inside the Dockerfile you can also build `steemd` to run on a Gentoo server.
+
+## Prerequisites
+
+The accessory scripts in this package assume a number of prerequisites, which are mostly
+generally already present. The exceptions are the python related stuff (not necessary for
+the basic witness operation but for the feeder and monitor they are required)
+
+For Ubuntu to install these prerequisites:
+
+    sudo apt install python3 python3-pip screen
+    sudo -H pip3 install asyncio requests piston-lib websockets
+
+For other distros, like Arch linux, the information will be added in the near future.

_config.yml

@@ -0,0 +1 @@
+theme: jekyll-theme-hacker

config.example

@@ -0,0 +1,38 @@
+#!/bin/bash
+### Configuration for l0k1's compleat witness management system
+
+# witness account name
+export WITNESSNAME=""
+
+# witness account private key
+export WIF=""
+
+# primary witness private key
+export PRIMARYPRIVKEY=""
+
+# primary witness public key
+export PRIMARYPUBKEY=""
+
+# secondary witness private key
+export SECONDARYPRIVKEY=""
+
+# secondary witness public key
+export SECONDARYPUBKEY=""
+
+# address of primary witness
+export PRIMARYURL=""
+
+# address of secondary witness
+export SECONDARYURL=""
+
+# location of log file on remote primary server
+export LOGFILE=""
+
+# port of primary server's SSH
+export PRIMARYPORT=""
+
+# threshold time in seconds since last log entry to trigger switch
+export THRESHOLD=45
+
+# witness thread for update_witness
+export WITNESSTHREAD=""

config.py.example

@@ -0,0 +1,17 @@
+import os
+# switch.py is invoked by monitor.sh which imports the 'config' file,
+# which contains the values below that are also used from 'config'
+# so these sensitive values are all in one place.     
+# The .gitignore by default will not upload the 'config' or this
+# file but only this example file.
+
+owner = os.environ [ 'WITNESSNAME' ] # Witness account username
+witnessthread = os.environ [ 'WITNESSTHREAD' ] # Witness announcement post
+account_creation_fee = "" # eg: '10.000 STEEM'
+maximum_block_size = "" # eg: '65536'
+sbd_interest_rate = "" # eg: '0000'
+fee = "" # eg: '10.000 STEEM'
+block_signing_public_key = os.environ [ 'SECONDARYPUBKEY' ]
+wif = os.environ [ 'WIF' ]
+quote = "" # eg: '1.010'
+node = "" # eg: 'wss://node.steem.ws'

data/witness_node_data_dir/config.ini.example

@@ -0,0 +1,36 @@
+seed-node = seed.riversteem.com:2001       # riverhead
+seed-node = steem-seed1.abit-more.com:2001 # abit
+seed-node = 52.74.152.79:2001              # smooth.witness
+seed-node = seed.steemd.com:34191          # roadscape
+seed-node = steemwitness.matthewniemerg.com:2001 # complexring
+seed-node = steemd.pharesim.me:2001        # pharesim
+seed-node = seed.jesta.us:2001             # jesta
+seed-node = 212.117.213.186:2016           # liondani
+seed-node = anyx.co:2001                   # anyx
+seed-node = seed.xeldal.com:12150          # xeldal
+seed-node = seed.steemnodes.com:2001       # wackou
+seed-node = steem.clawmap.com:2001         # steempty
+seed-node = gtg.steem.house:2001           # gtg
+seed-node = 192.99.3.29:2001               # joseph
+seed-node = 5.9.18.213:2001                # pfunk
+seed-node = lafonasteem.com:2001           # lafona
+seed-node = seed.rossco99.com:2001         # rossco99
+seed-node = 212.47.249.84:40696            # ihashfury
+seed-node = seed.steemfeeder.com:2001      # au1nethyb1
+seed-node = 52.175.211.168:2001            # aizensou
+seed-node = seed.roelandp.nl:2001          # roelandp
+seed-node = 81.89.101.133:2001             # cyrano.witness
+seed-node = steem.global:2001              # klye
+seed-node = seed.esteem.ws:2001            # good-karma
+
+shared-file-size = 20G
+
+enable-plugin = witness 
+
+# name of account in quotes
+witness = ""
+
+# primary (no quotes required for keys, two slots for a primary and secondary)
+private-key = 
+# secondary
+#private_key = 

dirtycache.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+    echo    75 | sudo tee /proc/sys/vm/dirty_background_ratio
+    echo  1000 | sudo tee /proc/sys/vm/dirty_expire_centisecs
+    echo    80 | sudo tee /proc/sys/vm/dirty_ratio
+    echo 30000 | sudo tee /proc/sys/vm/dirty_writeback_centisecs

dkr/Dockerfile

@@ -0,0 +1,61 @@
+FROM gentoo/stage3-amd64
+
+RUN cd /usr ; wget -c http://distfiles.gentoo.org/snapshots/portage-latest.tar.xz;\
+    tar xvf portage-latest.tar.xz; rm portage-latest.tar.xz
+
+RUN emerge --sync
+
+# Set compiler optimisation for minimising code size
+RUN sed -i 's/CFLAGS=\"-O2 -pipe\"/CFLAGS=\"-Os -pipe\"/g' /etc/portage/make.conf
+
+# Enable the elliptic curve functions in openssl
+RUN sed -i 's/bindist/-bindist context python tools/g' /etc/portage/make.conf
+
+RUN emerge --update world
+
+RUN emerge -v openssl:0 openssh
+
+RUN emerge dev-vcs/git 
+RUN emerge autoconf 
+RUN emerge cmake 
+RUN emerge libtool 
+RUN emerge make 
+RUN emerge pkg-config 
+RUN emerge app-arch/bzip2
+RUN emerge doxygen 
+RUN emerge ncurses 
+RUN emerge readline 
+RUN emerge perl 
+RUN emerge python
+
+# Build Boost from version 1.57.0 snapshot
+RUN cd /root; BOOST_ROOT=/usr/local; \
+    URL='http://sourceforge.net/projects/boost/files/boost/1.57.0/boost_1_57_0.tar.bz2/download'; \
+    wget -c "$URL" -O boost_1_57_0.tar.bz2; \
+    [ $( sha256sum boost_1_57_0.tar.bz2 | cut -d ' ' -f 1 ) == \
+      "910c8c022a33ccec7f088bd65d4f14b466588dda94ba2124e78b8c57db264967" ] \
+      || ( echo 'Corrupt download' ; exit 1 ); \
+    tar xjf boost_1_57_0.tar.bz2; \
+    cd boost_1_57_0; \
+    ./bootstrap.sh "--prefix=$BOOST_ROOT"; \
+    ./b2 -j$(nproc) install; \
+    cd ..\
+    rm -rf boost* \
+    exit 0
+
+RUN echo
+
+RUN cd /root; git clone https://github.com/steemit/steem
+RUN cd /root/steem; git submodule update --init --recursive
+RUN cd /root/steem; mkdir build
+RUN cd /root/steem/build; cmake -DCMAKE_BUILD_TYPE=Release ..
+RUN cd /root/steem/build; make -j$(nproc) steemd
+RUN cd /root/steem/build; make -j$(nproc) cli_wallet
+RUN cd /root/steem/build; make install
+
+RUN emerge zsh
+
+VOLUME /work
+WORKDIR /work                  
+
+CMD steemd 1>>/work/steemd.log 2>>/work/steemd.log

dkr/init.sh

@@ -0,0 +1,54 @@
+#!/bin/bash #NOPRINT
+export NAME="docker-gentoo-steemdrpc" #NOPRINT
+export DATADIR="`dirname $(realpath $(dirname $0))`" #NOPRINT
+source $DATADIR/config #NOPRINT
+echo "l0k1's compleat witness management system v0.1"
+echo
+echo "Loading command aliases... Type 'halp' to see available commands" #NOPRINT
+
+### HALP! How to control your $NAME docker container
+
+alias      dkr="sudo docker"
+         ### [ shortcut to run docker with sudo ]
+alias   .where="echo $DATADIR"
+         ### [ show where the current instance activated by init.sh lives ]
+alias      .cd="cd $DATADIR"
+         ### [ change working directory to instance folder ]
+alias     .run="sudo docker run -v $DATADIR/data:/work -d=true --name $NAME $NAME"
+         ### [ start up the container (after building, to restart. for a'.stop'ed container, use '.start') ]
+alias   .start="sudo docker start $NAME"
+         ### [ start the container that was previously '.stop'ed ]
+alias    .stop="sudo docker stop $NAME"
+         ### [ stop the container, start it again with '.start' ]
+alias  .replay="sudo docker stop $NAME;sudo docker run -v $DATADIR/data:/work -d=true $NAME sh -c 'steemd --replay 1>>/work/steemd.log 2>>/work/steemd.log'"
+         ### [ replay blockchain (for after upgrade, after unclean shutdown or starting from recent block_log) ]
+alias   .steem="sudo docker exec -it $NAME steemd"
+         ### [ start up steemd inside the container attached to current terminal ]
+alias  .status="ps avx|grep steemd|grep -v grep|grep -v zsh|grep -v docker"
+         ### [ display process information about all steemd's running on this server
+alias   .enter="sudo docker exec -it $NAME zsh"
+         ### [ open a shell inside the container ]
+alias     .log="sudo tail -f $DATADIR/data/steemd.log"
+         ### [ show the current output from the primary process in the container ]
+alias   .build="sudo docker build -t $NAME $DATADIR/dkr"
+         ### [ build the container from the Dockerfile ]
+alias      .rm="sudo docker rm $NAME"
+         ### [ remove the current container (for rebuilding) ]
+alias .editdkr="nano $DATADIR/dkr/Dockerfile"
+         ### [ edit the Dockerfile ]
+alias  .editsh="nano $DATADIR/dkr/init.sh;source $DATADIR/dkr/init.sh"
+         ### [ edit init.sh with nano then reload ]
+alias .editcfg="nano $DATADIR/config"
+         ### [ edit environment variables ]
+alias .editwit="nano $DATADIR/config.py"
+         ### [ edit witness failover configuration ]
+alias .monitor="screen -d -S monitor -m $DATADIR/monitor.sh" 
+         ### [ start up primary witness failover script in a detached screen session ]
+alias  .feeder="screen -d -S feeder -m $DATADIR/feeder.py"
+         ### [ start up feed setter script in a detached screen session ]
+alias  .screen="screen -r" # monitor or feeder <
+         ### [ view feeder or monitor, name in the parameter. Ctrl-A then D to exit, Ctrl-C to kill process ]
+alias   .dirty="$DATADIR/dirtycache.sh"
+         ### [ set kernel disk cache parameters to decrease disk I/O ]
+alias  halp="sed 's/\$NAME/$NAME/g' $DATADIR/dkr/init.sh|sed 's#\$DATADIR#$DATADIR#g'|grep -v NOPRINT|sed 's/alias //g'|sed 's/=\"/     \"/g'|sed 's/#/>/g'|less"
+######### hit the 'q' key to exit help viewer <<<<<<<<<

feeder.py

@@ -0,0 +1,57 @@
+#!/usr/bin/python3
+import sys, os
+import random
+from pprint import pprint
+import requests
+import simplejson as json
+import asyncio
+import websockets
+import time
+from pistonbase import operations
+from collections import OrderedDict
+from piston.transactionbuilder import TransactionBuilder
+from graphenebase import base58
+from piston import Steem
+
+import config
+
+st = Steem ( keys = [ config.wif ], node = config.node )
+
+old_price = "999999.999"
+
+while True:
+  print ( "[" + str ( time.time() ) + "] Checking coinmarketcap..." )
+
+  r_cmc = requests.get ( "https://api.coinmarketcap.com/v1/ticker/" )
+
+  result_cmc = json.loads ( r_cmc.text ) 
+
+  for i in result_cmc:
+    if ( i [ 'id' ] == 'steem' ):
+      price_usd = i [ 'price_usd' ]
+      formatted_price = "{0:.3f}".format ( float ( i [ 'price_usd' ] ) )
+      if ( formatted_price != old_price ):
+        print ( "Setting price feed to: " + formatted_price )
+        tx = TransactionBuilder ()
+        tx.appendOps (
+          operations.Feed_publish (
+            **{ "publisher": config.owner,
+                "exchange_rate": {
+                  "base": formatted_price + " SBD",
+                  "quote": config.quote + " STEEM"
+                  }
+              }
+            )
+          )
+        tx.appendSigner ( config.owner, "active" )
+        tx.sign()
+        try:
+          tx.broadcast ()
+          print ( "Successfully set feed to:" )
+          print ( "base = " + formatted_price )
+          print ( "quote = " + config.quote + " STEEM" )
+        except:
+          print ( "Transaction broadcast failed" )
+        old_price = formatted_price
+  print ("Pausing for around 1 hour")
+  time.sleep ( int ( random.random() * 7200 ) )

monitor.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+source config
+
+PINGFAIL=0
+
+while true
+do
+  if ! ping -c1 $PRIMARYURL >/dev/null
+  then
+    PINGFAIL=$(( PINGFAIL + 1 ))
+    echo "`date +%Y-%m-%d_%H:%M:%S` Witness server not responding to ping. Failover after 25 seconds"
+    if (( $PINGFAIL > 5 ))
+    then
+      echo "`date +%Y-%m-%d_%H:%M:%S` Witness server not responding to ping for 25 seconds. Switching to secondary"
+      ./switch.py
+      exit 1
+    fi
+  else
+    PINGFAIL=0
+    if (( (( `date +%s` - `ssh -p $PRIMARYPORT $PRIMARYURL date +%s -r $LOGFILE` )) > $THRESHOLD ))
+    then
+      echo "`date +%Y-%m-%d_%H:%M:%S` Log file is more than $THRESHOLD seconds old. Switching to secondary"
+      ./switch.py
+      exit 1
+    else
+      echo "`date +%Y-%m-%d_%H:%M:%S` Witness server up and log less than $THRESHOLD seconds old"
+    fi
+  fi
+  sleep 5
+done

switch.py

@@ -0,0 +1,44 @@
+#!/usr/bin/python3
+#
+# witness toggle
+#
+# Configuration requires environment variables pulled in by config.py
+# monitor.sh script pulls these in from 'config' and config.py takes
+# the ones needed for this script
+
+import asyncio
+import websockets
+import time
+import sys
+import random
+import os
+from pistonbase import operations
+from collections import OrderedDict
+from piston.transactionbuilder import TransactionBuilder
+from graphenebase import base58
+from piston import Steem
+
+import config
+
+st = Steem ( keys = config.wif )
+
+tx = TransactionBuilder ()
+tx.appendOps (
+  operations.Witness_update (
+    **{ "owner": config.owner,
+        "url": config.witnessthread,
+        "block_signing_key": config.block_signing_public_key,
+        "props": { "account_creation_fee": config.account_creation_fee,
+                   "maximum_block_size": config.maximum_block_size,
+                   "sbd_interest_rate": config.sbd_interest_rate},
+         "fee": config.fee,
+      }
+    )
+  )
+tx.appendSigner ( config.owner, "active" )
+tx.sign()
+try:
+  tx.broadcast ()
+  print ( "Successfully switched to secondary witness" )
+except:
+  print ( "Transaction broadcast failed" )