wiki/services/Distributed-Wiki.md

The idea is to deploy mirrors across dn42 using [anycast](https://en.wikipedia.org/wiki/Anycast) addressing (BGP), thus providing redundancy, load-balancing and improved access times to the site.
The local webserver is monitored with a simple [[shell script|Distributed-Wiki#exabgp_watchdog-script]] working [[in conjunction|Distributed-Wiki#exabgp]] with [ExaBGP](https://github.com/Exa-Networks/exabgp), announcing/withdrawing the assigned route if the service is up/down.  

## Network

 - Install wiki anycast address `172.23.0.80/32` on the system
 - Setup tunnel(s) to the dn42 network (routing daemon not required)

## Setup gollum

 - Install [gollum](https://github.com/gollum/gollum)
 - Clone the dn42 wiki repo:

    `git clone ssh://git@dn42.us/dn42/wiki <path>`

 - Contact [XUU-DN42](https://io.nixnodes.net?t=person&l=XUU-DN42) and ask for write access to the repo
 - Setup cron for periodic pull/push jobs for the repo
 - Generate a [CSR](/services/Certificate-Authority) and send to `xuu@dn42.us`. Wait for a reply containing internal.dn42/wiki.dn42 certificates.
 - Start two gollum instances, read-only and read/write on `127.0.0.1`:
 
   Read/write (SSL only):
    ```
gollum --css <path>/custom.css --gollum-path <path> --host 127.0.0.1  --port 4568
    ```
   Read-only:
    ```
gollum --css <path>/custom.css --gollum-path <path> --host 127.0.0.1  --port 4567 --no-edit
    ```

## Nginx proxy

#### Header

A custom header `X-SiteID` identifies the site you're connecting to:

  + X-SiteID: `AS number`-`ISO country code`


#### Config example

```
ssl_protocols  TLSv1.2 TLSv1.1 TLSv1;
ssl_session_cache shared:SSL:2m;
 
ssl_ciphers   ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA;

ssl_prefer_server_ciphers   on;

upstream wiki { server 127.0.0.1:4567; }

server { 
        server_name internal.dn42 wiki.dn42;

        listen 172.23.0.80:80 default;

        add_header strict-transport-security  "max-age=0; includeSubDomains";
        add_header X-SiteID                   '<aut-num>-<cc>';

        location / {
                proxy_pass http://wiki;
        }
}

upstream wikirw { server 127.0.0.1:4568; }


server { 
        server_name internal.dn42 wiki.dn42;

        listen 172.23.0.80:443 ssl default;

	ssl on;
        ssl_certificate      <path>/ssl.crt;  
        ssl_certificate_key  <path>/ssl.key;

        add_header strict-transport-security  "max-age=0; includeSubDomains";
        add_header Public-Key-Pins            'pin-sha256="mJ1xUCzfru8Ckq2+M6VkNKGOGgSETImRAHBF24mjalw="; pin-sha256="/gOyi7syRMR+d2jZoB/MzcSD++8ciZkSl/hZAQgzWws="; max-age=0; includeSubDomains';
        add_header X-SiteID                   '<aut-num>-<cc>';

        location / {
                proxy_pass http://wikirw;
        }
}

```

## [ExaBGP](https://github.com/Exa-Networks/exabgp)

#### Announcing

The prefix AS-PATH should show the announcement is originating from your AS. After peering ExaBGP to the nearest speaker(s), check if the prefix is routing properly inside your network. Try not to blackhole the passing traffic (e.g. no static routes to `172.23.0.80/28`). Test the whole thing by shutting down nginx/gollum and watch what happens.

#### Configuration

```
# exabgp.conf

group gollum-watchdog {
  neighbor <peer1> {
    router-id x.x.x.x;
    local-address <source-address>;
    local-as <ownas>;
    peer-as <peeras>;
  }

  ## (example) peer with one of our iBGP speakers:
  neighbor <172.22.0.1> {
    router-id 172.23.0.80;
    local-address 172.22.0.2;
    local-as 123456;
    peer-as 123456;
  }

  ## ...

  process watch-gollum {
     run <path>/gollum-watchdog.sh;
  }
}

```

#### Watchdog script

Watchdog runs in an infinite loop, sending the appropriate commands to stdout. [ExaBGP](https://github.com/Exa-Networks/exabgp) attaches to the process' stdout and listens for instructions. Watchdog sends either a route announce or widthdraw.

Run `gollum-watchdog.sh` in a shell first to validate it's working:

```sh
#!/bin/bash

CURL=curl

## url's to check (all listed must be alive to send announce)
URL=( "http://172.23.0.80" "https://172.23.0.80" )

## the anycast route (/28 due to prefix size limits)
ROUTE='172.23.0.80/28'

## the next-hop we'll be advertising to neighbor(s)
NEXTHOP='<source-address>' 

## regex match this keyword against HTTP response from curl
VALIDATE_KEYWORD='gollum'

INTERVAL=60

###########################

RUN_STATE=0

check_urls() {
	for url in "${URL[@]}"; do

                ## workaround curl errno 23 when piping
                http_response=`${CURL} --insecure -L -o - "${url}"`

		echo "${http_response}" | egrep -q "${VALIDATE_KEYWORD}" || {
			return 1
		}

                ## add more checks

	done
	return 0
}

while [ 1 ]; do
	if [ ${RUN_STATE} -eq 0 ]; then
		check_urls && {
			RUN_STATE=1
			echo "announce route ${ROUTE} next-hop ${NEXTHOP}"
		}
	else
		check_urls || {
			RUN_STATE=0
			echo "withdraw route ${ROUTE} next-hop ${NEXTHOP}"
		}
	fi

	sleep ${INTERVAL}

done

exit 0


```

#### Run

  Normally SIGUSR1 to the exabgp process triggers a configuration update, but at occasion the process might need to be restarted - since its gracefull shutdown can be glitchy , this might be a bit difficult. Sending SIGKILL to the child(ren) and immediately after, the parent, does the job (quick-and-dirty). 

`USAGE: /etc/exabgp/run.sh [start|stop|restart]`

```sh
#!/bin/bash

PID_FILE=/var/run/exaBGP/exabgp_PID

######################################

EXABGP=<path>/sbin/exabgp
EXA_LOG=/var/log/exabgp.log
CONF=/etc/exabgp/exabgp.conf


start() {
	[ -f ${PID_FILE} ] && {
		echo "WARNING: `cat ${PID_FILE}`: exabgp already running"; return 1
	}
	${EXABGP} ${CONF} &> ${EXA_LOG} &
	cpid=$!
	[ ${cpid} -eq 0 ] && {
		echo "ERROR: could not start process"; return 1
		
	}
        echo $! > ${PID_FILE}
}

stop(){
	[ -f ${PID_FILE} ] || return 1 
	pkill -9 -P $(cat ${PID_FILE})
        kill -9  $(cat ${PID_FILE})
	rm -f ${PID_FILE}
}

case ${1} in
    start )
	start
    ;;
    stop )
	stop
   ;;
    restart )
	stop
	sleep 1
	start
   ;;
esac

exit 0

```