6.6 KiB
The idea is to deploy mirrors across dn42 using anycast addressing (BGP), thus providing redundancy, load-balancing and improved access times to the site. The local webserver is monitored with a simple shell script working in conjunction with ExaBGP, announcing/withdrawing the assigned route if the service is up/down.
Network
- Install wiki anycast address
172.23.0.80/32on the system - Setup tunnel(s) to the dn42 network (routing daemon not required)
Setup gollum
-
Install gollum
-
Clone the dn42 wiki repo:
git clone ssh://git@dn42.us/dn42/wiki <path> -
Contact XUU-DN42 and ask for write access to the repo
-
Setup cron for periodic pull/push jobs for the repo
-
Generate a CSR and send to
xuu@dn42.us. Wait for a reply containing internal.dn42/wiki.dn42 certificates. -
Start two gollum instances, read-only and read/write on
127.0.0.1:Read/write (SSL only):
gollum --css /custom.css --gollum-path --host 127.0.0.1 --port 4568
Read-only:
gollum --css /custom.css --gollum-path --host 127.0.0.1 --port 4567 --no-edit
```
Nginx proxy
Header
A custom header X-SiteID identifies the site you're connecting to:
- X-SiteID:
AS number-ISO country code
Config example
ssl_protocols TLSv1.2 TLSv1.1 TLSv1;
ssl_session_cache shared:SSL:2m;
ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA;
ssl_prefer_server_ciphers on;
upstream wiki { server 127.0.0.1:4567; }
server {
server_name internal.dn42 wiki.dn42;
listen 172.23.0.80:80 default;
add_header strict-transport-security "max-age=0; includeSubDomains";
add_header X-SiteID '<aut-num>-<cc>';
location / {
proxy_pass http://wiki;
}
}
upstream wikirw { server 127.0.0.1:4568; }
server {
server_name internal.dn42 wiki.dn42;
listen 172.23.0.80:443 ssl default;
ssl on;
ssl_certificate <path>/ssl.crt;
ssl_certificate_key <path>/ssl.key;
add_header strict-transport-security "max-age=0; includeSubDomains";
add_header Public-Key-Pins 'pin-sha256="mJ1xUCzfru8Ckq2+M6VkNKGOGgSETImRAHBF24mjalw="; pin-sha256="/gOyi7syRMR+d2jZoB/MzcSD++8ciZkSl/hZAQgzWws="; max-age=0; includeSubDomains';
add_header X-SiteID '<aut-num>-<cc>';
location / {
proxy_pass http://wikirw;
}
}
ExaBGP
Announcing
The prefix AS-PATH should show the announcement is originating from your AS. After peering ExaBGP to the nearest speaker(s), check if the prefix is routing properly inside your network. Try not to blackhole the passing traffic (e.g. no static routes to 172.23.0.80/28). Test the whole thing by shutting down nginx/gollum and watch what happens.
Configuration
# exabgp.conf
group gollum-watchdog {
neighbor <peer1> {
router-id x.x.x.x;
local-address <source-address>;
local-as <ownas>;
peer-as <peeras>;
}
## (example) peer with one of our iBGP speakers:
neighbor <172.22.0.1> {
router-id 172.23.0.80;
local-address 172.22.0.2;
local-as 123456;
peer-as 123456;
}
## ...
process watch-gollum {
run <path>/gollum-watchdog.sh;
}
}
Watchdog script
Watchdog runs in an infinite loop, sending the appropriate commands to stdout. ExaBGP attaches to the process' stdout and listens for instructions. Watchdog sends either a route announce or widthdraw.
Run gollum-watchdog.sh in a shell first to validate it's working:
#!/bin/bash
CURL=curl
## url's to check (all listed must be alive to send announce)
URL=( "http://172.23.0.80" "https://172.23.0.80" )
## the anycast route (/28 due to prefix size limits)
ROUTE='172.23.0.80/28'
## the next-hop we'll be advertising to neighbor(s)
NEXTHOP='<source-address>'
## regex match this keyword against HTTP response from curl
VALIDATE_KEYWORD='gollum'
INTERVAL=60
###########################
RUN_STATE=0
check_urls() {
for url in "${URL[@]}"; do
## workaround curl errno 23 when piping
http_response=`${CURL} --insecure -L -o - "${url}"`
echo "${http_response}" | egrep -q "${VALIDATE_KEYWORD}" || {
return 1
}
## add more checks
done
return 0
}
while [ 1 ]; do
if [ ${RUN_STATE} -eq 0 ]; then
check_urls && {
RUN_STATE=1
echo "announce route ${ROUTE} next-hop ${NEXTHOP}"
}
else
check_urls || {
RUN_STATE=0
echo "withdraw route ${ROUTE} next-hop ${NEXTHOP}"
}
fi
sleep ${INTERVAL}
done
exit 0
Run
Normally SIGUSR1 to the exabgp process triggers a configuration update, but at occasion the process might need to be restarted - since its gracefull shutdown can be glitchy , this might be a bit difficult. Sending SIGKILL to the child(ren) and immediately after, the parent, does the job (quick-and-dirty).
USAGE: /etc/exabgp/run.sh [start|stop|restart]
#!/bin/bash
PID_FILE=/var/run/exaBGP/exabgp_PID
######################################
EXABGP=<path>/sbin/exabgp
EXA_LOG=/var/log/exabgp.log
CONF=/etc/exabgp/exabgp.conf
start() {
[ -f ${PID_FILE} ] && {
echo "WARNING: `cat ${PID_FILE}`: exabgp already running"; return 1
}
${EXABGP} ${CONF} &> ${EXA_LOG} &
cpid=$!
[ ${cpid} -eq 0 ] && {
echo "ERROR: could not start process"; return 1
}
echo $! > ${PID_FILE}
}
stop(){
[ -f ${PID_FILE} ] || return 1
pkill -9 -P $(cat ${PID_FILE})
kill -9 $(cat ${PID_FILE})
rm -f ${PID_FILE}
}
case ${1} in
start )
start
;;
stop )
stop
;;
restart )
stop
sleep 1
start
;;
esac
exit 0