Initial Commit

API.md

@@ -0,0 +1,83 @@
+# dn42regsrv API Description
+
+## GET /<file>
+
+If the StaticRoot configuration option points to a readable directory, files from
+the directory will be served under /
+
+The git repository contains a sample StaticRoot directory with a simple registry
+explorer web app.
+
+## GET /api/registry/
+
+Returns a JSON object, with keys for each registry type and values containing a count
+of the number of registry objects for each type.
+
+Example:
+```
+http://localhost:8042/api/registry/
+
+# sample output
+{"as-block":8,"as-set":34,"aut-num":1482,"domain":451,"inet6num":744,"inetnum":1270,"key-cert":7,"mntner":1378,"organisation":275,"person":1387,"registry":4,"role":14,"route":886,"route-set":2,"route6":594,"schema":18,"tinc-key":25,"tinc-keyset":3}
+```
+
+
+## GET /api/registry/<type>?match
+
+Returns a JSON object listing all objects for the matched types.
+
+Keys for the returned object are registry types, the value for each type is an
+array of object names
+
+If the match parameter is provided, the <type> is substring matched against
+all registry types, otherwise an exact type name is required.
+
+A special type of '*' returns all types and objects in the registry.
+
+Example:
+```
+http://localhost:8042/api/registry/aut-num      # list aut-num objects
+http://localhost:8042/api/registry/*            # list all types and objects
+http://localhost:8042/api/registry/route?match  # list route and route6 objects
+
+# sample output
+{"role":["ALENAN-DN42","FLHB-ABUSE-DN42","ORG-SHACK-ADMIN-DN42","PACKETPUSHERS-DN42","CCCHB-ABUSE-DN42","ORG-NETRAVNEN-DN42","ORG-SHACK-ABUSE-DN42","MAGLAB-DN42","NIXNODES-DN42","SOURIS-DN42","CCCKC-DN42","NL-ZUID-DN42","ORG-SHACK-TECH-DN42","ORG-YANE-DN42"]}
+
+```
+
+## GET /api/registry/<type>/<object>?match&raw
+
+Return a JSON object with the registry data for each matching object.
+
+The keys for the object are the object paths in the form <type>/<object name>. The values depends on the raw parameter.
+
+if the raw parameter is provided, the returned object consists of a single key 'Attributes'
+which will be an array of key/value pairs exactly as held within the registry.
+
+If the raw parameter is not provided, the returned Attributes are decorated with markdown
+style links depending the relations defined in the DN42 schema. In addition a
+'Backlinks' key is added which provides an array of registry objects that
+reference this one.
+
+If the match parameter is provided, the <object> is substring matched against all
+objects in the <type>. Matching is case insensitive.
+
+If the match parameter is not provided, an exact, case sensitive object name is required.
+
+A special object of '*' returns all objects in the type
+
+Example:
+```
+http://localhost:8042/api/registry/domain/burble.dn42?raw # return object in raw format
+http://localhost:8042/api/registry/mntner/BURBLE-MNT      # return object in decorated format
+http://localhost:8042/api/registry/aut-num/2601?match     # return all aut-num objects matching 2601
+http://localhost:8042/api/registry/schema/*               # return all schema objects
+
+# sample output (raw)
+{"domain/burble.dn42":[["domain","burble.dn42"],["descr","burble.dn42 https://dn42.burble.com/"],["admin-c","BURBLE-DN42"],["tech-c","BURBLE-DN42"],["mnt-by","BURBLE-MNT"],["nserver","ns1.burble.dn42 172.20.129.161"],["nserver","ns1.burble.dn42 fd42:4242:2601:ac53::1"],["ds-rdata","61857 13 2 bd35e3efe3325d2029fb652e01604a48b677cc2f44226eeabee54b456c67680c"],["source","DN42"]]}
+
+# sample output (decorated)
+{"mntner/BURBLE-MNT":{"Attributes":[["mntner","BURBLE-MNT"],["descr","burble.dn42 https://dn42.burble.com/"],["admin-c","[BURBLE-DN42](person/BURBLE-DN42)"],["tech-c","[BURBLE-DN42](person/BURBLE-DN42)"],["auth","pgp-fingerprint 1C08F282095CCDA432AECC657B9FE8780CFB6593"],["mnt-by","[BURBLE-MNT](mntner/BURBLE-MNT)"],["source","[DN42](registry/DN42)"]],"Backlinks":["as-set/AS4242422601:AS-DOWNSTREAM","as-set/AS4242422601:AS-TRANSIT","inetnum/172.20.129.160_27","person/BURBLE-DN42","route/172.20.129.160_27","inet6num/fd42:4242:2601::_48","mntner/BURBLE-MNT","aut-num/AS4242422601","aut-num/AS4242422602","route6/fd42:4242:2601::_48","domain/collector.dn42","domain/burble.dn42"]}}
+
+```
+

README.md

@@ -0,0 +1,66 @@
+# dn42regsrv
+
+A REST API for the DN42 registry, written in Go, to provide a bridge between
+interactive applications and the DN42 registry.
+
+## Features
+
+- REST API for querying DN42 registry objects
+- Able to decorate objects with relationship information based on SCHEMA type definitions
+- Includes a simple webserver for delivering static files which can be used to deliver
+  basic web applications utilising the API (such as the included DN42 Registry Explorer)
+- Automatic pull from the DN42 git repository to keep the registry up to date
+- Included responsive web app for exploring the registry
+
+## Building
+
+Requires [git](https://git-scm.com/) and [go](https://golang.org)
+
+```
+go get https://git.dn42.us/burble/dn42regsrv
+```
+
+## Running
+
+Use --help to view configurable options
+```
+./dn42regsrv --help
+```
+
+The server requires access to a clone of the DN42 registry and for the git executable
+to be accessible.  
+If you want to use the auto pull feature then the registry must
+also be writable by the server.
+
+```
+cd ${GOROOT}/src/dn42regsrv
+git clone http://git.dn42.us/dn42/registry.git
+./dn42regsrv --help
+./dn42regsrv
+```
+
+A sample service file is included for running the server under systemd
+
+## Using
+
+By default the server will be listening on port 8042.  
+See the [API.md](API.md) file for a detailed description of the API.
+
+
+## Support
+
+Please feel free to raise issues or create pull requests for the project git repository.
+
+## #ToDo
+
+### Server
+
+- Add WHOIS interface
+- Add endpoints for ROA data
+- Add attribute searches
+
+### DN42 Registry Explorer Web App
+
+- Add search history and fix going back
+- Allow for attribute searches
+

StaticRoot/explorer.js

@@ -0,0 +1,245 @@
+//////////////////////////////////////////////////////////////////////////
+// DN42 Registry Explorer
+//////////////////////////////////////////////////////////////////////////
+
+//////////////////////////////////////////////////////////////////////////
+// registry-stats component
+
+Vue.component('registry-stats', {
+    template: '#registry-stats-template',
+    data() {
+        return {
+            state: "loading",
+            error: "",
+            types: null,
+        }
+    },
+    methods: {
+        updateSearch: function(str) {
+            vm.updateSearch(str)
+        },
+        
+        reload: function(event) {
+            this.types = null,
+            this.state = "loading"
+
+            axios
+                .get('/api/registry/')
+                .then(response => {
+                    this.types = response.data
+                    this.state = 'complete'
+                })
+                .catch(error => {
+                    this.error = error
+                    this.state = 'error'
+                    console.log(error)                    
+                })
+        }
+    },
+    mounted() {
+        this.reload()
+    }
+})
+
+//////////////////////////////////////////////////////////////////////////
+// registry object component
+
+Vue.component('reg-object', {
+    template: '#reg-object-template',
+    props: [ 'link' ],
+    data() {
+        return { }
+    },
+    methods: {
+        updateSearch: function(str) {
+            vm.updateSearch(str)
+        }
+    },
+    computed: {
+        rtype: function() {
+            var ix = this.link.indexOf("/")
+            return this.link.substring(0, ix)
+        },
+        obj: function() {
+            var ix = this.link.indexOf("/")
+            return this.link.substring(ix + 1)            
+        }
+    }
+})
+
+//////////////////////////////////////////////////////////////////////////
+// reg-attribute component
+
+Vue.component('reg-attribute', {
+    template: '#reg-attribute-template',
+    props: [ 'content' ],
+    data() {
+        return { }
+    },
+    methods: {
+        isRegObject: function(str) {
+            return (this.content.match(/^\[.*?\]\(.*?\)/) != null)
+        }
+    },
+    computed: {
+        objectLink: function() {
+            reg = this.content.match(/^\[(.*?)\]\((.*?)\)/)
+            return reg[2]
+        },
+        decorated: function() {
+            return anchorme(this.content, {
+                truncate: 40,
+                ips: false,
+                attributes: [ { name: "target", value: "_blank" } ]                
+            })
+        }
+    }
+})
+
+//////////////////////////////////////////////////////////////////////////
+// construct a search URL from a search term
+
+function matchObjects(objects, rtype, term) {
+
+    var results = [ ]
+    
+    for (const obj in objects) {
+        var s = objects[obj].toLowerCase()
+        var pos = s.indexOf(term)
+        if (pos != -1) {
+            if ((pos == 0) && (s == term)) {
+                // exact match, return just this result
+                return [[ rtype, objects[obj] ]]
+            }
+            results.push([ rtype, objects[obj] ])
+        }
+    }
+    
+    return results
+}
+
+
+function searchFilter(index, term) {
+
+    var results = [ ]
+
+    // comparisons are lowercase
+    term = term.toLowerCase()
+
+    // includes a '/' ? search only in that type
+    var slash = term.indexOf('/')
+    if (slash != -1) {
+        var rtype = term.substring(0, slash)
+        var term = term.substring(slash + 1)
+        objects = index[rtype]
+        if (objects != null) {
+            results = matchObjects(objects, rtype, term)
+        }
+    }
+    else {
+        // walk though the entire index
+        for (const rtype in index) {
+            results = results.concat(matchObjects(index[rtype], rtype, term))
+        }
+    }
+
+    return results
+}
+
+//////////////////////////////////////////////////////////////////////////
+// main application
+
+// application data
+var appData = {
+    searchInput: '',
+    searchTimeout: 0,
+    state: '',
+    debug: "",
+    index: null,
+    filtered: null,
+    result: null
+}
+
+// methods
+var appMethods = {
+
+    loadIndex: function(event) {
+        axios
+            .get('/api/registry/*')
+            .then(response => {
+                this.index = response.data
+            })
+            .catch(error => {
+                // what to do here ?
+                console.log(error)
+            })
+    },
+
+    // called on every search input change
+    debounceSearchInput: function(value) {
+        
+        if (this.search_timeout) {
+            clearTimeout(this.search_timeout)
+        }
+
+        // reset if searchbox is empty
+        if (value == "") {
+            this.state = ""
+            this.searchInput = ""
+            this.filtered = null
+            this.results = null
+            return
+        }
+        
+        this.search_timeout =
+            setTimeout(this.updateSearch.bind(this,value),500)
+    },
+
+    // called after the search input has been debounced
+    updateSearch: function(value) {
+        this.searchInput = value
+        this.filtered = searchFilter(this.index, value)
+        if (this.filtered.length == 0) {
+            this.state = "noresults"
+        }
+        else if (this.filtered.length == 1) {
+            this.state = "loading"
+            var details = this.filtered[0]
+
+            query = '/api/registry/' + details[0] + '/' + details[1]
+            
+            axios
+                .get(query)
+                .then(response => {
+                    this.state = 'result'
+                    this.result = response.data
+                })
+                .catch(error => {
+                    this.error = error
+                    this.state = 'error'
+                })
+        }
+        else {
+            this.state = "resultlist"
+            this.result = this.filtered
+        }
+    }
+
+}
+
+
+// intialise Vue instance
+
+var vm = new Vue({
+    el: '#explorer',
+    data: appData,
+    methods: appMethods,
+    mounted() {
+        this.loadIndex()
+    }
+})
+
+
+
+//////////////////////////////////////////////////////////////////////////
+// end of code

StaticRoot/index.html

@@ -0,0 +1,184 @@
+<!doctype html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
+  <link rel="stylesheet" href="bootstrap.min.css">
+  <link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons">
+  <style>
+    .material-icons { display:inline-flex;vertical-align:middle }
+    .table th, .table td { padding: 0.2rem; border: none }
+    pre { margin-bottom: 0px }
+    .regref span {
+    padding: 0.3em 1em 0.3em 1em; margin: 0.4em;
+    white-space: nowrap; display:inline-block;
+    }
+    a { cursor: pointer }
+  </style>
+  <title>DN42 Registry Explorer</title>
+</head>
+<body style="box-shadow: inset 0 2em 10em rgba(0,0,0,0.4); min-height: 100vh">
+<div id="explorer">
+<nav class="navbar navbar-fixed-top navbar-expand-md navbar-dark bg-dark">
+  <form class="form-inline" id="SearchForm">
+    <input v-bind:value="searchInput"
+           v-on:input="debounceSearchInput($event.target.value)"
+           class="form-control-lg" size="30" type="search"
+           placeholder="Search the registry" aria-label="Search"/>
+  </form>
+  <div class="collapse navbar-collapse w-100 ml-auto">
+    <div class="ml-auto"><a class="navbar-brand"
+     href="/">Registry Explorer</a>&nbsp;<a class="pull-right navbar-brand"
+     href="https://dn42.us/"><img src="/dn42_logo.png" width="173"
+     height="60"/></a></div>
+</nav>
+<div style="padding: 1em">
+
+<div v-show="searchInput == ''">
+
+<div class="jumbotron">
+  <h1>DN42 Registry Explorer</h1>
+  <p class="lead">Just start typing in the search box to start searching the registry</p>
+  <hr/>
+  <p>
+  <p>Search Tips</p>
+  <ul>
+    <li>Searches are case independent
+    <li>No need to hit enter, searches will start immediately
+    <li>Prefixing the search by a registry type followed by / will narrow the search to
+      just that type (e.g. <a v-on:click="updateSearch('domain/.dn42')"
+                                          class="text-success">domain/.dn42</a>&nbsp;)
+    <li>Searching for <b>type/</b> will return all the objects for that type (e.g.
+      <a v-on:click="updateSearch('schema/')" class="text-success">schema/</a>&nbsp;)
+    <li>A blank search box will return you to these instructions
+    <li>Searches are made on object names; searching the content of objects
+      is not supported (yet!).
+    <li>Going back (or any search history) is also not supported yet.
+  </ul>
+  <hr/>
+  <p>The registry explorer is a simple web app using
+    <a href="https://git.dn42.us/burble/dn42regsrv">dn42regsrv</a>;
+    a REST API for the DN42 registry built with <a href="https://golang.org/">Go</a>
+</div>    
+  
+<registry-stats/>
+
+</div>
+
+<section v-if="state == 'loading'">
+  <div class="alert alert-info" role="alert">
+    Loading data ...
+  </div>
+</section>
+
+<section v-else-if="state == 'error'">
+  <div class="alert alert-primary clearfix" role="alert">
+    An error recurred whilst retrieving data <button type="button" class="float-right btn btn-warning" v-on:click="reload"><span class="material-icons">refresh</span>&nbsp;Refresh</button>
+  </div>
+</section>  
+
+<section v-else-if="state == 'noresults'">
+  <h2>Searching for "{{ searchInput }}" ...</h2>
+  <div class="alert alert-dark" role="alert">
+    Sorry, no results found
+  </div>
+</section>
+
+<section v-else-if="state == 'resultlist'">
+  <h2>Listing results for "{{ searchInput }}" ...</h2>
+  <div class="container d-flex flex-row flex-wrap">
+    <div style="text-align: center">
+      <span v-for="value in result" style="margin: 0.5em 1em 0.5em 1em; display:inline-block">
+        <reg-object v-bind:link="value[0] + '/' + value[1]"></reg-object>
+        </span>
+      </div>
+  </div>
+</section>
+
+<section v-else-if="state == 'result'">
+  <div v-for="(val, key) in result">
+  <h2><reg-object v-bind:link="key"></reg-object></h2>
+  <div style="padding-left: 2em">
+  <table class="table">
+    <thead>
+      <tr><th scope="col">Key</th><th scope="col">Value</th></tr>
+    </thead>
+    <tbody>
+      <tr v-for="a in val.Attributes">
+        <th scope="row" class="text-primary" style="white-space:nowrap">{{ a[0] }}</th>
+        <td><reg-attribute v-bind:content="a[1]"></reg-attribute></td>
+      </tr>
+    </tbody>
+  </table>
+  </div>
+  <section v-if="val.Backlinks.length != 0">
+    <p>Referenced by</p>
+    <div style="padding-left: 2em">
+    <table class="table">
+      <tbody>
+        <tr v-for="r in val.Backlinks">
+          <td><reg-object v-bind:link="r"></reg-object></td>
+        </tr>
+      </tbody>
+    </table>
+    </div>
+  </section>
+  </div>
+</section>
+
+</div>
+</div>
+<footer class="page-footer font-small">
+<div style="margin-top: 20px; padding: 5px">
+<a href="https://git.dn42.us/burble/dn42regsrv">Source Code</a>.
+Powered by
+<a href="https://getbootstrap.com/">Bootstrap</a>,
+<a href="https://vuejs.org">Vue.js</a>,
+<a href="https://github.com/axios/axios">axios</a>,
+<a href="http://alexcorvi.github.io/anchorme.js/">Anchorme</a>.
+</div>
+</footer>
+</div>
+
+<script type="text/x-template" id="registry-stats-template">
+<div class="container d-flex flex-column w-75">
+  <h5>Registry Stats</h5>
+<section v-if="state == 'error'">
+  <div class="alert alert-primary clearfix" role="alert">
+    An error recurred whilst retrieving data <button type="button" class="float-right btn btn-warning" v-on:click="reload"><span class="material-icons">refresh</span>&nbsp;Refresh</button>
+  </div>
+</section>
+<section v-else-if="state == 'loading'">
+  <div class="alert alert-info" role="alert">
+    Loading data ...
+  </div>
+</section>
+<section v-else>
+  <p style="padding:1em;text-align:center">
+    <span v-for="(value, key) in types" style="margin-left:0.5em;margin-right:0.5em;white-space:nowrap;display:inline-block"><a v-on:click="updateSearch(key + '/')" class="text-success">{{ key }}</a>:&nbsp;<b>{{ value }}</b>&nbsp;records</span>
+  </p>
+</section>
+</div>
+</script>
+
+<script type="text/x-template" id="reg-object-template">
+<span class="regref"><a v-on:click="updateSearch(rtype + '/' + obj)" class="text-success" 
+style="margin-right: 0.4em">{{ obj }}</a>&nbsp;<span 
+class="badge badge-pill badge-dark text-muted">{{ rtype }}</span></span></span>  
+</script>
+
+<script type="text/x-template" id="reg-attribute-template">
+  <span style="word-break: break-all">
+    <reg-object v-if="isRegObject()" v-bind:link="objectLink"></reg-object>
+    <span v-else class="text-monospace" v-html="decorated"></span>
+</span>
+</script>
+
+<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.2.1/js/bootstrap.min.js" integrity="sha384-B0UglyR+jN6CkvvICOB2joaf5I4l3gm9GU6Hc1og6Ls7i6U/mkkaduKaBhlAXv9k" crossorigin="anonymous"></script>
+<script src="https://cdn.jsdelivr.net/npm/vue@2.6.2/dist/vue.js"></script>
+<!-- <script src="https://cdn.jsdelivr.net/npm/vue@2.6.2/dist/vue.min.js"></script> -->
+<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
+<script src="anchorme.min.js"></script>
+<script src="explorer.js"></script>
+</body>
+</html>

dn42regsrv.go

@@ -0,0 +1,169 @@
+//////////////////////////////////////////////////////////////////////////
+// DN42 Registry API Server
+//////////////////////////////////////////////////////////////////////////
+
+package main
+
+//////////////////////////////////////////////////////////////////////////
+
+import (
+	"context"
+	"github.com/gorilla/mux"
+	log "github.com/sirupsen/logrus"
+	flag "github.com/spf13/pflag"
+	"net/http"
+	"os"
+	"os/signal"
+	"time"
+)
+
+//////////////////////////////////////////////////////////////////////////
+// list of API endpoints
+
+type InitEndpointFunc = func(route *mux.Router)
+
+var apiEndpoints = make([]InitEndpointFunc, 0)
+
+func RegisterAPIEndpoint(f InitEndpointFunc) {
+	apiEndpoints = append(apiEndpoints, f)
+}
+
+//////////////////////////////////////////////////////////////////////////
+// utility function to set the log level
+
+func setLogLevel(levelStr string) {
+
+	if level, err := log.ParseLevel(levelStr); err != nil {
+		// failed to set the level
+
+		// set a sensible default and, of course, log the error
+		log.SetLevel(log.InfoLevel)
+		log.WithFields(log.Fields{
+			"loglevel": levelStr,
+			"error":    err,
+		}).Error("Failed to set requested log level")
+
+	} else {
+
+		// set the requested level
+		log.SetLevel(level)
+
+	}
+}
+
+//////////////////////////////////////////////////////////////////////////
+// http request logger
+
+func requestLogger(next http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+
+		log.WithFields(log.Fields{
+			"method": r.Method,
+			"URL":    r.URL.String(),
+			"Remote": r.RemoteAddr,
+		}).Debug("HTTP Request")
+
+		next.ServeHTTP(w, r)
+	})
+}
+
+//////////////////////////////////////////////////////////////////////////
+// everything starts here
+
+func main() {
+
+	// set a default log level, so that logging can be used immediately
+	// the level will be overidden later on once the command line
+	// options are loaded
+	log.SetLevel(log.InfoLevel)
+	log.Info("DN42 Registry API Server Starting")
+
+	// declare cmd line options
+	var (
+		logLevel        = flag.StringP("LogLevel", "l", "Info", "Log level")
+		regDir          = flag.StringP("RegDir", "d", "registry", "Registry data directory")
+		bindAddress     = flag.StringP("BindAddress", "b", "[::]:8042", "Server bind address")
+		staticRoot      = flag.StringP("StaticRoot", "s", "StaticRoot", "Static page directory")
+		refreshInterval = flag.StringP("Refresh", "i", "60m", "Refresh interval")
+		gitPath         = flag.StringP("GitPath", "g", "/usr/bin/git", "Path to git executable")
+		autoPull        = flag.BoolP("AutoPull", "a", true, "Automatically pull the registry")
+		pullURL         = flag.StringP("PullURL", "p", "origin", "URL to auto pull")
+	)
+	flag.Parse()
+
+	// now initialise logging properly based on the cmd line options
+	setLogLevel(*logLevel)
+
+	// parse the refreshInterval and start data collection
+	interval, err := time.ParseDuration(*refreshInterval)
+	if err != nil {
+		log.WithFields(log.Fields{
+			"error":    err,
+			"interval": *refreshInterval,
+		}).Fatal("Unable to parse registry refresh interval")
+	}
+
+	InitialiseRegistryData(*regDir, interval,
+		*gitPath, *autoPull, *pullURL)
+
+	// initialise router
+	router := mux.NewRouter()
+	// log all access
+	router.Use(requestLogger)
+
+	// initialise API routes
+	subr := router.PathPrefix("/api").Subrouter()
+	for _, epInit := range apiEndpoints {
+		epInit(subr)
+	}
+
+	// initialise static routes
+	InstallStaticRoutes(router, *staticRoot)
+
+	// initialise http server
+	server := &http.Server{
+		Addr:         *bindAddress,
+		WriteTimeout: time.Second * 15,
+		ReadTimeout:  time.Second * 15,
+		IdleTimeout:  time.Second * 60,
+		Handler:      router,
+	}
+
+	// run the server in a non-blocking goroutine
+
+	log.WithFields(log.Fields{
+		"BindAddress": *bindAddress,
+	}).Info("Starting server")
+
+	go func() {
+		if err := server.ListenAndServe(); err != nil {
+			log.WithFields(log.Fields{
+				"error":       err,
+				"BindAddress": *bindAddress,
+			}).Fatal("Unable to start server")
+		}
+	}()
+
+	// graceful shutdown via SIGINT (^C)
+	csig := make(chan os.Signal, 1)
+	signal.Notify(csig, os.Interrupt)
+
+	// and block
+	<-csig
+
+	log.Info("Server shutting down")
+
+	// deadline for server to shutdown
+	ctx, cancel := context.WithTimeout(context.Background(), 10)
+	defer cancel()
+
+	// shutdown the server
+	server.Shutdown(ctx)
+
+	// nothing left to do
+	log.Info("Shutdown complete, all done")
+	os.Exit(0)
+}
+
+//////////////////////////////////////////////////////////////////////////
+// end of code

regapi.go

@@ -0,0 +1,267 @@
+//////////////////////////////////////////////////////////////////////////
+// DN42 Registry API Server
+//////////////////////////////////////////////////////////////////////////
+
+package main
+
+//////////////////////////////////////////////////////////////////////////
+
+import (
+	"encoding/json"
+	"github.com/gorilla/mux"
+	log "github.com/sirupsen/logrus"
+	"net/http"
+	"strings"
+	//	"time"
+)
+
+//////////////////////////////////////////////////////////////////////////
+// register the api
+
+func init() {
+	RegisterAPIEndpoint(InitRegAPI)
+}
+
+//////////////////////////////////////////////////////////////////////////
+// called from main to initialise the API routing
+
+func InitRegAPI(router *mux.Router) {
+
+	s := router.
+		Methods("GET").
+		PathPrefix("/registry").
+		Subrouter()
+
+	s.HandleFunc("/", regRootHandler)
+	//s.HandleFunc("/.schema", rTypeListHandler)
+	//s.HandleFunc("/.meta/", rTypeListHandler)
+
+	s.HandleFunc("/{type}", regTypeHandler)
+	s.HandleFunc("/{type}/{object}", regObjectHandler)
+
+	log.Info("Registry API installed")
+}
+
+//////////////////////////////////////////////////////////////////////////
+// handler utility funcs
+
+func responseJSON(w http.ResponseWriter, v interface{}) {
+
+	// for response time testing
+	//time.Sleep(time.Second)
+
+	// marshal the JSON string
+	data, err := json.Marshal(v)
+	if err != nil {
+		log.WithFields(log.Fields{
+			"error": err,
+		}).Error("Failed to marshal JSON")
+	}
+
+	// write back to http handler
+	w.Header().Set("Content-Type", "application/json")
+	w.Write(data)
+}
+
+//////////////////////////////////////////////////////////////////////////
+// root handler, lists all types within the registry
+
+func regRootHandler(w http.ResponseWriter, r *http.Request) {
+
+	response := make(map[string]int)
+	for _, rType := range RegistryData.Types {
+		response[rType.Ref] = len(rType.Objects)
+	}
+	responseJSON(w, response)
+
+}
+
+//////////////////////////////////////////////////////////////////////////
+// type handler returns list of objects that match the type
+
+func regTypeHandler(w http.ResponseWriter, r *http.Request) {
+
+	// request parameters
+	vars := mux.Vars(r)
+	query := r.URL.Query()
+
+	typeName := vars["type"] // type name to list
+	match := query["match"]  // single query or match
+
+	// special case to return all types
+	all := false
+	if typeName == "*" {
+		match = []string{}
+		all = true
+	}
+
+	// results will hold the types to return
+	var results []*RegType
+
+	//	check match type
+	if match == nil {
+		// exact match
+
+		// check the type object exists
+		rType := RegistryData.Types[typeName]
+		if rType == nil {
+			http.Error(w, "No types matching '"+typeName+"' found", http.StatusNotFound)
+			return
+		}
+
+		// return just a single result
+		results = []*RegType{rType}
+
+	} else {
+		// substring match
+
+		// comparisons are lower case
+		typeName = strings.ToLower(typeName)
+
+		// walk through the types and filter to the results list
+		results = make([]*RegType, 0)
+		for key, rType := range RegistryData.Types {
+			if all || strings.Contains(strings.ToLower(key), typeName) {
+				// match found, add to the list
+				results = append(results, rType)
+			}
+		}
+
+	}
+
+	// construct the response
+	response := make(map[string][]string)
+	for _, rType := range results {
+
+		objects := make([]string, 0, len(rType.Objects))
+		for key := range rType.Objects {
+			objects = append(objects, key)
+		}
+
+		response[rType.Ref] = objects
+	}
+
+	responseJSON(w, response)
+}
+
+//////////////////////////////////////////////////////////////////////////
+// object handler returns object data
+
+// per object response structure
+type RegObjectResponse struct {
+	Attributes [][2]string
+	Backlinks  []string
+}
+
+func regObjectHandler(w http.ResponseWriter, r *http.Request) {
+
+	// request parameters
+	vars := mux.Vars(r)
+	query := r.URL.Query()
+
+	typeName := vars["type"]  // object type
+	objName := vars["object"] // object name or match
+	match := query["match"]   // single query or match
+	raw := query["raw"]       // raw or decorated results
+
+	// special case to return all objects
+	all := false
+	if objName == "*" {
+		match = []string{}
+		all = true
+	}
+
+	// verify the type exists
+	rType := RegistryData.Types[typeName]
+	if rType == nil {
+		http.Error(w, "No types matching '"+typeName+"' found",
+			http.StatusNotFound)
+		return
+	}
+
+	// results will hold the objects to return
+	var results []*RegObject
+
+	// check match type
+	if match == nil {
+		// exact match
+
+		// check the object exists
+		object := rType.Objects[objName]
+		if object == nil {
+			http.Error(w, "No objects matching '"+objName+"' found",
+				http.StatusNotFound)
+			return
+		}
+
+		// then just create a results list with one object
+		results = []*RegObject{object}
+
+	} else {
+		// substring matching
+
+		// comparisons are lower case
+		objName = strings.ToLower(objName)
+
+		// walk through the type objects and filter to the results list
+		results = make([]*RegObject, 0)
+		for key, object := range rType.Objects {
+			if all || strings.Contains(strings.ToLower(key), objName) {
+				// match found, add to the list
+				results = append(results, object)
+			}
+		}
+	}
+
+	// collate the results in to the response data
+	if raw == nil {
+		// provide a decorated response
+		response := make(map[string]RegObjectResponse)
+
+		// for each object in the results
+		for _, object := range results {
+
+			// copy the raw attributes
+			attributes := make([][2]string, len(object.Data))
+			for ix, attribute := range object.Data {
+				attributes[ix] = [2]string{attribute.Key, attribute.Value}
+			}
+
+			// construct the backlinks
+			backlinks := make([]string, len(object.Backlinks))
+			for ix, object := range object.Backlinks {
+				backlinks[ix] = object.Ref
+			}
+
+			// add to the response
+			response[object.Ref] = RegObjectResponse{
+				Attributes: attributes,
+				Backlinks:  backlinks,
+			}
+		}
+
+		responseJSON(w, response)
+
+	} else {
+		// provide a response with just the raw registry data
+		response := make(map[string][][2]string)
+
+		// for each object in the results
+		for _, object := range results {
+
+			attributes := make([][2]string, len(object.Data))
+			response[object.Ref] = attributes
+
+			// copy the raw attributes
+			for ix, attribute := range object.Data {
+				attributes[ix] = [2]string{attribute.Key, attribute.RawValue}
+			}
+		}
+
+		responseJSON(w, response)
+	}
+
+}
+
+//////////////////////////////////////////////////////////////////////////
+// end of code

registry.go

@@ -0,0 +1,638 @@
+//////////////////////////////////////////////////////////////////////////
+// DN42 Registry API Server
+//////////////////////////////////////////////////////////////////////////
+
+package main
+
+//////////////////////////////////////////////////////////////////////////
+
+import (
+	"bufio"
+	//	"errors"
+	"fmt"
+	log "github.com/sirupsen/logrus"
+	"io/ioutil"
+	"os"
+	"os/exec"
+	"strings"
+	"time"
+)
+
+//////////////////////////////////////////////////////////////////////////
+// registry data model
+
+// registry data
+
+// Attributes within Objects
+type RegAttribute struct {
+	Key      string
+	Value    string // this is a post-processed, or decorated value
+	RawValue string // the raw value as read from the registry
+}
+
+type RegObject struct {
+	Ref       string          // the ref contains the full path for this object
+	Data      []*RegAttribute // the key/value data for this object
+	Backlinks []*RegObject    // other objects that reference this one
+}
+
+// types are collections of objects
+type RegType struct {
+	Ref     string                // full path for this type
+	Objects map[string]*RegObject // the objects in this type
+}
+
+// registry meta data
+
+type RegAttributeSchema struct {
+	Fields    []string
+	Relations []*RegType
+}
+
+type RegTypeSchema struct {
+	Ref        string
+	Attributes map[string]*RegAttributeSchema
+}
+
+// the registry itself
+
+type Registry struct {
+	Schema map[string]*RegTypeSchema
+	Types  map[string]*RegType
+}
+
+// and a variable for the actual data
+var RegistryData *Registry
+
+// store the current commit has
+var previousCommit string
+
+//////////////////////////////////////////////////////////////////////////
+// utility and manipulation functions
+
+// general functions
+
+func RegistryMakePath(t string, o string) string {
+	return t + "/" + o
+}
+
+// attribute functions
+
+// return a pointer to a RegType from a decorated attribute value
+func (*RegAttribute) ExtractRegType() *RegType {
+	return nil
+}
+
+// object functions
+
+// return attributes exactly matching a specific key
+func (object *RegObject) GetKey(key string) []*RegAttribute {
+
+	attributes := make([]*RegAttribute, 0)
+	for _, a := range object.Data {
+		if a.Key == key {
+			attributes = append(attributes, a)
+		}
+	}
+
+	return attributes
+}
+
+// return a single key
+func (object *RegObject) GetSingleKey(key string) *RegAttribute {
+
+	attributes := object.GetKey(key)
+	if len(attributes) != 1 {
+		log.WithFields(log.Fields{
+			"key":    key,
+			"object": object.Ref,
+		}).Error("Unable to find unique key in object")
+
+		// can't register the object
+		return nil
+	}
+	return attributes[0]
+}
+
+// schema functions
+
+// validate a set of attributes against a schema
+func (schema *RegTypeSchema) validate(attributes []*RegAttribute) []*RegAttribute {
+
+	validated := make([]*RegAttribute, 0, len(attributes))
+	for _, attribute := range attributes {
+
+		// keys beginning with 'x-' are user defined, skip validation
+		if !strings.HasPrefix(attribute.Key, "x-") {
+			if schema.Attributes[attribute.Key] == nil {
+				// couldn't find a schema attribute
+
+				log.WithFields(log.Fields{
+					"key":    attribute.Key,
+					"schema": schema.Ref,
+				}).Error("Schema validation failed")
+
+				// don't add to the validated list
+				continue
+			}
+		}
+
+		// all ok
+		validated = append(validated, attribute)
+	}
+
+	return validated
+}
+
+//////////////////////////////////////////////////////////////////////////
+// reload the registry
+
+func reloadRegistry(path string) {
+
+	log.Debug("Reloading registry")
+
+	// r will become the new registry data
+	registry := &Registry{
+		Schema: make(map[string]*RegTypeSchema),
+		Types:  make(map[string]*RegType),
+	}
+
+	// bootstrap the schema registry type
+	registry.Types["schema"] = &RegType{
+		Ref:     "schema",
+		Objects: make(map[string]*RegObject),
+	}
+	registry.loadType("schema", path)
+
+	// and parse the schema to get the remaining types
+	registry.parseSchema()
+
+	// now load the remaining types
+	for _, rType := range registry.Types {
+		registry.loadType(rType.Ref, path)
+	}
+
+	// mark relationships
+	registry.decorate()
+
+	// swap in the new registry data
+	RegistryData = registry
+}
+
+//////////////////////////////////////////////////////////////////////////
+// create and load the raw data for a registry type
+
+func (registry *Registry) loadType(typeName string, path string) {
+
+	// the type will already have been created
+	rType := registry.Types[typeName]
+
+	// as will the schema (unless attempting to load the schema itself)
+	schema := registry.Schema[typeName]
+
+	// special case for DNS as the directory
+	// doesn't match the type name
+	if typeName == "domain" {
+		path += "/dns"
+	} else {
+		path += "/" + typeName
+	}
+
+	// and load all the objects in this type
+	rType.loadObjects(schema, path)
+
+}
+
+//////////////////////////////////////////////////////////////////////////
+// load all the objects associated with a type
+
+func (rType *RegType) loadObjects(schema *RegTypeSchema, path string) {
+
+	entries, err := ioutil.ReadDir(path)
+	if err != nil {
+		log.WithFields(log.Fields{
+			"error": err,
+			"path":  path,
+			"type":  rType.Ref,
+		}).Error("Failed to read registry type directory")
+		return
+	}
+
+	// for each entry in the directory
+	for _, entry := range entries {
+
+		// each file maps to a registry object
+		if !entry.IsDir() {
+
+			filename := entry.Name()
+			// ignore dotfiles
+			if !strings.HasPrefix(filename, ".") {
+
+				// load the attributes from file
+				attributes := loadAttributes(path + "/" + filename)
+
+				// basic validation of attributes against the schema
+				// schema may be nil if we are actually loading the schema itself
+				if schema != nil {
+					attributes = schema.validate(attributes)
+				}
+
+				// make the object
+				object := &RegObject{
+					Ref:       RegistryMakePath(rType.Ref, filename),
+					Data:      attributes,
+					Backlinks: make([]*RegObject, 0),
+				}
+
+				// add to type
+				rType.Objects[filename] = object
+			}
+		}
+	}
+
+	log.WithFields(log.Fields{
+		"ref":   rType.Ref,
+		"path":  path,
+		"count": len(rType.Objects),
+	}).Debug("Loaded registry type")
+
+}
+
+//////////////////////////////////////////////////////////////////////////
+// read attributes from a file
+
+func loadAttributes(path string) []*RegAttribute {
+
+	attributes := make([]*RegAttribute, 0)
+
+	// open the file to start reading it
+	file, err := os.Open(path)
+	if err != nil {
+		log.WithFields(log.Fields{
+			"error": err,
+			"path":  path,
+		}).Error("Failed to read attributes from file")
+		return attributes
+	}
+	defer file.Close()
+
+	// read the file line by line using the bufio scanner
+	scanner := bufio.NewScanner(file)
+	for scanner.Scan() {
+
+		line := strings.TrimRight(scanner.Text(), "\r\n")
+		runes := []rune(line)
+
+		// lines starting with '+' denote an empty line
+		if runes[0] == rune('+') {
+
+			// concatenate a \n on to the previous attribute value
+			attributes[len(attributes)-1].RawValue += "\n"
+
+		} else {
+
+			// look for a : separator in the first 20 characters
+			ix := strings.IndexByte(line, ':')
+			if ix == -1 || ix >= 20 {
+				// couldn't find one
+
+				if len(runes) <= 20 {
+					// hmmm, the line was shorter than 20 characters
+					// something is amiss
+
+					log.WithFields(log.Fields{
+						"length": len(runes),
+						"path":   path,
+						"line":   line,
+					}).Warn("Short line detected")
+
+				} else {
+
+					// line is a continuation of the previous line, so
+					// concatenate the value on to the previous attribute value
+					attributes[len(attributes)-1].RawValue +=
+						"\n" + string(runes[20:])
+
+				}
+			} else {
+				// found a key and : separator
+
+				// is there actually a value ?
+				var value string
+				if len(runes) <= 20 {
+					// blank value
+					value = ""
+				} else {
+					value = string(runes[20:])
+				}
+
+				// create a new attribute
+				a := &RegAttribute{
+					Key:      string(runes[:ix]),
+					RawValue: value,
+				}
+				attributes = append(attributes, a)
+			}
+		}
+	}
+
+	return attributes
+}
+
+//////////////////////////////////////////////////////////////////////////
+// parse schema files to extract keys and for attribute relations
+
+func (registry *Registry) parseSchema() {
+
+	// for each object in the schema type
+	for _, object := range registry.Types["schema"].Objects {
+
+		// look up the ref attribute
+		ref := object.GetSingleKey("ref")
+		if ref == nil {
+			log.WithFields(log.Fields{
+				"object": object.Ref,
+			}).Error("Schema record without ref")
+
+			// can't process this object
+			continue
+		}
+
+		// create the type schema object
+		typeName := strings.TrimPrefix(ref.RawValue, "dn42.")
+		typeSchema := &RegTypeSchema{
+			Ref:        typeName,
+			Attributes: make(map[string]*RegAttributeSchema),
+		}
+
+		// ensure the type exists
+		rType := registry.Types[typeName]
+		if rType == nil {
+			rType := &RegType{
+				Ref:     typeName,
+				Objects: make(map[string]*RegObject),
+			}
+			registry.Types[typeName] = rType
+		}
+
+		// for each key attribute in the schema
+		attributes := object.GetKey("key")
+		for _, attribute := range attributes {
+
+			// split the value on whitespace
+			fields := strings.Fields(attribute.RawValue)
+			keyName := fields[0]
+
+			typeSchema.Attributes[keyName] = &RegAttributeSchema{
+				Fields: fields[1:],
+			}
+		}
+
+		// register the type schema
+		registry.Schema[typeName] = typeSchema
+
+	}
+
+	// scan the fields of each schema attribute to determine relationships
+	// this needs to be second step to allow pre-creation of the types
+	for _, typeSchema := range registry.Schema {
+		for attribName, attribSchema := range typeSchema.Attributes {
+			for _, field := range attribSchema.Fields {
+				if strings.HasPrefix(field, "lookup=") {
+
+					// the relationships may be a multivalue, separated by ,
+					rels := strings.Split(strings.
+						TrimPrefix(field, "lookup="), ",")
+
+					// map to a regtype
+					relations := make([]*RegType, 0, len(rels))
+					for ix := range rels {
+						relName := strings.TrimPrefix(rels[ix], "dn42.")
+						relation := registry.Types[relName]
+
+						// log if unable to look up the type
+						if relation == nil {
+							// log unless this is the schema def lookup=str '>' [spec]...
+							if typeSchema.Ref != "schema" {
+								log.WithFields(log.Fields{
+									"relation":  relName,
+									"attribute": attribName,
+									"type":      typeSchema.Ref,
+								}).Error("Relation to type that does not exist")
+							}
+
+						} else {
+							// store the relationship
+							relations = append(relations, relation)
+						}
+					}
+
+					// register the relations
+					attribSchema.Relations = relations
+
+					// assume only 1 lookup= per key
+					break
+				}
+			}
+		}
+	}
+
+	log.Debug("Schema parsing complete")
+}
+
+//////////////////////////////////////////////////////////////////////////
+// parse all attributes and decorate them
+
+func (registry *Registry) decorate() {
+
+	cattribs := 0
+	cmatched := 0
+
+	// walk each attribute value
+	for _, rType := range registry.Types {
+		schema := registry.Schema[rType.Ref]
+		for _, object := range rType.Objects {
+			for _, attribute := range object.Data {
+				cattribs += 1
+
+				attribSchema := schema.Attributes[attribute.Key]
+				// are there relations defined for this attribute ?
+				// attribSchema may be null if this attribute is user defined (x-*)
+				if (attribSchema != nil) && attribute.matchRelation(object,
+					attribSchema.Relations) {
+					// matched
+					cmatched += 1
+				} else {
+					// no match, just copy the attribute data
+					attribute.Value = attribute.RawValue
+				}
+			}
+		}
+	}
+
+	log.WithFields(log.Fields{
+		"attributes": cattribs,
+		"matched":    cmatched,
+	}).Debug("Decoration complete")
+
+}
+
+//////////////////////////////////////////////////////////////////////////
+// match an attribute against schema relations
+
+func (attribute *RegAttribute) matchRelation(parent *RegObject,
+	relations []*RegType) bool {
+
+	// it's not going to match if relations is empty
+	if relations == nil {
+		return false
+	}
+
+	// check each relation
+	for _, relation := range relations {
+
+		object := relation.Objects[attribute.RawValue]
+		if object != nil {
+			// found a match !
+
+			// decorate the attribute value
+			attribute.Value = fmt.Sprintf("[%s](%s)",
+				attribute.RawValue, object.Ref)
+
+			// and add a back reference to the related object
+			object.Backlinks = append(object.Backlinks, parent)
+			return true
+		}
+
+	}
+
+	// didn't find anything
+	return false
+}
+
+//////////////////////////////////////////////////////////////////////////
+// fetch the current commit hash
+
+func getCommitHash(regDir string, gitPath string) string {
+
+	// run git to get the latest commit hash
+	cmd := exec.Command(gitPath, "log", "-1", "--format=%H")
+	cmd.Dir = regDir
+	// execute
+	out, err := cmd.Output()
+	if err != nil {
+		log.WithFields(log.Fields{
+			"error":   err,
+			"gitPath": gitPath,
+			"regDir":  regDir,
+		}).Error("Failed to execute git log")
+	}
+
+	return strings.TrimSpace(string(out))
+}
+
+//////////////////////////////////////////////////////////////////////////
+// refresh the registry
+
+func refreshRegistry(regDir string, gitPath string, pullURL string) {
+
+	// run git to get the latest commit hash
+	cmd := exec.Command(gitPath, "pull", pullURL)
+	cmd.Dir = regDir
+	// execute
+	out, err := cmd.Output()
+	if err != nil {
+		log.WithFields(log.Fields{
+			"error":   err,
+			"gitPath": gitPath,
+			"regDir":  regDir,
+			"pullURL": pullURL,
+		}).Error("Failed to execute git log")
+	}
+
+	fmt.Println(string(out))
+}
+
+//////////////////////////////////////////////////////////////////////////
+// called from main to initialse the registry data and syncing
+
+func InitialiseRegistryData(regDir string, refresh time.Duration,
+	gitPath string, autoPull bool, pullURL string) {
+
+	// validate that the regDir/data path exists
+	dataPath := regDir + "/data"
+	regStat, err := os.Stat(dataPath)
+	if err != nil {
+		log.WithFields(log.Fields{
+			"error": err,
+			"path":  dataPath,
+		}).Fatal("Unable to find registry directory")
+	}
+
+	// and it is a directory
+	if !regStat.IsDir() {
+		log.WithFields(log.Fields{
+			"error": err,
+			"path":  dataPath,
+		}).Fatal("Registry path is not a directory")
+	}
+
+	// check that git exists
+	_, err = os.Stat(gitPath)
+	if err != nil {
+		log.WithFields(log.Fields{
+			"error": err,
+			"path":  gitPath,
+		}).Fatal("Unable to find git executable")
+	}
+
+	// enforce a minimum update time
+	minTime := 10 * time.Minute
+	if refresh < minTime {
+		log.WithFields(log.Fields{
+			"interval": refresh,
+		}).Error("Enforcing minimum update time of 10 minutes")
+
+		refresh = minTime
+	}
+
+	// initialise the previous commit hash
+	// and do initial load from registry
+	previousCommit = getCommitHash(regDir, gitPath)
+	reloadRegistry(dataPath)
+
+	go func() {
+
+		// every refresh interval
+		for range time.Tick(refresh) {
+			log.Debug("Refresh Timer")
+
+			// automatically try to refresh the registry ?
+			if autoPull {
+				refreshRegistry(regDir, gitPath, pullURL)
+			}
+
+			// get the latest hash
+			currentCommit := getCommitHash(regDir, gitPath)
+
+			// has the registry been updated ?
+			if currentCommit != previousCommit {
+				log.WithFields(log.Fields{
+					"current":  currentCommit,
+					"previous": previousCommit,
+				}).Info("Registry has changed, refresh started")
+
+				// refresh
+				reloadRegistry(dataPath)
+
+				// update commit
+				previousCommit = currentCommit
+			}
+
+		}
+	}()
+
+}
+
+//////////////////////////////////////////////////////////////////////////
+// end of code

static.go

@@ -0,0 +1,55 @@
+//////////////////////////////////////////////////////////////////////////
+// DN42 Registry API Server
+//////////////////////////////////////////////////////////////////////////
+
+package main
+
+//////////////////////////////////////////////////////////////////////////
+
+import (
+	"github.com/gorilla/mux"
+	log "github.com/sirupsen/logrus"
+	"net/http"
+	"os"
+)
+
+//////////////////////////////////////////////////////////////////////////
+// called from main to initialise the API routing
+
+func InstallStaticRoutes(router *mux.Router, staticPath string) {
+
+	// an empty path disables static route serving
+	if staticPath == "" {
+		log.Info("Disabling static route serving")
+		return
+	}
+
+	// validate that the staticPath exists
+	stat, err := os.Stat(staticPath)
+	if err != nil {
+		log.WithFields(log.Fields{
+			"error": err,
+			"path":  staticPath,
+		}).Fatal("Unable to find static page directory")
+	}
+
+	// and it is a directory
+	if !stat.IsDir() {
+		log.WithFields(log.Fields{
+			"error": err,
+			"path":  staticPath,
+		}).Fatal("Static path is not a directory")
+	}
+
+	// install a file server for the static route
+	router.PathPrefix("/").Handler(http.StripPrefix("/",
+		http.FileServer(http.Dir(staticPath)))).Methods("GET")
+
+	log.WithFields(log.Fields{
+		"path": staticPath,
+	}).Info("Static route installed")
+
+}
+
+//////////////////////////////////////////////////////////////////////////
+// end of code