Create new Source Based Routing plugin

This creates a new plugin (sbr) which sets up source based routing, for use
as a chained plugin for multi-network environments.

README.md

@@ -26,6 +26,7 @@ Read [CONTRIBUTING](CONTRIBUTING.md) for build and test instructions.
 * `tuning`: Tweaks sysctl parameters of an existing interface
 * `portmap`: An iptables-based portmapping plugin. Maps ports from the host's address space to the container.
 * `bandwidth`: Allows bandwidth-limiting through use of traffic control tbf (ingress/egress).
+* `sbr`: A plugin that configures source based routing for an interface (from which it is chained).
 
 ### Sample
 The sample plugin provides an example for building your own plugin.

plugins/meta/sbr/README.md

@@ -0,0 +1,127 @@
+# Source based routing plugin
+
+## Introduction
+
+This plugin performs Source Based Routing (SBR). The most common and standard way to
+perform routing is to base it purely on the destination. However, in some
+applications which are using network separation for traffic management and
+security, there is no way to tell *a priori* which interface should be used,
+but the application is capable of making the decision.
+
+As an example, a Telco application might have two networks, a management
+network and a SIP (telephony) network for traffic, with rules which state that:
+
+- SIP traffic (only) must be routed over the SIP network;
+
+- all other traffic (but no SIP traffic) must be routed over the management
+  network.
+
+There is no way of configuring this based on destination IP, since there is no
+way of telling whether a destination IP on the internet is (say) an address
+used for downloading updated software packages or a remote SIP endpoint.
+
+Hence Source Based Routing is used.
+
+- The application explicitly listens on the correct interface for incoming
+  traffic.
+
+- When the application wishes to send to an address via the SIP network, it
+  explicitly binds to the IP of the device on that network.
+
+- Routes for the SIP interface are configured in a separate routing table, and
+  a rule is configured to use that table based on the source IP address.
+
+Note that in most cases there is a management device (the first one) and that
+the SBR plugin is called once for each device after the first, leaving the
+default routing table applied to the management device. However, this not
+mandatory, and source based routing may be configured on any or all of the
+devices as appropriate.
+
+## Usage
+
+This plugin runs as a chained plugin, and requires the following information
+passed in from the previous plugin (which has just set up the network device):
+
+- What is the network interface in question?
+
+- What is the IP address (or addresses) of that network interface?
+
+- What is the default gateway for that interface (if any)?
+
+It then reads all routes to the network interface in use.
+
+Here is an example of what the plugin would do. (The `ip` based commands are
+implemented in go, but easier to describe via the command line.) Suppose that
+it reads that:
+
+- The interface is `net1`.
+
+- The IP address on that interface is `192.168.1.209`.
+
+- The default gateway on that interface is `192.168.1.1`.
+
+- There is one route configured on that network interface, which is
+  `192.168.1.0/24`.
+
+Then the actions it takes are the following.
+
+- It creates a new routing table, and sets a rule to use it for the IP address in question.
+
+        ip rule add from 192.168.1.209/32 table 100
+
+- It adds a route to the default gateway for the relevant table.
+
+        ip route add default via 192.168.1.1 dev net1 table 100
+
+- It moves every existing route on the device to the new table.
+
+        ip route del 192.168.1.0/24 dev net1 src 192.168.1.209
+        ip route add 192.168.1.0/24 dev net1 src 192.168.1.209 table 100
+
+On deletion it:
+
+- deletes the rule (`ip rule del from 192.168.1.209/32 table 100`), which it
+  finds by deleting rules relating to IPs on the device which is about to be
+  deleted.
+
+- does nothing with routes (since the kernel automatically removes routes when
+  the device with which they are associated is deleted).
+
+## Future enhancements and known limitations
+
+The following are possible future enhancements.
+
+- The table number is currently selected by starting at 100, then incrementing
+  the value until an unused table number is found. It might be nice to have an
+  option to pass the table number as an input.
+
+- There is no log severity, and there is no logging to file (pending changes to
+  CNI logging generally).
+
+- This plugin sets up Source Based Routing, as described above. In future,
+  there may be a need for a VRF plugin (that uses
+  [VRF routing](https://www.kernel.org/doc/Documentation/networking/vrf.txt)
+  instead of source based routing). If and when this happens, it is likely that
+  the logic would be virtually identical for the plugin, and so the same plugin
+  might offer either SBR or VRF routing depending on configuration.
+
+## Configuration
+
+This plugin must be used as a chained plugin. There are no specific configuration parameters.
+
+A sample configuration for this plugin acting as a chained plugin after flannel
+is the following.
+
+~~~json
+{
+  "name": "flannel-sbr",
+  "cniVersion": "0.3.0",
+  "plugins":
+    [
+        { "type": "flannel" },
+        {
+          "type": "sbr",
+        }
+    ]
+}
+~~~

plugins/meta/sbr/main.go

@@ -0,0 +1,378 @@
+// Copyright 2017 CNI authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// This is the Source Based Routing plugin that sets up source based routing.
+package main
+
+import (
+	"encoding/json"
+	"fmt"
+	"log"
+	"net"
+
+	"github.com/alexflint/go-filemutex"
+	"github.com/containernetworking/cni/pkg/skel"
+	"github.com/containernetworking/cni/pkg/types"
+	"github.com/containernetworking/cni/pkg/types/current"
+	"github.com/containernetworking/cni/pkg/version"
+	"github.com/containernetworking/plugins/pkg/ns"
+
+	"github.com/vishvananda/netlink"
+)
+
+const firstTableID = 100
+
+// PluginConf is the configuration document passed in.
+type PluginConf struct {
+	types.NetConf
+
+	// This is the previous result, when called in the context of a chained
+	// plugin. Because this plugin supports multiple versions, we'll have to
+	// parse this in two passes. If your plugin is not chained, this can be
+	// removed (though you may wish to error if a non-chainable plugin is
+	// chained).
+	RawPrevResult *map[string]interface{} `json:"prevResult"`
+	PrevResult    *current.Result         `json:"-"`
+
+	// Add plugin-specific flags here
+}
+
+// Wrapper that does a lock before and unlock after operations to serialise
+// this plugin.
+func withLockAndNetNS(nspath string, toRun func(_ ns.NetNS) error) error {
+	// We lock on the network namespace to ensure that no other instance
+	// clashes with this one.
+	log.Printf("Network namespace to use and lock: %s", nspath)
+	lock, err := filemutex.New(nspath)
+	if err != nil {
+		return err
+	}
+
+	err = lock.Lock()
+	if err != nil {
+		return err
+	}
+
+	err = ns.WithNetNSPath(nspath, toRun)
+
+	if err != nil {
+		return err
+	}
+
+	// Cleaner to unlock even though about to exit
+	err = lock.Unlock()
+
+	return err
+}
+
+// parseConfig parses the supplied configuration (and prevResult) from stdin.
+func parseConfig(stdin []byte) (*PluginConf, error) {
+	conf := PluginConf{}
+
+	if err := json.Unmarshal(stdin, &conf); err != nil {
+		return nil, fmt.Errorf("failed to parse network configuration: %v", err)
+	}
+
+	// Parse previous result.
+	if conf.RawPrevResult != nil {
+		resultBytes, err := json.Marshal(conf.RawPrevResult)
+		if err != nil {
+			return nil, fmt.Errorf("could not serialize prevResult: %v", err)
+		}
+		res, err := version.NewResult(conf.CNIVersion, resultBytes)
+		if err != nil {
+			return nil, fmt.Errorf("could not parse prevResult: %v", err)
+		}
+		conf.RawPrevResult = nil
+		conf.PrevResult, err = current.NewResultFromResult(res)
+		if err != nil {
+			return nil, fmt.Errorf("could not convert result to current version: %v", err)
+		}
+	}
+	// End previous result parsing
+
+	return &conf, nil
+}
+
+// getIPCfgs finds the IPs on the supplied interface, returning as IPConfig structures
+func getIPCfgs(iface string, prevResult *current.Result) ([]*current.IPConfig, error) {
+
+	if len(prevResult.IPs) == 0 {
+		// No IP addresses; that makes no sense. Pack it in.
+		return nil, fmt.Errorf("No IP addresses supplied on interface: %s", iface)
+	}
+
+	// We do a single interface name, stored in args.IfName
+	log.Printf("Checking for relevant interface: %s", iface)
+
+	// ips contains the IPConfig structures that were passed, filtered somewhat
+	ipCfgs := make([]*current.IPConfig, 0, len(prevResult.IPs))
+
+	for _, ipCfg := range prevResult.IPs {
+		// IPs have an interface that is an index into the interfaces array.
+		// We assume a match if this index is missing.
+		if ipCfg.Interface == nil {
+			log.Printf("No interface for IP address %s", ipCfg.Address.IP)
+			ipCfgs = append(ipCfgs, ipCfg)
+			continue
+		}
+
+		// Skip all IPs we know belong to an interface with the wrong name.
+		intIdx := *ipCfg.Interface
+		if intIdx >= 0 && intIdx < len(prevResult.Interfaces) && prevResult.Interfaces[intIdx].Name != iface {
+			log.Printf("Incorrect interface for IP address %s", ipCfg.Address.IP)
+			continue
+		}
+
+		log.Printf("Found IP address %s", ipCfg.Address.IP.String())
+		ipCfgs = append(ipCfgs, ipCfg)
+	}
+
+	return ipCfgs, nil
+}
+
+// cmdAdd is called for ADD requests
+func cmdAdd(args *skel.CmdArgs) error {
+	conf, err := parseConfig(args.StdinData)
+	if err != nil {
+		return err
+	}
+
+	log.Printf("Configure SBR for new interface %s - previous result: %v",
+		args.IfName, conf.PrevResult)
+
+	if conf.PrevResult == nil {
+		return fmt.Errorf("This plugin must be called as chained plugin")
+	}
+
+	// Get the list of relevant IPs.
+	ipCfgs, err := getIPCfgs(args.IfName, conf.PrevResult)
+	if err != nil {
+		return err
+	}
+
+	// Do the actual work.
+	err = withLockAndNetNS(args.Netns, func(_ ns.NetNS) error {
+		return doRoutes(ipCfgs, conf.PrevResult.Routes, args.IfName)
+	})
+	if err != nil {
+		return err
+	}
+
+	// Pass through the result for the next plugin
+	return types.PrintResult(conf.PrevResult, conf.CNIVersion)
+}
+
+// doRoutes does all the work to set up routes and rules during an add.
+func doRoutes(ipCfgs []*current.IPConfig, origRoutes []*types.Route, iface string) error {
+	// Get a list of rules and routes ready.
+	rules, err := netlink.RuleList(netlink.FAMILY_ALL)
+	if err != nil {
+		return fmt.Errorf("Failed to list all rules: %v", err)
+	}
+
+	routes, err := netlink.RouteList(nil, netlink.FAMILY_ALL)
+	if err != nil {
+		return fmt.Errorf("Failed to list all routes: %v", err)
+	}
+
+	// Pick a table ID to use. We pick the first table ID from firstTableID
+	// on that has no existing rules mapping to it and no existing routes in
+	// it.
+	table := firstTableID
+	for {
+		foundExisting := false
+		for _, rule := range rules {
+			if rule.Table == table {
+				foundExisting = true
+				break
+			}
+		}
+
+		for _, route := range routes {
+			if route.Table == table {
+				foundExisting = true
+				break
+			}
+		}
+
+		if foundExisting {
+			table++
+		} else {
+			break
+		}
+	}
+
+	log.Printf("First unreferenced table: %d", table)
+
+	link, err := netlink.LinkByName(iface)
+	if err != nil {
+		return fmt.Errorf("Cannot find network interface %s: %v", iface, err)
+	}
+
+	linkIndex := link.Attrs().Index
+
+	// Loop through setting up source based rules and default routes.
+	for _, ipCfg := range ipCfgs {
+		log.Printf("Set rule for source %s", ipCfg.String())
+		rule := netlink.NewRule()
+		rule.Table = table
+
+		// Source must be restricted to a single IP, not a full subnet
+		var src net.IPNet
+		src.IP = ipCfg.Address.IP
+		if ipCfg.Version == "4" {
+			src.Mask = net.CIDRMask(32, 32)
+		} else {
+			src.Mask = net.CIDRMask(64, 64)
+		}
+
+		log.Printf("Source to use %s", src.String())
+		rule.Src = &src
+
+		if err = netlink.RuleAdd(rule); err != nil {
+			return fmt.Errorf("Failed to add rule: %v", err)
+		}
+
+		// Add a default route, since this may have been removed by previous
+		// plugin.
+		if ipCfg.Gateway != nil {
+			log.Printf("Adding default route to gateway %s", ipCfg.Gateway.String())
+
+			var dest net.IPNet
+			if ipCfg.Version == "4" {
+				dest.IP = net.IPv4zero
+				dest.Mask = net.CIDRMask(0, 32)
+			} else {
+				dest.IP = net.IPv6zero
+				dest.Mask = net.CIDRMask(0, 64)
+			}
+
+			route := netlink.Route{
+				Dst:       &dest,
+				Gw:        ipCfg.Gateway,
+				Table:     table,
+				LinkIndex: linkIndex}
+
+			err = netlink.RouteAdd(&route)
+			if err != nil {
+				return fmt.Errorf("Failed to add default route to %s: %v",
+					ipCfg.Gateway.String(),
+					err)
+			}
+		}
+	}
+
+	// Move all routes into the correct table. We are taking a shortcut; all
+	// the routes have been added to the interface anyway but in the wrong
+	// table, so instead of removing them we just move them to the table we
+	// want them in.
+	routes, err = netlink.RouteList(link, netlink.FAMILY_ALL)
+	if err != nil {
+		return fmt.Errorf("Unable to list routes: %v", err)
+	}
+
+	for _, route := range routes {
+		log.Printf("Moving route %s from table %d to %d",
+			route.String(), route.Table, table)
+
+		err := netlink.RouteDel(&route)
+		if err != nil {
+			return fmt.Errorf("Failed to delete route: %v", err)
+		}
+
+		route.Table = table
+
+		// We use route replace in case the route already exists, which
+		// is possible for the default gateway we added above.
+		err = netlink.RouteReplace(&route)
+		if err != nil {
+			return fmt.Errorf("Failed to readd route: %v", err)
+		}
+	}
+
+	return nil
+}
+
+// cmdDel is called for DELETE requests
+func cmdDel(args *skel.CmdArgs) error {
+	// We care a bit about config because it sets log level.
+	_, err := parseConfig(args.StdinData)
+	if err != nil {
+		return err
+	}
+
+	log.Printf("Cleaning up SBR for %s", args.IfName)
+	err = withLockAndNetNS(args.Netns, func(_ ns.NetNS) error {
+		return tidyRules(args.IfName)
+	})
+
+	return err
+}
+
+// Tidy up the rules for the deleted interface
+func tidyRules(iface string) error {
+
+	// We keep on going on rule deletion error, but return the last failure.
+	var errReturn error
+
+	rules, err := netlink.RuleList(netlink.FAMILY_ALL)
+	if err != nil {
+		log.Printf("Failed to list all rules to tidy: %v", err)
+		return fmt.Errorf("Failed to list all rules to tidy: %v", err)
+	}
+
+	link, err := netlink.LinkByName(iface)
+	if err != nil {
+		log.Printf("Failed to get link %s: %v", iface, err)
+		return fmt.Errorf("Failed to get link %s: %v", iface, err)
+	}
+
+	addrs, err := netlink.AddrList(link, netlink.FAMILY_ALL)
+	if err != nil {
+		log.Printf("Failed to list all addrs: %v", err)
+		return fmt.Errorf("Failed to list all addrs: %v", err)
+	}
+
+RULE_LOOP:
+	for _, rule := range rules {
+		log.Printf("Check rule: %v", rule)
+		if rule.Src == nil {
+			continue
+		}
+
+		for _, addr := range addrs {
+			if rule.Src.IP.Equal(addr.IP) {
+				log.Printf("Delete rule %v", rule)
+				err := netlink.RuleDel(&rule)
+				if err != nil {
+					errReturn = fmt.Errorf("Failed to delete rule %v", err)
+					log.Printf("... Failed! %v", err)
+				}
+				continue RULE_LOOP
+			}
+		}
+
+	}
+
+	return errReturn
+}
+
+func main() {
+	skel.PluginMain(cmdAdd, cmdGet, cmdDel, version.All, "TODO")
+}
+
+func cmdGet(args *skel.CmdArgs) error {
+	return fmt.Errorf("not implemented")
+}

plugins/meta/sbr/sbr_linux_test.go

@@ -0,0 +1,430 @@
+// Copyright 2017 CNI authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+	"fmt"
+	"log"
+	"net"
+
+	"github.com/containernetworking/cni/pkg/skel"
+	"github.com/containernetworking/plugins/pkg/ns"
+	"github.com/containernetworking/plugins/pkg/testutils"
+
+	"github.com/vishvananda/netlink"
+	"golang.org/x/sys/unix"
+
+	. "github.com/onsi/ginkgo"
+	. "github.com/onsi/gomega"
+)
+
+// Structures specifying state at start.
+type Device struct {
+	Name   string
+	Addrs  []net.IPNet
+	Routes []netlink.Route
+}
+
+type Rule struct {
+	Src   string
+	Table int
+}
+
+type netStatus struct {
+	Devices []Device
+	Rules   []netlink.Rule
+}
+
+// Create a link, shove some IP addresses on it, and push it into the network
+// namespace.
+func setup(targetNs ns.NetNS, status netStatus) error {
+	// Get the status right
+	err := targetNs.Do(func(_ ns.NetNS) error {
+		for _, dev := range status.Devices {
+			log.Printf("Adding dev %s\n", dev.Name)
+			link := &netlink.Dummy{LinkAttrs: netlink.LinkAttrs{Name: dev.Name}}
+			err := netlink.LinkAdd(link)
+			if err != nil {
+				return err
+			}
+
+			err = netlink.LinkSetUp(link)
+			if err != nil {
+				return err
+			}
+
+			for _, addr := range dev.Addrs {
+				log.Printf("Adding address %v to device %s\n", addr, dev.Name)
+				err = netlink.AddrAdd(link, &netlink.Addr{IPNet: &addr})
+				if err != nil {
+					return err
+				}
+			}
+
+			for _, route := range dev.Routes {
+				log.Printf("Adding route %v to device %s\n", route, dev.Name)
+				route.LinkIndex = link.Attrs().Index
+				err = netlink.RouteAdd(&route)
+				if err != nil {
+					return err
+				}
+			}
+		}
+		return nil
+	})
+
+	return err
+}
+
+// Readback the routes and rules.
+func readback(targetNs ns.NetNS, devNames []string) (netStatus, error) {
+	// Get the status right.
+	var retVal netStatus
+
+	err := targetNs.Do(func(_ ns.NetNS) error {
+		retVal.Devices = make([]Device, 2)
+
+		for i, name := range devNames {
+			log.Printf("Checking device %s", name)
+			retVal.Devices[i].Name = name
+
+			link, err := netlink.LinkByName(name)
+			if err != nil {
+				return err
+			}
+
+			// Need to read all tables, so cannot use RouteList
+			routeFilter := &netlink.Route{
+				LinkIndex: link.Attrs().Index,
+				Table:     unix.RT_TABLE_UNSPEC,
+			}
+
+			routes, err := netlink.RouteListFiltered(netlink.FAMILY_ALL,
+				routeFilter,
+				netlink.RT_FILTER_OIF|netlink.RT_FILTER_TABLE)
+
+			if err != nil {
+				return err
+			}
+
+			for _, route := range routes {
+				log.Printf("Got %s route %v", name, route)
+			}
+
+			retVal.Devices[i].Routes = routes
+		}
+
+		rules, err := netlink.RuleList(netlink.FAMILY_ALL)
+		if err != nil {
+			return err
+		}
+
+		retVal.Rules = make([]netlink.Rule, 0, len(rules))
+
+		for _, rule := range rules {
+			// Rules over 250 are the kernel defaults, that we ignore.
+			if rule.Table < 250 {
+				log.Printf("Got interesting rule %v", rule)
+				retVal.Rules = append(retVal.Rules, rule)
+			}
+		}
+
+		return nil
+	})
+
+	return retVal, err
+}
+
+func equalRoutes(expected, actual []netlink.Route) bool {
+	// Compare two sets of routes, comparing only destination, gateway and
+	// table. Return true if equal.
+	match := true
+
+	// Map used to make comparisons easy
+	expMap := make(map[string]bool)
+
+	for _, route := range expected {
+		expMap[route.String()] = true
+	}
+
+	for _, route := range actual {
+		routeString := route.String()
+		if expMap[routeString] {
+			log.Printf("Route %s expected and found", routeString)
+			delete(expMap, routeString)
+		} else {
+			log.Printf("Route %s found, not expected", routeString)
+			match = false
+		}
+	}
+
+	for expRoute := range expMap {
+		log.Printf("Route %s expected, not found", expRoute)
+		match = false
+	}
+
+	return match
+}
+
+func createDefaultStatus() netStatus {
+	// Useful default status.
+	devs := make([]Device, 2)
+	rules := make([]netlink.Rule, 0)
+
+	devs[0] = Device{Name: "eth0"}
+	devs[0].Addrs = make([]net.IPNet, 1)
+	devs[0].Addrs[0] = net.IPNet{
+		IP:   net.IPv4(10, 0, 0, 2),
+		Mask: net.IPv4Mask(255, 255, 255, 0),
+	}
+	devs[0].Routes = make([]netlink.Route, 2)
+	devs[0].Routes[0] = netlink.Route{
+		Dst: &net.IPNet{
+			IP:   net.IPv4(10, 2, 0, 0),
+			Mask: net.IPv4Mask(255, 255, 255, 0),
+		},
+		Gw: net.IPv4(10, 0, 0, 5),
+	}
+	devs[0].Routes[1] = netlink.Route{
+		Gw: net.IPv4(10, 0, 0, 1),
+	}
+
+	devs[1] = Device{Name: "net1"}
+	devs[1].Addrs = make([]net.IPNet, 1)
+	devs[1].Addrs[0] = net.IPNet{
+		IP:   net.IPv4(192, 168, 1, 209),
+		Mask: net.IPv4Mask(255, 255, 255, 0),
+	}
+	devs[1].Routes = make([]netlink.Route, 1)
+	devs[1].Routes[0] = netlink.Route{
+		Dst: &net.IPNet{
+			IP:   net.IPv4(192, 168, 2, 0),
+			Mask: net.IPv4Mask(255, 255, 255, 0),
+		},
+		Gw: net.IPv4(192, 168, 1, 2),
+	}
+
+	return netStatus{
+		Devices: devs,
+		Rules:   rules}
+}
+
+var _ = Describe("sbr test", func() {
+	var targetNs ns.NetNS
+
+	BeforeEach(func() {
+		var err error
+		targetNs, err = testutils.NewNS()
+		Expect(err).NotTo(HaveOccurred())
+	})
+
+	AfterEach(func() {
+		targetNs.Close()
+	})
+
+	It("Works with a 0.3.0 config", func() {
+		ifname := "net1"
+		conf := `{
+	"cniVersion": "0.3.0",
+	"name": "cni-plugin-sbr-test",
+	"type": "sbr",
+	"prevResult": {
+		"interfaces": [
+			{
+				"name": "%s",
+				"sandbox": "%s"
+			}
+		],
+		"ips": [
+			{
+				"version": "4",
+				"address": "192.168.1.209/24",
+				"gateway": "192.168.1.1",
+				"interface": 0
+			}
+		],
+		"routes": []
+	}
+}`
+		conf = fmt.Sprintf(conf, ifname, targetNs.Path())
+		args := &skel.CmdArgs{
+			ContainerID: "dummy",
+			Netns:       targetNs.Path(),
+			IfName:      ifname,
+			StdinData:   []byte(conf),
+		}
+
+		err := setup(targetNs, createDefaultStatus())
+		Expect(err).NotTo(HaveOccurred())
+
+		oldStatus, err := readback(targetNs, []string{"net1", "eth0"})
+		Expect(err).NotTo(HaveOccurred())
+
+		_, _, err = testutils.CmdAddWithArgs(args, func() error { return cmdAdd(args) })
+		Expect(err).NotTo(HaveOccurred())
+
+		newStatus, err := readback(targetNs, []string{"net1", "eth0"})
+		Expect(err).NotTo(HaveOccurred())
+
+		// Check results. We expect all the routes on net1 to have moved to
+		// table 100 except for local routes (table 255); a new default gateway
+		// route to have been created; and a single rule to exist.
+		expNet1 := oldStatus.Devices[0]
+		expEth0 := oldStatus.Devices[1]
+		for i := range expNet1.Routes {
+			if expNet1.Routes[i].Table != 255 {
+				expNet1.Routes[i].Table = 100
+			}
+		}
+		expNet1.Routes = append(expNet1.Routes,
+			netlink.Route{
+				Gw:        net.IPv4(192, 168, 1, 1),
+				Table:     100,
+				LinkIndex: expNet1.Routes[0].LinkIndex})
+
+		Expect(len(newStatus.Rules)).To(Equal(1))
+		Expect(newStatus.Rules[0].Table).To(Equal(100))
+		Expect(newStatus.Rules[0].Src.String()).To(Equal("192.168.1.209/32"))
+		devNet1 := newStatus.Devices[0]
+		devEth0 := newStatus.Devices[1]
+		Expect(equalRoutes(expNet1.Routes, devNet1.Routes)).To(BeTrue())
+		Expect(equalRoutes(expEth0.Routes, devEth0.Routes)).To(BeTrue())
+
+		conf = `{
+	"cniVersion": "0.3.0",
+	"name": "cni-plugin-sbr-test",
+	"type": "sbr"
+}`
+
+		// And now check that we can back it all out.
+		args = &skel.CmdArgs{
+			ContainerID: "dummy",
+			Netns:       targetNs.Path(),
+			IfName:      ifname,
+			StdinData:   []byte(conf),
+		}
+		err = testutils.CmdDelWithArgs(args, func() error { return cmdDel(args) })
+		Expect(err).NotTo(HaveOccurred())
+
+		retVal, err := readback(targetNs, []string{"net1", "eth0"})
+		Expect(err).NotTo(HaveOccurred())
+
+		// Check results. We expect the rule to have been removed.
+		Expect(len(retVal.Rules)).To(Equal(0))
+	})
+
+	It("Works with a default route already set", func() {
+		ifname := "net1"
+		conf := `{
+	"cniVersion": "0.3.0",
+	"name": "cni-plugin-sbr-test",
+	"type": "sbr",
+	"prevResult": {
+		"interfaces": [
+			{
+				"name": "%s",
+				"sandbox": "%s"
+			}
+		],
+		"ips": [
+			{
+				"version": "4",
+				"address": "192.168.1.209/24",
+				"gateway": "192.168.1.1",
+				"interface": 0
+			}
+		],
+		"routes": []
+	}
+}`
+		conf = fmt.Sprintf(conf, ifname, targetNs.Path())
+		args := &skel.CmdArgs{
+			ContainerID: "dummy",
+			Netns:       targetNs.Path(),
+			IfName:      ifname,
+			StdinData:   []byte(conf),
+		}
+
+		preStatus := createDefaultStatus()
+		// Remove final (default) route from eth0, then add another default
+		// route to net1
+		preStatus.Devices[0].Routes = preStatus.Devices[0].Routes[:0]
+		routes := preStatus.Devices[1].Routes
+		preStatus.Devices[1].Routes = append(preStatus.Devices[1].Routes,
+			netlink.Route{
+				Gw:        net.IPv4(192, 168, 1, 1),
+				LinkIndex: routes[0].LinkIndex})
+
+		err := setup(targetNs, preStatus)
+		Expect(err).NotTo(HaveOccurred())
+
+		oldStatus, err := readback(targetNs, []string{"net1", "eth0"})
+		Expect(err).NotTo(HaveOccurred())
+
+		_, _, err = testutils.CmdAddWithArgs(args, func() error { return cmdAdd(args) })
+		Expect(err).NotTo(HaveOccurred())
+
+		newStatus, err := readback(targetNs, []string{"net1", "eth0"})
+		Expect(err).NotTo(HaveOccurred())
+
+		// Check results. We expect all the routes on net1 to have moved to
+		// table 100 except for local routes (table 255); a new default gateway
+		// route to have been created; and a single rule to exist.
+		expNet1 := oldStatus.Devices[0]
+		expEth0 := oldStatus.Devices[1]
+		for i := range expNet1.Routes {
+			if expNet1.Routes[i].Table != 255 {
+				expNet1.Routes[i].Table = 100
+			}
+		}
+
+		Expect(len(newStatus.Rules)).To(Equal(1))
+		Expect(newStatus.Rules[0].Table).To(Equal(100))
+		Expect(newStatus.Rules[0].Src.String()).To(Equal("192.168.1.209/32"))
+		devNet1 := newStatus.Devices[0]
+		devEth0 := newStatus.Devices[1]
+		Expect(equalRoutes(expEth0.Routes, devEth0.Routes)).To(BeTrue())
+		Expect(equalRoutes(expNet1.Routes, devNet1.Routes)).To(BeTrue())
+	})
+
+	It("works with a 0.2.0 config", func() {
+		conf := `{
+	"cniVersion": "0.2.0",
+	"name": "cni-plugin-sbr-test",
+	"type": "sbr",
+	"anotherAwesomeArg": "foo",
+	"prevResult": {
+		"ip4": {
+        	"ip": "192.168.1.209/24",
+		    "gateway": "192.168.1.1",
+			"routes": []
+		}
+	}
+}`
+
+		args := &skel.CmdArgs{
+			ContainerID: "dummy",
+			Netns:       targetNs.Path(),
+			IfName:      "net1",
+			StdinData:   []byte(conf),
+		}
+		err := setup(targetNs, createDefaultStatus())
+		Expect(err).NotTo(HaveOccurred())
+
+		_, _, err = testutils.CmdAddWithArgs(args, func() error { return cmdAdd(args) })
+		Expect(err).NotTo(HaveOccurred())
+	})
+
+})

plugins/meta/sbr/sbr_suite_test.go

@@ -0,0 +1,15 @@
+// The boilerplate needed for Ginkgo
+
+package main
+
+import (
+	. "github.com/onsi/ginkgo"
+	. "github.com/onsi/gomega"
+
+	"testing"
+)
+
+func TestSample(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "plugins/sbr")
+}