OpenVSwitch (OVS) is the pillar used by several emblazoned software, such as OpenStack or Red Hat's OpenShift to set up their Software Defined Networks (SDN): it enables users to quickly and easily implement multiple bridges to which connect Virtual Machines or Containers.
These bridges can be left standalone, creating isolated networks, or interconnected to the machine (or VM) NICs, providing bidirectional access to the network segment the NIC is connected to. In addition to that, it also enables the set up VxLAN Tunnel EndPoint (VTEP) on these bridges, enabling interconnecting OVS bridges from different machines. Last but not least, it also enforces traffic policies defined using OpenFlow.
The SDN tutorial - OpenFlow with OpenVSwitch on Oracle Linux, starts from where we left in the "Free Range Routing (FRR) And OpenVSwitch On OracleLinux" post, extending its Lab and provides a practical guide on how to write and set OpenFlow rules on OpenVSwitch.
Provision The Lab
In this post we are using the following networks we already set up in the "Free Range Routing (FRR) And OpenVSwitch On OracleLinux" post:
Name
Subnet CIDR
Domain
Description
Application Servers Security Tier 1
192.168.0.0/24
as-t1.carcano.local
The network used to home Test Security Tier 1 environment's Application Servers
Database Servers Security Tier 1
192.168.6.0/24
db-t1.carcano.local
The network used to home Test Security Tier 1 environment's database servers
Management Security Tier 1
10.0.0.0/8
mgmt-t1.carcano.local
The network used for operating and management every Test Security Tier 1 environment's server
In order to play a little bit with OpenFLow we add the following hosts:
- as-ca-ut1a001
- pg-ca-ut1a001
So the set of the Lab's hosts used in this post is:
Hostname
Services Subnet/Domain(s)
Management Subnet/Domain
Description
gw-ca-ut1a001
- as-t1.carcano.local
- dbt-1.carcano.local
mgmt-t1.carcano.local
the Test Security Tier 1 environment's gateway: it provides routing and enforce network policies set through OpenFlow.
as-ca-ut1a001
as-t1.carcano.local
mgmt-t1.carcano.local
a Test Security Tier 1 environment's Application Server - in this post we actually don't install anything on it: we use it only as a source host that needs to connect to a database machine on the Database ServersSecurity Tier 1 network
pg-ca-ut1a001
dbt-1.carcano.local
mgmt-t1.carcano.local
a Test Security Tier 1 environment's PostgreSQL Database server
Deploying Using Vagrant
In order to add the two above VM, it is necessary to extend the Vagrantfile shown in the previous post as follows:
- add the setup_service_interface provisioning SHELL script - it is used to automatically configure networking, including the VLAN
- add the install_postgresql_client and install_postgresql_server provisioning SHELL script - it is used to automatically PostgreSQL Server
- add the "as-ca-ut1a001" and "pg-ca-ut1a001" VMs to the "host_vms" list of dictionaries
- add some conditionals to the statement lists, so to customize the VM properly, using the service_class property as the matching criteria
For your convenience, this is how the whole Vagrantfile must now look like after these changes:
# -*- mode: ruby -*-
# vi: set ft=ruby :
$wipe_network_interface = <<-'SCRIPT'
DEV=$(ip -4 addr | grep -v "altname" | grep ${1} -B 1 | head -n 1 | awk '{print $2}' |tr -d :)
[[ -z "${DEV}" ]] && exit 0
CONN_NAME=$(nmcli -t -f NAME,DEVICE con | grep "${DEV}" | grep -v "br-" | cut -d : -f 1)
nmcli conn modify "${CONN_NAME}" ipv4.method auto
nmcli conn modify "${CONN_NAME}" ipv4.address ''
nmcli conn modify "${CONN_NAME}" ipv4.method disabled
nmcli conn modify "${CONN_NAME}" autoconnect no
SCRIPT
$setup_services_interface = <<-'SCRIPT'
DEV=$(ip -4 addr | grep -v "altname" | grep ${1} -B 1 | head -n 1 | awk '{print $2}' |tr -d :)
VLAN=$2
CONN_UUID=$(nmcli -t -f UUID,DEVICE con | grep "${DEV}" | grep -v "\.${VLAN}" | cut -d : -f 1)
nmcli conn delete "${CONN_UUID}"
CONN_UUID=$(nmcli -t -f UUID,DEVICE con | grep "${DEV}" | grep -v "\.${VLAN}" | cut -d : -f 1)
[[ -n "${CONN_UUID}" ]] && nmcli conn delete "${CONN_UUID}"
[[ $(nmcli -t -f NAME conn | grep ${DEV}-vlan.${VLAN}) ]] && exit 0
IP=$3
MASK=$4
ROUTES=$5
nmcli con add type vlan con-name ${DEV}-vlan.${VLAN} ifname ${DEV}.${VLAN} dev ${DEV} id ${VLAN} ip4 ${IP}/${MASK}
nmcli con mod "${DEV}-vlan.${VLAN}" +ipv4.routes "${ROUTES}"
nmcli con up "${DEV}-vlan.${VLAN}"
[[ "$(systemctl is-enabled firewalld)" == "enabled" ]] || systemctl enable firewalld
[[ "$(systemctl is-active firewalld)" == "active" ]] || systemctl start firewalld
firewall-cmd --permanent --new-zone=services1
firewall-cmd --permanent --zone=services1 --add-interface=${DEV}.${VLAN}
firewall-cmd --reload
SCRIPT
$install_postgresql_client = <<-'SCRIPT'
ARCH=$(uname -i)
dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-${ARCH}/pgdg-redhat-repo-latest.noarch.rpm
dnf install -y net-tools postgresql16
SCRIPT
$install_postgresql_server = <<-'SCRIPT'
ARCH=$(uname -i)
dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-${ARCH}/pgdg-redhat-repo-latest.noarch.rpm
dnf install -y net-tools postgresql16 postgresql16-server
firewall-cmd --permanent --zone=services1 --add-service postgresql
firewall-cmd --reload
/usr/pgsql-16/bin/postgresql-16-setup initdb
echo "listen_addresses = '*'" >> /var/lib/pgsql/16/data/postgresql.conf
echo "host all all 192.168.0.0/24 scram-sha-256" >> /var/lib/pgsql/16/data/pg_hba.conf
systemctl enable --now postgresql-16
sudo -u postgres psql -c "alter user postgres with password 'grimoire'"
SCRIPT
$reboot = <<-'SCRIPT'
sudo shutdown -r now
SCRIPT
VAGRANTFILE_API_VERSION = "2"
Vagrant.require_version ">= 1.5"
host_vms=[
{
:hostname => "gw-ca-ut1a001",
:domain => "netdevs.carcano.local",
:infra_net_ip => "172.16.0.11",
:core_net_temporary_ip => "192.168.253.253",
:box => "grimoire/ol92",
:ram => 2048,
:cpu => 2,
:service_class => "netdev"
},
{
:hostname => "gw-ca-up1a001",
:domain => "netdevs.carcano.local",
:infra_net_ip => "172.16.0.12",
:core_net_temporary_ip => "192.168.254.254",
:box => "grimoire/ol92",
:ram => 2048,
:cpu => 2,
:service_class => "netdev"
},
{
:hostname => "as-ca-ut1a001",
:domain => "netdevs.carcano.local",
:core_net_temporary_ip => "192.168.253.11",
:services_net_ip => "192.168.0.10",
:services_net_mask => "24",
:services_net_vlan => "100",
:summary_route => "192.168.0.0/16 192.168.0.254",
:box => "grimoire/ol92",
:ram => 2048,
:cpu => 2,
:service_class => "ws"
},
{
:hostname => "pg-ca-ut1a001",
:domain => "netdevs.carcano.local",
:core_net_temporary_ip => "192.168.253.12",
:services_net_ip => "192.168.6.10",
:services_net_mask => "24",
:services_net_vlan => "101",
:summary_route => "192.168.0.0/16 192.168.6.254",
:box => "grimoire/ol92",
:ram => 2048,
:cpu => 2,
:service_class => "postgresql"
}
]
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
host_vms.each do |machine|
config.vm.define machine[:hostname] do |node |
node.vm.box = machine[:box]
node.vm.hostname = "#{machine[:hostname]}.#{machine[:domain]}"
node.vm.network "private_network", ip: machine[:core_net_temporary_ip]
if machine[:service_class] == "netdev"
node.vm.network "private_network", ip: machine[:infra_net_ip]
end
node.vm.provider :virtualbox do |vm|
vm.name = "grimoire_#{machine[:hostname]}"
vm.cpus = machine[:cpu]
vm.customize [ "modifyvm", :id, "--memory", machine[:ram] ]
if machine[:service_class] == "netdev"
vm.customize [ "modifyvm", :id, "--nicpromisc2", "allow-vms" ]
end
end
node.vm.provider :parallels do |vm|
vm.name = "grimoire_#{machine[:hostname]}"
vm.memory = machine[:ram]
vm.cpus = machine[:cpu]
vm.update_guest_tools = false
vm.optimize_power_consumption = false
end
if machine[:service_class] == "netdev"
node.vm.provision :shell, :args => machine[:core_net_temporary_ip], inline: $wipe_network_interface, run: "always"
node.vm.provision :shell, inline: $reboot, run: "always"
end
if machine[:service_class] == "ws"
node.vm.provision :shell, :args => [ machine[:core_net_temporary_ip], machine[:services_net_vlan], machine[:services_net_ip], machine[:services_net_mask], machine[:summary_route] ], inline: $setup_services_interface, run: "always"
node.vm.provision :shell, inline: $install_postgresql_client
end
if machine[:service_class] == "postgresql"
node.vm.provision :shell, :args => [ machine[:core_net_temporary_ip], machine[:services_net_vlan], machine[:services_net_ip], machine[:services_net_mask], machine[:summary_route] ], inline: $setup_services_interface, run: "always"
node.vm.provision :shell, inline: $install_postgresql_server
end
end
end
end
we can now provision the new VMs by simply running:
vagrant up as-ca-ut1a001 pg-ca-ut1a001 gw-ca-ut1a001
The Example Use Case
Vagrant takes care of fully provisioning both the "as-ca-ut1a001" and "pg-ca-ut1a001" VMs: besides configuring networking,
- on "pg-ca-ut1a001", it installs and configure PostgreSQL server
- on "as-ca-ut1a001" it installs the PostgreSQL client components
Our example use case is configuring OpenFlow traffic policies that:
- permit ARP traffic
- permit ICMP (ping)
- permit connecting from the "as-ca-ut1a001" host to the PostgreSQL instance on the "pg-ca-ut1a001" host
As we said the default Openflow flow on OpenVSwitch permits everything: this means that PostgreSQL connection is already permitted - we can easily check it by SSH connecting to "as-ca-ut1a001" using Vagrant:
vagrant ssh as-ca-ut1a001
then connect to the PostgreSQL instance running on the "pg-ca-ut1a001" server:
psql -h 192.168.6.10 -U postgres
when asked, just enter '"grimoire" as the password.
Once connected, just try listing the available databases:
\l
the list is as follows:
Name | Owner | Encoding | Locale Provider | Collate | Ctype | ICU Locale | ICU Rules | Access privileges
-----------+----------+----------+-----------------+-------------+-------------+------------+-----------+-----------------------
postgres | postgres | UTF8 | libc | en_US.UTF-8 | en_US.UTF-8 | | |
template0 | postgres | UTF8 | libc | en_US.UTF-8 | en_US.UTF-8 | | | =c/postgres +
| | | | | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | libc | en_US.UTF-8 | en_US.UTF-8 | | | =c/postgres +
| | | | | | | | postgres=CTc/postgres
(3 rows)
We successfully reached the PostgreSQL instance because the default OpenFlow flow in OpenVSwitch is set to permit everything, but this is not acceptable by the security perspective in a segregated networking environment.
Software Defined Networks
Before specifically talking about OpenFlow, since it is a protocol used by Software Defined Network (SDN), it is certainly worth the effort to spend a few words on them too.
Software Defined Network (SDN) is an architectural paradigm that separates the logic that controls how to forward the traffic (it calls it the control plane) from the underlying system that forwards traffic (that calls the data plane). Everything is then managed through the management plane.
Control separation has many benefits like:
- Light-weighted devices: since intelligence is at the controller side, network equipments such switches and routers can be slimmed down, reducing their CAPEX (CAPital EXpenses) compared to over-priced high-end routing and switching equipments
- Central management: everything can be configured, monitored from a central controller: this eases also getting a complete view and troubleshooting, reducing OPEX (OPerational and maintenance EXpenses) too.
Lowering CAPEX and OPEX is obviously a good deal to succeed when asking for budgets to C-Levels.
As of the architectural separation, we distinguish three layers:
the networking equipments (router, switches) which forms the network that actually forward the traffic
it is the brain of the SDN infrastructure: it exchanges:
- protocol updates
- system management messages
it also maintains:
- the Routing Information Base (RIB), that is the routing table used to exchange route updates with the routers
- the Forwarding Information Base (FIB), that is an ordered list with the most specific route for each IP prefix at the top, created using the same data of the stable RIB table
So long story short is that the control plane manages and maintains a lot of things, such as details about link state, topology statistics details, and more.
It is here that are implemented real world network use-cases like switching, routing, L2 VPN, L3 VPN, firewall security rules, DNS, DHCP, and clustering.
All of these are implemented into a component called SDN controller: it exposes two types of interfaces:
- Northbound interface: APIs (typically REST based) is meant for communication with upper, Application layer and would be in general realized through REST APIs of SDN controllers
- Southbound interface: is meant for communication with lower, Infrastructure layer of network elements and would be in general realized through southbound protocols – Openflow, Netconf, Ovsdb, etc.
Examples of SDN Controllers are OpenDaylight, ONOS (Open Network Operating System), NOX/POX, Floodlight - they are all open source. An example of a commercial SDN controller is Cisco Open SDN Controller – it's based on OpenDaylight .
an area open to develop innovative network applications such as automation, configuration and management, monitoring, troubleshooting, policies and security
OpenFlow
Since SDN works with heterogeneous hardware of different vendors, a standard protocol is needed, and here is where OpenFlow comes into play: Openflow is a standard protocol (at the time of writing this post the current version is 1.5.1.) provided by Open Networking Foundation (ONF) that states specification for communication between SDN controllers and network equipment: routing decisions are taken by SDN controllers and let forwarding rules, security rules being pushed on switches of the underlying network.
Within OpenFlow, it's the switch that connects to the OpenFlow controller (default port is TCP/6653) either plain or using TLS, although the opposite is still possible.
The following example shows the command you would issue to connect the br-test bridge created on the gw-ca-ut1a001 VM to a OpenDaylight SDN controller listening on port TCP/6640 of a host with IP 172.16.10.11.
ovs-vsctl set-controller br-test tcp:172.16.10.11:6640
OpenFlow's Flows
Since the VM interconnecting the networks and implementing network policies is the "gw-ca-ut1a001" host, SSH connect to it using Vagrant:
vagrant ssh gw-ca-ut1a001
then switch to the "root" user:
sudo su -
Dumping The Currently Set Flows
Let's start by having a look to the default OpenFlow flow set on OVS - type the following statement
ovs-ofctl -O OpenFlow13 dump-flows br-test
the outcome is:
cookie=0x0, duration=5943.914s, table=0, n_packets=1956, n_bytes=209514, priority=0 actions=NORMAL
as you probably guessed, this is a "permit ALL" flow we talked about a while ago: we are about to configure a more restrictive policy.
Delete Every Flow
Before configuring a new policy, we must flush (delete all) the already defined flows:
ovs-ofctl del-flows --protocols=OpenFlow13 br-test
By now, since there are no flows defined, OpenVSwitch stops forwarding packages - we can easily verify if from the "as-ca-ut1a001" host.
Once connected, flush every network cache:
sudo ip -s -s neigh flush all
then try to ping the IP (192.168.0.254) of the "gw-ca-ut1a001" gateway on the same subnet where "as-ca-ut1a001" host is homed:
ping -c 1 192.168.0.254
the outcome is as follows:
PING 192.168.0.254 (192.168.0.254) 56(84) bytes of data.
--- 192.168.0.254 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
as expected, every packet is lost.
Mind that not only IP packets are dropped, ... even Layer 2 packages, such as ARP are.
Indeed, if we try to use the "arping" command line tool:
arping -c1 192.168.0.254
we receive no response:
ARPING 192.168.0.254 from 192.168.0.10 enp0s6.100
Sent 1 probes (1 broadcast(s))
Received 0 response(s)
and if we have a look to the ARP table:
arp -a
the outcome will look like is as follows:
? (192.168.0.254) at <incomplete> on enp0s6.100
? (10.211.55.2) at be:d0:74:a0:4b:64 [ether] on enp0s5
prl-local-ns-server.shared (10.211.55.1) at 00:1c:42:00:00:18 [ether] on enp0s5
The "<incomplete>" MAC address next to the "192.168.0.254" IP means that the ARP resolution is not working.
The Structure Of a Flow
Before writing flows, it is best to know the structure of a flow.
OpenFlow Flows basically carry three types of information:
- Match fields: it is the criteria to match packets based on header fields
- ARP fields, ICMP fields, MPLS field
- L2 (source destination ethernet addresses, VLAN ID, VLAN priority, etc.)
- L3 (IPv4/IPv6 source destination address, protocol type, DSCP, etc.)
- L4 fields (TCP/UDP/SCTP source destination port), .
- Actions: what to do with a packet matching the criteria
- drop
- forward on some port of switch
- modify packet (push/pop VLAN ID, push/pop MPLS label, increment/decrement IP TTL)
- forward to a specific queue of ports, etc.
- Counters: track how many packets matched the flow
Adding Flows
Let's see now how to add flows to OpenVSwitch using the "ovs-ofctl" command line utility:
Deny Everything Flow
Let's add an explicit drop flow for every packet sent to any of the subnets among the "192.168.0.0/16" supernet:
ovs-ofctl -O OpenFlow13 add-flow br-test --protocols=OpenFlow13 \
"cookie=0x1, priority=1, table=0, tcp, nw_dst=192.168.0.0/16, nw_proto=6, actions=drop"
Permit ARP Flow
The very first thing to permit is ARP, otherwise the hosts will not be able to guess MAC addresses from IP addresses and nothing will work.
Just add the following flows:
ovs-ofctl -O OpenFlow13 add-flow br-test --protocols=OpenFlow13 \
"cookie=0x2, priority=65535, table=0, priority=65535, arp,action=normal"
let's try again the same arping from the "as-ca-ut1a001" host:
arping -c1 192.168.0.254
this time we get 1 response:
ARPING 192.168.0.254 from 192.168.0.10 enp0s6.100
Unicast reply from 192.168.0.254 [0A:DE:DC:E1:27:8C] 3.634ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)
of course, if for now, if we try to ping the IP "192.168.0.254":
ping -c 1 192.168.0.254
the outcome is:
PING 192.168.0.254 (192.168.0.254) 56(84) bytes of data.
--- 192.168.0.254 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
So, as expected, every packet is lost. But if this time we look at the ARP table:
arp -a
the outcome looks like as follows:
? (192.168.0.254) at 0a:de:dc:e1:27:8c [ether] on enp0s6.100
? (10.211.55.2) at be:d0:74:a0:4b:64 [ether] on enp0s5
prl-local-ns-server.shared (10.211.55.1) at 00:1c:42:00:00:18 [ether] on enp0s5
so, as expected, this time the MAC address for the "192.168.0.254" IP is now resolved.
Permit ICMP Flows
Despite we are setting up quite a strict environment, since this is a Lab, it is safe to enable ICMP packages anyway - just add the following flows:
ovs-ofctl -O OpenFlow13 add-flow br-test --protocols=OpenFlow13 \
"cookie=0x3, priority=65535, table=0, icmp, icmp_type=0, icmp_code=0 actions=normal"
ovs-ofctl -O OpenFlow13 add-flow br-test --protocols=OpenFlow13 \
"cookie=0x4, priority=65535, table=0, icmp, icmp_type=8, icmp_code=0 actions=normal"
this time, from the "as-ca-ut1a001" host, we try pinging the "gw-ca-ut1a001" interface on the database subnet :
ping -c 1 192.168.36.254
as expected, this time it works:
PING 192.168.36.254 (192.168.36.254) 56(84) bytes of data.
64 bytes from 192.168.36.254: icmp_seq=1 ttl=63 time=1.31 ms
--- 192.168.36.254 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.312/1.312/1.312/0.000 ms
let's have a look to the flows defined on the "br-test" of the gw-ca-ut1a001 host:
ovs-ofctl -O OpenFlow13 dump-flows br-test
the outcome is:
cookie=0x2, duration=17.119s, table=0, n_packets=4, n_bytes=176, priority=65535,arp actions=NORMAL
cookie=0x3, duration=17.107s, table=0, n_packets=3, n_bytes=294, priority=65535,icmp,icmp_type=0,icmp_code=0 actions=NORMAL
cookie=0x4, duration=17.095s, table=0, n_packets=3, n_bytes=306, priority=65535,icmp,icmp_type=8,icmp_code=0 actions=NORMAL
cookie=0x1, duration=153.049s, table=0, n_packets=0, n_bytes=0, priority=1,tcp,nw_dst=192.168.0.0/16 actions=drop
Permit PostgreSQL Flows
We are still missing a flow to enable access to the PostgreSQL service on the "pg-ca-ut1a001" host - from the "as-ca-ut1a001" host, we try running:
psql -h 192.168.6.10 -U postgres
this time the connection hangs and we have to press "CTRL+C" to terminate it.
This is because of the "deny everything" flow we added.
To permit access again, on the gw-ca-ut1a001 host, we must add the following couple of flows:
ovs-ofctl -O OpenFlow13 add-flow br-test --protocols=OpenFlow13 \
"cookie=0x5, priority=65000, table=0, tcp, nw_src=192.168.0.0/24, nw_dst=192.168.6.0/24, tp_dst=5432, nw_proto=6, actions=normal"
ovs-ofctl -O OpenFlow13 add-flow br-test --protocols=OpenFlow13 \
"cookie=0x6, priority=65000, table=0, tcp, nw_src=192.168.6.0/24, nw_dst=192.168.0.0/24, tp_src=5432, tcp_flags=+ack, nw_proto=6, actions=normal"
on the "as-ca-ut1a001" host, run again the following statement:
psql -h 192.168.6.10 -U postgres
this time you must be able to connect to the PostgreSQL service on the "pg-ca-ut1a001" host - the password is "grimoire".
OpenFlow Flows Life time
As you probably are inferring, the flows added using the "ovs-ofctl" command line tool are ephemeral, and so they get lost after a system restart: as we saw, OpenFlow flows are fetched from the configured SDN controller.
Indeed, if you restart the "gw-ca-ut1a001" VM:
shutdown -r now
then log in again, switch to the "root" user, and dump the currently set flows:
ovs-ofctl -O OpenFlow13 dump-flows br-test
you get again the default "permit all" flow:
cookie=0x0, duration=5943.914s, table=0, n_packets=1956, n_bytes=209514, priority=0 actions=NORMAL
A Script To Automatically Load Openflow Flows From A File
For the sake of completeness - but consider it just a learning exercise, here is an example of a script that loads Openflow flows from a file and that can even be set as a one-shot Systemd service unit so to have them loaded at boot time.
This script and the related Systemd unit are just a toy - I wrote them only to enable my labs to implement network policies making use of Openflow without having to provision a VM dedicated to a SDN controller such as OpenDayLight.
Prerequisites
The script detects the changes applied to the file containing the Openflow flows to be loaded by using the "git" command line tool.
Because of this requisite, it is necessary to install git as follows:
dnf install -y git
then just create the directories where to store the script and the file containing the Openflow's flows:
mkdir -m 755 /opt/grimoire /opt/grimoire/bin /opt/grimoire/etc
OpenFlow Rules Loader Script
Create the "/opt/grimoire/bin/of-flow-load.sh" with the following contents:
#!/bin/bash
BRIDGE="${1}"
SCRIPT=$(readlink -f $0)
SCRIPTPATH=$(dirname ${SCRIPT})
CFG_FILE="${SCRIPTPATH}/../etc/of-flows.txt"
cd ${SCRIPTPATH}/../etc
if [[ ! -d .git ]]; then
echo "Initializing the Openflow configuration repository"
[ -f /tmp/openflow.lastcommitted ] && rm /tmp/openflow.lastcommitted
git init
git config user.name "System"
git config user.email "me@foo.org"
echo "foo.sh" > .gitignore
git add .gitignore
fi
git add ${CFG_FILE}
git commit -m "fix"
CURRENT=$(git log -1 --pretty=format:"%h")
if [[ ! -f /tmp/openflow.lastcommitted ]]; then
echo "Performing a full load ..."
ovs-ofctl del-flows --protocols=OpenFlow10,OpenFlow13 ${BRIDGE}
while IFS= read -r line
do
[[ "$line" =~ ^#.*$ ]] && continue
COMMAND=$(echo $line | sed "s/^\"/ovs-ofctl add-flow ${BRIDGE} --protocols=OpenFlow10,OpenFlow13 \"/g")
[ -n "${COMMAND}" ] && echo "Executing '${COMMAND}'"
eval ${COMMAND}
done < ${CFG_FILE}
else
PREVIOUS=$(cat /tmp/openflow.lastcommitted)
#echo PREVIOUIS=$PREVIOUS
#=$(git log -2 --pretty=format:"%h"|tail -n 1)
if [[ "${CURRENT}" == "$PREVIOUS" ]]; then
echo "We are alread at $PREVIOUS, so there's nothing to do, ... exiting"
exit 0
fi
ADD=$(git diff $PREVIOUS..$CURRENT |grep -- '+"')
REMOVE=$(git diff $PREVIOUS..$CURRENT |grep -- '-"')
#echo CURRENT=$CURRENT
#echo PREVIOUS=$PREVIOUS
while IFS= read -r line
do
COMMAND=$(echo $line| sed "s+^-\"\(cookie=0x[0-9]*\).*+ovs-ofctl del-flows ${BRIDGE} --protocols=OpenFlow10,OpenFlow13 \"\1/-1\"+")
[ -n "${COMMAND}" ] && echo "Executing '${COMMAND}'"
eval ${COMMAND}
done <<< "$REMOVE"
while IFS= read -r line
do
#echo $line
COMMAND=$(echo $line | sed "s/^+\"/ovs-ofctl add-flow ${BRIDGE} --protocols=OpenFlow10,OpenFlow13 \"/g")
[ -n "${COMMAND}" ] && echo "Executing '${COMMAND}'"
eval ${COMMAND}
done <<< "$ADD"
fi
echo ${CURRENT} > /tmp/openflow.lastcommitted
this script loads the flows defined in the "/opt/grimoire/etc/of-flows.txt" file in the OpenVSwitch's bridge specified as the first argument when launching it.
chmod 755 /opt/grimoire/bin/of-flow-load.sh
OpenFlow Service
The last step is creating the "/etc/systemd/system/openflow.service" with the following contents:
[Unit]
Description=Open vSwitch
Before=network.target network.service
After=network-pre.target ovsdb-server.service ovs-vswitchd.service openvswitch.service
PartOf=network.target
Requires=ovsdb-server.service
Requires=ovs-vswitchd.service
Requires=openvswitch
[Service]
Type=idle
ExecStartPre=-rm /tmp/openflow.lastcommitted
ExecStart=/opt/grimoire/bin/of-flow-load.sh br-test
[Install]
WantedBy=multi-user.target
and of course reload Systemd:
systemctl daemon-reload
then, enable the "openflow" service to start at boot time:
systemctl enable openflow
OpenFlow Flows File
The last missing bit is just the "/opt/grimoire/etc/of-flows.txt" file - create it with the flows we saw so far:
# drop everything sent to our whole /16 private subnet
"cookie=0x1, priority=1, table=0, tcp, nw_dst=192.168.0.0/16, nw_proto=6, actions=drop"
# permit ARP
"cookie=0x2, priority=65535, table=0, priority=65535, arp,action=normal"
# permit ICMP (PING)
"cookie=0x3, priority=65535, table=0, icmp, icmp_type=0, icmp_code=0 actions=normal"
"cookie=0x4, priority=65535, table=0, icmp, icmp_type=8, icmp_code=0 actions=normal"
# permit access to PostgreSQL service from 192.168.0.0/24 (www-t1) to 192.168.6.0/23 (db-t1)
"cookie=0x5, priority=65000, table=0, tcp, nw_src=192.168.0.0/24, nw_dst=192.168.6.0/24, tp_dst=5432, nw_proto=6, actions=normal"
"cookie=0x6, priority=65000, table=0, tcp, nw_src=192.168.6.0/24, nw_dst=192.168.0.0/24, tp_src=5432, tcp_flags=+ack, nw_proto=6, actions=normal"
then start the "openflow" service to have them loaded:
systemctl start openflow
Since the service has been configured to be loaded at boot, let's try a system reboot to see if it is really working as expected:
shutdown -r now
Once logged in again, and switched to the "root" user, if we dump the flows set on the "br-test":
ovs-ofctl -O OpenFlow13 dump-flows br-test
we will see the flows loaded by the script:
cookie=0x2, duration=52.946s, table=0, n_packets=4, n_bytes=168, priority=65535,arp actions=NORMAL
cookie=0x3, duration=52.933s, table=0, n_packets=0, n_bytes=0, priority=65535,icmp,icmp_type=0,icmp_code=0 actions=NORMAL
cookie=0x4, duration=52.921s, table=0, n_packets=0, n_bytes=0, priority=65535,icmp,icmp_type=8,icmp_code=0 actions=NORMAL
cookie=0x5, duration=52.909s, table=0, n_packets=0, n_bytes=0, priority=65000,tcp,nw_src=192.168.0.0/24,nw_dst=192.168.6.0/24,tp_dst=5432 actions=NORMAL
cookie=0x6, duration=52.896s, table=0, n_packets=0, n_bytes=0, priority=65000,tcp,nw_src=192.168.6.0/24,nw_dst=192.168.0.0/24,tp_src=5432,tcp_flags=+ack actions=NORMAL
cookie=0x1, duration=52.959s, table=0, n_packets=0, n_bytes=0, priority=1,tcp,nw_dst=192.168.0.0/16 actions=drop
OpenFlow Flows Sample Snippets
The last part of this post is a cheatsheet of snippets you can use as a reference when writing flows:
SSH From A Jump Station To A Subnet
A very common use case is to permit SSH access from a jump host to a subnet - the following snippet permits SSH from the host with IP 192.168.254.253 to the hosts in the 192.168.149.0/24 subnet.
# SSH from jump-ci-upa002 to any host of 192.168.149.0/24 subnet
"cookie=0x11, priority=65000, table=0, tcp, nw_src=192.168.254.253, nw_dst=192.168.149.0/24, tp_dst=22, nw_proto=6, actions=normal"
"cookie=0x12, priority=65000, table=0, tcp, nw_src=192.168.149.0/24, nw_dst=192.168.254.253, tp_src=22, tcp_flags=+ack, nw_proto=6, actions=normal"
Please note the use of the "+ack" flag in the second flow.
Access To A FreeIPA (Red Hat Directory Server)
Another common use case is permit to any host access to a Red Hat Directory Server (FreeIPA) - this time, since the amount of services, there are necessary more flows:
# FreeIPA Services on dir-ci-up3a001
"cookie=0x31, priority=65000, table=0, tcp, nw_dst=192.168.150.10, tp_dst=80, nw_proto=6, actions=normal"
"cookie=0x32, priority=65000, table=0, tcp, nw_src=192.168.150.10, tp_src=80, tcp_flags=+ack, nw_proto=6, actions=normal"
"cookie=0x33, priority=65000, table=0, tcp, nw_dst=192.168.150.10, tp_dst=443, nw_proto=6, actions=normal"
"cookie=0x34, priority=65000, table=0, tcp, nw_src=192.168.150.10, tp_src=443, tcp_flags=+ack, nw_proto=6, actions=normal"
"cookie=0x35, priority=65000, table=0, tcp, nw_dst=192.168.150.10, tp_dst=389, nw_proto=6, actions=normal"
"cookie=0x36, priority=65000, table=0, tcp, nw_src=192.168.150.10, tp_src=389, tcp_flags=+ack, nw_proto=6, actions=normal"
"cookie=0x37, priority=65000, table=0, tcp, nw_dst=192.168.150.10, tp_dst=636, nw_proto=6, actions=normal"
"cookie=0x38, priority=65000, table=0, tcp, nw_src=192.168.150.10, tp_src=636, tcp_flags=+ack, nw_proto=6, actions=normal"
"cookie=0x39, priority=65000, table=0, tcp, nw_dst=192.168.150.10, tp_dst=88, nw_proto=6, actions=normal"
"cookie=0x40, priority=65000, table=0, tcp, nw_src=192.168.150.10, tp_src=88, tcp_flags=+ack, nw_proto=6, actions=normal"
"cookie=0x41, priority=65000, table=0, tcp, nw_dst=192.168.150.10, tp_dst=464, nw_proto=6, actions=normal"
"cookie=0x42, priority=65000, table=0, tcp, nw_src=192.168.150.10, tp_src=464, tcp_flags=+ack, nw_proto=6, actions=normal"
"cookie=0x43, priority=65000, table=0, tcp, nw_dst=192.168.150.10, tp_dst=53, nw_proto=6, actions=normal"
"cookie=0x44, priority=65000, table=0, tcp, nw_src=192.168.150.10, tp_src=53, tcp_flags=+ack, nw_proto=6, actions=normal"
"cookie=0x45, priority=65000, table=0, udp, nw_dst=192.168.150.10, tp_dst=53, nw_proto=17, actions=normal"
"cookie=0x46, priority=65000, table=0, udp, nw_src=192.168.150.10, tp_src=53, nw_proto=17, actions=normal"
"cookie=0x47, priority=65000, table=0, udp, nw_dst=192.168.150.10, tp_dst=88, nw_proto=17, actions=normal"
"cookie=0x48, priority=65000, table=0, udp, nw_src=192.168.150.10, tp_src=88, nw_proto=17, actions=normal"
"cookie=0x49, priority=65000, table=0, udp, nw_dst=192.168.150.10, tp_dst=464, nw_proto=17, actions=normal"
"cookie=0x50, priority=65000, table=0, udp, nw_src=192.168.150.10, tp_src=464, nw_proto=17, actions=normal"
"cookie=0x51, priority=65000, table=0, udp, nw_dst=192.168.150.10, tp_dst=123, nw_proto=17, actions=normal"
"cookie=0x52, priority=65000, table=0, udp, nw_src=192.168.150.10, tp_src=123, nw_proto=17, actions=normal"
the flows in the above snippet permit access from everywhere to the IPA services of the host with IP 192.168.150.10.
Access To A NFS Server
Another very common use case is permitting access to a NFS server:
# NFS Services on fss-ci-up3a001
"cookie=0x75, priority=65000, table=0, tcp, nw_dst=192.168.152.10, tp_dst=111, nw_proto=6, actions=normal"
"cookie=0x76, priority=65000, table=0, tcp, nw_src=192.168.152.10, tp_src=111, tcp_flags=+ack, nw_proto=6, actions=normal"
"cookie=0x77, priority=65000, table=0, udp, nw_dst=192.168.152.10, tp_dst=111, nw_proto=17, actions=normal"
"cookie=0x78, priority=65000, table=0, udp, nw_src=192.168.152.10, tp_src=111, nw_proto=17, actions=normal"
"cookie=0x79, priority=65000, table=0, tcp, nw_dst=192.168.152.10, tp_dst=20048, nw_proto=6, actions=normal"
"cookie=0x80, priority=65000, table=0, tcp, nw_src=192.168.152.10, tp_src=20048, tcp_flags=+ack, nw_proto=6, actions=normal"
"cookie=0x81, priority=65000, table=0, udp, nw_dst=192.168.152.10, tp_dst=20048, nw_proto=17, actions=normal"
"cookie=0x82, priority=65000, table=0, udp, nw_src=192.168.152.10, tp_src=20048, nw_proto=17, actions=normal"
"cookie=0x83, priority=65000, table=0, tcp, nw_dst=192.168.152.10, tp_dst=662, nw_proto=6, actions=normal"
"cookie=0x84, priority=65000, table=0, tcp, nw_src=192.168.152.10, tp_src=662, tcp_flags=+ack, nw_proto=6, actions=normal"
"cookie=0x85, priority=65000, table=0, udp, nw_dst=192.168.152.10, tp_dst=662, nw_proto=17, actions=normal"
"cookie=0x86, priority=65000, table=0, udp, nw_src=192.168.152.10, tp_src=662, nw_proto=17, actions=normal"
"cookie=0x87, priority=65000, table=0, tcp, nw_dst=192.168.152.10, tp_dst=2049, nw_proto=6, actions=normal"
"cookie=0x88, priority=65000, table=0, tcp, nw_src=192.168.152.10, tp_src=2049, tcp_flags=+ack, nw_proto=6, actions=normal"
"cookie=0x89, priority=65000, table=0, tcp, nw_dst=192.168.152.10, tp_dst=32803, nw_proto=6, actions=normal"
"cookie=0x90, priority=65000, table=0, tcp, nw_src=192.168.152.10, tp_src=32803, tcp_flags=+ack, nw_proto=6, actions=normal"
"cookie=0x91, priority=65000, table=0, udp, nw_dst=192.168.152.10, tp_dst=32769, nw_proto=17, actions=normal"
"cookie=0x92, priority=65000, table=0, udp, nw_src=192.168.152.10, tp_src=32769, nw_proto=17, actions=normal"
"cookie=0x93, priority=65000, table=0, tcp, nw_dst=192.168.152.10, tp_dst=875, nw_proto=6, actions=normal"
"cookie=0x94, priority=65000, table=0, tcp, nw_src=192.168.152.10, tp_src=875, tcp_flags=+ack, nw_proto=6, actions=normal"
"cookie=0x95, priority=65000, table=0, udp, nw_dst=192.168.152.10, tp_dst=875, nw_proto=17, actions=normal"
"cookie=0x96, priority=65000, table=0, udp, nw_src=192.168.152.10, tp_src=875, nw_proto=17, actions=normal"
the flows in the above snippet permit access from everywhere to the NFS services of the host with IP 192.168.152.10: even here, because of the amount of services, there are necessary more flows.
Footnotes
Here it ends this post dedicated to configuring OpenFlow flows: we gradually learned how to set up flows using the "ovs-ofctl" command line tool, and even how to persist them for playing with our labs.
In the next post we will talk about VxLAN, going through the various available technologies, as usual seeing everything in action.
I hate blogs with pop-ups, ads and all the (even worse) other stuff that distracts from the topics you're reading and violates your privacy. I want to offer my readers the best experience possible for free, ... but please be wary that for me it's not really free: on top of the raw costs of running the blog, I usually spend on average 50-60 hours writing each post. I offer all this for free because I think it's nice to help people, but if you think something in this blog has helped you professionally and you want to give concrete support, your contribution is very much appreciated: you can just use the above button.