DISCLAIMER: I’m still getting my feet wet with Open vSwitch. This post is just a cleaned up version of my scratchpad.
Open vSwitch has a few ways of providing rate limiting - this deep dive will go into the internals of reverse engineering an existing virtual interface’s egress rate limits applied with tc-htb. Hierarchy Token Bucket (htb) is a standard linux packet scheduling implementation. More reading on HTB can be done on the author’s site - I found the implementation and theory pretty interesting.
This is current as of Open vSwitch 1.9.
The information needed to retrieve htb rate limits mostly lives in the ovsdb:
Open vSwitch Schema ( http://openvswitch.org/ovs-vswitchd.conf.db.5.pdf)
Things can get complex depending on how your vifs plug into your physical interfaces. In my case, OpenStack Quantum requires an integration bridge which I’ve attempted to diagram:
- On instance boot, vifs are plugged into xapi0. xapi0’s controller nodes pull down information including flows and logical queues.
- The flows pulled from (1) set the destination queue on all traffic for the source IP address for the interface.
- The queue which the traffic gets sent to goes to a linux-htb ring where the packets are scheduled.
Let’s take a look at an example. I want to retrieve the rate limit according to the hypervisor for vif2.1 which connects to xapi0, xenbr1, and the physical interface eth1. The IP address is 10.0.0.37.
Steps:
- Find the QoS used by the physical interface:
# ovs-vsctl find Port name=eth1 | grep qos qos : 678567ed-9f71-432b-99a2-2f28efced79c - Determine which queue is being used for your virtual interface. The value after set_queue is our queue_id.
# ovs-ofctl dump-flows xapi0 | grep 10.0.0.37 | grep "set_queue" ... ,nw_src=10.0.0.37 actions=set_queue:13947, ... - List the QoS from the first step and its type. NOTE: This command outputs every single OpenFlow queue_id/OVS Queue UUID for the physical interface. The queue_id from the previous step will be the key we’re interested in and the value is our Queue’s UUID
# ovs-vsctl list Qos 678567ed-9f71-432b-99a2-2f28efced79c | egrep 'queues|type' queues : { ... 13947=787b609b-417c-459f-b9df-9fb5b362e815,... } type : linux-htb - Use the Queue UUID from the previous step to list the Queue:
# ovs-vsctl list Queue 787b609b-417c-459f-b9df-9fb5b362e815 | grep other_config other_config : {... max-rate="614400000" ...} - In order to tie it back to tc-htb we have to convert the OpenFlow queue_id+1 to hexadecimal (367c). I think it’s happening here in the OVS code, but I’d love to have a definitive answer.
# tc -s -d class show dev eth1 | grep 367c | grep ceil # Queue ID + 1 in Hex class htb 1:367c ... ceil 614400Kbit