Several Google Employees published The Datacenter as a Computer. It’s freely available. The text covers a broad area, from monitoring to cost modelling for power but it’s quite digestable. At a light ~130 pages it’s an easy read. The bibliography is worth digging into if you want deeper dives on specific topics.

Operating Open vSwitch brings a new set of challenges. One of those challenges is managing Open vSwitch itself and making sure you’re up to date with performance and stability fixes. For example, in late 2013 there were significant performance improvements with the release of 1.11 ( flow wildcarding!) and in the 2.x series there are even more improvements coming. This means everyone running those old versions of OVS (I’m looking at you, <=1....

Measure all the things! Graphite & statsd are my weapons of choice. One set of metrics in particular that we wanted to measure are the various TCP stats, including TCP Retransmit rate. We crafted a Python script to send all of the metrics in a single UDP packet and hit a weird scenario. The python script was all ready to roll except that StatsD was only logging one metric. All of the metric packets were arriving at the StatsD instance, but only one was being processed....

This post is a deep dive into what happens in Nova (and where in the code) when a console URL is retrieved via the nova API for a Nova configuration backed by XVP and XenServer/XenAPI. Hopefully the methods used in Nova’s code will not change over time, and this guide will remain good starting point. Example nova client call: [code]nova get-vnc-console [uuid] xvpvnc[/code] And the call returns: +--------+-------------------------------------------------------------------------------------------------------+ | Type | Url | +--------+-------------------------------------------------------------------------------------------------------+ | xvpvnc | https://URL:PORT/console?...

This post is a collection of useful articles/videos that I’ve collected about networking on XenServer and Linux. XenServer Xen Network Throughput and Performance Guide (Technical Overview) XenServer: Under the Hood < Specifically device -> PIF -> network -> VIF relationship Linux (video) Through the Ether and Back Again < discusses python and the Linux Sockets API How SKBs work Queueing in the Linux Network Stack Linux Advanced Routing & Traffic Control HOWTO Linux Device Drivers 3rd Edition < specifically chapter 17 As you can see, there are a multitude of elements to consider when looking into host networking issues for a Linux VM running on XenServer (which is Linux underneath the covers anyway)....

There’s a good talk given by Gabe Westmaas at the HK OpenStack Summit: The talk describes what Rackspace monitors in the public cloud OpenStack deployment, how responses are handled, and some of the integration points that are used. I recommend watching it for OpenStack specific monitoring and a little context around this post. In this post I am going to discuss how the sausage gets made - how the underlying Nagios configuration is managed....

Similar to this thread, I wanted to see what VLANs were allowed for a trunked port as reported by SNMP with Python. With the help of a couple of colleagues, I made some progress. [code language=“python”] vlan_value = ‘000000000020000000000000000000000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000’ for key,value in enumerate(format(int(vlan_value, 16), “0100b”).rjust(len(vlan_value) * 4, ‘0’)): … if value == ‘1’: … print key … … … 42 146 [/code] Convert the string returned to Hex Convert that to Binary Right fill 0s to the appropriate length to give offset (determined by the size of the string) Loop through the resulting value and each character that is a 1 is an enabled VLAN on the port In conjunction with LLDP, I’m able to query each switch/port and interface is connected to and determine if the VLANs are set properly on the port....

A month or two ago I finally went through all the old hard drives I’ve accumulated over the past decade. I mounted each of the disks and moved a bunch of files onto my desktop’s drive. There were lots of photos from the drives that I don’t want to lose so I decided to get a little more serious about backups. I decided to give Duply a go. Duply is a wrapper for duplicity, which underneath it all uses the tried and trusted rsync....

In the physical world when you power on a server it’s already cabled (hopefully). With VMs things are a bit different. Here’s the sequence of events when a VM is started in Nova and what happens on XenServer to wire it up with Open vSwitch. nova-compute starts the VM via XenAPI XenAPI VM.start creates a domain and creates the VM’s vifs on the hypervisor The Linux user device manager manages receives this event, and scripts within /etc/udev/rules....

After discussing [ovs-discuss] ovs performance on ‘worst case scenario’ with ovs-vswitchd up to 100%. One of my colleagues had a good idea: tcpdump the physical interface and the vif at the same time. The difference between when the packet reaches the vif and the packet reaches the physical device can help measure the amount of time in a userspace->kernelspace transit. Of course, virtual switches aren’t the only culprit in virtual networking overhead- virtual networking is a very complex topic....