Posts

Capacity Management/Provisioning (Cloud’s full, Can’t build here) - Video, Slides As a service provider, Rackspace is constantly bringing new OpenStack capacity online. In this session, we will detail a myriad of challenges around adding new compute capacity. These include: planning, automation, organizational, quality assurance, monitoring, security, networking, integration, and more. Managing Open vSwitch Across a Large Heterogenous Fleet - Video, Slides Open vSwitch (OVS) is one of the more popular ways to provide VM connectivity in OpenStack....

Recently I got to work with the NSX API and write a tool to do a quick health check of NSX networks. nsxchecker is a valuable operational tool to quickly report a NSX network’s health. One of the promises of SDN is automated tooling for operational teams and with the NSX API I was quickly able to deliver. nsxchecker accepts a NSX lswitch UUID or a neutron_net_id. Rackspace’s Neutron plugin, quark, tags created lports with a neutron_net_id....

At the OpenStack Operators meetup the question was asked about monitoring issues that are related to RabbitMQ. Lots of OpenStack components use a message broker and the most commonly used one among operators is RabbitMQ. For this post I’m going to concentrate on Nova and a couple of scenarios I’ve seen in production. It’s important to understand the flow of messages amongst the various components and break things down into a couple of categories:...

Over the last few months I’ve done a bit of work around monitoring, Open vSwitch, and XenServer. This post lists some of the networking/Open vSwitch specific items to monitor on hypervisors. Link Status: Nagios SNMP Interfaces plugin works well for reporting a failed link as well as reporting error rates and inbound/outbound bandwidth. Open vSwitch Manager and Controller Status: Transport Node Status is a quick and dirty python script which can be used with extended SNMP to alert when OVS loses a connection to a manager/controller....

A couple of interesting research papers around failure, found in The Datacenter as a Computer. Failure Trends in a Large Disk Drive Population (2007) Out of all failed drives, over 56% of them have no count in any of the four strong SMART signals, namely scan errors, reallocation count, offline reallocation, and probational count. In other words, models based only on those signals can never predict more than half of the failed drives....

In late March I relocated from San Antonio, TX to Lexington, KY. Same awesome job just with a twist…REMOTE WORK! I am mainly collaborating via IRC, tmux, 1:1/M:M TeamSpeak, and 1:1/M:M video conferencing. My takeaways after the first month: Obvious, but it took me a while: When you’re talking, LOOK AT THE CAMERA - multiple monitors and a MacBook make this a little awkward Quality matters: appropriate microphones & video, particularly conference rooms, make a huge difference....

Several Google Employees published The Datacenter as a Computer. It’s freely available. The text covers a broad area, from monitoring to cost modelling for power but it’s quite digestable. At a light ~130 pages it’s an easy read. The bibliography is worth digging into if you want deeper dives on specific topics.

Operating Open vSwitch brings a new set of challenges. One of those challenges is managing Open vSwitch itself and making sure you’re up to date with performance and stability fixes. For example, in late 2013 there were significant performance improvements with the release of 1.11 ( flow wildcarding!) and in the 2.x series there are even more improvements coming. This means everyone running those old versions of OVS (I’m looking at you, <=1....

Measure all the things! Graphite & statsd are my weapons of choice. One set of metrics in particular that we wanted to measure are the various TCP stats, including TCP Retransmit rate. We crafted a Python script to send all of the metrics in a single UDP packet and hit a weird scenario. The python script was all ready to roll except that StatsD was only logging one metric. All of the metric packets were arriving at the StatsD instance, but only one was being processed....

This post is a deep dive into what happens in Nova (and where in the code) when a console URL is retrieved via the nova API for a Nova configuration backed by XVP and XenServer/XenAPI. Hopefully the methods used in Nova’s code will not change over time, and this guide will remain good starting point. Example nova client call: [code]nova get-vnc-console [uuid] xvpvnc[/code] And the call returns: +--------+-------------------------------------------------------------------------------------------------------+ | Type | Url | +--------+-------------------------------------------------------------------------------------------------------+ | xvpvnc | https://URL:PORT/console?...