Troubleshoot Openstack Networking with Python

As an Openstack Administrator for a while, I found the most complicated topic to be understood in openstack project is Networking and how instances ( formerly virtual machines) are communicated with each other and with external world.

Unlike VMWare ESXI, where you can just create vSwitch and attach a VM to it, Openstack Networking is much more complex that that. You need first to define Network type itself(Flat, VLAN, VxLAN, GRE), attach it to the subnet with IPv4 or IPv6 Block, Create a Floating IP address if this network will be connected externally to a provider network and optionally create an internal router to route between different networks and subnets. lots of steps!

Floating ip and neutron router in nutshell

Also Neutron itself doesn’t provide an actual networking to the instances. it just a wrapper to a drivers called “Mechanism Drivers” on which they provide the actual networking(switching, routing and so on). The most famous one is the OpenVswitch which provide basic and advanced switching between instances and external world

But openvswtich lack the capability of enforcing security policy over incoming and outgoing packets, That’s why Openstack community choose to connect the linuxBridge with OpenVswitch to solve this problem which introduce another layer of complexity!

Imagine that you need to attach one ethernet interface to instance. Openstack creates additional Four different interfaces to satisfy the need of OpenVswitch and LinuxBridge. Very complex approach really!. Below is an example of these interfaces. You can find more about it by clicking on image itslef!

So Where’s the problem?

when you face a problem in openstack networking like instance is not pingable from outside world, You can’t reach the instance gateway, You can’t get an IP address from DHCP pool or even you don’t see any incoming or outgoing traffic from it. Chances are you’ve missed something in networking configuration and you need to fix it.

Let’s start by answering the following questions:

1- How many interfaces assigned to an instance

2- What’s  the MAC address of each interface?

3- What’s the IP address of each interface?

4- What’s the internal VLAN assign by OVS to our interface?

5- How will the External Network (Provider Network) treat the traffic from each interface? (Strip VLAN, Add a VLAN, Modify a VLAN..etc)

6-Which ports in integration bridge (br-int) and External Bridge (br-ex) are connecting our instance ? and which flow table rules are applied on them

Answering the above questions will help us , a lot , in troubleshooting any networking problem in openstack.

You can use some useful commands like (ip a , ovs-vsctl show, neutron port-list ..etc) in your troubleshooting. However you will spent a lot of time trying to connecting everything together especially if you’ve environment with hundreds of instances and hundreds of networks.

So let’s Automate this job by using Python!

I wrote a python script that can do this job easily. it utilize two famous python libraries requests and netmiko  to connect to Openstack Keystone API service , grep the required information from it and parse the returned info and finally connecting the dots . it will print a nice report with all detailed information.

So How Does it Work?

First we define the Openstack Credentials (if you’ve multi-node installation then define the keystone ip address)

Then will send API request to Openstack Keystone to generate a Token. Token will be used later to authenticate us against any other openstack service

You can see the Openstack Token workflow in below picture. Don’t forget that Token has an expiration time so you have to use it before that date.

Next we will parse the returned output to find out the MAC address and IP address. I wrote a function on which you give it an instance name and it will do the rest. I thought it would be better to write it like that in case I need to use it later in any of my other projects

Unfortunately, OVS doesn’t provide an API interface like openstack. So I had to use the netmiko  library to send the required commands and parse the output using the linux text stream like cut and grep commands. The returned output require additional handling on which I choose to do from Python itself.

Finally I defined a function that use the above two methods to generate the required report. The returned output is concatenated together and grouped per physical (or should I say Virtual!) network interface

This is where you connect the dot for each part of neutron project either neutron itself or any defined mechanism drivers.

Testing:

Running this code against one my Openstack environment, I can easily identify how openstack networking handle and forward the traffic from each interface in my instance.

You can answer the above mentioned questions, whether the IP address, MAC , VLAN tagging and OVS bridge handling per Network interface attached to the instance

You can even visualize it

Finally you can Find the code in my GitHub repo here

Wrapping Up

Many people find openstack is complex and hard to understand, I partially agree with them. However it provides a lot of tools and interfaces on which can be used to get the job done and make your life easy. Neutron is a great and modular project under openstack umbrella and you can automate a lot of neutron tasks by using Python. The Sky is your only limit.

I hope this has been informative for your and I’d like to thank you for reading. Feel free to comment or share your experience in troubleshooting problems in openstack