Using Python Multi-Processing for Networking–Part 1

Python became the de-facto standard for Network Automation  nowadays, Many Network Engineers already use it on daily basis to automate networking tasks starting from configuration to operation till troubleshooting the network problems. in This post, we will visit one of advanced -yet- topics in python and scratch the multi-processing nature of python and learn how to use it to accelerate the script execution time.

First, We need to understand, How Computer execute a python script?


1- When you type #python <your_awesome_automation_script>.py in the shell, Python (which run as a process)  instruct your computer processor to schedule a thread  (which is the smallest unit of processing)


2- The allocated thread will start to execute your script , line by line. Thread can do anything, starting from interacting with I/O devices, connecting to router, printing output, doing mathematical stuff..anything

3- Once The script hit the EOF (End of File), the thread will be terminated and returned to the free pool to be used by other processes

in linux, You can use #strace –p <pid> to trace a specific thread execution

The more threads you assigned to script (and permitted by your processor or OS), the faster your script will run. Actually threads sometimes called “Workers” or “Slaves

I have a feeling that you’ve that little idea in your head, Why wouldn’t we assign a LOT of threads from all cores to python script in order to get the job done,  Quickly!

The problem with assigning a lot of threads to one process without special handling is what’s called “Race Condition”. The operating systems will allocate memory to your process (in this case it’s python process) to be used at the runtime and accessed by all Threads, ALL OF THEM AT THE SAME TIME. Now imagine one of those threads is reading a data before it’s actually written by another thread!.you don’t know the order in which the threads will attempt to access the shared data and this is called Race Condition.


You can read about the race condition and how to avoid it in this link

one of the available solutions is to make thread acquire lock, in fact, Python, by default is optimized to run as a single threaded process and has something called GIL(Global Interpreter Lock). GIL does not allow multiple threads to execute Python code at the same time.

but rather than have Multi-threads, Why Don’t we have Multi-processes?!!

The beauty of the multi-processes over multi-thread is you won’t be afraid from data corruption due to shared data among the threads.  Each spawned process will have their own allocated memory that won’t accessed by other python processes. This will allow us to execute parallel tasks at the same time!


Also, from Python PoV, each process has it’s owned GIL. so there’s no resource conflict or race condition here.

Enough Talk, Let’s jump to the code.

First you need to import the module to your python script

import multiprocessing as mp

Second, you need to wrap your code with a python function, this will allow the process to “target” this function and mark it as a parallel execution

let’s say we have a code that connect to router and execute command on it using netmiko library and we want to connect to all devices in parallel, This is a sample “serial” code


We need to assign number of process equal to number of devices (one process will connect to one device and execute the command) and set the target of the process to execute the function we wrapped around our previous code. This is an example


Finally we need to launch the process


behind the scene, the main thread that execute the main script will start to “Fork” a number of processes equal to number of devices and each of them targeting one function that execute “show arp” on all devices at the same time and store the output in a variable without affecting each other, brilliant!

this is a sample view for the processes inside the python now when you execute the full code.


one final thing need to be done, is to “join” the “forked” process again to the main thread/truck in order to finish the program execution smoothly.



I have a script used to push some initial configuration to lab devices (9 routers) and I decided to re-write it again using same above procedures and measure the time and here’s the findings (the value that we’re looking for is “real” row)

With Multi-Processing


Without Multi-Processing


You can spot the difference, it’s 6 time faster than the serial execution! and the gap will be increased as you add more devices.

Wrapping Up

in one simple word,

Don’t just Automate the task,  make it fast!

In next post, we will explore some additional flags for multi-processing library inside the python

Finally , I hope this has been informative for your and I’d like to thank you for reading.


BGP Visualization Using Python


During my Network study, I always admire the way that BGP works and operate. The black magic that handle how the packets are exit from one country (Autonomous System=ASN for short!) and enter the another without any “Boarding-Pass” or “Visa”. Not just that, but BGP strive to make the travel time and path is the shortest one amongst the available routes to destination using of exchanged path attributes between those ASN. very clever, robust and old Winking smile

But as the internet routing table grows and number of assigned ASN increase each day, I think it’s become harder to visualize the interconnection between ASNs. Everyday a new ASN is being connected to a bunch of other ASNs and it’s really hard to trace those connections with just a raw data provided by Route Glass Servers or RIRs.

So I think it’s time to involve Python to solve this problem. Basically I tried to build a python module that answer the below questions:

1- How ASNs are being connected to each other given a list of ASNs?

2- How ASNs are being connected in one country?

3- Which ASNs are considered as an Service Provider or IXP? (has more than 15 BGP peering with other ASN)

4- Which ASN is considered an Upstream to a specific ASN? That will help in defining the ASN gateway for a country for example.

5- Which ASN is considered a Downstream (Customer) to a specific Service Provider (Operator)? and how are they distributed comparing to other Service Providers?

6- Finally I need all of that in one picture, visually! You know the old say,

A picture is worth a thousand words

and those thousand words are being stored in a lot of  RIRs like RIPE and AFRNIC that provide useful BGP data publically but in raw format.

I started developing a python module to address the above questions and in two days I have a promising prototype and able to visualize the first country then later on added a capability to visualize a portion set of ASNs. Final stage is adding some console logs for troubleshooting and publishing the package into PyPI . I called it  (bgp_visualize) , Yeah, Couldn’t find a better one Open-mouthed smile.

I tried to design the bgp_visualize to work with a minimum set of possible data. For  example in case of visualizing BGP in specific country, you need to just provide the country code. However you can customize the way and look for the generated graph by providing few parameters like node_color , node_size , desired_background  and so on.

Working on Module

First you need to install it (and install python 2.7 of course if you didn’t have it already in your machine)

Using CLI

pip install bgp_visualize

Using GUI (Pycharm IDE)

Open Settings | Project Interpreter | | Add New

Then search for bgp_visualize python module and install it

Then Run the below code to visualize a set of ASNs (You can run it also from Python native IDLE if you’re using Windows OS)

from bgp_visualize import bgp_visualize_asn
ASNs= bgp_visualize_asn.bgp_visualize(asns=[8452,24835],dark=True)

The resulting graph will be something like below (Click on Image for better resolution) and to visualize all autonomous systems in specific country, you need to provide the country code to the object

from bgp_visualize import bgp_visualize_asn
country= bgp_visualize_asn.bgp_visualize(country='sa')

There’re a lot of screenshots for different BGP graphs are available in my Github page, so please check them out!. Also you can send me your generated graph and I’ll add it to the Github

Color Map

bgp_visualize module use different colors to represent the Autonomous System role in the graph. below is the list of colors and meaning of each in the generated graph

First if AS is considered to be  a service provider or IXP, then it will be colored with one of below colors

if AS is an upstream for specific ASN, then it will be colored as blue

ASN is Downstream:

Transit or not defined

Wrapping Up

I really enjoyed working on this package!.You can use it to troubleshoot and visualize any ASN in your network or in your country and understand the upstreams and downstreams for each one and easily identify the service providers, All in one graph!

For me, I had that idea long time ago to visualize every ASN, every connection, every prefix in the planet and draw them in nice and presentable way and I think this package is a good start, That’s my dream!.

Finally , I hope this has been informative for your and I’d like to thank you for reading.

Visualizing Python Module for Network Libraries (Netmiko and PyEZ)

Ever wondering How a python custom module or class is manufactured ? How does the developer write the python code and glue it together to create this nice and amazing “x” module ? What’s going on under-the-hood?

Documentation is good start of course, but we all know that it’s not usually updated with every new step or detail that developer add.

For Example,

we all know the powerful netmiko library created and maintained by Kirk Byers that utilize another popular SSH library called Paramiko. but we don’t understand the details and how the classes are connected together. we just write the below code  to execute a specific command in Cisco IOS platform

from netmiko import ConnectHandler

device = {"device_type":"cisco_ios",


net_connect = ConnectHandler(**device)
output = net_connect.send_command("show arp")

and booom, it’s working like charm

if you need to understand the magic behind the “self.charm” that netmiko use to return the result,  Please follow the below steps (require Pycharm IDE)

Netmiko Module


Open the Netmiko location inside the python library location folder (usually C:\Python27\Lib\site-packages) in Pycharm IDE


Step 2:

Right click on python module and choose Diagrams then Show Diagram


it will take some time to generate the diagram based on Your Java xms settings ( usually I assign 1024MB for it to work properly)

Step 3:

Now save the resulting  image into your desktop



And Here’s it (click on it to enlarge and zoom as much as you can)


Understand the result UML graph

Based on the resulting graph you can see that Netmiko is supporting a lot of vendors  like HP Comware, entrasys , Cisco ASA, Force10, Arista, Avaya,..etc and all of these classes are are inheriting from parent netmiko.cisco_base_connection.CicsoSSHConnection class (I think because they use the same SSH style as Cisco) which in turn inherit from another Big parent Class called netmiko.cisco_base_connection.BaseConnection

Also you can see that Juniper has it’s own class ( that connect directly to the Big parent

And finally we connect to the parent of all parents in python. The Object class (remember everything in Python is an object at the end)

Also you can find a lot of interesting things like SCP transfer class, SNMP class and with each one your will find the methods and parameters used to initialize this class

So the ConnectHandler method is primarily  used to check the device_type availability in above vendor classes and based on it , it will use the corresponding SSH Class. A well designed module really, Thanks Kirk Byers !



Juniper PyEZ Module

Applying the same procedure in Juniper Python wrapper (PyEZ) used to connect to JunOS based platforms. we can get another useful diagram as below (Again click to enlarge)


Also you can see the support for the netconf, serial and telnet connection and Views and Tables classes that Juniper use to define the facts about the device. Very clever.

That’s all. Please share any other interesting findings in python modules

I hope this has been informative for your and I’d like to thank you for reading.

Install Cacti on CentOS 7– The Definitive Guide in 2017


Cacti is one of the most robust  monitoring tool available in the market. It has a lot of features and options that able to give you a complete visibility for your infrastructure. In this guide I will walk through installing Cacti on CentOS 7, Configuring the MariaDB (The new Database for the CentOS 7 ), Adding few devices and finally plotting the result. Also  I will cover some troubleshooting points during the configuration. So Let’s start


Read More »

Troubleshoot Openstack Networking with Python

As an Openstack Administrator for a while, I found the most complicated topic to be understood in openstack project is Networking and how instances ( formerly virtual machines) are communicated with each other and with external world.

Unlike VMWare ESXI, where you can just create vSwitch and attach a VM to it, Openstack Networking is much more complex that that. You need first to define Network type itself(Flat, VLAN, VxLAN, GRE), attach it to the subnet with IPv4 or IPv6 Block, Create a Floating IP address if this network will be connected externally to a provider network and optionally create an internal router to route between different networks and subnets. lots of steps!


Floating ip and neutron router in nutshellimage

Also Neutron itself doesn’t provide an actual networking to the instances. it just a wrapper to a drivers called “Mechanism Drivers” on which they provide the actual networking(switching, routing and so on). The most famous one is the OpenVswitch which provide basic and advanced switching between instances and external world


But openvswtich lack the capability of enforcing security policy over incoming and outgoing packets, That’s why Openstack community choose to connect the linuxBridge with OpenVswitch to solve this problem which introduce another layer of complexity!

Imagine that you need to attach one ethernet interface to instance. Openstack creates additional Four different interfaces to satisfy the need of OpenVswitch and LinuxBridge. Very complex approach really!. Below is an example of these interfaces. You can find more about it by clicking on image itslef!

So Where’s the problem?

when you face a problem in openstack networking like instance is not pingable from outside world, You can’t reach the instance gateway, You can’t get an IP address from DHCP pool or even you don’t see any incoming or outgoing traffic from it. Chances are you’ve missed something in networking configuration and you need to fix it.

Let’s start by answering the following questions:

1- How many interfaces assigned to an instance

2- What’s  the MAC address of each interface?

3- What’s the IP address of each interface?

4- What’s the internal VLAN assign by OVS to our interface?

5- How will the External Network (Provider Network) treat the traffic from each interface? (Strip VLAN, Add a VLAN, Modify a VLAN..etc)

6-Which ports in integration bridge (br-int) and External Bridge (br-ex) are connecting our instance ? and which flow table rules are applied on them

Answering the above questions will help us , a lot , in troubleshooting any networking problem in openstack.

You can use some useful commands like (ip a , ovs-vsctl show, neutron port-list ..etc) in your troubleshooting. However you will spent a lot of time trying to connecting everything together especially if you’ve environment with hundreds of instances and hundreds of networks.

So let’s Automate this job by using Python!

I wrote a python script that can do this job easily. it utilize two famous python libraries requests and netmiko  to connect to Openstack Keystone API service , grep the required information from it and parse the returned info and finally connecting the dots . it will print a nice report with all detailed information.

So How Does it Work?

First we define the Openstack Credentials (if you’ve multi-node installation then define the keystone ip address)


Then will send API request to Openstack Keystone to generate a Token. Token will be used later to authenticate us against any other openstack service


You can see the Openstack Token workflow in below picture. Don’t forget that Token has an expiration time so you have to use it before that date.

Next we will parse the returned output to find out the MAC address and IP address. I wrote a function on which you give it an instance name and it will do the rest. I thought it would be better to write it like that in case I need to use it later in any of my other projects


Unfortunately, OVS doesn’t provide an API interface like openstack. So I had to use the netmiko  library to send the required commands and parse the output using the linux text stream like cut and grep commands. The returned output require additional handling on which I choose to do from Python itself.


Finally I defined a function that use the above two methods to generate the required report. The returned output is concatenated together and grouped per physical (or should I say Virtual!) network interface


This is where you connect the dot for each part of neutron project either neutron itself or any defined mechanism drivers.



Running this code against one my Openstack environment, I can easily identify how openstack networking handle and forward the traffic from each interface in my instance.

You can answer the above mentioned questions, whether the IP address, MAC , VLAN tagging and OVS bridge handling per Network interface attached to the instance

You can even visualize it Smile


Finally you can Find the code in my GitHub repo here

Wrapping Up

Many people find openstack is complex and hard to understand, I partially agree with them. However it provides a lot of tools and interfaces on which can be used to get the job done and make your life easy. Neutron is a great and modular project under openstack umbrella and you can automate a lot of neutron tasks by using Python. The Sky is your only limit.

I hope this has been informative for your and I’d like to thank you for reading. Feel free to comment or share your experience in troubleshooting problems in openstack

Introduction to SDN and NFV

If you’re confused about what’s the difference between SDN, NFV, Overlays and automation or what’s the role of each technology and how they’re connected

These are introductory slides for explaining the SDN and NFV technologies. what’s the difference between them and when each one is used. Also it talk about some of Cisco products in each area either SDN or NFV or the Automation with some of real use cases deployed in today’s service provider network.

Hope it’s useful and you like it.


NFV ETSI Lab in Egypt

in last few weeks, I’ve been involved on building and designing the NFV lab according to ETSI standard in my company. The ETSI standard is shown in below snapshotimage

The concept of NFV is simple. it tends to convert the functions that exist in your “physical network” to a virtual. functions like DPI, BNG and route reflectors will be converted to Virtual machines.

The left hand side of the picture is called “MANO = Management and Orchestration” where the right hand side of the standard is the real hardware and bunch of hypervisors (KVM, ESXI)


This allow to create many use cases such as service chaining on which subscriber traffic could be easily passed by any type of VNFs (Virtual Network Functions) regardless of it’s physical location. For example the below subscriber traffic is passed by virtual firewall, DDoS and virtual DPI before sending it out to the internet. Other subscriber traffic could be passed by a different “chains”


Cisco has a wide portfolio that cover most of the ETSI components, Let’s explore them in brief


NFV-O : Orchestrator

Cisco has the NSO product (Network Service Orchestrator, before it’s called Tail-F NCS). Tail-F has a huge contribution in defining the YANG language used in service modeling in the NFV. Cisco acquire the Tail-F company two years ago and it’s one of the most successful acquisition in cisco. You can read more about the Tail-F in thislink. The orchestrator job is to orchestrate the service creation over the VNFs and push the correct configuration on them based on many triggers

NSO use a concept of NEDs (Network Element Drivers) that capable of communicate with many many vendors like Sandvine, Palo-alto, Juniper and of course Cisco. it also capable of communicate using the NetConf protocol that allow it to not only orchestrate the VNF but also the PNF (Physical Network Functions – The real hardware and ASICs).

VNF-M: VNF Manager

because your network functions will be a bunch of VMs (Firewall, DPI,..etc). You will need to have a “manager” that manage the CRUD(Creation, Redeploy, Update and Deletion) operation of those VMs. Cisco has a product that called ESC (Elastic Service Controller) that integrate very well with NSO and any type of orchestrator in northbound. In southbound it ‘s capable to communicate with Openstack and VMware through standard RESful services.


The virtual infrastructure manager (openstack or vcenter) are responsible of creating the actual HDD, RAM and CPU for the VNF. Cisco recommend to integrate with RedHat Openstack (RDO)

EMS (or VNF)

This is the network function that become virtual! . I used in this lab the Cisco Cloud Service Router (CSRv) that capable on running most of the ASR functions without a problem (side note: I used it to build a complete SP-WiFi lab for one of the operator here in Egypt and it work very well in EAP-SIM and Portal based scenarios). below is the available VNFs from cisco


My NFV Lab


First I tried  to integrate the Cisco Elastic Service controller with Vmware vCenter but not having much luck on this integration. I stucked in starting the orchestrator service in vCenter process on which I’m thinking it’s one of the component that used by cisco ESC for communicating with VMware infrastructure. also vCenter seems complex solution to me on which limiting my options


Although the ESC is connected to vCenter, and able to read all “tenants” or VMs from it, but it was unable to administrate them



Hmmm, Ok. let’s seal to the other destination, The Openstack Smile

Second I imported the ESC to the openstack and installed it using the python script.


Great. next step is to integrate the ESC with Openstack that took only two minutes! (Thank you VMware for wasting my time!!)

Third, Once Integration is done, You can see that ESC can successfully retrieve all the tenants from the openstack


Also it’s capable of communicating with Openstack services like nova, cinder


And finally it’s capable of reading all compute hosts and hosted instances


Fourth, Push the configuration from orchestrator to ESC and watch Openstack create the images, flavors and attach all networks to CSRv (through the ESC)



Instances page


if you check the ESC portal, you will see immediately the CSR VNF Active, up and running in the ESC


And you can access the CSRv console directly from the ESC through the built-in VNC utility


But really, what’s the job of the ESC?

ESC play a vital role as a VNF manager in monitoring the VNF operations.  for example if one of the VNF that created through the ESC is deleted by mistake. the ESC will detect this event and immediately re-deploy the impacted machine without any intervention from your side. You can program it with many events to be monitored like overloading the VM, underloading, License experience..etc


The ESC communicate with openstack through REST messages over HTTP and order it to create VM (VNF) with specific flavor, Image ID and attached networks) as shown in blow snoop between ESC and Openstack



NFV is one of the hottest topics in service provider area and it soonest they will convert to this model to save much power and space in datacenter and more important is to introduce agility and harmony in today’s complex network. I really recommend you to choose open standard solutions and not limit your options to propriety software. Learn openstack and YANG modeling and be open minded to automation mythology

Finally I’ve the complete NFV lab integrated components (Orchestrator, VNF-Manager, Openstack and VNF) up and running in my company lab. I think it’s the first lab of it’s type here in Egypt to the best of my knowledge Smile

for any questions, please post a comment and we can discuss it together

Thank You