Introduction
Building Telco cloud is the first step in CSP (Communication Service Provider) transformation journey into self-operated; self-organised and self-healing network (I’m not talking here about LTE/Radio concept but in general).
Second step is to operate the environment efficiently. Telco operators usually tend to integrate multiple products from different vendors in any solution. Take ETSI NFV standard as an example; You will find many vendors exist in each block and they have to be integrated together
However this integration come at a price. multiple teams from different vendors are working on the environment at the same time and many changes could be rolled out that affecting the whole service if it’s not properly testing continuously. Does this word ring a bell or sound familiar? Yes continuous testing become one of the new building block in modern DevOps lifecycle
When Should you execute Tests in Cloud?
This philosophy dictates that testing can’t be treated as a separate stage at the end of the lifecycle. it should be early as possible and in every stage to collect feedback, correct the system behavior.
The more you “Shift left” in your planned acceptance testing strategy the more you can discover bugs and fix it before rollout to production with a “release”. if you applied the same concept in Telco cloud then you need to do the acceptance testing at following stages (Given you already have a functional Telco cloud environment)
1- Do acceptance testing on existing services (end to end) after any new change introduced to your openstack installer templates (TripleO or Kolla)
2- Do acceptance testing on existing services (end to end) after any new nodes added/deleted to/from the cloud (which include the servers, storage, fabric nodes, switches…etc)
3- Do acceptance testing on existing services (end to end) after any major changes on external networks such as Transport/Core/MPLS/IGW layers..etc
4- Do acceptance testing on existing services (end to end) on regular basis
Introducing RobotFramework
Robotframework is an open source automation tool developed using Python. it’s used in acceptance test driven development (ATDD), and robotic process automation (RPA). the robot is generic tool that could be used for any SUT (System Under Test, in our case it’s the Telco Cloud).
The power of this framework came in its readability. You think python syntax is designed to be readable?, think again when you see how easy to develop a test case for robot. it will not limit you with spaces, tabs, delimiters, parentheses and case sensitive words. you can use any words to describe your intention and use the pre-defined keywords to explain how to reach to that with rich set of libraries already available. even if you don’t find your keyword in any of them then you can easily develop a code and robot will load it. let’s take a quick demo example
The demo application is a very simple login page shown below. With user name demo and password mode you get into a welcome page, and otherwise you end up to an error page. How to start and stop the application yourself is explained in the Starting demo application section.
The robot file that do the testing scenario for us will be as following:
Yes, no code :). Just plain and clear english that describe your intention (to have a Valid Login) and how to reach to it(Test Case Procedures)!. but how this will be transformed in the backend? actually Robotframework core shipped with libraries that contains keywords which could be used to write any use case to test SUT (System Under Test)
without further ado, let’s jump to lab setup and execute our test cases
Lab Setup
Lab is consisting of normal 3-stages IP-CLOS fabric, Red Hat openstack 10 (Newton) and Juniper SDN Contrail 4.1.2
Test Scenario
My test scenario that I want to verify after any change in cloud is simple. I want to create Virtual Network in Contrail with route target, create a port with fixed IP address over this network, spawn virtual machine over certain compute node (availability zone) then verify this virtual machine IP is learnt to the SDN gateway and pingable from it under specific VRF. After test is passed, I want to clean everything (VM, port, virtual network…etc). This will verify end-to-end communication by touching different components such as Openstack, SDN and Fabric EVPN/VxLAN functionality
Step 1: Install Requirements
we will start by installing the robotframework from PyPI
pip install robotframework
and since we will interact with Openstack services and Juniper devices, then we will need some additional packages to communicate with API
pip install python-novaclient \
python-keystoneclient \
python-neutronclient \
python-glanceclient \
junos-eznc
Finally we need the SDN API bindings for Juniper contrail (VNC_API) which could be installed from Contrail Controller
Step 2: Create robot file
we will need two files to execute our tests. The first one contains the use cases description while the second one is the python code that robot will use to execute that use case. let’s start by robot file
All the following code will be found in my GitHub Repo
It contains 4 main parts, Settings , Variables, Test Cases , Keywords . in Settings you can call external libraries and specify procedure to run before/after any test
in Test cases section you can also define how to start the test case and tag each one with specific value that will be used to run this test only. (similar to ansible playbook tags!)
You can move some of these sections such as variables and keywords to external files called resources to have more organised/scalable file structure
Finally the Keywords will be called from your libraries that you imported in at beginning of the file
Step 3: Create PyBot file
Here where the fun begins, we need to create three python classes for interacting with VIM, SDN and Fabric.
All the following code will be found in my GitHub Repo
Each one will contains the “Keywords” located under test case. if robot fail to run any one of them (for example the API server is not reachable), it will show this test case as Fail. otherwise if the code run without any exception then it will be Pass
The below code for example will connect to Contrail API and try to create virtual-network with specific parameters such as Name, prefix, route target..etc
Same also for openstack class and juniper fabric class, each is containing keywords to do specific procedure inside the test case. The below function is designed to verify if the VM IP address is advertised to SDN gateways or not by leveraging the JunOS PyEZ module …you get the idea
Step 4: Run the test case
it’s time now to run the robot test. Launch your terminal and execute the following command. just don’t forget to change the directory location to where you stored you robot and pybot files.
/usr/local/bin/robot \
--pythonpath Scripts/TelcoCloud_Testing\
--outputdir Scripts/TelcoCloud_Testing\
-l DEBUG \
--include create_pod \
Scripts/TelcoCloud_Testing/test_basic_vnf_connectivity.robot
The output show how many tests are passed and how many failed. The ping failed in my case because of firewall filter applied on that network (which could be ignored for now). let’s check the SDN controller and Openstack to see if data is already exist or not
Starting by SDN controller you will find the network & ports is created with parameters passed by robot
Also in Openstack you will find the instance created with the desired name and attached to created port
Finally you should see the VM IP address is advertised to SDN gateway through BGP under respective routing-instances (VRFs) that share the route-target and the gateway build overlay tunnel to the compute node hosting that VM
Great! 🙂
Moreover you can check the report generated by the robotframework that contains more details about the test cases, execution time for each and statistics
Step 5: Destroy test elements
After you finish evaluating the result of each test case, it’s recommended to delete the created elements to clean up the fabric from non-production workloads. This could be done by calling delete_pod tag that will execute this cleanup
/usr/local/bin/robot \
--pythonpath Scripts/TelcoCloud_Testing\
--outputdir Scripts/TelcoCloud_Testing\
-l DEBUG \
--include delete_pod \
Scripts/TelcoCloud_Testing/test_basic_vnf_connectivity.robot
Wrapping Up
when we perform Acceptance Testing to the cloud then we need to make sure our cloud answers the acceptance criteria, defined by end users needs and requirements. we need to verify if the whole system as a single unit could be used or not after ANY change.
So given you already have multi-vendors environment and you should do testing on frequently & continuously then you will find yourself need to adapt some “Automated” Acceptance Testing framework and here it come RobotFramework. just you need a little effort on developing the test cases then you can use it everywhere to perform & orchestrate your tests
Build Once use Many!
Finally, I hope this will be informative for you, and I’d like to thank you for reading.
The Networker 
Share you opinion to benefit others :)