Monday, January 26, 2015

Validating BGP Announcements by Automating Filter Generation with Python: Part2

At first part of the post, i wrote that some tools use “whois”  client to query RIR databases. So it is possible to use a “whois cli tool” to gather the necessary data of your interest.


You can find whois flags for RIPE on  to elaborate the whois queries.  Lets pull routes of an AS34984 in Linux command Line:

root@test:~# whois -h -i or as34984 | grep route:
…output omitted

In order to get route data as a variable, to play with, you still need to need to parse the related part of the data from  the output. A better approach, which i generally prefer , is to use a native library, so that your script won’t depend on  a tool,  as a result it would be executable both  on Windows, or or Linux/Mac.  For a native client library check

Restful Web Services API

Even with a native library, you will still need to parse the output data. There is a better way, which is getting the data in a format like XML, YAML, JSON etc  so that it is easier to get related parts systematically.  Fortunately RIPE/ARIN has a RESTFul Web Services API  which is a REST interface to their WhoIS Database which are invoked via http requests. Here are the documentation links:

For ARIN documentation check: 

Querying RIPE  Web API and Processing  XML Input With Python

By looking at the documentation, it can be seen that for AS5400, which is BritishTelecom, url  could be used to get all routes that has an origin of AS5400. The ony thing to get prefixes for another AS is to change the “as5400” value in the url itelf. If you open the url in your web browser you will see an xml output like below: 

So if we could open this url in python, and get the output, then parse it as xml, everything will be ok. To do so, lets examine the code i wrote, part by part:

Examining the code: 

At first part of the code we need to import libraries, which are: urllib, urllib2 and xml.etree.ElementTree   (you may also use lxml instead of xml.etree.ElementTree)

import urllib,urllib2
    import xml.etree.cElementTree as ET
except ImportError:
    import xml.etree.ElementTree as ET

We determined how to create a URL to pull prefixes that is originated from an AS. Creating an AS variable is logical so that in further usages you can change the AS number or get the value as an input.

as_to_pull_prefixes = "AS5400"
url = "" % as_to_pull_prefixes

As url is ready we need to open the url and assign the output to a string variable:

fp = urllib2.urlopen (url)
response =

Then we need to parse XML from string into an element:

tree = ET.fromstring(response)

Last part is to get the prefix values from the xml output, which we turned into an element by using xml.etree.ElementTree. First we need to determine the hierarchy so that we can write a path argument. Having a look at the xml output, the hierarchy should be:

  • inside <objects> 
  • inside <object> that has a type value of  "route" (for ipv4 prefixes) 
  • inside <primary-key>
  • every <attribute> which has a name value of "route" 

To write such a path argument, you may wanted to have a look at Here is the code:

interested =  tree.findall("./objects/object[@type='route']/primary-key/attribute[@name='route']")

Our job is still not done yet, cause inside the tree we created, which is "interested" variable, we need to take the values of "value": Here is the last piece of the code:

for child in interested:
    print child.get('value')

Trying It Out 

Here is the output when i executed the script:

To download the script, click here, or copy-paste the full code below:

import urllib,urllib2
    import xml.etree.cElementTree as ET
except ImportError:
    import xml.etree.ElementTree as ET
###Variables which changes per request
as_to_pull_prefixes = "AS5400"
url = "" % as_to_pull_prefixes
###Pull Info From IRR(RIPE)and assign it to a variable as string
fp = urllib2.urlopen (url)
response =
###parse xml from  from string into an element 
tree = ET.fromstring(response)
###get interested data from element 
interested =  tree.findall("./objects/object[@type='route']/primary-key/attribute[@name='route']")
for child in interested:
    print child.get('value')
##for more info on how to select interested data from xml

Wednesday, January 21, 2015

Validating BGP Announcements by Automating Filter Generation with Python: Part1

Securing Internet Routing infrastructure has been a hot topic for a long time as hijack events occurs again and again either mistakenly or on purpose. Operators use different  techniques to validate BGP announcements. I will try to explain creating an ip/as prefix-list  by pulling information from a RIR database, in our case RIPE, using python. But first, lets take a look at other methods in Part1: 


Probabily  the best  way to prevent BGP/IP hijacks is to use RPKI infrastructure, , which is a  PKI framework (trust anchor mechanism). Unfortunately the method needs legitamate AS/IP owners to register their sources. needless to say that no one is registering their sources, as a result the method is still not applicable. By 21.01.2015 the current validation states for all IPv4 prefixes is only %5.59, You can check validation states from


For RPKI is not applicable, we need another way to validate the BGP announcements, where IRR RPSL database entries comes into play. RPSL is a language commonly used by ISPs to describe their routing policies. ISP's could store their routing policies either on their own server or on open public whois databases including RIPE, RADB, APNIC etc.  RIPE (RIR for Europe) transformed its database to RPSL format in 2001.  Example entries in RIPE database are:

as-set:          AS-FUNET
descr:           Macro with all ASes exported by FUNET
members:         AS1741
members:         AS1739
members:         AS565
members:         AS15496
members:         AS30754
members:         AS39098
members:         AS39857
members:         AS39662
tech-c:          FH437-RIPE
admin-c:         FA1183-RIPE
mnt-by:          AS1741-MNT
source:          RIPE # Filtered
As-set object which was created by FUNET, and includes their members

aut-num:         AS5400
as-name:         BT
descr:           British Telecommunications plc
org:             ORG-CNS3-RIPE
import:          from AS1741 action pref=20; accept AS-FUNET
British Telecoms RPSL entry which accepts the informations in AS-FUNET  as-set entry.
Route−set: AS4763:RS−ROUTES:AS681
descr:     prefix filter for AS681
tech−c:    JA39
changed: 19991118
source:    RADB
Route-set sample

Using these databases it is possible to get infos for AS-SETs, AS entries, route objects etc. We know that RPKI validation states are really low, so what about the RPSL entries?  Unfortunately Inter Route Registries are not that accurate also. BGPmon analysis which is done at 2010, shows that only %46 of the global routing table entries has a  matching route-object. 

If  you wanted to update a BGP import policy automatically then you should force the facing AS owner to update their IRR entries (route-objects, as-sets…). Some providers area also very strict in the IRR usage , so  why shouldn’t you ? 

Getting  Info From RIR Databases


There are many great tools that are ready for use which does the whois queries behind, which some of are: 
  • IRR Toolset
  • IRR Power Tools
  • Bgpq3
  • Md
  • P2BGPTool
Lets take a loot at one of these tools, namely  Bgpq3, before writing our own python script: 

Bgpq3 Installation and Usage

Download compressed bgpq3 file from , extract and install it (below steps for Ubuntu):

admin@ubuntu:~# wget 
admin@ubuntu:~#  tar -xvzf bgpq3-0.1.21.tgz 
admin@ubuntu:~#  cd bgpq3-0.1.21/
admin@ubuntu:~/bgpq3-0.1.21# ./configure
admin@ubuntu:~/bgpq3-0.1.21# sudo make && sudo make install

After installing we could create a prefix-list from command line using the tool. Below is a sample for creating Junos prefix list for as-set: AS-FUNET

You may check bgpq3 man page for  more parameters (cisco, juniper, as-path, ipv4, ipv6)

Definitely command line tools are useful  but if you wanted get data from command line output in  python script it means that you need to parse the data.  There are better ways which you may continue reading in Part 2. 

Monday, January 12, 2015

Huawei NetStream (Netflow) RFC Compability ?

Netflow, which has become a de-facto industry standard, is a network protocol developed by Cisco for collecting IP information and monitoring network traffic. Other vendors also support other alternative flow protocols like: 

  •  Juniper® (Jflow)
  •  3Com/HP® , Dell® , and Netgear® (s-flow)
  •  Huawei® (NetStream)
  •  Alcatel-Lucent® (Cflow)
  •  Ericsson® (Rflow)

By analizing the flow data you can draw a map of your traffic, sources and destinations, determine the normal flow values thus analyze ddos attacks, and much more. 

A sample configuration for Huawei is:

ip netstream export version 9
ip netstream sampler fix-packets 1000 inbound
ip netstream sampler fix-packets 1000 outbound
ip netstream export source 
ip netstream export host  9996
ip netstream export template timeout-rate 1
ip netstream timeout active 1
ip netstream timeout inactive 5
ip netstream tcp-flag enable
ip netstream mpls-aware label-and-ip

 ip netstream inbound
 ip netstream sampler fix-packets 1000 inbound

 ip netstream sampler to slot self

In production, based on the configuration above, we realized that packet sequence numbers, from a NE40X series router, does not increase sequentially as defined in RFC3954:

   Sequence Number
         Incremental sequence counter of all Export Packets sent from
         the current Observation Domain by the Exporter.  This value
         MUST be cumulative, and SHOULD be used by the Collector to
         identify whether any Export Packets have been missed. were .....

Instead the sequence numbers were increasing by former flow record (21, 9 21, 16,21 ...) count,

NetStream V9:
1          0.000000        CFLOW           1326   SEQ:510215546       total: 21 (v9) records
3          0.100533        CFLOW           606    SEQ:510215567       total: 9 (v9) records
4          0.163794        CFLOW           1326   SEQ:510215576       total: 21 (v9) records
5          0.264529        CFLOW           1026   SEQ:510215597       total: 16 (v9) records
6          0.327319        CFLOW           1326   SEQ:510215613       total: 21 (v9) records

which can be formalized as:

  • "SEQ2=SEQ1+counter"

In VRP, for Netstream to be compatible with RFC, sequence-mode configuration must be configured exclusively as it is not the default behavior.

 [HUAWEI] slot x
 [HUAWEI-slot-x] ip netstream export sequence-mode packet

As sequence numbers are followed by SourceID in collectors, using "cflow.source_id==4" filter in wireshark (for this specific case) we verified that the problem is gone.

If you examine the capture carefully, you might see that there still is a little problem which is: even the sequence ID is the same, there is an extra packet which's flow number is not lined up.

If you are working with a Huawei router, be careful that many features that you expect it to work in default, must be exclusively configured.  I will share these commands in another post, hopefully.

You could contiue reading Part 2:

Monday, January 5, 2015

Controlling Junos with Python & PyEZ, Part 2: Getting Info, RPC on Demand

At first part of this post, i've tried to explain how to install PyEZ module on Windows, and how to get the device name. Surely the power of python, PyEZ framework and XML API of Junos is not limited with that, so let's get into detail.

RPC on Demand

For any command you can execute on Junos CLI, you can invoke it's RPC equivalent. To determine an RPC equivalent of a command, you use the | display xml rpc mechanism as illustrated:

etorun@as34984-lg> show route | display xml rpc 
<rpc-reply xmlns:junos="">

The content between the "<rpc>...</rpc>" is the XML RPC command, in this case "get_route_information". Be careful you need to swap dashes (-) to underscores(_) 

show_route_output = testdevice.rpc.get_route_information(destination="")

If you wanted search active route for in inet.0 table, then you need to add parameters to the same command, but the command you need to execute to get the rpc equivalent does not change.
etorun@as34984-lg> show route table inet.0 active-path | display xml rpc 
<rpc-reply xmlns:junos="">

However you need to add these extra parameters to python equivalent

show_route_output = testdevice.rpc.get_route_information(destination="", table="inet.0", active_path=True) 

The "active-path" parameter does not take any value, so just assign it to True

To determine the  RPC response use the | display xml mechanism

etorun@as34984-lg> show route table inet.0 active-path | display xml        
<rpc-reply xmlns:junos="">
    <route-information xmlns="">
            <rt junos:style="brief">
                    <age junos:seconds="791658">1w2d 03:54:18</age>
                    <as-path>15169 I

Trying It Out

Imagine you need to take the active as-path value of Then the python script should be:

from jnpr.junos import Device
from jnpr.junos import Device
router = Device(user='admin', host='', password='logmein')
# invoke the RPC equivalent to "show route table inet.0 active-path"
show_route_output = router.rpc.get_route_information(destination="", table="inet.0", active_path=True) 
# use XPath expressions to extract the data from the Junos/XML response# the :show_route_output: variable is an lxml Element object
print "the as-path value is: %s" % show_route_output.findtext('route-table/rt/rt-entry/as-path')

Here is the output when i run the .py file from windows powershell:

If you wanted to have a look, I did benefit from RPC On Demand page of Juniper Tech Wiki when writing this post.

Thursday, January 1, 2015

Controlling Junos with Python & PyEZ, Part 1: Installation

You can manage or automate routers by using python. Having an XML API is the benefit of using a Juniper router as it is easier than other routers to manage it with python. Moreover Juniper wrote a Python library, which is actually a "micro-framework", called PyEZ and it makes easier for non-proggramers to manage a Junos OS router. 


I am taking the exact explanation from the "juniper tech wiki":

Junos PyEZ is designed to provide the same capabilities as a user would have on the Junos OS CLI, but in an environment built for automation tasks. These capabilities include, but are not limited to:

  • Provide "facts" about the device such as software version, serial number, and so on
  • Retrieve the "operational" or "run-state" information (think "show" commands) using Tables/Views
  • Retrieve the configuration using Tables/Views
  • Make unstructured configuration changes with "snippets" and "templates"
  • Make structured configuration changes with modeled abstractions
  • Provide common utilities for tasks such as secure copy of files and software updates

Installation Python, PIP and PyEZ (Windows) 

To use PyEz, you must be using a python version 2.7 or later. I am not getting  into details of installing Python on to Windows as it is as easy as installing an .exe file. You may download the latest build from When installing python select "pip" and "setuptools" to get installed. Also to register the version which you are installing select "" as seen in the capture below:

You may need to reload Windows inorder for python to work. If PIP is not installed and you do not wanted to install python from scratch then you have the option to only install PIP. (PIP is a package itself which install packages.). Download,  save it as a .py file then, run it from the command prompt "python".

Behind the scenes PyEZ uses lxml, ncclient, paramiko, pycrypto, jinja2, PyYAML, netaddr, ecdsa and markupsafe python packages.  For ease of installation, we will use PyPI, which is a Python package repository. But before  you need to check if  Microsoft Visual C++ Compiler for Python 2.7 is installed on your PC and if not install it before executing the command "pip install junos-eznc" command in powershell/cmd.

Configuring Router 

For PyEz to work, ssh and netconf must be enabled on router which means you must be using "domestic" version of Junos. Basic configuration to enable ssh and netconf: 


If everything is right, then you should be able to connect to a router using PyEZ and get some outputs. Here is a basic script which gets version info from a router:

from jnpr.junos import Device

dev = Device(host='',user='admin',password='password')

print dev.facts['version']
print dev.cli("show configuration system services)


Here is the output:

For more you should visit Junipers PyEZ TechWiki page. You may also have a look at Juniper Python TeckWiki page. 

Happy New Year!!


Internetworking Hints Copyright © 2011 -- Template created by O Pregador -- Powered by Blogger