Parsing

Before starting, it’s important to look at the parser API and parse_indented_config. It may look like a lot and some parts may not make sense at this point but it will help understanding the tutorial.

We are going to see how to parse configuration by example. To do that we are going to use a modified/simplified version of the openconfig-interfaces and openconfig-vlan models. Note they have been slightly modified for simplicity and brevity.

Let’s start by looking at the ASCII tree representation of the model:

+--rw openconfig-interfaces:interfaces
|  +--rw interface* [name]
|     +--rw config
|     |  +--rw description? <string>
|     |  +--rw enabled? <boolean>
|     |  +--rw loopback-mode? <boolean>
|     |  +--rw mtu? <uint16>
|     |  +--rw name? <string>
|     |  +--rw type <identityref>
|     +--rw name <leafref>
|     +--rw subinterfaces
|        +--rw subinterface* [index]
|           +--rw config
|           |  +--rw description? <string>
|           |  +--rw enabled? <boolean>
|           |  +--rw index? <uint32>
|           +--rw index <leafref>
+--rw openconfig-vlan:vlans
   +--rw vlan* [vlan-id]
      +--rw config
      |  +--rw name? <string>
      |  +--rw status? <enumeration>
      |  +--rw vlan-id? <vlan-id(uint16)>
      +--rw vlan-id <leafref>

To write a parser you are going to need to write a class following the rules below:

1. A grouping (either a list or a container) is represented by a class that inherits from yangify.parsers.Parser.
2. Classes that implement a part of the tree are nested in the parent object and are named as the grouping it implements.
3. Each class may have a nested class named Yangify that inherits from yangify.parsers.ParserData. This class may implement code to help with the processing. See the API documentation for details.
4. Finally, leaves are processed by a function named after the leaf and will have to return an object of the correct type.

Using the openconfig-vlans:vlans as an example, a parser may look like this:

class Vlans(Parser):
    class vlan(Parser):
        class Yangify(ParserData):
            def extract_elements(self):
                # code to extra each element from the config goes here

        class config(Parser):
            def vlan_id(self):
                # code to parse the vlan id goes here

            def name(self):
                # code to parse the name goes here

            def status(self):
                # code to parse the vlan status goes here

        def vlan_id(self):
                # code to parse the vlan id goes here

This format may be a bit cumbersome in deeply nested models, hence, an alternative maybe to use classes in the global namespace instead, for instance:

class VlanConfig(Parser):
    def vlan_id(self):
        # code to parse the vlan id goes here

    def name(self):
        # code to parse the name goes here

    def status(self):
        # code to parse the vlan status goes here

class Vlan(Parser):
    class Yangify(ParserData):
        def extract_elements(self):
            # code to extra each element from the config goes here

    config = VlanConfig

    def vlan_id(self):
            # code to parse the vlan id goes here

class Vlans(Parser):
    vlan = Vlan

Both are equivalent and both have their advantages and disadvantages when it comes to readibility. We are going to use the second method for this demo.

The openconfig-interfaces parser

To explain how this works we are going to write a parser that reads a configuration belonging to an IOS device and maps it into the model. Let’s look at the configuration first:

%cat data/ios/config.txt

interface FastEthernet1
   description This is Fa1
   shutdown
   exit
!
interface FastEthernet1.1
   description This is Fa1.1
   exit
!
interface FastEthernet1.2
   description This is Fa1.2
   exit
!
interface FastEthernet3
   description This is Fa3
   no shutdown
   exit
!
interface FastEthernet4
   shutdown
   exit
!
vlan 10
   name prod
   no shutdown
   exit
!
vlan 20
   name dev
   shutdown
   exit
!

Nothing very complex, just a few interfaces and subinterfaces and a couple of vlans. The code for this tutorial is in tutorial_parser.py, so let’s start by importing it:

import tutorial_parser

Now that we have imported the code let’s start looking at the code piece by piece. The starting point is going to be the class Interfaces which will be used to parse openconfig-interfaces:interfaces:

show_code(tutorial_parser.Interfaces)

class Interfaces(Parser):
    """
    Implements openconfig-interfaces:interfaces
    """

    interface = Interface

Not much here. The container openconfig-interfaces:interfaces only has a YANG list in the interface node. As interface is a grouping a different Parser class is used. Let’s look at it:

show_code(tutorial_parser.Interface)

class Interface(Parser):
    """
    Implements openconfig-interfaces:interfaces/interface
    """

    class Yangify(ParserData):
        def extract_elements(self) -> Iterator[Tuple[str, Dict[str, Any]]]:
            """
            IOS interfaces are in the root of the configuration. However,
            subinterfaces are also in the root of the configuration. For
            that reason we will have to make sure that we filter the
            subinterfaces as we iterate over all the interfaces in the root
            of the configuration. That's should be as easy as checking that
            the interface name has not dot in it.
            """
            for k, v in self.native["interface"].items():
                # k == "#text" is due to a harmless artifact in the
                # parse_indented_config function that needs to be addressed
                if k == "#text" or "." in k:
                    continue
                yield k, v

    config = InterfaceConfig
    subinterfaces = Subinterfaces

    def name(self) -> str:
        return self.yy.key

This one is more interesting. It has a few things:

1. A Yangify class that implements extract_elements. This is mandatory as openconfig-interfaces:interfaces/interface is a YANG list.
2. It has a config attribute that corresponds to the container openconfig-interfaces:interfaces/interface/config and points to the Parser class InterfaceConfig
3. It also has a subinterfaces attribute that correspononds to the container openconfig-interfaces:interfaces/interface/subinterfaces
4. Finally, it has a function to extract the leaf corresponding to openconfig-interfaces:interfaces/interface/name.

Two things to note here:

1. As explained in the API section linked in the note block early in this document, Yangify.extract_elements is used to extract each element returning its key and relevant configuration data.
2. The key is automatically added to the dictionary inside self.yy.keys using the path to the node as the key and made easily available via the shortcut property self.yy.key. This can be used, for instance, when returning the name of the interface, which we already know as it’s the same as key.

Now let’s look at the InterfaceConfig class that implements openconfig-interfaces:interfaces/interface/config:

show_code(tutorial_parser.InterfaceConfig)

class InterfaceConfig(Parser):
    """
    Implements openconfig-interfaces:interfaces/interface/config
    """

    def description(self) -> Optional[str]:
        return cast(Optional[str], self.yy.native.get("description", {}).get("#text"))

    def enabled(self) -> bool:
        shutdown = self.yy.native.get("shutdown", {}).get("#standalone")
        if shutdown:
            return False
        else:
            return True

    def name(self) -> str:
        return self.yy.key

    def type(self) -> str:
        if "Ethernet" in self.yy.key:
            return "iana-if-type:ethernetCsmacd"
        elif "Loopback" in self.yy.key:
            return "iana-if-type:softwareLoopback"
        else:
            raise ValueError(f"don't know type for interface {self.yy.key}")

This container only has leaves so we only have functions named after those leaves that will return the extracted data.

Now let’s circle back to openconfig-interfaces:interfaces/interface and head down to the subinterfaces container, which was parsed with the Subinterfaces class:

show_code(tutorial_parser.Subinterfaces)

class Subinterfaces(Parser):
    """
    Implements openconfig-interfaces:interfaces/interface/subinterfaces
    """

    subinterface = Subinterface

Like the Interface class, not much to look at here, let’s head down to the Subinterface class:

show_code(tutorial_parser.Subinterface)

class Subinterface(Parser):
    """
    Implements openconfig-interfaces:interfaces/interface/subinterfaces/subinterface
    """

    class Yangify(ParserData):
        def extract_elements(self) -> Iterator[Tuple[str, Dict[str, Any]]]:
            """
            IOS subinterfaces are in the root of the configuration and named following
            the format ``$parent_interface.$index``. The model specifies the key is
            the $index, which is just a number. These means we will need to:

            1. Iterate over all the interfaces
            2. Filter the ones that don't start by `$parent_interface.`
            3. Extract the $index and return it
            """
            # self.keys keeps a record of all the keys found so far in the current
            # object. To access them you can use the full YANG path
            parent_key = self.keys["/openconfig-interfaces:interfaces/interface"]
            for k, v in self.root_native["interface"].items():
                if k.startswith(f"{parent_key}."):
                    yield k, v

    config = SubinterfaceConfig

    def index(self) -> int:
        return int(self.yy.key.split(".")[-1])

This almost looks identical to the Interface class. Extracting the elements is a bit tricker as it’s explained in the code but it’s still quite similar. The index function that extracts is also similar to the name function in the Interface class, however, the index is just what’s after the . in the interface name, so we need to extract that from the key and convert it into an integer as that’s what the mode expects.

Finally, let’s look at the SubinterfaceConfig class:

show_code(tutorial_parser.SubinterfaceConfig)

class SubinterfaceConfig(Parser):
    """
    Implements openconfig-interfaces:interfaces/interface/subinterfaces/subinterface/config
    """

    def description(self) -> Optional[str]:
        return cast(Optional[str], self.yy.native.get("description", {}).get("#text"))

    def index(self) -> int:
        return int(self.yy.key.split(".")[-1])

At this point I am sure there isn’t much to add here :)

Now that we have the parser classes we need to create the root parser. The root parser has the following functions:

1. Allow the user pick and choose which parsers to use
2. Initialize the configuration, if needed.
3. Perform some post operations, if needed.

Using the parser

Our root class is going to load the Interfaces parser we explored before and use parse_indented_config to prepare the configuration:

from yangify import parser
from yangify.parser.text_tree import parse_indented_config

class IOSParser(parser.RootParser):
    class Yangify(parser.ParserData):
        def init(self) -> None:
            self.root_native = parse_indented_config(self.native.splitlines())
            self.native = self.root_native

    interfaces = tutorial_parser.Interfaces

Now we need to load the configuration file:

with open("data/ios/config.txt", "r") as f:
    config = f.read()

Now we will create the datamodel as we will need it later on:

from yangson.datamodel import DataModel
dm = DataModel.from_file("yang/yang-library-data.json", ["yang/yang-modules/ietf", "yang/yang-modules/openconfig"])

Finally, we are going to instantiate the IOSParser and call the process method:

p = IOSParser(dm, native=config)
result = p.process()

Now that we got the processed object, let’s see the result:

import json
print(json.dumps(result.raw_value(), indent=4))

{
    "openconfig-interfaces:interfaces": {
        "interface": [
            {
                "name": "FastEthernet1",
                "config": {
                    "name": "FastEthernet1",
                    "type": "iana-if-type:ethernetCsmacd",
                    "description": "This is Fa1",
                    "enabled": false
                },
                "subinterfaces": {
                    "subinterface": [
                        {
                            "index": 1,
                            "config": {
                                "index": 1,
                                "description": "This is Fa1.1"
                            }
                        },
                        {
                            "index": 2,
                            "config": {
                                "index": 2,
                                "description": "This is Fa1.2"
                            }
                        }
                    ]
                }
            },
            {
                "name": "FastEthernet3",
                "config": {
                    "name": "FastEthernet3",
                    "type": "iana-if-type:ethernetCsmacd",
                    "description": "This is Fa3",
                    "enabled": true
                }
            },
            {
                "name": "FastEthernet4",
                "config": {
                    "name": "FastEthernet4",
                    "type": "iana-if-type:ethernetCsmacd",
                    "enabled": false
                }
            }
        ]
    }
}

Adding a second parser

In the previous example we created a parser that only parses the openconfig-interfaces model, however, our tutorial_parser.py contains code to also parse the openconfig-vlan model, let’s create a second RootParser class that can parse both models:

from yangify import parser
from yangify.parser.text_tree import parse_indented_config

class IOSParser2(parser.RootParser):
    class Yangify(parser.ParserData):
        def init(self) -> None:
            self.root_native = parse_indented_config(self.native.splitlines())
            self.native = self.root_native

    interfaces = tutorial_parser.Interfaces
    vlans = tutorial_parser.Vlans

p = IOSParser2(dm, native=config)
result = p.process()

import json
print(json.dumps(result.raw_value(), indent=4))

{
    "openconfig-interfaces:interfaces": {
        "interface": [
            {
                "name": "FastEthernet1",
                "config": {
                    "name": "FastEthernet1",
                    "type": "iana-if-type:ethernetCsmacd",
                    "description": "This is Fa1",
                    "enabled": false
                },
                "subinterfaces": {
                    "subinterface": [
                        {
                            "index": 1,
                            "config": {
                                "index": 1,
                                "description": "This is Fa1.1"
                            }
                        },
                        {
                            "index": 2,
                            "config": {
                                "index": 2,
                                "description": "This is Fa1.2"
                            }
                        }
                    ]
                }
            },
            {
                "name": "FastEthernet3",
                "config": {
                    "name": "FastEthernet3",
                    "type": "iana-if-type:ethernetCsmacd",
                    "description": "This is Fa3",
                    "enabled": true
                }
            },
            {
                "name": "FastEthernet4",
                "config": {
                    "name": "FastEthernet4",
                    "type": "iana-if-type:ethernetCsmacd",
                    "enabled": false
                }
            }
        ]
    },
    "openconfig-vlan:vlans": {
        "vlan": [
            {
                "vlan-id": 10,
                "config": {
                    "vlan-id": 10,
                    "name": "prod",
                    "status": "ACTIVE"
                }
            },
            {
                "vlan-id": 20,
                "config": {
                    "vlan-id": 20,
                    "name": "dev",
                    "status": "SUSPENDED"
                }
            }
        ]
    }
}