ENSM Products are Commodities?

On your quest to put in network and systems management capabilities, you have to figure in several explicit and implicit factors related to your end goals.  What I mean is that while it's easy to go to your Framework Vendor of choice, break out the Bill of Materials spreadsheet, and sit down with the Sales person and go through the elements you would need for your environment, it may be filled with hidden challenges. And some challenges may be harder to overcome than others once you have signed the check.

Don't forget, these products don't magically install and run themselves. They take care and feeding. Some more than others. And the more complex it is, the more complex it is to figure out when something goes awry.

Sounds so easy! After all, all of these products are commodities. And buying from a single vendor gives you a single point of support... and blame. In effect, a single "throat to choke".  NOTHING could be further from the truth!

Most of the big vendor's product frameworks are aggregations and conglomerations of products that have been acquired, some overlapping, into what looks like a somewhat unified solution.  In many cases, it is only after you buy the product framework that you discover stuff like there are different portals with different products and these portals don't effectively integrate together. Or you may find the north bound interface of one product is a kludge to somewhat loosely fit the two products together. Or two products use competing Java versions.

Some vendors product suites have become more and more complex as new releases are GAed.   In many cases, these new levels of complexity have a profound impact on your ability to install, administer, or diagnose issues as they arise.

First up - Where are your requirements? Do you know the numbers and types of elements in your environment? What about the applications? How do these apply to Service Level Agreements? Do you have varying levels of maintenance and support for the components in your environment?

Do you know who the users will be?  Have you defined your support model? Which groups need access to what elements of information?  Do you have or have you prepared a proposed workflow of how users, managers, and even customers are going to interact with the new capabilities?

Who is going to take care of the management systems and applications? Have you aligned your organization to be successful in deployment? Do you have the skill sets? Do you have adequate skills coverage?

Have you defined the event flow? What about performance reports needs and distributions?  And ad hoc reporting needs? Have you defined any baseline thresholds?

Do you have SNMP access? What about ICMP? SSH? Have you considered the implications of management traffic across your security zones?

Product Choices

While there are a plethora of choices available to you, many do not want to go through the hassle of doing due diligence.  But be forewarned, failure to do due diligence can wreak mayhem in you environment.  I know, the big guns say that "our product works in your competitors" but does it really? You don't know?  As is your competition that undifferentiated from you? (May not be a good thing!)

When you go through product selection, you need to realize the support needed to administer the new management applications.  Do you need specialists just to install it? What about training? Are you going to need other resources like Business Intelligence Analysts, Web Developers, Database Administrators, Script Developers, or even additional Analysts or Engineers.

Here are some signs you may experience:

 If the product takes longer than a couple of days to install and integrate,  here's your sign.

If two or more products in your big vendor product suite need a significant amount of customization to work together, here's your sign.

 If the installation document for the product deviates from the actual installation, here's your sign.

  If you find out you actually have to install additional product as discovered during the installation, here's your sign.

  If you end up realizing that the recommended hardware specs are either overkill or under-speced, here's your sign.

 If you end up having to deal with libraries and utilities that are not included or resolved with the product installation, here's your sign.

Missed it by THAT much!

 If you find yourself opening up support tickets in the middle of the installation, here's your sign.

 If you find that the product breaks your security model AFTER you do the installation, here's your sign.

   If it takes Vendor specific Engineering to install the product, here's your sign.

   If you cannot see value in the first day after the installation of a product, here's your sign.

   If you find that you need to restructure and build out your support team AFTER the installation, here's your sign.

Systems Management

Systems Management brings whole new challenges to your environment.  Some of the things you need to evaluate up front are:

• Agent deployment - Level of Difficulty - OS Coverage - consistent data across agents. Manual,  Automatic, or  distribute able
• Agent-less - Browser specific? Adequate coverage?  Full transactions? Handles redirection?
• Agent run time - Resource utilization - memory footprint - stability - Security.
• Data collection - Pull or push model? Resiliency? Effect on run time resources?
• External Restrictions - Java versions? Perl versions? Python versions?
• Adequate application coverage?
• Thresholds - Level of difficulty? Binary only or degrees of utilization/capacity/performance? Stateful? Dynamic thresholds? Northbound traps already defined or do you have to do your own?

Summary

Enterprise Management does not have to be that difficult.   There are products out there that work very well for what they do and are easy to deploy and maintain. For example, go do an OpenNMS installation.  Even though OpenNMS runs on just about any platform (a testament to their developer community and product maturity), you go to their wiki page http://www.opennms.org/documentation/installguide.html , pick out your platform of choice, and follow the procedure.  Most of the time, you are looking at maybe an hour. In an hour, you're starting discovery and picking up inventory to monitor and manage.

Solarwinds isn't too bad either. Nice, clean install on Windows.

Splunk is awesome and up in running in no time. http://www.splunk.com/

Hyperic HQ wasn't a bad installation either. Pretty simple. However, it is time sensitive on the agents.  Kind of thick (I think its the Struts), Java wise. http://www.hyperic.com/

eGInnovations is cake.  One agent everywhere for OS and applications. Handles VMWare, Xen and others. And the UI is straight forward. A Ton of value across both system and application monitoring and performance.   http://www.eginnovations.com/

Appliance based solutions take a bit more time in the planning phase up front but take the sting out of installation.  Some of these include:

http://www.sevone.com/ (SevOne does offer a software download for evaluation)
http://www.sciencelogic.com/
http://www.loglogic.com/ (They also offer a virtual appliance download)

One solution I dig is Tavve ZoneRanger for solving those access issues like UDP/SNMP across firewalls, SSH access across a firewall, etc.,  without having to run through proxies upon proxies and still maintain consistent auditing and logging.  It deploys as an appliance of virtual appliance.  http://www.tavve.com/

Another aspect you may consider include hosted applications. ServiceNow is easy to deploy because it is a hosted solution. http://www.servicenow.com/

Simplifying topology

I have been looking at monitoring and how its typically implemented. Part of my look is to drive visualization but also how can I leverage the data in a way that organizes people's thoughts on the desk.

Part of my thought process is around OpenNMS.  What can I contribute to make the project better.

What I came to realize is that Nodes are monitored on a Node / IP address basis by the majority of products available today.  All of the alarms and events are aligned by node - even the sub-object based events get aggregated back to the node level.  And for the most part, this is OK.  You dispatch a tech to the Node level, right?

When you look at topology at a general sense, you can see the relationship between the poller and the Node under test.  Between the poller and the end node, there is a list of elements that make up the lineage of network service components. So, from a service perspective, a simple traceroute between the poller and the end node produces a simple network "lineage".


Extending this a bit further, knowing that traceroute is typically done in ICMP, this gives you an IP level perspective of the network.  Note also that because traceroute exploits the time to Live parameter of IP, it can be accomplished in any transport layer protocol. For example, traceroute could work on  TCP port 80 or 8080.  The importance is that you place a protocol specific responder on the end of the code to see if the service is actually working beyond just responding to a connection request.

And while traceroute is a one way street, it still derives a lineage of path between the poller and the Node under test - and now the protocol or SERVICE under test. And it is still a simple lineage.

The significance of the path lineage is that in order to do some level of path correlation, you need to understand what is connected to what.  given that this can be very volatile and change very quickly, topology based correlation can be somewhat problematic - especially if your "facts" change on the fly.  and IP based networks do that.  They are supposed to do that.  They are a best efffort communications methodology that needs to adapt to various conditions.

Traceroute doesn't give you ALL of the topology.  By far. Consider the case of a simple frame relay circuit. A Frame Relay circuit is mapped end to end by a Circuit provider but uses T carrier access to the local exchange.  Traceroute only captures the IP level access and doesn't capture elements below that. In fact, if you have ISDN backup enabled for a Frame Relay circuit, your end points for the circuit will change in most cases, for the access.  And the hop count may change as well.

The good part about tracerouteing via a legitimate protocol is that you get to visualize any administrative access issues up front. For example, if port 8080 is blocked between the poller and the end node, the traceroute will fail. Additionally, you may see ICMP administratively prohibited messages as well. In effect, by positioning the poller according to end users populations, you get to see the service access pathing.

Now, think about this... From a basic service perspective, if you poll via the service, you get a basic understanding of the service you are providing via that connection.  When something breaks, you also have a BASELINE with which to diagnose the problem. So, if the poll fails, rerun the traceroute via the protocol and see where it stops.

Here are the interesting things to note about this approach:

• You are simply replicating human expert knowledge in software.  Easy to explain.  Easy to transition to personnel.
• You get to derive path breakage points pretty quickly.
• You get to discern the perspective of the end user.
• You are now managing your Enterprise via SERVICE!

Topology really doesn't mean ANYTHING until you evolve to manage by Service and not by individual nodes.  You can have all the pretty maps you want.  It doesn't mean crapola until you start managing by service.

This approach is an absolute NATURAL for OpenNMS.  Let me explain...

Look at the Path Outages tab. While it is currently manually configured, using the traceroute by service lineage here provides a way of visualizing the path lineage.

OpenNMS supports services pollers natively.  There are alot of different services out of the box and its easy to do more if you find something different from what they already do.

Look at the difference between Alarms versus Events. Service outages could directly be related to an Alarm while the things that are eventing underneath may affect the service, are presented as events.

What if you took the reports and charts and aligned the elements to the service lineage?  For example, if you had a difference in service response, you could align all of the IO graphs for everything in the service lineage.  You could also align all of the CPU utilizations as well.

In elements where there are subobjects abstracted in the lineage, if you discover them, you could add those in the lineage.  For example, if you discovered the Frame Relay PVCs and LEC access circuits, these could be included in with your visualization underneath the path where they are present.

The other part is that the way you work may need to evolve as well.  For example, if you've traditionally ticketed outages on Nodes, now you may need to transition to a Service based model. And while you may issue tickets on a node, your ticket on a Service becomes the overlying dominant ticket  in that multiple node problems may be present in a service problem.

And the important thing.  You become aware of the customer and Service first, then elements underneath that.  It becomes easier to manage to service along with impact assessments, when you manage to a service versus manage to a node.  And when you throw in the portability, agility, and abstractness of Cloud computing, this approach is a very logical fit.