ENSM Products are Commodities?

On your quest to put in network and systems management capabilities, you have to figure in several explicit and implicit factors related to your end goals.  What I mean is that while it's easy to go to your Framework Vendor of choice, break out the Bill of Materials spreadsheet, and sit down with the Sales person and go through the elements you would need for your environment, it may be filled with hidden challenges. And some challenges may be harder to overcome than others once you have signed the check.

Don't forget, these products don't magically install and run themselves. They take care and feeding. Some more than others. And the more complex it is, the more complex it is to figure out when something goes awry.

Sounds so easy! After all, all of these products are commodities. And buying from a single vendor gives you a single point of support... and blame. In effect, a single "throat to choke".  NOTHING could be further from the truth!

Most of the big vendor's product frameworks are aggregations and conglomerations of products that have been acquired, some overlapping, into what looks like a somewhat unified solution.  In many cases, it is only after you buy the product framework that you discover stuff like there are different portals with different products and these portals don't effectively integrate together. Or you may find the north bound interface of one product is a kludge to somewhat loosely fit the two products together. Or two products use competing Java versions.

Some vendors product suites have become more and more complex as new releases are GAed.   In many cases, these new levels of complexity have a profound impact on your ability to install, administer, or diagnose issues as they arise.

First up - Where are your requirements? Do you know the numbers and types of elements in your environment? What about the applications? How do these apply to Service Level Agreements? Do you have varying levels of maintenance and support for the components in your environment?

Do you know who the users will be?  Have you defined your support model? Which groups need access to what elements of information?  Do you have or have you prepared a proposed workflow of how users, managers, and even customers are going to interact with the new capabilities?

Who is going to take care of the management systems and applications? Have you aligned your organization to be successful in deployment? Do you have the skill sets? Do you have adequate skills coverage?

Have you defined the event flow? What about performance reports needs and distributions?  And ad hoc reporting needs? Have you defined any baseline thresholds?

Do you have SNMP access? What about ICMP? SSH? Have you considered the implications of management traffic across your security zones?

Product Choices

While there are a plethora of choices available to you, many do not want to go through the hassle of doing due diligence.  But be forewarned, failure to do due diligence can wreak mayhem in you environment.  I know, the big guns say that "our product works in your competitors" but does it really? You don't know?  As is your competition that undifferentiated from you? (May not be a good thing!)

When you go through product selection, you need to realize the support needed to administer the new management applications.  Do you need specialists just to install it? What about training? Are you going to need other resources like Business Intelligence Analysts, Web Developers, Database Administrators, Script Developers, or even additional Analysts or Engineers.

Here are some signs you may experience:

 If the product takes longer than a couple of days to install and integrate,  here's your sign.

If two or more products in your big vendor product suite need a significant amount of customization to work together, here's your sign.

 If the installation document for the product deviates from the actual installation, here's your sign.

  If you find out you actually have to install additional product as discovered during the installation, here's your sign.

  If you end up realizing that the recommended hardware specs are either overkill or under-speced, here's your sign.

 If you end up having to deal with libraries and utilities that are not included or resolved with the product installation, here's your sign.

Missed it by THAT much!

 If you find yourself opening up support tickets in the middle of the installation, here's your sign.

 If you find that the product breaks your security model AFTER you do the installation, here's your sign.

   If it takes Vendor specific Engineering to install the product, here's your sign.

   If you cannot see value in the first day after the installation of a product, here's your sign.

   If you find that you need to restructure and build out your support team AFTER the installation, here's your sign.

Systems Management

Systems Management brings whole new challenges to your environment.  Some of the things you need to evaluate up front are:

• Agent deployment - Level of Difficulty - OS Coverage - consistent data across agents. Manual,  Automatic, or  distribute able
• Agent-less - Browser specific? Adequate coverage?  Full transactions? Handles redirection?
• Agent run time - Resource utilization - memory footprint - stability - Security.
• Data collection - Pull or push model? Resiliency? Effect on run time resources?
• External Restrictions - Java versions? Perl versions? Python versions?
• Adequate application coverage?
• Thresholds - Level of difficulty? Binary only or degrees of utilization/capacity/performance? Stateful? Dynamic thresholds? Northbound traps already defined or do you have to do your own?

Summary

Enterprise Management does not have to be that difficult.   There are products out there that work very well for what they do and are easy to deploy and maintain. For example, go do an OpenNMS installation.  Even though OpenNMS runs on just about any platform (a testament to their developer community and product maturity), you go to their wiki page http://www.opennms.org/documentation/installguide.html , pick out your platform of choice, and follow the procedure.  Most of the time, you are looking at maybe an hour. In an hour, you're starting discovery and picking up inventory to monitor and manage.

Solarwinds isn't too bad either. Nice, clean install on Windows.

Splunk is awesome and up in running in no time. http://www.splunk.com/

Hyperic HQ wasn't a bad installation either. Pretty simple. However, it is time sensitive on the agents.  Kind of thick (I think its the Struts), Java wise. http://www.hyperic.com/

eGInnovations is cake.  One agent everywhere for OS and applications. Handles VMWare, Xen and others. And the UI is straight forward. A Ton of value across both system and application monitoring and performance.   http://www.eginnovations.com/

Appliance based solutions take a bit more time in the planning phase up front but take the sting out of installation.  Some of these include:

http://www.sevone.com/ (SevOne does offer a software download for evaluation)
http://www.sciencelogic.com/
http://www.loglogic.com/ (They also offer a virtual appliance download)

One solution I dig is Tavve ZoneRanger for solving those access issues like UDP/SNMP across firewalls, SSH access across a firewall, etc.,  without having to run through proxies upon proxies and still maintain consistent auditing and logging.  It deploys as an appliance of virtual appliance.  http://www.tavve.com/

Another aspect you may consider include hosted applications. ServiceNow is easy to deploy because it is a hosted solution. http://www.servicenow.com/

Dante's Inferno in IT...

We’ve all been through Dante’s Inferno. I know I have.

I was at one place where they developed an in house SNMP agent to be deployed on all managed systems. Because of the variety of operating systems supported and the amount of bugs in the software, many customers hated the agent and would request that the agent be taken off of their systems. But because the agent was “free”, it kept living in a miserable existence. Turns out there were many different versions supported and deployed. Additionally, the design of the agents’ sub-agents capabilities deviated significantly from industry standards such that it hamstrung the open source openness of the base agent.

Dante’s Inferno came in when you had to deal with the agent capabilities as an Architect. Saying anything negative related to the agent was Heresy. The Manager that owned the Agent would resort to Anger, Fraud, and Treachery in order to divert any negative attention to their baby. Part of the reason for hanging on to this Agent was that the ownership of more developed products promoted the Manager’s gluttony and greed. It was his silo of management technology.

While I was busy circumnavigating the Machiavellian Urinary Olympics, that Manager was working hard to put me in Limbo. Any requirements that I put forth were immediately in negotiation such that I could not finish requirements. Finally, in total frustration, I sent out a Final version of the requirements. Doing this sent the Manager into a frenzy of new Machiavellian Urinary Olympics such that my actions were elevated all the way up to a Sr. VP. Alas, I could not overcome the Marijuana Principle of Management (Harder you suck ,the higher you get!)

I left shortly afterward. So did several of my coworkers. Some are still there. All with the common experience that we’ve all been through Dante’s Inferno.

Lessons for the Architect :

• Be careful in calling someone’s baby ugly. Given the “embeddedness” of a given politician, there may be some things you cannot change. 

• Some Silos can only break down through years of pain and years of continued failure.

• In moving toward Cloud computing models, some folks may have an inclination to bring with them all of the bad habits they have currently.

• If a person has only ever seen one place, they may not understand that success looks totally different in other places.

• There is a direct cost and an indirect cost to supporting internally developed products. If your internally developed product is holding back progress and new business, it is a danger sign…

As an Architect, be wary of consensus. “Where there is no vision, the people perish. (Proverbs 29:18)”

Politically based Engineering Organizations

When an organization is resistant to change and evolution, it can in many cases be attributed to weak technical leadership. Now this weakness can be because of the politicalization of Engineering and Development organizations.

Some of the warning signs include:

- Political assasinations to protect the current system
- Senior personnel and very experienced people intimidate the management structure and are discounted
- Inability to deliver
- An unwillingness to question existing process or function.


People that function at a very technical level find places where politics prevalent, a difficult place to work.

- Management doesn't want to know whats wrong with their product.
- They don't want to change.
- They shun and avoid senior technical folks and experience is shunned.

Political tips and tricks

- Put up processes and negotiations to stop progress.
- Random changing processes.
- Sandbagging of information.
- When something is brought to the attention of the Manager, the technical
person's motivations are called into question. What VALUE does the person bring to the Company?
- The person's looks or mannerisms are called into question.
- The persons heritage or background is called into question.
- The person's ability to be part of a team is called into question.
- Diversions are put in place to stop progress at any cost.

General Rules of Thumb

+ Politically oriented Supervisors kill technical organizations.
+ Image becomes more important than capability.
+ Politicians cannot fail or admit failure. Therefore, risks are avoided.
+ Plausible deniability is prevalent in politics.
+ Blame-metrics is prevalent. "You said..."

Given strong ENGINEERING leadership, technical folks will grow very quickly. Consequently, their product will become better and better as you learn new techniques and share them, everyone gets smarter. True Engineers have a willingness to help others, solve problems, and do great things. A little autonomy and simple recognitions and you're off to the races.

Politicians are more suited to Sales jobs. Sales is about making the numbers at whatever cost. In fact, Sales people will do just about anything to close a deal. They are more apt to help themselves than to help others unless it also helps themselves. Engineers need to know that their managers have their backs. Sales folks have allegiance to the sale and themselves... A bad combination for Engineers.

One of the best and most visionary Vice Presidents I've ever worked under, John Schanz once told us in a Group meeting that he wanted us to fail. Failure is the ability to discover what doesn't work. So, Fail only once. And be willing to share your lessons, good and bad. And dialog is good. Keep asking new questions.


In Operations, Sales people can be very dangerous. They have the "instant gratification" mentality in many cases. Sign the PO, stand up the product, and the problem is solved in their minds. They lack the understanding that true integration comes at a personal level with each and every user. This level of integration is hard to achieve. And when things don't work or are not accepted, folks are quick to blame someone or some vendor.

The best Engineers, Developers, and Architects are the ones that have come up through the ranks. They have worked in a NOC or Operations Center. They have fielded customer calls and problems. They understand how to span from the known to and unknown even when the customer is right there. And they learn to think on their feet.

Product Evaluations...

I consider product competition as a good thing. It keeps everyone working to be the best in breed, deliver the best and most cost effective solution to the customer, and drives the value proposition.

In fact, in product evaluations I like to pit vendors products against each other so that my end customer gets the best solution and the most cost effective. For example, I use capabilities that may not have been in the original requirements to further the customer capability refinement. If they run across something that makes their life better, why not leverage that in my product evaluations? In the end, I get a much more effective solution and my customer gets the best product for them.

When faced with using internal resources to develop a capability and using an outside, best of breed solution, danger exists in that if you grade on a curve for internally developed product, you take away competition and ultimately the competitive leadership associated with a Best of Breed product implementation.

It is too easy to start to minimize requirements to the bare necessities and to further segregate these requirements into phases. When you do, you lose the benefit of competition and you lose the edge you get when you tell the vendors to bring the best they have.

Its akin to looking at the problem space and asking what is th bare minimum needed to do this. Or asking what is the best solution for this problem set? Two completely different approaches.

If you evaluate on bare minimums, you get bare minimums. You will always be behind the technology curve in that you will never consider new approaches, capabilities, or technology in your evaluation. And your customer is always left wanting.

It becomes even more dangerous when you evaluate internally developed product versus COTS in that, if you apply the minimum curve gradient to only the internally developed product, the end customer only gets bare minimum capabilities within the development window. No new capabilities. No new technology. No new functionality.

It is not a fair and balanced evaluation anyway if you only apply bare minimums to evaluations. I want the BEST solution for my customer. Bare minimums are not the BEST for my customer. They are best for the development team because now, they don't have to be the best. They can slow down innovation through development processes. And the customer suffers.

If you're using developer in house, it is an ABSOLUTE WASTE of company resources and money to develop commodity software that does not provide clear business discriminators. Free is not a business discriminator in that FREE doesn't deliver any new capabilities - capabilities that commodity software doesn't already have.

Inherently, there are two mindsets that evolve. You take away or you empower the customer. A Gatekeeper or a Provider.

If you do bare minimums, you take away capabilities that the customer wants but because it may not be a bare minimum, the capability is taken away.

If you evaluate on Best of Breed, you ultimately bring capabilities to them.