tag:blogger.com,1999:blog-55900498886037300462024-02-02T00:15:25.859-06:00Dougie's Enterprise Management WorldDougie Stevenson's thoughts, ramblings, and ideas concerning Enterprise Management technology and application.Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.comBlogger52125tag:blogger.com,1999:blog-5590049888603730046.post-8491293535588053452017-08-14T05:00:00.004-05:002017-08-14T07:36:48.679-05:00Machine Learning and Status Determination<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.7); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Serif Pro", serif; font-size: 21px; line-height: 32px; margin-bottom: 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
In thinking about status polling in networks, most of the software we use utilizes ping lists of IP addresses or host names to perform the polling. We set standard timeouts across the board as well as poll rates, statically. Not that ICMP ping is all that accurate in the determination of a specific Node REALLY being up or down. It is rife with misconceptions by those that lack an in depth understanding of the ICMP protocol and its purpose in life. Especially if there are devices in the network that are not pingable but can affect latency and connectivity.</div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.7); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Serif Pro", serif; font-size: 21px; line-height: 32px; margin-bottom: 3.2rem; margin-top: 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
What if we are thinking about this wrong? </div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.7); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Serif Pro", serif; font-size: 21px; line-height: 32px; margin-bottom: 3.2rem; margin-top: 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
Artificial intelligence and machine learning techniques may enable us to do a much better job in status "determination" versus just status polling. </div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.7); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Serif Pro", serif; font-size: 21px; line-height: 32px; margin-bottom: 3.2rem; margin-top: 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
First, I would think that we should reset the objectives for status polling. Status polling used to be goals like ping every device every 2 minutes in the network. While this seems OK, is it really what you want? What if you set the objective to:</div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.7); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Serif Pro", serif; font-size: 21px; line-height: 32px; margin-bottom: 3.2rem; margin-top: 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
Establish a status interval of 2 minutes for every managed device. Or I need to know when a device is no longer responding within 2 minutes. And what about latency and thresholds?</div>
<h3 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.85); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Sans Pro", Helvetica, Arial, sans-serif; font-size: 21px; line-height: 28px; margin: 2.4rem 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
Heres a thought</h3>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.7); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Serif Pro", serif; font-size: 21px; line-height: 32px; margin-bottom: 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
I have 10000 device IPs and 2 pollers. </div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.7); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Serif Pro", serif; font-size: 21px; line-height: 32px; margin-bottom: 3.2rem; margin-top: 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
Give the list to both pollers. Have them go through the list to establish:</div>
<ol style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.7); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Serif Pro", serif; font-size: 21px; line-height: 32px; margin: 3.2rem 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
<li style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; counter-increment: list-num 1; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; list-style-type: inherit; margin: 2.4rem 0px 2.4rem 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline;">The latency distance from the poller.</li>
<li style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; counter-increment: list-num 1; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; list-style-type: inherit; margin: 2.4rem 0px 2.4rem 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline;">Establish the baseline root mean and standard deviation for latency.</li>
</ol>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.7); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Serif Pro", serif; font-size: 21px; line-height: 32px; margin-bottom: 3.2rem; margin-top: 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
Now, organize your latency distances from the longest to the shortest into groups or bands. Work to distribute the list as primary and secondary based on best results but evenly.</div>
<h3 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.85); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Sans Pro", Helvetica, Arial, sans-serif; font-size: 21px; line-height: 28px; margin: 2.4rem 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
Assumptions</h3>
<ol style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.7); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Serif Pro", serif; font-size: 21px; line-height: 32px; margin: 0px 0px 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
<li style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; counter-increment: list-num 1; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; list-style-type: inherit; margin: 2.4rem 0px 2.4rem 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline;">The farthest latency distance may or may not be the farthest physical distance</li>
<li style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; counter-increment: list-num 1; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; list-style-type: inherit; margin: 2.4rem 0px 2.4rem 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline;">Non-responsive IPs may or may not be down.</li>
<li style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; counter-increment: list-num 1; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; list-style-type: inherit; margin: 2.4rem 0px 2.4rem 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline;">Polling the farthest latencies should be more synchronous than closer IPs.</li>
</ol>
<h3 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.85); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Sans Pro", Helvetica, Arial, sans-serif; font-size: 21px; line-height: 28px; margin: 2.4rem 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
What if...?</h3>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.7); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Serif Pro", serif; font-size: 21px; line-height: 32px; margin-bottom: 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
What if I organize the data for each band into a heat map? Could I use this to visualize clusters of changes or anomalies in latency on the network? The problem today is that we respond to ping failures individually and we work those tickets that way. In some cases, by the time the engineer gets around to diagnosis, the problem has magically disappeared. So, we do what everyone else does - we give away the SLA to let only the persistent problems bubble up.</div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.7); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Serif Pro", serif; font-size: 21px; line-height: 32px; margin-bottom: 3.2rem; margin-top: 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
By organizing in bands, in the heat map, what you will see is:</div>
<ul style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.7); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Serif Pro", serif; font-size: 21px; line-height: 32px; margin: 3.2rem 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
<li style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; list-style-type: inherit; margin: 2.4rem 0px 2.4rem 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline;">Changes in closer latency bands may affect other more distant bands. So, diagnose the closer first, then work your way out. (Kind of helps to prioritize the list when things get rough)</li>
<li style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; list-style-type: inherit; margin: 2.4rem 0px 2.4rem 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline;">The outages are as important as the changes. When one or more IPs change enough latency wise to transition from one band to another, iit illuminates something in the network that may be indicative of something much bigger. Or may be precursors to outages. Or may be indicative of static or dynamic changes in the network.</li>
</ul>
<h3 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.85); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Sans Pro", Helvetica, Arial, sans-serif; font-size: 21px; line-height: 28px; margin: 2.4rem 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
Machine Learning possibilities</h3>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.7); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Serif Pro", serif; font-size: 21px; line-height: 32px; margin-bottom: 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
Look for groupings in the managed elements. For example, each subnet has 1 or more IP addresses that are a part of that subnet. </div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.7); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Serif Pro", serif; font-size: 21px; line-height: 32px; margin-bottom: 3.2rem; margin-top: 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
IPs in some subnets may display different latency distances. For example a 255.255.255.252 subnet has 2 host IPs and is commonly used for WAN links. If the distance to one is longer than the other, you can discern that its probably the distant end. (If you can poll the more distant one, would that not imply that the nearer IP is also up?) Interesting is that subnets like this may be able to be visualized as "bridges" between bands.</div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.7); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Serif Pro", serif; font-size: 21px; line-height: 32px; margin-bottom: 3.2rem; margin-top: 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
On networks that have a larger mask, one can assume that each shared network has one or more routers that provide service to that network. And they have one or more switches used to interconnect devices at the logical link layer. It may not be the case that you can imply that a node being up on a LAN also means the switch and router are up. But, when things are broken, you would want to check the routers first, then the switches, then the nodes.</div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.7); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Serif Pro", serif; font-size: 21px; line-height: 32px; margin-bottom: 3.2rem; margin-top: 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
Use machine learning to capture and hypothesize failures as patterns. When something changes, what also changed around it? What broke further downstream? After seeing a few failures, you should be able to pick up a probability on the causes and side effects in the network. (What a brilliant insight this would be for engineering teams to identify and solve for availability, performance, and redundancy issues based on priors?) </div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.7); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Serif Pro", serif; font-size: 21px; line-height: 32px; margin-bottom: 3.2rem; margin-top: 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
Let machine learning determine the frequency based on the goals and objectives.<br />
<br />
When you overlay other data elements in with this, the picture becomes a much more effective story. For example, in the heat map identify any node / IP that has experienced configuration changes. What about has an open ticket on it? What about event counts beyond the ping?</div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: 0px 0px; background-repeat: initial; background-size: initial; border: 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.7); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: "Source Serif Pro", serif; font-size: 21px; line-height: 32px; margin-bottom: 3.2rem; margin-top: 3.2rem; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
What if I use RUM and a bit of creative data from IPFIX to overlay onto the heat map application performance? Interesting...</div>
Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-65400324168315038592017-08-13T14:59:00.000-05:002017-08-14T07:39:48.104-05:00Machine Learning and Event Classification<div style="margin-top: 0.17in; page-break-after: avoid;">
<br /></div>
<div style="margin-bottom: 0in;">
When you look at the Netcool probe
systems, there are a couple of things you can learn
</div>
<div style="margin-bottom: 0in;">
from the rules files as well as the
events and raw data. Really, there is a tremendous amount of data at your fingertips that could be leveraged in new ways.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
When you look at basic Naive Bayes as a
function, it is an effective algorithm to use to classify</div>
<div style="margin-bottom: 0in;">
text elements. While I like the
approach, I chose to simplify it a bit here to make it more
understandable and usable without undue complication. Lets go on a
data exploration journey and see what we can derive from the mental
exercise...</div>
<h2>
Heres a thought</h2>
<div style="margin-bottom: 0in;">
Given I have a set of rules, and event
example, and I have raw event data. One of the first thing I'd like
to train on is severity. This is important because it is an
organizationally defined tag depicting a user perception.<br />
<br />
On a side note, I seem to always revert back to the Golden Rule of correlation:<br />
<br />
ENTERPRISE:NODE:SUBOBJECT.INSTANCE:PROPERTY<br />
<br />
The Scope applies to how much of the record is applied to delineate the Managed Object. For example a Node Scope means that correlation elements need to match on Enterprise and Node to apply together. </div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Naive Bayes is about intelligently
"guessing" elements that make up a fact as to whether they
are goodor bad. For a given small sample of events, I'm going to use
4 events as my reference training to illustrate potential learning.
</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Summary = Interface GE1 is down ,
Severity = 4</div>
<div style="margin-bottom: 0in;">
Summary = Interface GE1 is up ,
Severity = 0</div>
<div style="margin-bottom: 0in;">
Summary = Node down ping fail, Severity
= 5</div>
<div style="margin-bottom: 0in;">
Summary = Node up, Severity = 0</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
So,for the training process, I break
down each word in the Summary as follows:</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
<table border="0" cellspacing="0" cols="7" frame="VOID" rules="NONE">
<colgroup><col width="99"></col><col width="66"></col><col width="50"></col><col width="59"></col><col width="52"></col><col width="53"></col><col width="55"></col></colgroup>
<tbody>
<tr>
<td align="LEFT" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;" width="99"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;" width="66"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;" width="50"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;" width="59"><b>Word hit count</b></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;" width="52"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;" width="53"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;" width="55"><br /></td>
</tr>
<tr>
<td align="LEFT" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><b>Clear</b></td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><b>Indeterminate</b></td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><b>Warning</b></td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><b>Minor </b></td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><b>Major</b></td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><b>Critical</b></td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Interface</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">GE1</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">is</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">down</td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">up</td>
<td align="RIGHT" sdnum="1033;" sdval="2" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">2</td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Node</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">ping</td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">fail</td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
</tr>
</tbody>
</table>
</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
It should be noted that some words may
be a bit nebulous and have no differentiation in the determination of
severity. For example, check out the following table.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
<table border="0" cellspacing="0" cols="9" frame="VOID" rules="NONE">
<colgroup><col width="99"></col><col width="66"></col><col width="50"></col><col width="59"></col><col width="52"></col><col width="53"></col><col width="55"></col><col width="50"></col><col width="52"></col></colgroup>
<tbody>
<tr>
<td align="LEFT" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;" width="99"><br /></td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;" width="66">Interface </td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;" width="50">GE1</td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;" width="59">is</td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;" width="52">down</td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;" width="53">up</td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;" width="55">fail</td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;" width="50">ping</td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;" width="52">Node</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Clear</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="2" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">2</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Indeterminate</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Warning</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Minor</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Major</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Critical</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
</tr>
<tr>
<td align="LEFT" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" bgcolor="#CCCCCC" colspan="4" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Ratio of occurrence by Severity</td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Clear</td>
<td align="RIGHT" sdnum="1033;" sdval="0.125" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.125</td>
<td align="RIGHT" sdnum="1033;" sdval="0.125" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.125</td>
<td align="RIGHT" sdnum="1033;" sdval="0.125" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.125</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0.25" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.25</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0.125" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.125</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Indeterminate</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Warning</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Minor</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Major</td>
<td align="RIGHT" sdnum="1033;" sdval="0.125" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.125</td>
<td align="RIGHT" sdnum="1033;" sdval="0.125" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.125</td>
<td align="RIGHT" sdnum="1033;" sdval="0.125" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.125</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0.125" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.125</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Critical</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0.125" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.125</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0.125" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.125</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0.125" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.125</td>
<td align="RIGHT" sdnum="1033;" sdval="0.125" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.125</td>
</tr>
<tr>
<td align="LEFT" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
</tr>
<tr>
<td align="LEFT" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
<td align="LEFT" bgcolor="#CCCCCC" colspan="5" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><b>Ratio of Non-occurrence</b></td>
<td align="LEFT" bgcolor="#CCCCCC" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;"><br /></td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Clear</td>
<td align="RIGHT" sdnum="1033;" sdval="0.875" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.875</td>
<td align="RIGHT" sdnum="1033;" sdval="0.875" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.875</td>
<td align="RIGHT" sdnum="1033;" sdval="0.875" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.875</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0.75" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.75</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" sdnum="1033;" sdval="0.875" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.875</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Indeterminate</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Warning</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Minor</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Major</td>
<td align="RIGHT" sdnum="1033;" sdval="0.875" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.875</td>
<td align="RIGHT" sdnum="1033;" sdval="0.875" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.875</td>
<td align="RIGHT" sdnum="1033;" sdval="0.875" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.875</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0.875" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.875</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
</tr>
<tr>
<td align="LEFT" bgcolor="#FFCC99" height="17" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">Critical</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0.875" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.875</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="1" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">1</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0.875" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.875</td>
<td align="RIGHT" bgcolor="#23FF23" sdnum="1033;" sdval="0.875" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.875</td>
<td align="RIGHT" sdnum="1033;" sdval="0.875" style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: 1px solid #000000;">0.875</td>
</tr>
</tbody>
</table>
</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Normally, naive Bayes specifies a
simple true of false kind of thing. With 6 different severities, one
could classify the true as severities 2-5 and false 0 and 1. In this
case, you look for words that “differentiate” True or false. In
the example, the differentiating words are down, up, ping, and fail.
In this first iteration, I would be tempted to either drop ping or
add it to the Up event.</div>
<div style="margin-bottom: 0in;">
</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
In analyzing the words and distribution
of severities, we can discern that 4 words are differentiators. If
the distribution of words like “is” are across multiple
severities, they aren't so relevant in use as a predictor or a prior
in bayesian terms.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
The ratios are calculated per the total
word count for the entire sample. Out of each word,what ratio would
that word play on determination of severity.
</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
The ratio of non-occurrence is
interesting in that it shows you the ratio of how often it does not
occur and clearly illustrates severities that can be ignored by a
specific words presence.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
This is a very basic machine learning
event word classification mechanism spawned mainly out of the
simplification of Naive Bayes theory. While Naive Bayes is very
rudimentary, a lot of folks get hung up in the math that goes with it
in its truest sense. Here are the formulas I use:</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Ratio of occurrence is the number of
times a word is seen via a given severity divided by the total number
of unique words.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Ratio of non-Occurrence is Total number
of words minus the occurrences in this severity for this word,
divided by the total number of words.
</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
The benefit is that Severity ratings
are a perception of importance. The more accurate and consistent
these simple classifications become, the better your more intensive
Inferences and deep learning will be going forward.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Interestingly enough,these are just
words that are independent of one another. Other that occurrence and
perception, is is no different than down.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Now, what if I look for words in the
event text that are also in the AlertKey? Normally, AlertKey is used
to identify the Subobject and instance of the event. Would this not
readily identify the noun/pronoun/object this event relates to?
Could it be skipped in the calculations to make things more accurate?</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
What if you used AlertGroup to do
associative vectors to only group together like events? Like a
defined cluster of events by type.
</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
What if you did the same thing for
Type? Melding in the dimension of whether an event relates to a
Problem or Resolution could help drive the accuracy of your event
processing system.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-top: 0.17in; page-break-after: avoid;">
<span style="font-family: "arial" , sans-serif;"><span style="font-size: medium;">Solving
a Problem</span></span></div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
The questions you may be looking to
answer are:</div>
<ol>
<li><div style="margin-bottom: 0in;">
Are my event wordings consistent
to be able to guess a severity if we didn't have one already?</div>
</li>
<li><div style="margin-bottom: 0in;">
Is my severity selection
consistent?</div>
</li>
<li><div style="margin-bottom: 0in;">
Event object definition accurate?</div>
</li>
</ol>
<div style="margin-bottom: 0in;">
Lets say, I do this exercise on my top
20 events seen in the environment. Use these events to hone the
“dictionary” of initial training. Then, lets apply it to the
rest of the events.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
There are some good news elements here.
These include:</div>
<ol>
<li><div style="margin-bottom: 0in;">
Identify and build in consistency
into your event processing.</div>
</li>
<li><div style="margin-bottom: 0in;">
Use the results to identify the
severity of new and existing events.</div>
</li>
<li><div style="margin-bottom: 0in;">
You can store the results by event
(using Identifier as the key) so that you don't have to recalculate
each time.</div>
</li>
<li><div style="margin-bottom: 0in;">
You can use history to build this.
</div>
</li>
</ol>
<div style="margin-bottom: 0in;">
Another very interesting “feature”
here is that many of the events processed are SNMP based. Within the
rules used to process the event, in many cases, you get the OID of
the enterprise, OIDs or varbinds, and enumerations used to translated
the numbers into text values. And what about the MIB object
descriptions? And variable bindings that are instanced usually point
ot a given subobject and instance. (ifEntry.4 as an example)</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Some may even ask why do this. What
if, programmatically, I can determine that performance and error
conditions related to a given sub-object are in fact, supporting
evidence to a most probable cause event that resulted in an outage.
Now, we are getting some where. Think beyond the Root Cause and Side Effect paradigm into the realm of recognizing and understanding event clusters.</div>
<div style="margin-top: 0.17in; page-break-after: avoid;">
<span style="font-family: "arial" , sans-serif;"><span style="font-size: medium;">Summary</span></span></div>
<div style="margin-bottom: 0in;">
I should note that while I use Netcool
as an example, one should not constrain these techniques to a single
product. I could see this same sort of techniques used in HPE Node
Manager, ScienceLogic EM7, Splunk, OpenNMS, Monolith, and others.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Machine learning is about exploring
your data, classifying it, and producing better information.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<br />
<div style="margin-bottom: 0in;">
<br /></div>
Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-25255775545748283202015-10-13T21:26:00.001-05:002015-10-14T22:25:42.749-05:00Getting ahead of the Customer Experience Perception CaseI have read several articles on how users are the alarm of prevalence in many environments. How we should be looking at customer experience.<br />
<br />
This is so true and appropo. If you are truly looking to provide customer service, the customer experience should be at the forefront of your service philosophy.<br />
<h3>
Why?</h3>
In the beginning, help desk staff listened for a call. They waited on the phone to ring. In fact, in some (MANY!) environments, they still do. They also use the Call routing information to determine if there is a problem in a specific area, neighborhood, or part of the infrastructure. Wild huh? If your NOC is using call statistics to do correlation, you are definitely managing alarms and alerts in the user perception space.<br />
<br />
Even in many modern day Operations centers, Operations operates in a mode of being purely reactive to incoming events. Furthermore, in cases where the inputs overwhelm the staff's ability to discern real problems and prioritize them as time evolves,you see people that will wait for the loudest problem to surface with incoming calls.<br />
<br />
If your layer 1 support is doing dispatch only,in almost all cases, you are operating in a purely reactive environment.<br />
<br />
If you only allow events to be presented that are predefined and actionable, you are simulated that phone call in software. Same thing,different media.<br />
<h3>
User Perception Window</h3>
It is the time interval from when the customer is affected to the time it gets too painful to go on without reporting the problem.<br />
<br />
In some environments, this can be hours. Especially if the end user has not seen results from previous service outages. Or they have had negative experiences in calling in problems. Some will commence to doing their own troubleshooting like rebooting everything. Some will merely wait, go take a break, or go to lunch in hopes that the problem heals or gets fixed.<br />
<br />
When folks transition from in house support to an outsourced arrangement, one of the factors that is common is the need for better support. More responsiveness. Better up time. Better awareness.<br />
<br />
In some instances, the time has become so critical, end users will introduce problems just to test and see how long it takes for the managed services provider to respond. This results in a very short window and usually fares badly for the Service provider.<br />
<br />
Negative perceptions by your customers have a negative effect on your Net Promoter Scores and can be the most prevalent cause of customer churn. They affect the effectiveness of the support organization. And the ability to generate new revenue.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfLeGYic5GpYyU04dvgbesQ1_qv2e-ohk_QO6B3Q8leULAx4M76RhHdKyiNuA2elgOYUaHp9ME3dmbPhyRh4-jNa6xMNxZjpkuQyyetojzeBK11hX-EDOChWOXF1B3G-ZLn25oEgLFcP0/s1600/Slide1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfLeGYic5GpYyU04dvgbesQ1_qv2e-ohk_QO6B3Q8leULAx4M76RhHdKyiNuA2elgOYUaHp9ME3dmbPhyRh4-jNa6xMNxZjpkuQyyetojzeBK11hX-EDOChWOXF1B3G-ZLn25oEgLFcP0/s640/Slide1.png" width="640" /></a></div>
<h3>
Architecture</h3>
Most management architectures are designed wrong for the ability to migrate towards a proactive management stance. If you are waiting on Traps and syslog events, you are also waiting on the phone to ring. While this is cheap and easy, it carries with it the consequences of always being after the fact, always post-cognitive.<br />
<br />
And the problem is profoundly exacerbated by the introduction of agility in the enterprise. The migration towards constant updates, infrastructure movement and redefinition, migration of applications across cloud platforms and containers, even off premise.<br />
<br />
Consider this - changes in the environment can happen ANYWHERE in the Green, Red, or Yellow zones. In effect, a change can lead up to an event horizon, cause other effects after the event horizon, or change the effects by changing in the middle of a problem.<br />
<br />
If your architecture only looks at the red and yellow zones, you can never get AHEAD of the User Perception Window. You can get a better handle on how you handle problems, identification and prioritization of problems, even building better workflow and run book processes.<br />
<br />
<h3>
How Do You Get There?</h3>
In many cases,architects and management has chosen the path of least resistance in hopes that enterprise management as a technology, is a commodity. (Funny - This was a marketing ploy by wares vendors to circumvent having to compete!)<br />
<br />
Interesting thing about getting ahead of the customer is that this is the hard part. It is the part where you have to go through the data, the workflow, the results, and come up with solutions to designing and implementing around architectural and product shortcomings, improving the processes and automations, and building and putting in place more effective instrumentation.<br />
<br />
I'd like to warn you up front - if you're not willing to commit to the challenge, its better to admit that you will never get ahead of the customer experience perceptron. Maybe you can set expectations with customers. Maybe you can put some spin on it.<br />
<br />
There are several, very important Continuous Process Improvement sorts of tasks that need to be undertaken. These include:<br />
<br />
<h4>
1. Post Mortems.</h4>
<br />
What was the root cause of the problem? Was there more than a single cause?<br />
Did the organization mishandle the problem?<br />
Were there things that could have made the problem correction, better?<br />
Are the runbooks and processes in order?<br />
Has redundancy, DR, and HA been addressed properly?<br />
<br />
A post mortem analysis is imperative to go through and analyze the what happened and how the support organization responded. You need the data to be able to benchmark how information was derived and things were accomplished from the start to finish.<br />
<br />
<h4>
2. Failure Analysis</h4>
<br />
In the course of time, periodically, you need to go through your tickets and look for hardware and software that has failed over the reporting period. Look for patterns and inconsistencies in the products, services, and systems.<br />
<br />
An important gauge is to come up with a way of providing a cost of maintenance per device / Device type / Application. Analyze both Scheduled and unscheduled maintenance actions. This gives you an EXCELLENT way of illuminating problem areas in a way that non-technical people understand - dollars and cents. Doesn't have to be real but relative and relevant.<br />
<br />
Many Operations environments actually inflict a lot of pain on themselves by not doing failure analysis. You need to be ahead of the curve of equipments, systems, and software that fails more and more,takes more time to maintain, and causes more downtime.<br />
<br />
<h4>
3. Instrumentation</h4>
<br />
In the course of getting ahead of the customer perception window, you have to advance the instrument to seek out and illuminate issues before user perception is realized. If you are not increasing the instrumentation to be more predictive, you can not ever be able to visualize before the event horizon.<br />
<br />
With containerization and microarchitectures, you need to build in advanced monitoring capabilities. In fact, this advanced precognitive monitoring needs to be an integral part of the microarchitecture.<br />
<br />
If you are not fielding advanced correlation where you are ACTIVELY looking for pre-cognitive conditions - conditions that lead up to a potential failure, you will NEVER EVER get in front of the customer. If you are still waiting on a trap, a syslog event, or even a timed threshold, you are tragically on the wrong side of the timeline.<br />
<br />
You need to look at user transactions from the user perspective. a 3 second deviation,while not discernable to many, could yield huge insight into an oncoming disaster.<br />
<br />
What about IPFIX / Netflow data? What can you discern from this data to yield insights? Can you instrument the patterns into software to turn it into something to alert on?<br />
<br />
<h4>
4. Adaptive Analytics</h4>
<br />
You need to be able to sample through the combinations of configurations and analyze event streams, instrumentation, and workflow data to look for predictive patterns that point to a customer experience potential problem BEFORE the event horizon occurs.<br />
<br />
What things happened to illuminate an pending event horizon?<br />
<br />
Can you discern loading conditions and thresholds from you analytics?<br />
<br />
Linear regression? By time intervals? Related or not related. Causal or not?<br />
<br />
<h3>
Summary</h3>
<br />
While out of the box Enterprise Management applications say they are proactive, take a good look at where they function in the Customer perspective perceptron space. Could be, they are proactive after the customer perceptron.<br />
<br />
Until you instrument and threshold on things that are before the customer perception window, <br />
<br />
<span style="color: red; font-size: large;"><b>YOU CANNOT GET AHEAD OF THE CUSTOMER EXPERIENCE. </b></span><br />
<br />
In the comments, I'd be interested in hearing how your product / service fits in the Customer Perception window. Leave a comment!<br />
<br />
<br />Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-19236816932404734922015-09-12T21:39:00.004-05:002015-09-12T21:39:58.562-05:00To Build Versus BuyOver the course of time that Management systems have evolved - at least in my years of exposure - there seems to always be the question - Build vs. Buy.<br />
<br />
I have been in 3 different scenarios that show different perspectives as to where these views come from. These perspectives include:<br />
<br />
<ul>
<li>Product Company</li>
<li>Integrator</li>
<li>End User</li>
</ul>
<br />
<h2>
<b> Product Perspective</b></h2>
<br />
From a product perspective, many tend to believe that the product is close enough to the 80% rule that the question of build versus buy should never come up. In fact, some believe strongly enough to count the end user out as eccentric or lacking. Or they are intimidated by the Integrator.<br />
<br />
I always liked the possibilities I could bring about with Log Matrix NerveCenter. And the functions I put into the menus of HP OpenView.<br />
<br />
Some products painted themselves in a corner. Like Netcool OMNIbus. When they aligned to Telco oriented standards, they mandated to problems outside of the telco realm had to adapt to the Telco standards. For example, color coding of Severity. But then again, not everything has a real severity or conforms so much to event management.<br />
<br />
Some product companies are skeptical and somewhat intimidated by premier Integrators. Instead of listening to the requirements and approaches, then embracing change, they will shun it.<br />
<br />
As an Integrator, I was avoided, patronized, and shunned. A few select product companies embraced my approaches.<br />
<br />
As an end user, product companies seemed a bit taken aback when I told them their product doesn't scale. Or doesn't fit. Sometimes to the point of having their lawyers call me (Kinda weird). And I've seen Sales folks do all sorts of things in lieu of fixing the technical issues. Like a visit to your VP. Or a talk with your CEO. You know you've been sandbagged when your VP comes in with a bunch of glossies and tells you to evaluate this or that.<br />
<br />
<h2>
Integrator</h2>
<br />
As an Integrator, products are viewed as something they can use to deliver a value add. While there are a lot of Integrators that prefer to just do installation and setup, the premier Integrators are always looking for products that create a difference. That segregates them from others.<br />
<br />
From an Integrator perspective, I always strived to achieve success for both my team and my customer. Sometimes, that takes a bit of work. And some thinking out of the box.<br />
<br />
Sometimes, it was with a product and its capabilities. Other times, I did my own code.<br />
<br />
In the industry, there are a few folks out there that take products out of the box, put some code around them to integrate with other products or to add capabilities, and sell that and services around the extended capabilities and services. Product companies don't always know how to leverage these folks or even consider them viable.<br />
<br />
<h2>
End User</h2>
<br />
From an end user perspective, some products just don't scale enough to make it. Or they lack critical capabilities. And yes, price can be a factor.<br />
<br />
I've been in places that could not use commercial products without significant work to make it work. For example, I know of a place that had 8 separate and distinct eHealth instances. And they ended up with 8 people supporting the product.<br />
<br />
In instances where Build has evolved into a viable option, cost, capabilities, and scalability are the primary reasons.<br />
<br />
In the immortal words of Larry Wall -- "There's more than one way to do it!". Product companies don't have a stranglehold on innovation. In many cases, quite the opposite is true. Its hard to do something different if the product does 50% or less of what the requirement is if it means refactoring and redeveloping code. In fact, many developers consider that the product is done once coded. Of the ones that have evolved, you see a lot of refactoring and reworking to achieve more capabilities and scale.<br />
<br />
<h2>
In Summary... EMBRACE the Innovator.</h2>
<br />
Product Companies - Use these Innovators to expand horizons, empower repeatable integrations, and drive solutions over tools.<br />
<br />
Integrators - Want to get to the next level? Get you some Innovators and start productizing your Value Add.<br />
<br />
End Users - Use Innovators to work through problems and get to solutions. Tools aren't much it they don't fit your organization form and function and its workflow. Innovators do that. Embrace innovations that enhance your business in meaningful and distinctive ways. Keep driving efficiency and customer satisfaction up.<br />
<br />
<br />
<br />
<br />Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-51392808499334756312012-08-08T21:23:00.001-05:002012-08-09T04:32:37.702-05:00Enterprise Management - What's Missing?<div style="margin-bottom: 0in;">
<span>Enterprise Management has been a somewhat strange market over the years. The problem definitions are really tough but are rewritten to lessen the amount of development needed to achieve something that marketing can spin. Some elements like correlation and root cause analysis have been dumbed down and respun in an effort for many to tout a "me too" to the uninitiated.</span></div>
<div style="margin-bottom: 0in;">
<span><br /></span></div>
<div style="margin-bottom: 0in;">
<span>Other areas like performance management platforms have become huge report generators. After all, pretty graphs and charts sell. In some systems, it is amazing the amounts of graphs produced, 90% of which may never be looked at.</span></div>
<br />
<div style="margin-bottom: 0in;">
<span>The other amazing part is that even though a product is evolved and bought out, there comes a point where design decisions of the past dictate what can and cannot be done going forward. And this holds true for not just the older products but some new ones as well. It is incredible to see a product that in its relatively young life cycle that has already mandated the inability to extend the architecture further.</span></div>
<div style="margin-bottom: 0in;">
<span><br /></span></div>
<div style="margin-bottom: 0in;">
<span>The emerging technologies that have come to light most in the past 3 years or so are
provisioning, configuration management, and workflow automation. This
is especially true in the Cloud realm as Cloud needs standard
configuration items, a way of automating provisioning, and ways to
pre-provision systems overall. After all, virtualization is the
technology and Cloud is the process.</span></div>
<h1 class="western">
<span style="font-family: Times New Roman;">OK - What's
Missing?</span></h1>
<div style="margin-bottom: 0in;">
<span>I
know we've seen all of the sales pitches on performance management
with charts and graphs. And we've been barraged by the constant
pitch for business services management with its dashboards and
service trees. And we've seen the event and alarm lists, the maps,
and even ticketing.</span></div>
<div style="margin-bottom: 0in;">
<br />
</div>
<div style="margin-bottom: 0in;">
<span>Someone
once said that <b>"a fool with a tool is still a fool"</b>.
<span style="font-size: medium;"> <i><u>Effective</u></i> </span>Enterprise Management
Architectures and implementations have NEVER been about tools. Tools
are the technology and Workflow is the process. Tools do
nothing for the bottom line of an Enterprise or Service
Provider without someone using it. In fact, the tools that are not
used usually become shelfware or junkware. In many cases, you can't
even sell the stuff to some other fool looking for a tool.</span></div>
<div style="margin-bottom: 0in;">
<br />
</div>
<div style="margin-bottom: 0in;">
<span>One
of the most interesting assessments of emerging technologies is the
Gartner Hype Cycle. Here is the Wikipedia article on the Hype
Cycle.</span></div>
<div style="margin-bottom: 0in;">
<br />
</div>
<div style="margin-bottom: 0in;">
<a href="http://en.wikipedia.org/wiki/Hype_cycle">http://en.wikipedia.org/wiki/Hype_cycle</a></div>
<div style="margin-bottom: 0in;">
<br />
</div>
<div style="margin-bottom: 0in;">
<span>While
it is not so scientific and not really a cycle per se, it is a way to
describe the subjective nature of a technology and the phases it goes
through. Interestingly, there are five different stages in the
Gartner Hype Cycle. Following is an excerpt from Wikipedia describing
the Hype cycles.</span></div>
<div style="margin-bottom: 0in;">
<br />
</div>
<h3 class="western">
<a href="" name="Five_phases"></a>Five phases</h3>
<div style="margin-bottom: 0in;">
<a href="http://en.wikipedia.org/wiki/File:Gartner_Hype_Cycle_for_Emerging_Technologies.gif"><span style="color: white;"><img align="BOTTOM" border="1" height="264" name="graphics1" src="http://upload.wikimedia.org/wikipedia/commons/thumb/f/f1/Gartner_Hype_Cycle_for_Emerging_Technologies.gif/300px-Gartner_Hype_Cycle_for_Emerging_Technologies.gif" width="303" /></span></a></div>
<div style="margin-bottom: 0in;">
<a href="http://en.wikipedia.org/wiki/File:Gartner_Hype_Cycle_for_Emerging_Technologies.gif"><img align="BOTTOM" border="0" height="14" name="graphics2" src="http://bits.wikimedia.org/static-1.20wmf7/skins/common/images/magnify-clip.png" width="18" /></a></div>
<div style="margin-bottom: 0in;">
Hype cycle for emerging technologies as
of July, 2009</div>
<div style="margin-bottom: 0in;">
A hype cycle in Gartner's
interpretation comprises five phases:</div>
<ol>
<li><div style="margin-bottom: 0in;">
"Technology Trigger" —
The first phase of a hype cycle is the "technology trigger"
or breakthrough, product launch or other event that generates
significant press and interest.</div>
</li>
<li><div style="margin-bottom: 0in;">
"Peak of Inflated
Expectations" — In the next phase, a frenzy of publicity
typically generates over-enthusiasm and unrealistic expectations.
There may be some successful applications of a technology, but there
are typically more failures.</div>
</li>
<li><div style="margin-bottom: 0in;">
"Trough of Disillusionment"
— Technologies enter the "trough of disillusionment"
because they fail to meet expectations and quickly become
unfashionable. Consequently, the press usually abandons the topic
and the technology.</div>
</li>
<li><div style="margin-bottom: 0in;">
"Slope of Enlightenment"
— Although the press may have stopped covering the technology,
some businesses continue through the "slope of enlightenment"
and experiment to understand the benefits and practical application
of the technology.</div>
</li>
<li><div style="margin-bottom: 0in;">
"Plateau of Productivity"
— A technology reaches the "plateau of productivity" as
the benefits of it become widely demonstrated and accepted. The
technology becomes increasingly stable and evolves in
second and third generations. The final height of the plateau varies
according to whether the technology is broadly applicable or
benefits only a niche market.</div>
</li>
</ol>
<div style="margin-bottom: 0in;">
The term is now used more broadly in
the marketing of new technologies.</div>
<div style="border-bottom: 1px solid #000000; border-left: none; border-right: none; border-top: none; margin-bottom: 0in; padding-bottom: 0.03in; padding-left: 0in; padding-right: 0in; padding-top: 0in;">
<br />
</div>
<div style="margin-bottom: 0in;">
<br />
</div>
<div style="margin-bottom: 0in;">
Now keep in mind, the hype cycle is
very subjective. Dependent upon perspective, one person could place
a given technology in a different category than you would. (This is
why you enlist the expert analysis of the Gartner Analysts.)</div>
<h2 class="western">
Workflow Automation</h2>
<div style="margin-bottom: 0in;">
<span style="color: white"><span style="font-family: Times New Roman;">You
see a significant number of Workflow automation products that are
just now coming into Service Providers and Enterprises. Part of this
is being driven by requirements from cloud services. Cloud Services
NEED to be automated. The services need to be automated and user
driven as much as possible.</span></span></div>
<div style="margin-bottom: 0in;">
<br />
</div>
<div style="margin-bottom: 0in;">
<br />
</div>
<div style="margin-bottom: 0in;">
<span>Did
we ever get to be ITIL compliant? Or is there such a thing? Maybe
for ticketing systems. But do your organizations reall truly
follow ITIL Incident and problem management to the letter? Can your
organization?</span></div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
<span>ITIL
defines an Incident as : </span></div>
<div style="margin-bottom: 0in;">
--------------------------------
</div>
<span>Any
event which is not part of the standard operation of a service and
which causes, or may cause, an interruption to or a reduction in, the
quality of that service. The stated ITIL objective is to restore
normal operations as quickly as possible with the least possible
impact on either the business or the user, at a cost-effective price.</span><br />
<span>-------------------------------</span><br />
<div style="margin-bottom: 0in;">
<span>Effective
workflow automation and process optimization begins with a baseline
of the process. If you cannot define the process or measure it, how
will you improve it? In the spirit of getting ITIL Incident
Management to be optimized and effective, one needs to
implement effective business process management techniques.</span></div>
<div style="margin-bottom: 0in;">
<br />
</div>
<div style="margin-bottom: 0in;">
<span>But
there are psychological and philosophical problems with how processes
are approached and presented. Some of these "issues" include:</span></div>
<ul>
<li><div style="margin-bottom: 0in;">
The belief that documenting the
process is to train someone else for your job.</div>
</li>
<li><div style="margin-bottom: 0in;">
The belief that documenting some
technical processes is impossible as it takes into consideration
judgement and subjective decisions.</div>
</li>
<li>The belief that documenting processes gets in the way of
actual work.<br />
</li>
</ul>
<div style="margin-bottom: 0in;">
<span>A
significant amount of this is Fear, Uncertainty, and Doubt or FUD.
But it can block or kill effective process optimization. You
have to manage to and reward process optimization.</span></div>
<div style="margin-bottom: 0in;">
<br />
</div>
<h2 class="western">
Knowledge Management</h2>
<div style="margin-bottom: 0in;">
While you see
segmented implementations of Knowledge Management by various products
or as modules in ticketing systems, rarely do you see somewhat robust
implementations beyond the very large Service Providers.</div>
<div style="margin-bottom: 0in;">
<br />
</div>
<div style="margin-bottom: 0in;">
Knowledge
Management takes knowledge, facts, and references, organizes that
information and makes it available in the various processes as a way
or method of putting the knowledge to work. For example, an event is
presented to a level 1support person that denotes an actionable
event. In the course of this work, knowledge can be presented at
several different aspects to include:</div>
<div style="margin-bottom: 0in;">
<br />
</div>
<ul>
<li><div style="margin-bottom: 0in;">
Notification
instructions to contact the customer</div>
</li>
<li><div style="margin-bottom: 0in;">
Escalation
instructions to the appropriate Triage Team</div>
</li>
<li><div style="margin-bottom: 0in;">
Recent
histories of the elements in question</div>
</li>
<li><div style="margin-bottom: 0in;">
Scheduled
actions and maintenance actions</div>
</li>
<li><div style="margin-bottom: 0in;">
Recent
Configuration changes.</div>
</li>
<li><div style="margin-bottom: 0in;">
Checks and
validations process data.</div>
</li>
</ul>
<div style="margin-bottom: 0in;">
<br />
</div>
<div style="margin-bottom: 0in;">
Knowledge
Management takes information and creates confidence and situation
awareness in support organizations. The knowledge enables each and
every user or support member to work with common technical terms
regardless of level of expertise. Even simple knowledge management
tasks such as attaching linkages to vendor technical documentation to
configuration records,enables users to be able to reference and
communicate better regarding diagnosis, configuration, or
provisioning.</div>
<div style="margin-bottom: 0in;">
<br />
</div>
<h2 class="western">
Collaboration</h2>
<div style="margin-bottom: 0in;">
<br />
</div>
<div style="margin-bottom: 0in;">
When take a step
back and look at every application on the market today in the
Enterprise Management realm, the one thing that sticks out profoundly
is the lack of collaboration capabilities and multi-user awareness.
Not one single application has user awareness. For example, you open
a network map. Who else is in that map? When you open a ticket, who
else has that ticket open?</div>
<div style="margin-bottom: 0in;">
<br />
</div>
<div style="margin-bottom: 0in;">
Applications
don't really enable folks to work together. They are more geared
toward individual actions. Then when that person is done, it is
escalated to the next person. And when things move across products,
it is even worse.</div>
<div style="margin-bottom: 0in;">
<br />
</div>
<div style="margin-bottom: 0in;">
Consider the
customer who calls into a service Provider with a problem. The first
level takes down the ticket information, runs through some run book
elements, then escalates the ticket to the next level of support. At
this point,the customer is told they will get a call from the next
tier. So, the customer hangs up and waits on the next call. In this
scenario, who owns the problem? What if Tier 2 doesn't call back for
a long period of time?</div>
<div style="margin-bottom: 0in;">
<br />
</div>
<div style="margin-bottom: 0in;">
While this
scenario is all too common, who is collaborating? The customer is
stuck dealing with a bunch of individual contributors to get their
problem resolved. There is no team. The only way the customer gets
a team is to initiate an escalation to the Service Representative for
the Account and crank up a conference call with all of the Managers.</div>
<div style="margin-bottom: 0in;">
<br />
</div>
<div style="margin-bottom: 0in;">
Effective
customer service and customer support is a TEAM effort that requires
collaboration and socialization. It is not about fielding tickets.
It is about taking care of the customer.</div>
<div style="margin-bottom: 0in;">
<br />
</div>
<div style="margin-bottom: 0in;">
There are
products on the horizon that will empower collaboration as a whole
but still, a significant amount of work needs to be done on each
individual product.</div>
<div style="margin-bottom: 0in;">
<br />
</div>
<h2 class="western">
The Hype Cycle</h2>
<div style="margin-bottom: 0in;">
<br />
</div>
<div style="margin-bottom: 0in;">
<span>The Gartner folks are much more precise and much better at accurately listing the Hypecycle than I am capable of. However, I like the way it depicts a product or technology lifecycle. I am merely overlaying my opinion on the Gartner definitions. While I know its subjective and only my opinion, here's
where I put these technologies in the Gartner Hype Cycle:</span></div>
<div style="margin-bottom: 0in;">
<br />
</div>
<span style="font-size: medium;"><b>Workflow Automation</b></span> – Between the
“Peak of Inflated Expectations” and the “Trough of
Disillusionment”.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjeA0wR9CuFQwEuQxN6meG3rAjxMrrefQ3Wb9T2B9aFx2SU5HnNP_wic0Adbqc5RkD5c4yHeqf00RJeTagDzJMPnP44FpFUHK0QgWSRvoXHZbozF767fBn6v2Rn8lZwmaQFI2Bh-TxCmE/s1600/workflowautomation.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="356" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjeA0wR9CuFQwEuQxN6meG3rAjxMrrefQ3Wb9T2B9aFx2SU5HnNP_wic0Adbqc5RkD5c4yHeqf00RJeTagDzJMPnP44FpFUHK0QgWSRvoXHZbozF767fBn6v2Rn8lZwmaQFI2Bh-TxCmE/s640/workflowautomation.jpg" width="640" /></a></div>
<br />
<div style="margin-bottom: 0in;">
There are a significant number of
product offerings in the industry that do these functions. And
getting more and more each day. While we do see new product
offerings for the same sort of functionality, everyone seems to think
they do it better. Yet many fall short, are too complex to maintain,
or are too expensive.</div>
<div style="margin-bottom: 0in;">
<br />
</div>
<div style="margin-bottom: 0in;">
The big players have purchased many of
the initial workflow automation applications and next generation
applications are in process. Additionally, BPM related systems are
being enhanced to perform more of the workflow automation associated
with Enterprise Management.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Interesting links:</div>
<div style="margin-bottom: 0in;">
<a href="http://www.wfmc.org/">Workflow Management Coalition</a></div>
<div style="margin-bottom: 0in;">
<a href="http://cordys.com/business_operations_platform">Cordys Business Operations Platform</a></div>
<div style="margin-bottom: 0in;">
<a href="http://www8.hp.com/us/en/software-solutions/software.html?compURI=1170673#.UCMQfE1lQ0I">HP OO</a></div>
<div style="margin-bottom: 0in;">
<a href="http://www.bmc.com/products/product-listing/it-process-automation-software.html">BMC Atrium Orchestrator</a></div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
<br />
</div>
<span style="font-size: medium;"><b>Knowledge Management</b></span> -- “Technology
Trigger”<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhlCj0HkX2BLWFJNnv0yT4ElCIIGJ0-MDzS1DvjITJNGTeVw9Wod0haj8DAb_evt6Cnbph7WHDpZD1nzy8Jm9249Zllx4ndio7d_zUiGLWNX_Yo94ki0GotStrzB0yL_FW-AMdX7oZIcc0/s1600/knowledgemgmt.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="358" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhlCj0HkX2BLWFJNnv0yT4ElCIIGJ0-MDzS1DvjITJNGTeVw9Wod0haj8DAb_evt6Cnbph7WHDpZD1nzy8Jm9249Zllx4ndio7d_zUiGLWNX_Yo94ki0GotStrzB0yL_FW-AMdX7oZIcc0/s640/knowledgemgmt.jpg" width="640" /></a></div>
<br />
<br />
While there are a couple of KM systems offerings, most of the
implementations are released as product features or are developed by
large service providers for in house use only. For example, there are KM functions in SCOM/SCCM. Additionally,
there are KM modules for ticketing systems like Remedy.<br />
<br />
What you do see is that larger Service Providers are investing
heavily in KM systems. It is done for many reasons, chief among them
is driving down the cost of support and leveraging knowledge across
multiple customers. Makes sense as these large outsourcing and
service providers are usually a training ground for entry level
personnel. Implementing KM systems is an accelerator for learning in
these environments.<br />
<br />
Interesting links:<br />
<br />
<a href="http://www.wiikno.com/">WiiKno</a> - The company helps organizations put in place Knowledge Management systems and integration.<br />
<a href="http://msdn.microsoft.com/en-us/library/bb896740.aspx">Microsoft Knowledge Management Architecture paper</a><br />
<a href="http://www.bmc.com/products/product-listing/37811954-129049-1583.html">BMC Knowledge Management</a><br />
<br />
<br /><br />
<br />
<span style="font-size: medium;"><b>Collaboration – </b></span>“Technology
Trigger”<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPEjohJVXYUZtSw_IHapRIvSqe041QiR6R4cIvRAuscupJ1XqrrdDUvgvNvKJGbttU8IcwUr8RC4PDYGHX8u_XYZkzYPHxMDNARqTpIoqopdIUaRMkivaoGs3G3GmtUjTU2fheVj1ip8Y/s1600/collaboration.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="358" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPEjohJVXYUZtSw_IHapRIvSqe041QiR6R4cIvRAuscupJ1XqrrdDUvgvNvKJGbttU8IcwUr8RC4PDYGHX8u_XYZkzYPHxMDNARqTpIoqopdIUaRMkivaoGs3G3GmtUjTU2fheVj1ip8Y/s640/collaboration.jpg" width="640" /></a></div>
<br /><br />
<br />
There are a couple of emerging technologies here that could very
well redefine how IT organizations perform operations and support.
You may very well see some older technologies (Like Ticketing) take a
back seat to newer collaboration and socialization technologies.
<br />
<br />
Part of the problem is that ticketing systems rarely capture
enough information to do detailed operations analysis and
optimization. They are typically too user overbearing to facilitate
much of the real time notes that get missed or input after the fact.<br />
<br />
When problems are ongoing, do you get the full monte of whats
going on? Or do you have to aggregate your own picture? How
effective is your post mortem analysis data? Do you see and recognize involvement by different support
functions, business units, and the customers?<br />
<br />
<br />
Interesting links:<br />
<br />
<a href="http://www.moogsoft.com/">MOOGSoft</a> - Check out the guys behind the curtains.<br />
<br />
<br />Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-55408562747094167152012-07-15T01:02:00.000-05:002012-07-16T07:20:32.279-05:00Thoughts on Telepresense ManagementMany organizations have turned to Telepresense capabilities as a way to enable collaboration and teaming across very separated geographical locations. When you think about it, the cost of getting managers, directors, and engineers together to address challenges and work through solutions, Telepresense systems have the potential to not only save money but also empower organizations to respond much faster as an organization.<br />
<div>
<br /></div>
<div>
<div style="margin-bottom: 0in;">
Going further, the proliferation of
video streams is becoming more common place every day. Not only are
collaboration systems becoming more popular, many Service Providers are
offering video based services to both corporate and consumer markets.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Video conferencing technology has been
around for several years. It has evolved significantly as the Codecs
and MCUs have evolved significantly along with several standards.
Additionally, the transport of these streams is becoming common on
internet and private TCP/IP networks where in the past, dedicated DSx or ISDN circuits were used..</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Video traffic, by its very nature,
places a series of constraints on the network due to the nature and
behavior of the traffic. </div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
<b>Bandwidth</b></div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
First there are bandwidth requirements. High
motion digital MPEG video can present a significant amount of traffic
to the network. </div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
<b>QoS</b></div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Additionally, video traffic is very jitter sensitive
and dynamics like packet reordering can wreak havoc in the stream of
what you see on screen. In some cases, you may not even own portions
of the network you traverse and the symptoms may only be present for
a very short period of time.
</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
<b>Video Traffic</b></div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Alot of multimedia and streaming type traffic is facilitated by the pairing of RTP and RTCP channels. RTP usually runs on an even numbered port between 1024 and 65535 with the RTCP channel running on the next higherport number. This pairing is called a tuple. Additionally, while RTP and RTCP are protocol independent, they typically run on a UDP transport as TCP tends to sacrifice timeliness in favor of reliability.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Multipoint communications is facilitated by IP multicast.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
In digitized speech passing across the network, with G.711 and G.729, the two most common VoIP encoding methods, you get two samples of voice per packet. The size of voice packets tends to be slightly larger than 220 bytes per packet and therefore the overall behavior is the Voice is a constant bit stream constrained by the sample rates needed per interval. Because of sampling, the packets occur on a 20 msec interval.
</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
</div>
<div style="margin-bottom: 0in;">
Some video systems use ITU H.264 encoding which is equivalent to MPEG-4 Part 10, and uses 30 Frames per second. Each frame must be setup every 33 msec. But MPEG encoding per frame can have a variable number of bytes and packets depending upon the video content. MPEG video is accomplished using a reference frame and subsequent data sections provide updates to the original frame in what is called a GOP or Group of Pictures.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
A GOP starts off with an I frame which is the reference frame. A P frame represents a significant change in the initial I frame but continues to reference the I frame. B Frames update both I frames and P frames.</div>
<div style="margin-bottom: 0in;">
The types of frames and their location within a GOP can be defined in a temporal sequence. This temporal distance is the time or number of images between specific types of images in a digital video. “m” is the distance between successive P frames and “n” is the distance between I frames.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
In contrast to serial digital television, 30 frames per second is digitized where the entire screen is digitized and sent every frame. As compared to MPEG,the transmission requirements are significantly less than serial digital because only the changes to the I frame are updated using the P and B frames. </div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Following is a diagram of a typical Group of Pictures and how the frames are sequenced.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKAuXks7kCCD3KBnb6o4hp1DHXihGU75dMfJp84N0CVgz5_NK_BlW2smxJd9aWhJGZN7vFEO-G0Yq3ZmsDOxMpygBIK2avdpTx-sgnwexQQR2XbOQORrGEuAauNJUde4bwmAZ1okQOmJU/s1600/GOP.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="172" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKAuXks7kCCD3KBnb6o4hp1DHXihGU75dMfJp84N0CVgz5_NK_BlW2smxJd9aWhJGZN7vFEO-G0Yq3ZmsDOxMpygBIK2avdpTx-sgnwexQQR2XbOQORrGEuAauNJUde4bwmAZ1okQOmJU/s640/GOP.gif" width="640" /></a></div>
<div style="margin-bottom: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="margin-bottom: 0in;">
<a href="http://en.wikipedia.org/wiki/File:GOP.gif">http://en.wikipedia.org/wiki/File:GOP.gif</a></div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Following is another illustration of a GOP along with data requirements per frame.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7rm5JKasdgNBZyPOI5eKOreU-YPnigWx7dM5CvJIBJUO6BHP4zfxgQIy0t_XSaqLOKMeqiaQW3ZQdKATn7PPUu1MRR9smzPoKSQyb1fjo83fQBv3mvdPFPlpsA2K14QRF9E2rdUX5rB0/s1600/GOPlayout.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="558" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7rm5JKasdgNBZyPOI5eKOreU-YPnigWx7dM5CvJIBJUO6BHP4zfxgQIy0t_XSaqLOKMeqiaQW3ZQdKATn7PPUu1MRR9smzPoKSQyb1fjo83fQBv3mvdPFPlpsA2K14QRF9E2rdUX5rB0/s640/GOPlayout.jpg" width="640" /></a></div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
</div>
<div style="margin-bottom: 0in;">
This figure shows how different types of frames that can compose a group of pictures (GOP). A GOP can be characterized as the depth of compressed predicted frames (m) as compared to the total number of frames (n). This example shows that a GOP starts within an intra-frame (I-frame) and that intra-frames typically requires the largest number of bytes to represent the image (200 kB in this example). The depth m represents the number of frames that exist between the I-frames and P-frames.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
When you look at the data sizes, you realize that the transmission of data across a network as compared to the GOP diagram, you start to realize the breakdown of packets and packet streaming necessary to enable the GOP sequence to work in near real time. <span style="background-color: white;">For example, the first I frame must pass 200 kB sequentially which could turn out to be several hundred packets across the network. These packets are time and sequence sensitive. When things go wrong, you get video presentations that are frozen or pixelated.</span></div>
<br />
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
In Cisco TPS, latency is defined as the time it takes for audio and video input to go from input on one end to presentation on the other. It measures latency between two systems via time stamps in the RTP data as well as the RTCP return reports. It is only a measure of the network and does not take into consideration the codecs. The recommended latency target for Cisco TPs is 150 msec or less. However, this is not always possible. When latency is exceeds over 250 msec in a 10 second period, it generates an alarm, an on screen message, a syslog message and an SNMP Trap on the receiving system. The onscreen message is only displayed 15 seconds and isn’t displayed again unless the session is restarted or stopped and reinitiated.
</div>
<div style="margin-bottom: 0in;">
<span style="background-color: white;"><b><br /></b></span></div>
<div style="margin-bottom: 0in;">
<span style="background-color: white;"><b>Packet Loss</b></span></div>
<div style="margin-bottom: 0in;">
</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Packet Loss can occur across the network for a variety of reasons. It could be layer 1 errors, Ethernet duplex mismatches, overrunning the queue depth in routers, and even induced by jitter. In Cisco CTP systems, they recommend a loss less than .05 percent of traffic in each direction. When packet loss exceeds 1 percent averaged over a 10 second period, the system presents an on screen message, generates a syslog message and an SNMP Trap.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
When packet loss exceeds 10 percent average over a 10 second period, an additional on screen message is generated, syslog messages are logged and an SNMP Trap is generated.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
If the CTS system experiences packet loss greater than 10 percent averaged over a 60 second interval, it downgrades the quality of its outgoing video and puts up an onscreen message. Following is a table of key metrics.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<table border="1" bordercolor="#00000a" cellpadding="7" cellspacing="0" style="width: 565px;"><colgroup><col width="91"></col><col width="127"></col><col width="124"></col><col width="165"></col></colgroup><tbody>
<tr valign="TOP"><td bgcolor="#b6dde8" width="91"><span style="font-family: Cambria, serif;"><span style="font-size: small;"><i><b>Metric</b></i></span></span></td><td bgcolor="#b6dde8" width="127"><span style="font-family: Cambria, serif;"><span style="font-size: small;"><i><b>Target</b></i></span></span></td><td bgcolor="#b6dde8" width="124"><span style="font-family: Cambria, serif;"><span style="font-size: small;"><i><b>1</b></i></span></span><sup><span style="font-family: Cambria, serif;"><span style="font-size: small;"><i><b>st</b></i></span></span></sup><span style="font-family: Cambria, serif;"><span style="font-size: small;"><i><b> Threshold</b></i></span></span></td><td bgcolor="#b6dde8" width="165"><span style="font-family: Cambria, serif;"><span style="font-size: small;"><i><b>2</b></i></span></span><sup><span style="font-family: Cambria, serif;"><span style="font-size: small;"><i><b>nd</b></i></span></span></sup><span style="font-family: Cambria, serif;"><span style="font-size: small;"><i><b> Threshold</b></i></span></span></td></tr>
<tr valign="TOP"><td width="91">Latency</td><td width="127">150 ms</td><td width="124">250 ms (2 seconds for Satellite mode)</td><td width="165"><br /></td></tr>
<tr valign="TOP"><td width="91">Jitter</td><td width="127">10 ms Packet Jitter 50 ms Frame jitter</td><td width="124">125 ms of video frame jitter</td><td width="165">165 ms of video frame jitter</td></tr>
<tr valign="TOP"><td width="91">Packet Loss</td><td width="127">.05%</td><td width="124">1%</td><td width="165">10%</td></tr>
</tbody></table>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
If you are building KPIs around something like a Cisco CTS implementation, you need to look at areas where latency, jitter, and packet loss can occur. Sometimes, it is in your control and sometimes it is not. And there are a lot of different possibilities.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Some errors and performance conditions are persistent and some are intermittent and situationally specific. For example, a duplex mismatch on either end will result in a persistent packet loss during TPS calls. An overloaded WAN router in the path may be dropping packets during the call.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
While the CTS system gives your Operations personnel awareness that jitter, latency, or packet loss is occurring, it does not tell you where the problem is. And if it causes the call to reset, you may not be able to discern where the problem is either. Once the traffic is gone, the problem may disappear with it.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
<br /></div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://www.cisco.com/en/US/i/200001-300000/220001-230000/220001-221000/220194.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://www.cisco.com/en/US/i/200001-300000/220001-230000/220001-221000/220194.jpg" /></a></div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
When you look at the diagram, you see a Headquarters CTS system capable of connecting to both a Remote Campus CTS system and a Branch office equipped with a CTS system. Multiple redundant metwork connections are provided via Headquarters and the Remote Campus but a single Metro Ethernet connection connects the Branch office to the Headquarters and Remote campus facilities.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
</div>
<div style="margin-bottom: 0in;">
When you look at the potential instrumentation that could be applied, there are a lot of different points to sort out device by device. But diagnosis of something like this goes from end to end. The underlying performance and status data is really most useful when an engineer is drilling into the problem to look for root causes, capacity planning, or performing a post mortem on a specific problem.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Most Cisco TPS systems are high visibility as the users tend to be presented with the issues either by cue on screen or by performance during the call. It doesn’t take too many jerky or broken calls to render it a non-working technology in the minds of some.And these users tend to be high profile like senior management and business unit decision makers.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<b>So, What Do You Do?</b><br />
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
First of all, Operations personnel need to know proactively, when a CTS system is presenting errors and experiencing problems. On the elements you can monitor, setup and measure key performance metrics and thresholds, setup status change mechanisms, and threshold on mis-configured elements. Thresholds, traps, and events help you create situation awareness in your environment. This is a good start toward being able to recognize problems and conditions. Even creating awareness that a configuration change is made to any component in the CTS system enables Operations to be aware of elements in the service.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
In the monitoring and management of the infrastructure, you are probably receiving SNMP Traps and Syslog from the various components in the environment. This is a good start toward seeing failures and error conditions as they are sent as the events occur. However, you need performance data to be able to drill down into the data to discern and analyze conditions and situations that could affect streaming services.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
But when you think about it, you actually need to start with a concept of performance and availability of end to end, as a service. Having IO performance data on a specific router interface makes no seense until its put into a service perspective.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
<b>End to End Strategy</b></div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
First up, you need to look at an end to end monitoring strategy. And probably one of themost common steps to take is to employ something like Cisco IPSLA capabilities.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
</div>
<div style="margin-bottom: 0in;">
Cisco has a capability in many of its routers and switches that enables managing service levels using various measurements provided. There are a series of capabilities that can be utilized in your CTS Monitoring and management solution to make it more effective beyond the basic instrumentation. Cisco recommends that shadow devices be used as the IP SLA tests will circumvent routing and queuing mechanisms and can have an effect if done on production devices. </div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Not all devices support all tests. So, be sure to check with Cisco to ensure you can use a specific IOS version and platform for the given tests. Some service providers and enterprises use decommissioned components to reduce cost.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<b>Basic Connectivity</b><br />
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
One of the Cisco IP SLA most commonly used test is the ICMP Echo test. The ICMP Echo test sends a ping from the shadow router doing the test to an end IP. In looking over figure 2, we would need to setup an ICMP Echo test from the shadow router on one side to the CTS system on the other side. This is to establish a connectivity check in each direction. </div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
This will also provide a latency metric but at the IP level. ICMP is the control mechanism for the IP protocol. There may be added latency at higher level protocols depending upon the elements interacting on those protocols. For example, firewalls that maintain state at the transport layer (TCP) may introduce additional latency. Traffic shapers may do so as well.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
The collected data and status of the tests are in the CISCO-RTTMON-MIB.my MIB definition file. Pertinent data shows up in rttMonCtrlAdminEntry, rttMonCtrlOperEntry, rttMonStatsCaptureEntry, and rttMonStatsCollectEntry tables.</div>
<b><br /></b><br />
<b>Jitter</b><br />
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
We know that Jitter can have a profound effect on video streams as a GOP that arrives too late or out of sequence will be dropped by the receiving codec as it cannot go back in the video stream and redo the past GOP frame. We can also discern that at 30 frames a second, the GOP frame must be sent every 33 ms.</div>
<div style="margin-bottom: 0in;">
The jitter test is executed between a Source and Destination and subsequently, needs a Responder to operate. It is accomplished by sending packets separated by an interval. Both source to destination and destination to source is accommodated. </div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
The statistics available provide the types of information you are looking to threshold and extract. The Cisco IP SLA ICMP based Jitter test supports the following statistics:</div>
<ul>
<li><div style="margin-bottom: 0in;">
Jitter (Source to Destination and Destination to Source)</div>
</li>
<li><div style="margin-bottom: 0in;">
Latency (Source to Destination and Destination to Source</div>
</li>
<li><div style="margin-bottom: 0in;">
Round Trip Latency</div>
</li>
<li><div style="margin-bottom: 0in;">
Packet Loss</div>
</li>
<li><div style="margin-bottom: 0in;">
Successive Packet Loss</div>
</li>
<li><div style="margin-bottom: 0in;">
Out of Sequence Packets (Source to Destination and Destination to Source)</div>
</li>
<li><div style="margin-bottom: 0in;">
Late Packets</div>
</li>
</ul>
<div style="margin-bottom: 0in;">
A couple of factors need to be considered in setting up jitter with regards to video streams. These are:</div>
<ul>
<li><div style="margin-bottom: 0in;">
Jitter Interval</div>
</li>
<li><div style="margin-bottom: 0in;">
The number of packets</div>
</li>
<li><div style="margin-bottom: 0in;">
Test frequency</div>
</li>
<li><div style="margin-bottom: 0in;">
Dealing with Load Balancing and NAT</div>
</li>
</ul>
<div style="margin-bottom: 0in;">
The jitter interval needs to be set to the GOP frame interval of 33 ms.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
The number of packets needs to be set to something significant beyond single digits but below a value which would hammer the network. What you are looking for is just enough packets to understand you have an issue. I would probably recommend 20 at first even though the video packet numbers will be significantly higher. Not every network is created equal so you probably need to tune this. While the number of packets in a GOP frame can be highly variable, you just need to sample for the statistics that effect service.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Test frequency can have an effect on traffic. I would probably start with 5 minutes during active periods. But this too needs to be adjusted according to your environment.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
In the Jitter test setup, you can use either IP addresses and hostnames when you specify source and destination.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
The statistical data is collected in the rttMonJitterStatsEntry table.</div>
<b><br /></b><br />
<b>ICMP Path Echo</b><br />
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
When connectivity issues occur or changes in latency occur, one needs to be able to diagnose the path from both ends. The Cisco IP SLA ICMP Echo Path test is an ICMP based traceroute function. This test can help diagnose not only path connectivity issues but latency in the path, and asymmetric paths.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Traceroute uses the TTL field of IP to “walk” through a network. As a new hop is discovered, the TTL is incremented and attempted again. As each hop is discovered, an ICMP Echo test is performed to measure the response time.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Data will be presented in the rttMonStatsCaptureEntry table.</div>
<br />
<div style="margin-bottom: 0in;">
<br /></div>
<br />
<div style="margin-bottom: 0in;">
</div>
<b>Advanced Functions</b><br />
<b><br /></b><br />
<div style="margin-bottom: 0in;">
In looking over what we’ve discussed so far, we have reviewed in system diagnostic events, external events and conditions, and using IP SLA capabilities to test and evaluated in a shadow mode, between two or more telepresense systems. But when things happen, they do so in real time. Coupled with the facts of many areas that could affect degradation in services, in some instances, it may be necessary to provide enhanced levels of service.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Additionally, multiple problems can present confusing and anomalous event patterns and symptoms exacerbating the service restoral process. Of course, because you have high visibility, any confusion adds fuel to the fire.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<b>Media Delivery Index</b><br />
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Ineoquest has derived a service metric related to video streams services and provide a KPI for service delivery. The Media Delivery Index (MDI) can be used to monitor both the quality of a delivered video stream as well as to show system margin by providing an accurate measurement of jitter and delay. The MDI addresses a need to assure the health and quality of systems that are delivering ever higher numbers of streams simultaneously by providing a predictable, repeatable measurement rather than relying on subjective human observations. Use of the MDI further provides a network margin indication that warns system operators of impending operational issues with enough advance notice to allow corrective action before observed video is impaired.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
The MDI is also presented in RFC-4445 as jointly authored by Ineoquest and Cisco.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Ineoquest uses Probes and software to analyze streams in real time, to validate the actual video stream as it occurs. In fact, they can be used in a portspan, a network tap arrangement, or they can actually JOIN a conference! Why is this necessary?</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
The Singulus and Geminus probes look at each GOP and the packets that make up that frame. They capture and provide metrics on any data loss, jitter, and Program Clock Reference (PCR). They are also able to analyze the data stream and determine in real time, the quality, of the video. You can create awareness for increased jitter even before video is affected by analyzing the jitter margins. This takes the subjectiveness out of the measurement equation.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Additionally, you can record and play back video streams that are problematic. This is very important that in video encoding streams, the actual data patterns can be wildly different from one moment to the next depending upon the changes that occur in the GOP. For example, if the camera is displaying a white board, there a very few changes. However, if the camera is capturing 50 people in a room all moving, the GOP is going to be thick with updates to the I, P, and B frames.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
For instance, a Provider Network setup could have a cbQoS setup that’s perfectly acceptable for a Whiteboard camera or even moderate traffic. It may start buffering packets when times get tough. The bandwidth may be oversubscribed and reprioritized in areas where you have no visibility. This can introduce jitter and even packet loss only its specific to the conditions at hand. And how do you PROVE that back to your Service Provider?</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
You can go back and replay the video streams during a maintenance window to validate the infrastructure. And test, diagnose, and validate specific network components with your network folks and service providers alike to deliver a validated service as a known functional capability.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
And because you can replicate the captures, what a great way to test load and validate your end to end during maintenance windows!</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
The IQ Video Management System (iVMS) controls that probes, assembles the data into statistics and provides the technical diagnostic front end to access the detailed information elements available. Additionally, when problems and thresholds occur, it forwards SNMP Traps, syslogs, and potentially email regarding the issues at hand.</div>
<br />
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
<b>Analyzing the Data</b></div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
First and foremost, you have traps and syslog that tell you of impending conditions that are occuring in near real time. For example, due to the transmission quality of the stream, the end presentation may downgrade from 1080i to 720i to save on bandwidth.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
The IPSLA tests establish end to end connectivity and a portion of the path. In Traceroute, normally you only see one interface per device as traceroute walks the path. If only gives you one interface for the device and not the in and out interfaces. This is true because the Time to Live parameter wont be exceeded again until the next hop.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Keep in mind though that elements that can affect a video stream may not be in the direct path as a function. For example, if a router in the path has a low memory condition caused by the buffering from a fast network to a low speed WAN Link, this may affect the router's ability to correctly handle streaming data under load.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
What if path changes affect the order of packets? What if timing slips causes jitter on the circuit? Even BGP micro flaps may cause jitter and out of sequence frames. all in all, you have your work cut out for you.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
<b>Summary</b></div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Prepare yourself to approach the monitoring and management of video streams from a holistic point of view. You have to look at it first from the perspective of end to end to end. Also be prepared to be able to fill in the blanks with other configuration and performance data.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Because of the visibility of Video teleconferencing systems, be prepared to validate your environment on an ongoing basis. Things change constantly. Changes even only remotely related or out of your span of control, can affect the systems ability to perform well. Validate and test your QoS and performance from a holistic view on a recurring basis. A little recurring discipline will save you countless hours of heartbreak later.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Because of the situational aspects of video telepresense, be prepared to enable diagnosis in near real time. Specialized tools like Ineoquest can really empower you to support and understand whats going on and work through analysis,design changes, maintenance actions, and even vendor support to ensure you are on the road to an effective collaboration service.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
Sometimes, it pays to enlist the aid of an SME that is vendor independent. After all, in doing this Blog post, I credit a significant amount of education and technical validation on the GOP, RTP, RTVP, and other elements of streaming to Stephen Bavington of Bavington Consulting. <a href="http://bavingtonconsulting.com/">http://bavingtonconsulting.com/</a> </div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
<br /></div>
</div>Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-5594432343400645942012-07-11T10:26:00.000-05:002012-07-11T10:26:53.473-05:00Management and Cloud MigrationSo, you've been tasked with analyzing the systems you have and migrating them to the Cloud. Sounds like a bit of a challenge.
In part, you know there are some systems that will take longer to get to the Cloud than others. After all, you have systems like databases, ERP applications, and even Enterprise, Fault, and Performance systems that are critical to your operation. In many cases, email and DNS are very critical to your IT service. A lot of decisions, some harder than others.<br />
<br />
<b>Step Back</b><br />
<br />
All too often you find some Manager head over heels trying to push everything to the Cloud right up front. They've been to a couple of seminars and lurked on a few webcasts. It is so easy to just make a decision and hope for the best thinking it must be better in the cloud because everyone else is doing it.<br />
<br />
Even if you deploy your own virtualized environment, call it your private cloud, and start migrating things over wantonly, you are headed for more pitfalls than you think.
From a Management perspective, there are differences in legacy management applications and techniques and more "service oriented" approaches. Both have their form and function but migrating to cloud without considering the ramifications of management coverage could leave you exposed and vulnerable. (Translated as CLMs - Career Limiting Moves!)<br />
<br />
<b>What does Cloud Give you?</b><br />
<br />
<b>Mobility.</b><br />
<br />
First and foremost, it gives you the ability to move. Over the course of years, your process to put in place new computing assets has been increased and made more complex as it evolved. First, its the engineering process. Then the acquisition process. Then Development and prototyping. then migration into production. Each and every process along the way has put inplace more and more controls, check points, decision validations, and scheduling all to get something into production and working for a customer.<br />
<br />
If the IT department can't respond in short intervals, then why not hire a consultant to stand up a Web infrastructure in the cloud, upload our content, and take it down a month later when the need is diminished? In fact, you see alot of this in that the business needs Web assets for specific time periods and within a short time period and IT cannot even deliver a design in the time that the business unit needs the capability. A lot of business units will circumvent the IT department to make something happen. Only problem - When it breaks,who do they call?<br />
<br />
Being able to streamline your processes that enable new services to be delivered quickly, is a key benefit of cloud. You reserve your longer, legacy processes to developing the infrastructure and lining up the external cloud service contracts.<br />
<br />
<b>Standardization</b><br />
<br />
Because the unit of deployment can be whatever you decide, you must work to develop standard packages that can be pre-packaged, tested, and validated ahead of time. This saves countless hours in engineering as the engineering is done once and replicated. Additionally, patch management becomes more effective as you test against standard packages.<br />
<br />
Standardization also enables you as an IT Service provider, to provide a comprehensive catalog that business owners can use to satisfy their needs. Along with this, you can begin to standardize support costs more effectively.
Once you standardize the platforms and applications, you can begin to really standardize your processes.<br />
<br />
Think about your processes around monitoring and management and around support, care and feeding. Can you really start down the road of true ITIL Incident Management where corrective actions are defined?<br />
<br />
<b>Kill the Deadwood!</b><br />
<br />
Do you have elements / Silos in your environment that just get in the way of progress? Come on now, everyone does. Sometimes products / integrations just grow obsolete.<br />
<br />
Sometimes a product becomes in grained but the ability to support it or replace it is gone. At that point, people are afraid to touch it. It has become a boil on the rear of IT at that point. Nasty. Noone will touch it. And it festers on.<br />
<br />
Sometimes this happens because a person or group becomes the champion of the element and when you need something that the legacy product doesn't do, it means work. Theres an ongoing fallacy that in house developed products are free. Nothing is free. Someone has to continue development. Someone has to support it. Someone has to document it. Someone has to train others on it.<br />
<br />
Sometimes,an organization is just adverse to change. This is much more prevalent than one might think. There are folks that just don't do well with shifting sands. They prefer the status quo as it is more comfortable to avoid change.
<br />
<br />
Changing to Cloud and Virtualization changes the way you monitor and manage things. Instead of managing by IP address or a node, you must start with the service first. In contrast, a vMotion event can move your node to a completely different location, hardware, and even disk. And it happens very quickly.<br />
<br />
The legacy monitoring apps you've been used to may not be able to function effectively going forward. You must be willing to analyze and be real about the legacy products or you're going to get stuck with something that becomes a ball and chain.
Even if the product you have is free, a part of an ELA that doesn't cost anything, or you are living on maintenance only, if the product does not fit, you are wasting TIME AND MONEY.<br />
<br />
<b>Analyze and Define Your Needs</b><br />
<br />
Requirements is where you start and where you validate. If the solutions you put in place do not address the requirements, you need to analyze the risk of non-coverage and work through solutions. Wares vendors love to see valid requirements. They get an opportunity to address the things they do on a point by point basis.<br />
<br />
Because you are dealing with moving technology, keep the requirements open such that if a Wares vendor has additional capability that your requirements do not address, do not throw away these capabilities in your evaluations. Be extremely wary of vendors who want to minimize your requirements. It is often a sign of a lack of capability and a lack of the ability to effectively compete in their functional area.<br />
<br />
Be open to new things. If you are deploying VDI capabilities, look at VDI Management solutions. Some of these technologies are new enough that deployment without specialized management capabilities can leave to naked using legacy management technology. Consider this - SNMP support is waning in VMWare ESXi and they prefer you access their capabilities via their Web Services API.<br />
<br />
<b>Organize your Data Model</b><br />
<br />
Going forward, you need to look at your sources of truth - albeit configuration, performance, situational, and workflow data elements, need to be aligned and understood as a data model. What you do not want to end up with is multiple sources of the same data modeled in different ways, and not correlated. This will become painfully aware when you start organizing a service catalog.
How confusing would it be in your catalog to have an entry as Server - Redhat + MySQL
+ PHP + Apache and an adjacent entry that is LAMP + MySQL. Think about the differences in CPU Cores.<br />
<br />
<b>Validate your Security Posture</b><br />
<br />
Work diligently to identify and understand the risks of your systems, applications, and data as things migrate to a cloud or virtualized model. Consider where your data is, what its importance is, and what accesses you have to it. Moving to the Cloud or Virtualization may mean your data may move to a SAN.<br />
<br />
Network access protection changes a bit in Cloud. One may be limited in what can be done to protect access. And with vMotion this becomes even more complex.<br />
<br />
<b>Integration</b><br />
<br />
Use the move to Cloud and Virtualization to stress integration. You have requirements that may be addressed by multiple solutions. However, one must work to make things seamless. Because of the speed of deployment and deactivation, you need to work through how information and integration should work across products and data elements.<br />
<br />
When a service is provisioned, what happens next? Does it get put in your Service Desk solution? Is monitoring activated? Is billing activated? Customer notified? Work through defining and addressing the workflow and integration up front. Even if you setup a basic process, you can work to make it more efficient over time.<br />
<br />
<b>Cleanup your Applications</b><br />
<br />
Many applications you have may run inconsistently in constrained environments. What happens when you introduce latency into your application? What about running on constrained hardware? (1 CPU core a 1GB memory when the Java JVM takes 1.1GB of memory itself).
Some applications are terrible communicators across the network but network technology hides it. For example,open a Word document on a network share and watch it with a protocol analyzer. Now hit the page down key in your document and watch what happens. It downloads the entire document again and fseeks to the next page!<br />
<br />
Cloud and virtualized environments introduce new challenges. Combinations of applications may affect each other when presented on the same blade. Over subscription can wreak havoc if you put two dominant applications together.<br />
<br />
Invest in some APM applications, a protocol analyzer and some data gathering and reporting tools. I like to use both Class Loader based APM systems as well as network traffic capture based APMs systems as you get to analyze applications from multiple aspects.
<br />
<br />
<b>Summary</b><br />
<br />
Moving to the Cloud and toward virtualization does not have to be scary. But neither should you approach it blindly. Identify and understand your requirements and work through ensuring that whatever you implement, you understand what you are getting, what the risks are related to security, data, and availability, and what its going to take to implement and support.<br />
<br />
Moving to the Cloud is also an opportunity to clean up your legacy systems, applications, processes, and organization. Get rid of the functions that don't work. Work to eliminate all of the red tape, OSI Layer 8 infrastructures, and inefficiencies that have been dragging down your IT Departments ability to provide good service. If you migrate broken functions to the cloud, these will continue to break things, only much faster.Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com1tag:blogger.com,1999:blog-5590049888603730046.post-45847202259544750612012-07-01T11:23:00.002-05:002012-07-01T12:00:44.631-05:00ENSM Products are Commodities?On your quest to put in network and systems management capabilities, you have to figure in several explicit and implicit factors related to your end goals. What I mean is that while it's easy to go to your Framework Vendor of choice, break out the Bill of Materials spreadsheet, and sit down with the Sales person and go through the elements you would need for your environment, it may be filled with hidden challenges. And some challenges may be harder to overcome than others once you have signed the check.<br />
<br />
Don't forget, these products don't magically install and run themselves. They take care and feeding. Some more than others. And the more complex it is, the more complex it is to figure out when something goes awry.<br />
<br />
Sounds so easy! After all, all of these products are commodities. And buying from a single vendor gives you a single point of support... and blame. In effect, a single "throat to choke". NOTHING could be further from the truth!<br />
<br />
Most of the big vendor's product frameworks are aggregations and conglomerations of products that have been acquired, some overlapping, into what looks like a somewhat unified solution. In many cases, it is only after you buy the product framework that you discover stuff like there are different portals with different products and these portals don't effectively integrate together. Or you may find the north bound interface of one product is a kludge to somewhat loosely fit the two products together. Or two products use competing Java versions.<br />
<br />
Some vendors product suites have become more and more complex as new releases are GAed. In many cases, these new levels of complexity have a profound impact on your ability to install, administer, or diagnose issues as they arise.<br />
<br />
First up - Where are your requirements? Do you know the numbers and types of elements in your environment? What about the applications? How do these apply to Service Level Agreements? Do you have varying levels of maintenance and support for the components in your environment?<br />
<br />
Do you know who the users will be? Have you defined your support model? Which groups need access to what elements of information? Do you have or have you prepared a proposed workflow of how users, managers, and even customers are going to interact with the new capabilities?<br />
<br />
Who is going to take care of the management systems and applications? Have you aligned your organization to be successful in deployment? Do you have the skill sets? Do you have adequate skills coverage?<br />
<br />
Have you defined the event flow? What about performance reports needs and distributions? And ad hoc reporting needs? Have you defined any baseline thresholds?<br />
<br />
Do you have SNMP access? What about ICMP? SSH? Have you considered the implications of management traffic across your security zones?<br />
<br />
<h2>Product Choices</h2>
<br />
While there are a plethora of choices available to you, many do not want to go through the hassle of doing due diligence. But be forewarned, failure to do due diligence can wreak mayhem in you environment. I know, the big guns say that "our product works in your competitors" but does it really? You don't know? As is your competition that undifferentiated from you? (May not be a good thing!)<br />
<br />
When you go through product selection, you need to realize the support needed to administer the new management applications. Do you need specialists just to install it? What about training? Are you going to need other resources like Business Intelligence Analysts, Web Developers, Database Administrators, Script Developers, or even additional Analysts or Engineers.<br />
<br />
Here are some signs you may experience:<br />
<br />
<a href="http://farm1.staticflickr.com/81/264120060_2182c32691_s.jpg" imageanchor="1" style="background-color: white; clear: left; display: inline !important; margin-bottom: 1em; margin-right: 1em; text-align: center;"><img border="0" src="http://farm1.staticflickr.com/81/264120060_2182c32691_s.jpg" /></a> If the product takes longer than a couple of days to install and integrate, here's your sign.<br />
<br />
<a href="http://farm2.staticflickr.com/1390/742517244_1ea48919cd_s.jpg" imageanchor="1" style="background-color: white; clear: left; display: inline !important; margin-bottom: 1em; margin-right: 1em; text-align: center;"><img border="0" src="http://farm2.staticflickr.com/1390/742517244_1ea48919cd_s.jpg" /></a>If two or more products in your big vendor product suite need a significant amount of customization to work together, here's your sign.<br />
<br />
<a href="http://www.newbbc.accura.net/DangerWillRobinsonSign.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="147" src="http://www.newbbc.accura.net/DangerWillRobinsonSign.jpg" width="200" /></a> If the installation document for the product deviates from the actual installation, here's your sign.<br />
<br />
<span style="background-color: white;"> </span><a href="https://encrypted-tbn3.google.com/images?q=tbn:ANd9GcRBvKZ-0kQJ1orvF6JTo39XmksiBLQSsZr0aYxRP5DPZtLaWcNK" imageanchor="1" style="background-color: white; clear: left; display: inline !important; margin-bottom: 1em; margin-right: 1em; text-align: center;"><img border="0" height="200" src="https://encrypted-tbn3.google.com/images?q=tbn:ANd9GcRBvKZ-0kQJ1orvF6JTo39XmksiBLQSsZr0aYxRP5DPZtLaWcNK" width="134" /></a> If you find out you actually have to install additional product as discovered during the installation, here's your sign.<br />
<br />
<span style="background-color: white;"> </span><a href="http://farm1.staticflickr.com/87/264116794_3d76e5be15_s.jpg" imageanchor="1" style="background-color: white; clear: left; display: inline !important; margin-bottom: 1em; margin-right: 1em; text-align: center;"><img border="0" src="http://farm1.staticflickr.com/87/264116794_3d76e5be15_s.jpg" /></a> If you end up realizing that the recommended hardware specs are either overkill or under-speced, here's your sign.<br />
<br />
<a href="http://farm1.staticflickr.com/100/264120053_9278ad221e_s.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://farm1.staticflickr.com/100/264120053_9278ad221e_s.jpg" /></a> If you end up having to deal with libraries and utilities that are not included or resolved with the product installation, here's your sign.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="http://www.blogforarizona.com/.a/6a00d8341bf80c53ef015434690b98970c-120wi" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="http://www.blogforarizona.com/.a/6a00d8341bf80c53ef015434690b98970c-120wi" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Missed it by THAT much!</td></tr>
</tbody></table> If you find yourself opening up support tickets in the middle of the installation, here's your sign.<br />
<br />
<a href="http://farm1.staticflickr.com/89/264116790_4fb7f138e3_s.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://farm1.staticflickr.com/89/264116790_4fb7f138e3_s.jpg" /></a> If you find that the product breaks your security model AFTER you do the installation, here's your sign.<br />
<br />
<span style="background-color: white;"> </span><a href="http://farm1.staticflickr.com/110/264112998_ba1684ccc0_s.jpg" imageanchor="1" style="background-color: white; clear: left; display: inline !important; margin-bottom: 1em; margin-right: 1em; text-align: center;"><img border="0" src="http://farm1.staticflickr.com/110/264112998_ba1684ccc0_s.jpg" /></a> If it takes Vendor specific Engineering to install the product, here's your sign.<br />
<br />
<span style="background-color: white;"> </span><a href="http://farm1.staticflickr.com/109/264113001_f1ed3fb11e_s.jpg" imageanchor="1" style="background-color: white; clear: left; display: inline !important; margin-bottom: 1em; margin-right: 1em; text-align: center;"><img border="0" src="http://farm1.staticflickr.com/109/264113001_f1ed3fb11e_s.jpg" /></a> If you cannot see value in the first day after the installation of a product, here's your sign.<br />
<br />
<span style="background-color: white;"> </span><a href="http://farm1.staticflickr.com/112/264116796_35e8541107_s.jpg" imageanchor="1" style="background-color: white; clear: left; display: inline !important; margin-bottom: 1em; margin-right: 1em; text-align: center;"><img border="0" src="http://farm1.staticflickr.com/112/264116796_35e8541107_s.jpg" /></a> If you find that you need to restructure and build out your support team AFTER the installation, here's your sign.<br />
<br />
<h2>Systems Management</h2>
<br />
Systems Management brings whole new challenges to your environment. Some of the things you need to evaluate up front are:<br />
<br />
<br />
<ul>
<li>Agent deployment - Level of Difficulty - OS Coverage - consistent data across agents. Manual, Automatic, or distribute able</li>
<li>Agent-less - Browser specific? Adequate coverage? Full transactions? Handles redirection?</li>
<li>Agent run time - Resource utilization - memory footprint - stability - Security.</li>
<li>Data collection - Pull or push model? Resiliency? Effect on run time resources?</li>
<li>External Restrictions - Java versions? Perl versions? Python versions?</li>
<li>Adequate application coverage?</li>
<li>Thresholds - Level of difficulty? Binary only or degrees of utilization/capacity/performance? Stateful? Dynamic thresholds? Northbound traps already defined or do you have to do your own?</li>
</ul>
<br />
<h2>Summary</h2>
<br />
Enterprise Management does not have to be that difficult. There are products out there that work very well for what they do and are easy to deploy and maintain. For example, go do an OpenNMS installation. Even though OpenNMS runs on just about any platform (a testament to their developer community and product maturity), you go to their wiki page <a href="http://www.opennms.org/documentation/installguide.html" style="background-color: white;">http://www.opennms.org/documentation/installguide.html</a> , pick out your platform of choice, and follow the procedure. Most of the time, you are looking at maybe an hour. In an hour, you're starting discovery and picking up inventory to monitor and manage.<br />
<br />
Solarwinds isn't too bad either. Nice, clean install on Windows.<br />
<br />
Splunk is awesome and up in running in no time. <a href="http://www.splunk.com/">http://www.splunk.com/</a><br />
<br />
Hyperic HQ wasn't a bad installation either. Pretty simple. However, it is time sensitive on the agents. Kind of thick (I think its the Struts), Java wise. <a href="http://www.hyperic.com/">http://www.hyperic.com/</a><br />
<br />
eGInnovations is cake. One agent everywhere for OS and applications. Handles VMWare, Xen and others. And the UI is straight forward. A Ton of value across both system and application monitoring and performance. <a href="http://www.eginnovations.com/">http://www.eginnovations.com/</a><br />
<br />
Appliance based solutions take a bit more time in the planning phase up front but take the sting out of installation. Some of these include:<br />
<br />
<a href="http://www.sevone.com/">http://www.sevone.com/</a> (SevOne does offer a software download for evaluation)<br />
<a href="http://www.sciencelogic.com/">http://www.sciencelogic.com/</a>
<br />
<a href="http://www.loglogic.com/">http://www.loglogic.com/</a> (They also offer a virtual appliance download)<br />
<br />
One solution I dig is Tavve ZoneRanger for solving those access issues like UDP/SNMP across firewalls, SSH access across a firewall, etc., without having to run through proxies upon proxies and still maintain consistent auditing and logging. It deploys as an appliance of virtual appliance. <a href="http://www.tavve.com/">http://www.tavve.com/</a><br />
<br />
Another aspect you may consider include hosted applications. ServiceNow is easy to deploy because it is a hosted solution. <a href="http://www.servicenow.com/">http://www.servicenow.com/</a><br />
<br />Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-66722321344097100842012-06-25T21:51:00.001-05:002012-06-25T23:43:40.147-05:00Interview in Seattle - Perf Data in large environmentsI recently interviewed at a rather large company in the NorthWest and during the course of the interview, I discovered a few things.<br />
<br />
If you're limited to SQL Server for performance management data, there are a couple of things to consider:<br />
<ul>
<li><span style="background-color: white;">You're stuck with about a 25000-30000 insert per second maximum input rate.</span></li>
<li><span style="background-color: white;">While you are at this maximum, you're not getting any data out.</span></li>
<li><span style="background-color: white;">When you get to 100 million or so records, indexing and inserting slows down. Reporting slows down even more.</span></li>
</ul>
They had a 3 tier data model that kept 7 days of 2 or 5 minute data on the lowest tier database.<br />
This data was rolled up to a 1 hour average and stored at the second data tier for a period of 90 days.<br />
This second tier data was again rolled up to a 12 hour average and sent to the highest tier for a data retention of 1 year.<br />
<br />
Some of the head scratchers I came away with were:<br />
<br />
<br />
<ul>
<li><span style="background-color: white;">How do you do a graph that spans time periods?</span></li>
<li><span style="background-color: white;">If you're doing post mortems, how effective is any of the data in the second or third tier?</span></li>
<li><span style="background-color: white;">How useful is CPU utilization that has been averaged to 1 hour or 12 hours?</span></li>
<li><span style="background-color: white;">How do you effectively trend multiple elements across tiers without a significant amount of development just so you can provide a report?</span></li>
</ul>
<br />
<br />
So, what are you gonna do?<br />
<br />
What kind of data is it you're dealing with?<br />
<br />
When you look at performance related data, a SIGNIFICANT part of it is simply time series data. It has a unique identifier relating it to the managed object, attribute, and instance you are collecting data against and it has a timestamp of when it was collected. And then theres the value.<br />
<br />
So, in a relational database,you would probably set this up as:<br />
<br />
CREATE TABLE TSMETRICS {<br />
Metrickey varchar(64),<br />
timestamp datetime,<br />
value integer<br />
};<br />
<br />
You would probably want to create an index on Metrickey so that you can more efficiently grab metric values from a given Metrickey.<br />
<br />
When you consider you're collecting 10 metrics every 5 minutes for 10,000 nodes, you start to realize thatthe number of records starts to add up quickly. 288 specific records per metric for every day.<br />
So 10 metrics every 5 minutes turns into 2880 records per node times 10000 nodes equals 28,800,000 records per day. At the end of 4 days, you're looking at crossing that 100 million record boundary.<br />
<br />
<b>Scaling</b><br />
<br />
What if we changed TSMETRICS structure? We could change it to:<br />
<br />
CREATE TABLE TSMETRICS {<br />
Metrickey varchar(64),<br />
starttime datetime,<br />
endtime datetime,<br />
slot1 integer,<br />
...<br />
slot288 integer<br />
}<br />
<br />
This effectively flattens the table and reduces the duplicate string of Metrickey which would save a significant amount of repetitive record space. In effect,this is how Round Robin Data stores store metric data. But consider this, you either have to index each column or you have to process a row at a time to do so efficiently.<br />
<br />
This gets us into the 1000 days realm! 10000 nodes * 10 metrics each * 1000 days = 100000000 records.<br />
<br />
But the problem expands because the inserts and extracts become much more complex. And you have to go back to work on indexing.<br />
<br />
Sharding<br />
<br />
Some folks resort to sharding. What they will do is to take specific time ranges and move them onto their own table space. Sharding ends up being a HUGE nightmare. While it enables the DBA to control the Table spaces and number of records, getting data back out becomes another cryptic exercise is first finding the data, connecting to the appropriate database,and running your query. So, the first query is to find the data you want. Subsequent queries are used to go get that data. You're probably going to need to create a scratch database to use to aggregate the data from the multiple shards so that reporting tools can be more efficient.<br />
<br />
Another technique folks employ is to use a data hierarchy. Within the hierarchy, you keep high resolution data, say 7 days, in the lowest level. Roll the data up from a 5 minute interval to a 1 hour interval into a second data tier. Then roll up the 1 hour data to 12 hour data in a third data tier. I actually know of a large service provider that does exactly this sort of thing.<br />
<br />
Imagine mining through the data and you need to look at the CPU of 5 systems over the course of a year. How has the CPU load grown or declined over time? Now overlay the significant configuration changes and patch applications over that time period. Now, overlay the availability of the 5 systems over that time.<br />
<br />
All of a sudden, what looks like a simple reporting exercise becomes a major production issue. You have to get data from 3 separate databases, munge it together, handle the graphing of elements where the X axis is not linear, and it becomes mission impossible.<br />
<br />
<b>Suggestions</b><br />
<br />
If you're looking at moderate to heavy data spaces, consider the options you have. Do not automatically assume that everything fits in an RDBMS space effectively.<br />
<br />
The advantages of a Round Robin Data store are:<br />
<br />
<br />
<ul>
<li><span style="background-color: white;">Space is preallocated</span></li>
<li><span style="background-color: white;">It is already aligned to time series</span></li>
<li><span style="background-color: white;">Relatively compact</span></li>
<li><span style="background-color: white;">Handles missing data</span></li>
</ul>
<br />
<br />
Other considerations are that when you read an RRD type store,you store a copy in memory and read from that. Your data inserts do not get blocked.<br />
<br />
There are certain disadvantages to RRD stores as well to include:<br />
<br />
<br />
<ul>
<li><span style="background-color: white;">Concentration of RRD stores on given controllers can drive disk IO rather high.</span></li>
<li><span style="background-color: white;">Once you start down the road of RRD store distribution, how do you keep up with the when and where of your data in a transparent manner?</span></li>
<li><span style="background-color: white;">RRD doesn't fit the SQL paradigm.</span></li>
</ul>
<div>
If you need SQL, why not take a look at columnar databases?</div>
<div>
<br /></div>
<div>
Take a look see at Vertica or Calpont InfiniDB.</div>
<div>
<br /></div>
<div>
When you think about it, most time series data is ultra-simple. Yet when you do a graph or report, you are always comparing one time series element to another. A Columnar data LIVES here because the DB engine aligns the data by column and not row.</div>
<div>
<br /></div>
<div>
Another thought here is massive parallelism. If you can increase your IO and processing power, you can overcome large data challenges. Go check out Greenplum. While it is based on PostgresQL, it sets up as appliances based on VM instances. So, you start out with a few servers and as your data grows, you install another "appliance" and go. As you install more components, you add to the overall performance potential of the overall data warehouse.</div>
<div>
<br /></div>
<div>
If you can run without SQL, take a look atthe big Data and noSQL options like Casandra or Hadoop / MapReduce.</div>
<div>
<br /></div>
<div>
Links for you:</div>
<div>
<br /></div>
<div>
<a href="http://www.vertica.com/">http://www.vertica.com/</a>
</div>
<div>
<a href="http://infinidb.org/">http://infinidb.org/</a>
</div>
<div>
<a href="http://oss.oetiker.ch/rrdtool/">http://oss.oetiker.ch/rrdtool/</a>
</div>
<div>
<a href="http://www.jrobin.org/index.php/Main_Page">http://www.jrobin.org/index.php/Main_Page</a>
</div>
<div>
<a href="http://www.greenplum.com/">http://www.greenplum.com/</a>
</div>
<br />
<a href="http://hadoop.apache.org/">http://hadoop.apache.org/</a>
<br />
<a href="http://hadoop.apache.org/mapreduce/">http://hadoop.apache.org/mapreduce/</a>
<br />
<br />
An interesting experiment:<br />
<br />
Run RRD or JRobin stores under Hadoop and HDFS. Use MapReduce to index the RRD Stores.<br />
<br />
I wonder how that would scale compared to straight RRD stores, against an RDBMS, or a Columnar Database.<br />
<br />
<br />
<br />
<br />
<br />Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-77051293421042080832012-04-09T16:20:00.000-05:002012-05-08T09:04:44.317-05:00Product quality DilemmaAll too often, we have products that we have bought, put in production, and attempted to work through the shortcomings and obfuscated abnormalities prevalent in so many products. (I call this product "ISMs" and I use the term to describe specific product behaviors or personalities.) As part of this long living life cycle, changes, additions, deprecations, and behaviors change over time. Whether its fixing bugs or rewriting functions as upgrades or enhancements, things happen.<br />
<br />
All too often, developers tend to think of their creation in a way that may be significantly different than the deployed environments they go into. Its easy to get stuck in microcosms and walled off development environments. Sometimes you miss the urgency of need, the importance of the functionality, or the sense of mandate around the business.<br />
<br />
With performance management products, its all too easy just to gather everything and produce reports ad nauseum. With an overwhelming level of output, its easy to get caught up in the flash, glitz, and glamour of fancy graphs, pie charts, bar charts... Even Ishigawa diagrams!<br />
<br />
All this is a distraction of what the end user really really NEEDS. I'll give a shot at outlining some basic requirements pertinent to all performance management products.<br />
<br />
1. Don't keep trying to collect on broken access mechanisms.<br />
<br />
Many performance applications continue to collector attempt to collect, even when they haven't had any valid data in several hours or days. Its crazy as all of the errors just get in the way of valid data. And some applications will continue to generate reports even though no data has been collected! Why?<br />
<br />
SNMP Authentication failures are a HUGE clue your app is wasting resources or something simple. Listening for ICMP Source Quenches will tell you if you're hammering end devices.<br />
<br />
2. Migrate away from mass produced reports in favor of providing information.<br />
<br />
If no one is looking at the reports, you are wasting cycles, hardware,and personnel time on results that are meaningless.<br />
<br />
3. If you can't create new reports without code, its too complicated.<br />
<br />
All too often, products want to put glue code or even programming environments / IDEs in front of your reporting. Isn't it a stretch to assume that a developer will be the reporting person? Most of the time its someone more business process oriented.<br />
<br />
4. Data and indexes should be documented and manageable. If you have to BYODBA (Bring Your Own DBA), the wares vendor hasn't done their home work.<br />
<br />
How many times have we loaded up a big performance management application only to find out you have to do a significant amount of work tuning the data and the database parameters just to get the app to generate reports on time?<br />
<br />
And you end up having to dig through the logs to figure out what works and what doesn't.<br />
<br />
If you know what goes into the database, why do you not put in indexes,checks and balances, and even recommended functions when expansion occurs.<br />
<br />
In some instances, databases used by performance management applications are geared toward the polling and collection versus the reporting of information. In many cases, one needs to build data derivatives of multiple elements in order to facilitate information presentation. For example, a simple dynamic thresholding mechanism is to take a sample of a series of values and perform an average, root mean, and standard deviation derivative.<br />
<br />
If a reporting person has to do more than one join to get to their data elements, your data needs to be better organized, normalized, and accessible via either derivative tables or a view. Complex data access mechanisms tend to alienate BI and performance / Capacity Engineers. They would rather work the data than work your system.<br />
<br />
5. If the algorithm is too complex to explain without a PhD, it is not usable nor trustable.<br />
<br />
There are a couple of applications that use patented algorithms to extrapolate bandwidth, capacity, or effective usage. If you haven't simplified the explanation of how it works, you're going to alienate a large portion of your operations base.<br />
<br />
6. If an algorithm or method is held as SECRET, it works just until something breaks or is suspect. Then your problem is a SECRET too!<br />
<br />
Secrets are BAD. Cisco publishes all of its bugs online specifically because it eliminates the perception that they are keeping something from the customer.<br />
<br />
If one remembers Concord's eHealth Health Index... In the earlier days, it was SECRET SQUIRREL SAUCE. Many an Engineer got a bad review or lost their job because of the arrogance of not publishing the elements that made up the Health Index.<br />
<br />
7. Be prepared to handle BI types of access. Bulk transfers, ODBC and Excel/Access replication, ETL tools access, etc.<br />
<br />
If Engineers are REALLY using your data, they want to use it in their own applications, their own analysis work, and their own business activities. The more useful your data is, the more embedded and valuable your application is. Provide ways of providing shared tables,timed transfers, transformations, and data dumps.<br />
<br />
8. Reports are not just a graph on a splash page or a table of data. Reports to Operations personnel means they put text and formatting around the graphs, charts, tables, and data to relate the operational aspects of the environment in with the illustrations.<br />
<br />
9. In many cases, you need to transfer data in a transformed state from one system that reports to another. Without ETL tools, your reporting solution kind of misses the mark.<br />
<br />
Think about this... You have configuration data and you need this data in a multitude of applications. Netcool. Your CMDB. Your Operational Data Store. Your discovery tools. Your ticketing system. Your performance management system. And it seems that every one of this data elements may be text, XML, databases of various forms and flavors, even HTML. How do you get transformed from one place to another?<br />
<br />
10. If you cannot store, archive, and activate polling, collection, threshold, and reporting configurations accurately, you will drive away customization.<br />
<br />
As soon as a data source becomes difficult to work with, it gets in the way of progress. In essence, what happens in that when a data source becomes difficult to access, it quits being used beyond its own internal function. When this occurs, you start seeing separation and duplication of data.<br />
<br />
The definitions of the data can also morph over time. When this occurs and the data is shared, you can correct it pretty quickly. When data is isolated, many times the problem just continues until its a major ordeal to correct. Reconciliation when there are a significant number of discrepancies can be rough.<br />
<br />
Last but not least - If you develop an application and you move the configuration from test/QA to production and it does not stay EXACTLY the same, YOUR APPLICATION is <b><span style="color: red; font-size: large;">JUNK</span></b>. Its dangerous, haphazard, incredibly short sided, and should be avoided at all costs. Recently, I had a dear friend develop, test, and validate a performance management application upgrade. After a month in test and QA running many different use case validations, it was put into production. The application overloaded the paired down configurations to defaults upon placement into production, polled EVERYTHING and it caused major outages and major consternation for the business. In fact, heads rolled. The business lost customers. There were people that were terminated. And a lot of man power was expended trying to fix the issues. <br />
<br />
In no uncertain terms, I will never let my friends and customers be caught by this product.Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-53894074516216120942012-03-28T22:34:00.000-05:002012-03-28T22:34:07.289-05:00Time Planning - Simple is Good!I have been through the gamut of day planners and professional time planners. Like many software implementations - they tend to be overly complex. In fact, too complex to try to use it when you're busy and too inflexible to adapt to your environment.<br />
<br />
(I learned a bit from David, Matt, and Ben at OpenNMS as they are masters at refactoring and simplifying software.)<br />
<br />
There are certain inevitabilities that arise in every Techies world:<br />
<br />
<br />
<ul><li>Always way more work than can be humanly doable given the current resources.</li>
<li>No additional resources are coming to your rescue.</li>
<li>Always interruptions and priority situations.</li>
<li>Everything is an emergency.</li>
<li>You tend to deal with micro-management on occasion</li>
<li>Priorities change ad nauseum.</li>
</ul><br />
<br />
The goals - To track what you need to do as it gets done. Adapt to any new tasks and priorities. Produce concise status reports on a moments notice without missing anything.<br />
<br />
What I wanted was something simple and doable, and lets me focus on work versus the planner. Here's what I came up with:<br />
<br />
On your first day of the work week, take some time to make up a list of things that need to be done. This list doesn't have to be only for tasks this week either. You can document anything here that you consider a task.<br />
<br />
From the list, put a task / objective/goal on a sticky note. Make them one task per note.<br />
<br />
Prioritize the list. Stack them in the order you think is appropriate.<br />
<br />
Now, stick the list to your bulletin board in your cube. This is the TO DO stack.<br />
<br />
Begin working each task. For each thing you are currently working on, pull if off of the TO DO stack and put it in a "WORKING" stack. As you complete tasks, put them in a DONE Stack.<br />
<br />
Lets say Wednesday rolls around and you get a call to diagnose a problem or do some investigation work for a post mortem. Do a quick note on the task and ticket # and put it in your WORKING stack. As you finish, put in your DONE stack.<br />
<br />
On Friday afternoon, when its all said and done, list your DONE tasks in you weekly status. Make sure you list the things you're still working on from the WORKING stack and things to be done from the TO DO stack!Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-75543177573682177152012-03-28T10:34:00.000-05:002012-03-28T10:34:55.195-05:00Design by GlossyHow many folks do you know that live in a world of Design By Glossy? Inherently, some Director or VP goes to Interop or goes golfing with a bunch of Sales Weasels and ends up bringing in products without the first requirement documented and vetted. Now,all of a sudden, the implementation is on the schedule, its in your bucket list, and you are still trying to figure out what problem this thing solves.<br />
<br />
I cannot count how many times I've seen this and it is a travesty. It is devastating on morale, devastating on the services IT provides, and it is more disruptive than helpful. Many times, you end up with Shelfware, dumpware, or junkware and crippleware. Here is a list of damages these decisions cause:<br />
<br />
1. Without requirements, how do you do an effective test and acceptance? What criteria do you use? The applications glossy sheet? How smart is that?<br />
<br />
2. Has the workload been examined, vetted, and adjusted? All too often, products are piled up on other products such that the person cannot devote enough time to get the implementation close. Do you sacrifice people because of the new app and not even know it?<br />
<br />
3. What has been or is being sold to the customer? Many times the guy doing the selling has never seen it work. Now, someone else is responsible for making things work when even the vendor cannot respond.<br />
<br />
4. Who is responsible? Is it the buyer? Or is it the Implementer? Who is the Chicken and who is the pig? Remember the pig has more to lose in the game than the chicken. Be fair.<br />
<br />
Here are a few bad things that typically occur:<br />
<br />
The guy sticks around for a couple of years, then bolts, leaving a trail of tears. Thats OK. They going on to the next gig thinking that if only somebody had implemented what I said, it would have been perfect.<br />
<br />
Somebody somewhere in the organization is made a scapegoat for each one of these fiascos. The DBG person cannot be responsible. They claim they are the Sales person or the Director/VP at that point. Its up to someone else to be responsible.<br />
<br />
After going through a couple of these, there are people that consider a change in careers. It is so demoralizing that some even have to seek counselling. Many others end up hating what they do. Others tend to get as close as possible to the DBG in hopes that they become favored and not F@#$ed.(Kind of a behavior see in child and domestic abuse cases where the victim gets as close to the perp as possible to avoid more abuse.)<br />
<br />
The DBG person becomes a Troll Magnet for Sales people. They attract sales people because they have a history of foolish purchasing. No requirements. No POCs. Just run and Gun POs. Heres the check! It is almost the perfect storm for sales.<br />
<br />
They develop a history or legacy for technical people to be wary of. Strong technical people tend to push back. And when they do, the DBG person will resort to OSI Layer 8 - the Political Layer.Without a grounding in solid technical merit, their DBG choices have to resort to non-technical criteria.<br />
<br />
Conclusion - This is the danger of politicians becoming elevated into leadership roles in technical environments without paying their dues. They think they understand engineering and engineering concepts, but they only pay lip service. Even those with Engineering or Comp Sci backgrounds will do the same thing when they have not evolved technically over time. To them, its more important on who you know than it is for a product to actually solve some problems.<br />
<br />
Any sign of a DBG in charge can cost the organization a significant amount of money, time, and people.Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-55541659828519913892012-03-24T21:51:00.000-05:002012-03-24T21:51:12.517-05:00ENMS Integration - Scripting versus OOTBThe other day I was discussing integration work between products and how that takes a lot of scripting, data munging, and transformations to make an ENMS architecture work. In the course of the discussion, I was taken aback a bit when asked about supporting a bunch of scripts. Definitely caught me off guard.<br />
<br />
Part of the thing that caught me off guard was that some folks believe that products integrate tightly together without glueware and scripting. In fact, I got the impression that the products they had did enough out of the box for them.<br />
<br />
So, why do you script?<br />
<br />
To integrate products, tools, and technology. But most of all, INFORMATION. Scripts enable you to plumb data flow from product to product and feature to feature.<br />
<br />
Think about CGI tools, forms, and ASP pages. All scripting. Even the Javascript inside of your HTML that interfaces with Dojo libs on a server... SCRIPT.<br />
<br />
Think about the ETL tasks you have that grab data out of one application and fit it or compare it to data sets in other applications.<br />
<br />
Think about all of those mundane reports that you do. The configuration data. The performance data. The post mortems you do.<br />
<br />
Rules of Thumb<br />
<br />
1. There is no such thing as a temporary or disposable script. Scripts begin life as something simple and linear and end up living far longer than one would ever think.<br />
<br />
2.There will never be time to document a script after you put it in place. You have to document as you go. In fact, I really like to leave notes and design considerations within the script.<br />
<br />
3. You have to assume that sooner or later, someone else will need to maintain your script. You have to document egress and ingress points, expansion capabilities, and integrate in test cases.<br />
<br />
4. Assume that portions of your code may be usable by others. Work to make things modular, reusable, extensible, and portable. Probably 70% of all scripting done by System Administrators is done initially by reviewing someone else's code. Given this, you should strive to set the example.<br />
<br />
Things I like in Scripting<br />
<br />
perldoc- Perldoc is the stuff. Document your code inside of your own code. Your own module. Your script.<br />
<br />
perl -MCPAN -e shell Getting modules to perform things - PRICELESS!<br />
<br />
Templates. You need to build and use templates when developing code. For each function/ sub-routine/ Code Block / Whatever -You need to have documentation, test cases, logging, debugging, and return codes. Ultimately, it leads to much better consistency across the board. And Code Reviews get guaged around the template AND the functionality.<br />
<br />
Control ports - In long running or daemon processes, control ports save your Butt!<br />
<br />
getopt is your friend!!!<br />
<br />
STDERR is awesome for logging errors.<br />
<br />
POE - POE lets you organize your code into callbacks and subroutines around an event loop.<br />
<br />
/usr/bin/logger is AWESOME! I have used the LOCAL0 facility as an impromptu message bus as many apps only log to LOCAL1-7.<br />
<br />
Data::Dumper -- Nuff said!!!<br />
<br />
Date::Manip-- If you are dealing with date and time transformations, Date::Manip is your ace in the hole. It can translate last week from a "string" to a to and from date - time stamp and even on to a Unix time value.<br />
<br />
Spreadsheet::WriteExcel --I love this module! Lets me build Excel spreadsheets on the fly including multiple sheets, formulas, lookup tables and even charts and graphs. And using an .xls fie extension, most browsers know how to handle them. And EVERYONE knows how to work through a spreadsheet!<br />
<br />
ENMS products have a lot of scripting capabilities. Check out Impact. HP OO. BMC RBA. Logmatrix NerveCenter. Ionix SMARTs. The list goes on and on.<br />
<br />
Bottom line - If you have integration work to do, you will need to script. Could be perl, shell, python,or whatever. The products just don't have enough cross product functionality to fit themselves out of the box. In fact, there are several products that embrace scripting and scripting capabilities out of the box. Even products within the same product line will require scripting and glueware when you really start using the products. After all -> YOU ARE FITTING INFORMATION.Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-79422451095003855492011-10-10T08:21:00.000-05:002011-10-10T08:21:27.651-05:00Dante's Inferno in IT...We’ve all been through Dante’s Inferno. I know I have. <br />
<br />
I was at one place where they developed an in house SNMP agent to be deployed on all managed systems. Because of the variety of operating systems supported and the amount of bugs in the software, many customers hated the agent and would request that the agent be taken off of their systems. But because the agent was “free”, it kept living in a miserable existence. Turns out there were many different versions supported and deployed. Additionally, the design of the agents’ sub-agents capabilities deviated significantly from industry standards such that it hamstrung the open source openness of the base agent.<br />
<br />
Dante’s Inferno came in when you had to deal with the agent capabilities as an Architect. Saying anything negative related to the agent was<span class="Apple-style-span" style="color: red;"> Heresy</span>. The Manager that owned the Agent would resort to <span class="Apple-style-span" style="color: red;">Anger</span>, <span class="Apple-style-span" style="color: red;">Fraud</span>, and <span class="Apple-style-span" style="color: red;">Treachery</span> in order to divert any negative attention to their baby. Part of the reason for hanging on to this Agent was that the ownership of more developed products promoted the Manager’s <span class="Apple-style-span" style="color: red;">gluttony</span> and <span class="Apple-style-span" style="color: red;">greed</span>. It was his silo of management technology.<br />
<br />
While I was busy circumnavigating the Machiavellian Urinary Olympics, that Manager was working hard to put me in <span class="Apple-style-span" style="color: red;">Limbo</span>. Any requirements that I put forth were immediately in negotiation such that I could not finish requirements. Finally, in total frustration, I sent out a Final version of the requirements. Doing this sent the Manager into a frenzy of new Machiavellian Urinary Olympics such that my actions were elevated all the way up to a Sr. VP. Alas, I could not overcome the Marijuana Principle of Management (Harder you suck ,the higher you get!)<br />
<br />
I left shortly afterward. So did several of my coworkers. Some are still there. All with the common experience that we’ve all been through Dante’s Inferno. <br />
<br />
Lessons for the Architect :<br />
<br />
<ul><li>Be careful in calling someone’s baby ugly. Given the “embeddedness” of a given politician, there may be some things you cannot change. </li>
</ul><br />
<ul><li>Some Silos can only break down through years of pain and years of continued failure.</li>
</ul><br />
<ul><li>In moving toward Cloud computing models, some folks may have an inclination to bring with them all of the bad habits they have currently.</li>
</ul><br />
<ul><li>If a person has only ever seen one place, they may not understand that success looks totally different in other places.</li>
</ul><br />
<ul><li>There is a direct cost and an indirect cost to supporting internally developed products. If your internally developed product is holding back progress and new business, it is a danger sign…</li>
</ul><br />
As an Architect, be wary of consensus. “Where there is no vision, the people perish. (Proverbs 29:18)”Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-71992723487107804232011-09-05T06:40:00.000-05:002011-09-05T06:40:53.608-05:00Topology based CorrelationAmazing how many products have attempted just this sort of thing and for one reason or another, ended up with something a bit more complex than it really should be.<br />
<br />
Consider this... In an IP Network, when to have a connectivity issue, basic network diagnosis mandates a connectivity test like a ping and if that doesn't work, run a traceroute to the end node to see how far you get.<br />
<br />
What a traceroute doesn't give you is the interface by interface, blow by blow or the layer 2. If you turn on the verbose flag in ping or traceroute, you will see ICMP HOST_UNREACHABLE control messages. These come from the router that services the IP Subnet for the end device when an ARP request goes unnoticed.<br />
<br />
So, consider this. when you have a connectivity problem to an end node:<br />
<br />
Can you ping the end Node?<br />
If not, can you ping the router for that subnet?<br />
If yes, you have a Layer 2 problem.<br />
If not, you have a layer 3 problem. Perform a traceroute.<br />
<br />
Within the traceroute, it should bottom out where it cannot go further giving you some level of indication where the problem is.<br />
<br />
On a layer 2 problem, you need to check the path from the end node, through the switch or switches, on to the Router.<br />
<br />
Hows that for simplicity? Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com1tag:blogger.com,1999:blog-5590049888603730046.post-52095280861995089572011-06-12T20:58:00.000-05:002011-06-12T20:58:16.789-05:00Thinking Outside of the outsideTypical of any engineer, I like puzzles - problem sets. And as with any puzzle, you have to decide what the puzzle looks like when solved. Want to test your resolve and hone your pattern matching skills? Purchase a 1000 piece puzzle and solve it with picture down!<br />
<br />
I miss the scale and speed of working with problems sets in Global NetOps at AOL. What if you could experience a live event stream that was composed of over 1 million events a minute? How would you deal with the rate? Log turnover? Parsing and handling these as events? Handling decisions at this rate? I tell you, it gives you a whole new perspective.<br />
<br />
A couple of requirements we had with all of our applications were:<br />
<br />
Must run 24 By Forever.<br />
Must Scale<br />
<br />
24 By Forever meant that you had control ports in your processes. You could start, stop, change, failover, failback, reroute... Anything to keep from failing.<br />
<br />
Must scale meant that you could start small and get tall without intervention. Without buffering too much. Without rendering the information useless.<br />
<br />
I started off with a simple Netcool Syslog probe. It was filtered and only passed events every minute based upon a cron job that filtered the log and sumped to Netcool, only the pertinent events. Why? It just wasn't keeping up. Yet there were alot of syslog events that were not useful for Netcool. And yet, it was always behind. It was outdated before it even got to Netcool. <br />
<br />
Lesson 1 - Delaying Events from presentation can render your information useless.<br />
<br />
I built a little process called collatord that ran as a daemon, watched all of syslog logs, and pattern matched incoming lines for dispatch and formatting to Netcool. I subsequently moved from the venerable Syslog probe over to TCP Port probes which we had implemented behind load balancers. all I had to do was output my events in a value=pair manner with a \n\n line termination. (I subsequently found out that win or lose, it always returned and OK!)<br />
<br />
Lesson 2 - Always check returns!<br />
<br />
Little collatord had an ACTIONS hash that was made up of a regex pattern as a key and a Code reference to a subroutine. When a pattern "hit", it executed the subroutine passing in the line as an argument. Turns out, it ran pretty quickly. I was running in a POE kernel and only a single session. Even with 100K lines a minute, it still skimmed right along!<br />
<br />
One of the problems I had was that I would do a subroutine that parsed and handled a specific pattern for a given event and the syslog message would change over time. Maybe it was a field that moved from one position to another. Maybe it was a slightly different format. I found that if I took a sample line and put it in the subroutine as a comment, my whole subroutine became innately simpler in that I could see what the pattern was before and adapt the new pattern within the subroutine.<br />
<br />
Lesson 3 - Take care to make your app reentrant. The better you are at this, the less rewriting code you'll do trying to figure out how to change things.<br />
<br />
Now, with these samples, I got the bright idea that I could take any event in the sample, change the time and hostname to protect the innocent, and reinsert it as a Unit test.<br />
<br />
Lesson 4 - Having a repeatable Unit test. PRICELESS!<br />
<br />
Then I figured out the if I appended the word TRACEMESSAGE on any incoming event, I could profile each and every sample line. All I had to do was to recognize /TRACEMESSAGE$/ and log a microsecond timestamp along with what the function was doing.<br />
<br />
Lesson 5 - Being able to profile a specific function is INVALUABLE in a live system where you suspect something weird or intermittent and you don't have to restart it.<br />
<br />
After I ran into a couple of pattern / parser problems where I had to schedule downtime, I went and talked to my cohorts in crime. I got to looking at their stuff and found that they could change code on the fly. They didn't need downtime. (24 by Forever!) I went back to my desk and started putting together a control port.<br />
<br />
In the control port, I'd connect to the collatord process via a TCP socket, authenticate, and run commands. In Perl, you can even handle subroutines through an eval. So, I would pass in a new subroutine, eval it, and put the pattern and Code Reference in the %ACTIONS Hash.<br />
<br />
Lesson 6 - Being able to adapt on the fly without downtime. BRILLIANT<br />
<br />
Not all languages can be adapted to do this so your mileage may vary!Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-49842322545101108872011-04-17T13:22:00.003-05:002011-04-17T13:44:37.713-05:00Data Analytics and Decision Support - Continued<b>Data Interpolation</b><br />
<br />
In some performance management systems, the fact that you may have nulls or failed polls in your data series draws consternation in some products. If you are missing data polls, the software doesn't seem to know what to do.<br />
<br />
I think the important thing is to catch counter rollovers in instances where your data elements are counters. You want to catch a rollover in such a way as when you calculate the delta, you add up until the maximum counter value, then start adding again up to the new value. In effect you get a good delta in between two values. What you do not want is tainted counters that span more than one counter rollover.<br />
<br />
If you take the delta value between a missed poll as in:<br />
<br />
P1 500<br />
P2 Missed<br />
P3 1000<br />
<br />
In smoothing, you can take the difference between P3 and P1 and divide that by half as a delta. Simply add this to P1 to produce P2 which produces a derived time series of:<br />
<br />
P1 500<br />
P2 750<br />
P3 1000<br />
<br />
In gauges, you may have to sample the elements around the missed data element and use an average to smooth over the missed data element. Truth be known, I don't see why a graphing engine cannot connect from P1 to P4 without the interpolation! It would make graphs much simpler and more rich - If you don't see a data point on the time slot, guess it didn't happen! In the smoothing scenario, you cannot tell where the smoothing is.<br />
<br />
<b>Availability</b><br />
<br />
In my own thoughts, I view availability as a discreet event in time or a state over a time period. Both have valid uses but need to presented in the context that they are borne so that users do not get confused. For example, an ICMP ping that occurs every 5 minutes is a discreet event in time. Here is an example time series of pings:<br />
<br />
P1 Good<br />
P2 Good<br />
P3 Fail<br />
P4 Good<br />
P5 Fail<br />
P6 Good<br />
P7 Good<br />
P8 Good<br />
P9 Good<br />
P10 Good<br />
<br />
This time series denotes ping failures at P3 and P5 intervals. In theory, a line graph is not appropriate for this instance because it is boolean and it is discreet and not representative of the entire time period. If P0 is equal to 1303088400 on the local poller and the sysUptime value at P0 is 2500000, then the following SNMP polls yield:<br />
<br />
Period Poller uTime SysUptime<br />
P0 1303088400 2500000<br />
P1 1303088700 2503000<br />
P2 1303089000 2506000<br />
P3 1303089300 2509000<br />
P4 1303089600 2512000<br />
P5 1303089900 20000<br />
P6 1303090200 50000<br />
P7 1303090500 80000<br />
P8 1303090800 110000<br />
P9 1303091100 5000<br />
P10 1303091400 35000<br />
<br />
<span class="Apple-style-span" style="font-family: inherit;">utime is the number of seconds since January 1, 1970 and as such, it increments every second. SysUptime is the number of Clock ticks since the management system reinitialized. When you look hard at the TimeTicks data type, it is a modulo-s counter of the number of 1/100ths of a second for a given time epoch period. This integer will roll over at the value of <span class="Apple-style-span" style="white-space: pre;">4294967296 or approximately every 49.7 days.</span></span><br />
<span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="white-space: pre;"><br />
</span></span><br />
<span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="white-space: pre;">In looking for availability in the time series, if you derive the delta between the utime timestamps and multiply that times 100 (given that timeTicks is 1/100th of a second), you can see that if the sysUpTime value is less than the previous value plus the serived delta timeticks, you can clearly see where you have an </span></span><span class="Apple-style-span" style="white-space: pre;">availability</span><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="white-space: pre;"> issues. As compared to the previous time series using ICMP ping, P3 was meaningless and only P5 managed to ascertain some level of availability discrepancy. </span></span><br />
<span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="white-space: pre;"><br />
</span></span><br />
<span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="white-space: pre;">You can also derive from the irrant time series periods that P5 lost 10000 timeTicks from the minimum delta value of 30000 (300 seconds * 100). So, for period P5 we were not </span></span><span class="Apple-style-span" style="white-space: pre;">available</span><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="white-space: pre;"> for 10000 timeticks and not the full period. Also note that if you completely miss a poll or don't get a response, the delta of the last poll chould tell you where availability was affected or not even though the successful poll did not go as planned.</span></span><br />
<span class="Apple-style-span" style="font-family: monospace; font-size: 16px; white-space: pre;"><br />
</span><br />
From a pure time series data kind of conversion, one could translate sysUpTime to an accumulated number of available seconds over the time period. From a statistics perspective, this makes sense in that it becomes a count or integer.<br />
<br />
<b>Discreet to time Series Interpolation</b><br />
<br />
Let's say you have an event management system. And within this system, you receive a significant number of events If can categorize and number your events into instances counted per interval, you can convert this into a counter for a time period. for example, you received 4 CPU threshold events in 5 minutes for a given node.<br />
<br />
In other instances, you may have to covert discreet events into more of a stateful.approach toward conversions. For example, you have a link down and 10 seconds later, you get a link up. You have to translate this to a non-availability of the link for 10 seconds of a 5 minute interval. What is really interesting about this is that you have very finite time stamps. When you do, you can use this to compare to other elements that may have changed around the same period of time. Kind of a cause and effect analysis.<br />
<br />
This is especially pertinent when you analyze data from devices, event data, and workflow and business related elements. For example, what if you did a comparison between HTTP response times to Internet network IO rates, and Apache click rates on your web servers? What if you threw in trouble tickets and Change orders over this same period? What about query rates on your Oracle databases?<br />
<br />
Now, you can start translating real metrics with business metrics and even cause and effect elements because they are all done using a common denominator - Time. While many management platforms concentrate on the pure technical - like IO rates of Router interfaces, it may not mean as much to a business person. What if your web servers seem to run best when IO rates are between 10-15 percent. If that range is where the company performs best, I'd tune to that range. If you change your apps to increase efficiency, you can go back and look at these same metrics after you make changes and validate handily.<br />
<br />
This gets even more interesting when you get Netflow data for the intervals as well. But I'll cover that in a later post!Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-82781080524071558812011-04-10T11:16:00.002-05:002011-04-10T11:19:17.485-05:00Data Analytics and Decision Support.OK, The problem is that given a significant amount of raw performance data, I want to be able to determine some indication and potentially predictions, related to whether the metrics are growing or declining and what will potentially happen beyond the data we currently have, given the existing data trending.<br />
<br />
<i>Sounds pretty easy, doesn't it.</i><br />
<i><br />
</i><br />
When you look at the overall problem, what you're trying to do is to take raw data and turn it into information. In effect, it looks like a Business Intelligence system. Forrester Research defines Business Intelligence as<br />
<br />
<span class="Apple-style-span" style="background-color: #e06666; color: white;">"<span class="Apple-style-span" style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 13px;"><i>Forrester often defines BI in one of two ways. Typically, we use the following broad definition: Business intelligence is a set of methodologies, processes, architectures, and technologies that transform raw data into meaningful and useful information used to enable more effective strategic, tactical, and operational insights and decision-making. But when using this definition, BI also has to include technologies such as data integration, data quality, data warehousing, master data management, text, and content analytics, and many others that the market sometimes lumps into the information management segment. Therefore, we also refer to data preparation and data usage as two separate, but closely linked, segments of the BI architectural stack. We define the narrower BI market as: A set of methodologies, processes, architectures, and technologies that leverage the output of information management processes for analysis, reporting, performance management, and information delivery.</i>"</span></span><br />
<span class="Apple-style-span" style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 13px;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 13px;"><a href="http://www.forrester.com/ER/Glossary/Item/1,2425,834,00.html?Alpha=B">Here</a> is the link to their glossary for reference.</span><br />
<span class="Apple-style-span" style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 13px;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 13px;">When you look at this definition versus what you get in todays performance management systems, you see somewhat of a dichotomy. Most of the performance management systems are graph and report producing products. You get what you get and so its up to you to evolve towards a Decision Support System versus a Graphical Report builder.</span><br />
<span class="Apple-style-span" style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 13px;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 13px;">Many of the performance management applications lack the ability to produce and use derivative data sets. They do produce graphs and reports on canned data sets. It is left up to the user to sort through the graphs to determine what is going on in the environment.</span><br />
<span class="Apple-style-span" style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 13px;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 13px;">It should be noted that many products are hesitant to support extremely large data sets with OLTP oriented Relational Databases. When the database table sizes go up, you start seeing degradation or slowness in the response of <a href="http://en.wikipedia.org/wiki/Create,_read,_update_and_delete">CRUD</a> type transactions. Some vendors resort to batch loading. Others use complex table space schemes. </span><br />
<span class="Apple-style-span" style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 13px;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 13px;">Part of the problem with Row oriented databases is that it must conform to the <a href="http://en.wikipedia.org/wiki/ACID">ACID</a> model. Within the ACID model, a transaction must go or not go. Also, you cannot read at the same time as a write is occurring or your consistency is suspect. In performance management systems, there is typically a lot more writes than there a reads. Also, as the data set gets bigger and bigger, the indexes have to grow as well.</span><br />
<span class="Apple-style-span" style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 13px;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 13px;">Bottom line - If you want to build a Decision Support System around Enterprise Management and the corresponding Operational data, you have to have a database that can scale to the multiplication of data through new derivatives.</span><br />
<span class="Apple-style-span" style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 13px;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: x-small;">Consider this - a Terabyte of raw data may be able to produce more than 4 times that when performing analysis. You could end up with a significant number of derivative data sets and storing them in a database enables you to use them repeatedly. (Keep in mind, some OLAP databases are very good at data compression.)</span><br />
<br />
<b>Time - The Common Denominator</b><br />
<br />
In most graphs and in the data, the one common denominator about performance data is that it is intrinsically time series oriented. This provides a good foundation for comparisons. So now, what we need to do is to select samples in a context or SICs. For example, we may need to derive a sample of an hour over the context of a month.<br />
<br />
In instances where a time series is not directly implied, it can be derived. For example, the number of state changes for a given object can be counted for the time series interval. Another instance may be a derivate of the state for a given time interval or the state when the interval occurred. In this example, you use what ever state the object is in at the time stamp.<br />
<br />
Given that raw data may be collected at intervals much smaller than the sample period, we must look for a way to summarize the individual raw elements sufficiently enough to characterize the whole sample period. This gets one to contemplating about what would be interesting to look at. For example, probably the simplest way to characterize a counter is to Sum all of the raw data values into a sample (Simple Addition). For example, if you retrieved counts on 5 minute intervals, you could simply ad up the deltas over the course of the 1 hour sample into a derivative data element producing a total count for the sample. In some instances, you could look at the counter at the start and look at the end, if there are no resets or counter rollovers during the sample.<br />
<br />
In other instances, you may see the need to take in the deltas of the raw data for the sample to produce and average. This average is useful by itself in many cases. But in others, you could go back through the raw data and compare the delta to the average to produce an offset or deviation from the average. What this offset or deviation really does is to produce a metric to recognize activity within the sample. The more up and down off of the average, the greater the deviations will be.<br />
<br />
Another thing you need to be aware of is data set tainting. For example, lets say you have a period within your context where data is not collected, not available, or you suspect bad data. This sample could be considered tainted but in many cases, it can still be used. It is important sometimes to compare data collection to different elements like availability, change management windows, counter resets, or collection "holes" in the data.<br />
<br />
What these techniques provide is a tool bag of functions that you can use to characterize and compare time series data sets. What if you wanted to understand if there were any cause and effect correlation of any metric on a given device. There are elements you may find that are alike. Yet there are others that may be opposite from each other. For example, when one metric increases, the other decreases. The characterization enables you to programmatically determine relationships between metrics.<br />
<br />
The result of normalizing samples is to produce derivative data sets so that those can be used in different contexts and correlations. when you think about it, one should even be able to characterize and compare multiple SICs (Samples in Contexts). For example, one could compare day in month to week in month SICs.<br />
<br />
Back to our samples in context paradigm. What we need samples for is to look at the contexts over time and look for growth or decline behaviors. The interesting part is the samples in that samples are needed when performing Bayesian Belief Networks. BBNs take a series of samples and produce a ratio of occurrences and anti-occurrences, translated as a simple + 1 to -1.<br />
<br />
When you look at Operations as a whole, activity and work is accomplished around business hours. Weekdays become important as does shifts. So, it makes sense to align your Samples to Operational time domains.<br />
<br />
The first exercise we can do is to take a complex SIC like weekday by week by Month. For each weekday, we need to total up our metric for the weekday. For each weekday in week in month sample, is the metric growing or declining? If you create a derivative data set that captures the percent different of the previous value, you can start to visualize the change over the samples.<br />
<br />
When you start down the road of building the data sets, you need to be able to visualize the data in different ways. Graphs, charts, scatterplots, bubble charts, pie charts, and a plethora of other mechanisms provide ways you can see and use the data. One of the most common tools that BI professionals use is a Spreadsheet. You can access the data sources via ODBC and use the tools and functions within the spreadsheet to work through the analysis.Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com1tag:blogger.com,1999:blog-5590049888603730046.post-71974564254276617482010-11-27T03:25:00.000-06:002010-11-27T03:25:31.463-06:00EM7 Dynamic App DevelopmentDynamic Application development in EM7 is easy and a lot of fun.<br />
<br />
A Dynamic Application in EM7 is used to gather statistics, configuration data, performance data, set thresholds, and setup alerts and events.<br />
<br />
In the past, I've done alot of SNMP based applications where I would have to take SNMP Walks, the MIBs, and put together polling, thresholds, and triggers for events. I'd download the MIB, suck it into Excel, and start analyzing the data. A lot of time would be spent parsing MIBs, verifying tables, and looking at counters.<br />
<br />
Enter EM7.<br />
<br />
Login and go to System->Tools->MIB Compiler<br />
<br />
It provides a list of all MIBs it knows about.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtyVdUkGNj8ALVvrKt-q-ITzVFW9QjLWb03jIhQuQJAfy0rk0KpBiOx57_sAPVmspNYR0GUwhCgH_7x3u7szvyoXx8UOPunezHmfWZ3KU3HTX-56XoJ7ssjrZpSDqE-ECqd7E7CzM2iQk/s1600/EM7MIBCompiler.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtyVdUkGNj8ALVvrKt-q-ITzVFW9QjLWb03jIhQuQJAfy0rk0KpBiOx57_sAPVmspNYR0GUwhCgH_7x3u7szvyoXx8UOPunezHmfWZ3KU3HTX-56XoJ7ssjrZpSDqE-ECqd7E7CzM2iQk/s400/EM7MIBCompiler.jpg" width="400" /></a></div><br />
<div class="separator" style="clear: both; text-align: center;"></div>From this, you can see if the MIB you're looking for is there and compiled. If its not, you can hit the import button up on the top right portion of the window and it will prompt you to upload a new MIB file. (Pretty friggin straight forward if I do say so myself!)<br />
<br />
To compile the MIB, hit the lightning icon on the MIB entry. Simple as that.<br />
<br />
So, lets say you want to create a dynamic application around some configuration items associated with a couple of MIB tables in a MIB. For our example, we will use the CISCO-CCME-MIB applicable for the cisco Call Manager Express systems. You can view the MIB using the I icon, download or export hte MIB via the green disk icon on the right side of the entry, or compile it using the lightning bolt. I'm going to view the MIB so I select the I icon for the row for the MIB I want.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4JF-bU2KXN7PYCLib0lpbEwq_w7B-UDKSQo7awsWAGwIxZziw55MA9kNrLNEcu9AawMMK2TSEP50zy2m1gQqmoYoHE_8aOdXJvYBS1ZmO6EhyXfU7ckL7C8zFd6xbLppSNhUnCkqvkXE/s1600/EM7viewCCMEMIB.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4JF-bU2KXN7PYCLib0lpbEwq_w7B-UDKSQo7awsWAGwIxZziw55MA9kNrLNEcu9AawMMK2TSEP50zy2m1gQqmoYoHE_8aOdXJvYBS1ZmO6EhyXfU7ckL7C8zFd6xbLppSNhUnCkqvkXE/s320/EM7viewCCMEMIB.jpg" width="320" /></a></div>I hit the icon and it pulls up a file edit window. I scroll through to find an object I'd like to work with as an application. I highlight the object and copy it to my clipboard.<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
I go to System->Tools->OID Browser, select Where name is like in the search function and I paste the object I was looking at in the MIB. This pulls up a hierarchy of the OIDs and MIB objects below the object I input. All in all, we're into the second minute if you're a bit slow on the keyboard like me!<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_FJouTh5Eolegfdxq67gq0IEVzO900jQ29G-BpFJt3fkcHMhFHRJAQpW6Y1nuyh65h-VjpGmpC0ZtKnafxWZn2I7D4EGsF2XxUiRQvIcxwWUIKyY6q_trpcWL05XNlnwOkZqc-yQ5yoY/s1600/EM7OIDBrowserCCME.jpg" imageanchor="1" style="clear: left; display: inline !important; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_FJouTh5Eolegfdxq67gq0IEVzO900jQ29G-BpFJt3fkcHMhFHRJAQpW6Y1nuyh65h-VjpGmpC0ZtKnafxWZn2I7D4EGsF2XxUiRQvIcxwWUIKyY6q_trpcWL05XNlnwOkZqc-yQ5yoY/s320/EM7OIDBrowserCCME.jpg" width="320" /></a><br />
Notice on the top of the selection box is the MIB object path. This is handy to visualize where you are as you start to work with MIB objects.<br />
<br />
So, the first application I want to put together is a configuration type of application centered around parameters that are part of the Music on Hold system function. I select the MIB elements I want to work with in my configuration application by clicking on the box for each parameter.<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTcUWwMYrKkUqTQQDSIMS4Bs6G456wx6KXIrbUPA66Zr8_wXXOC7pIS9uQwsEKSayob0aR6pNe3ogmBbOS6NF_o350CAw6BpgnJD4b5XexAmSzyeQompzK-DwO9ijd_QvwjazII1he568/s1600/EM7CCMEMOH.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTcUWwMYrKkUqTQQDSIMS4Bs6G456wx6KXIrbUPA66Zr8_wXXOC7pIS9uQwsEKSayob0aR6pNe3ogmBbOS6NF_o350CAw6BpgnJD4b5XexAmSzyeQompzK-DwO9ijd_QvwjazII1he568/s320/EM7CCMEMOH.jpg" width="320" /></a></div><br />
<br />
On the bottom right corner of this window is a selection box with a default value of [Select Action]. Pull down the menu and select Add to a new Configuration Application. Select the go button directly to its right.<br />
<br />
You are then prompted if this is really what you want to do. Select Yes.<br />
<br />
<br />
<br />
<br />
What this does is to take the MIB objects and create a dynamic application with your collection objects setup already. Here is what it looks like.<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhC7k-3VrzsiIdnwHXUlqefOCD5qdqceRq51gQwkrCiq5GLx_Q3__P-4uHB5MS-t4c1vhFUb7UMt8eEHji1lfopqUVmQ8k6h_wM5cN8FPf0PompbHSWGMI_rjhWBsrKnc5-uO9trH92Muo/s1600/EM7CCMEMOHapp.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhC7k-3VrzsiIdnwHXUlqefOCD5qdqceRq51gQwkrCiq5GLx_Q3__P-4uHB5MS-t4c1vhFUb7UMt8eEHji1lfopqUVmQ8k6h_wM5cN8FPf0PompbHSWGMI_rjhWBsrKnc5-uO9trH92Muo/s320/EM7CCMEMOHapp.jpg" width="320" /></a></div><br />
If you do nothing else, go to the the Properties tab, input an application Name, then save it. You have just created your first dynamic application in its basic sense of form and function.<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
This is what the Properties screen looks like:<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZiKg6N24zfZV8d8uRUCIwuDjuyVyW7ZopNw7BoW7fKLhGyKlpL5q0cpDTjzSEalnSrtWWwwZQHP9E_tDfEtwH84Db5pMJjPEah8S2mBG2mpw9xRygoGeS-zkXpT_YIZ4q7XSPYl_VmUE/s1600/EM7CMME_MOHAppsave.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZiKg6N24zfZV8d8uRUCIwuDjuyVyW7ZopNw7BoW7fKLhGyKlpL5q0cpDTjzSEalnSrtWWwwZQHP9E_tDfEtwH84Db5pMJjPEah8S2mBG2mpw9xRygoGeS-zkXpT_YIZ4q7XSPYl_VmUE/s320/EM7CMME_MOHAppsave.jpg" width="320" /></a></div><br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Overall, you have a lot of work to do in that you probably need to cleanup the names of the Objects so that they show up better on reports. Setup polling intervals. And setup any thresholds, alerts, or events as needed. But you have an application in a few minutes. And it can cross multiple tables if you like! CLICK CLICK BOOM!Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-23459730020834081342010-11-27T01:54:00.000-06:002010-11-27T01:54:38.440-06:00ENMS ProjectsI have been looking over why a significant number of enterprise management implementations fail. And not only do they fail in implementation, they continue to live in a failed state year over year. Someone once said that 90% or greater of all enterprise management implementations fail. <br />
<br />
I cannot help but to think that part of this may be something to do with big projects and evolution during implementation.<br />
<br />
<br />
Another consideration I have seen is that lower level managers tend to manage to the green and are afraid of having to defend issues in their department or their products. When they manage to the Green, management above them loses awareness of how things are really working beneath them in the infrastructure.<br />
<br />
One article I really liked was by Frank Hayes called "Big Projects, Done Small" in a recent Computerworld issue. <a href="http://www.computerworld.com/s/article/352786/Big_Projects_Done_Small">Here</a> is the link to it. I like his way of thinking in that big projects need to be sliced and diced into smaller pieces in order to facilitate success more readily.<br />
<br />
Those of us in technical roles tend to operate and function in a 90 days are less role. We know from experience that if you have a project exceed 90 days, the "Dodge Syndrome" will rear its ugly head. (The rules have changed!) In all actuality, requirements tend to change and evolve around an implementation if the implementation takes over 90 days.<br />
<br />
The second part you realize is that in projects that need to span over 90 days, mid-level Managers tend to get nervous and lose faith on the project. Once this happens, you start seeing the Manager retract support and resources for the project. The representatives don't show up to meetings.<br />
<br />
But Tier 1 managers like the big, high cost projects as it is a way of keeping score. They like the big price tag and they like the enormous scope of global projects.<br />
<br />
It is the job of Architects to align projects into more manageable, chunks for implementation and integration. They need to know and plan the budget for these projects so that you get what needs to be done, in the proper sequence, with the proper hooks, and with the proper resources. If this is done by Project Managers, Contractors, or Development staff, you risk the project becoming tactical in nature and lose sight of strategic goals.<br />
<br />
When a project becomes tactical, the tasks become a cut list that has to happen every week. When something changes or impacts the schedule, tactical decisions are made to recover. For example, a decision to modify a configuration may be made today that kills or hampers an integration that needs to be done in a few months. These tactical decisions may save some time and resources today but yield a much larger need later on.<br />
<br />
It is the equivalent to painting yourself in a corner.<br />
<br />
Here in lies the situations that are oh so common:<br />
<br />
1) Tier 1 Management wants to show the shareholders and Board of Directors that they are leading the company in a new direction.<br />
<br />
2) Mid-level tiers of management can chose to use this project to promote their own enpires if possible.<br />
<br />
3) Mid-Level managers can chose just to accept the big project and support it as needed.<br />
<br />
4) Architects need to be keenly aware and on top of the project as a whole. They need to set, establish, and control direction.<br />
<br />
5) End users need to be empowered to use the product to solve problems and work with Architecture to get their requirements in.<br />
<br />
Conclusion<br />
<br />
A big project can be accomplished and can succeed. But the strategic direction needs to be set with a vision for both tactical and strategic goals alignment. Things need to be broken up by goals and objectives and costed accordingly. Simplify - simplify - simplify.Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-17427160165305142972010-10-12T09:50:00.000-05:002010-10-12T09:50:44.936-05:00The Case for EM7I have been monitoring alot of activity around various products that I have had my head wrapped around for years. See, I have been a Consultant, Engineer, end user, OSS Developer/Architect and even Chief Technologist. I've developed apps and I've done a TON of integration across the years and problem sets. I have worked with, implemented, and supported a ton of different products and functions.<br />
<br />
I think about all of the engagements I worked on where we spent 4-6 weeks just getting together the requirements doc and start down the road of doing a BOM. Then, after somebody approved the design, we get down to ordering hardware and software.<br />
<br />
Then, we'd go about bringing up the systems, loading the software, and tuning the systems.<br />
<br />
Before you knew it, we were 6 months on site.<br />
<br />
This has been standard operations for EONS. For as long as I remember.<br />
<br />
While the boom was on, this was acceptable. Business needed capabilities and that came from tools and technology. But now, money is tight and times are hard. However, companies still need IT capabilities that are managed and reliable.<br />
<br />
Now days, customers want results. Not in 6 months. Not even next month. NOW! When you look at the prospects of Cloud, what attracts business and IT alike is that in a cloud, you can deploy applications quickly!<br />
<br />
Enter EM7. What if I told you I could bring a couple of appliances (well, depends on the size of your environment...), rack and stack, and have you seeing results right away? Events. Performance. Ticketing. Discovery. Reports.<br />
<br />
EM7 lives in this Rapid Deployment, lots of capabilities up front sort of scenario. Literally, in a few hours, you have a management platform.<br />
<br />
Question: How long does it take for you to get changes into your network management platform? Days? Weeks? <br />
<br />
Question: How long does it take for you to get a new report in place on your existing system?<br />
<br />
Question: Are you dependent upon receiving traps and events only as your interface into receiving problem triggers? Do you even perform polls?<br />
<br />
Question: How many FTEs are used just to maintain the monitoring system? How many Consultants with that?<br />
<br />
Now, the question is WHY should you check out EM7?<br />
<br />
First and foremost, you want whats best for your environment. The term "Best" changes over time. And while many companies will tell you best of breed is dead and that you need a single throat to choke, its because they KNOW they are not Best of Breed and they HATE competition. But competition is good for customers. It keeps value in perspective.<br />
<br />
Without a bit of competition, you don't know best. Your environment has been evolving and so have the management capabilities and platforms (well some have!)<br />
<br />
In some environments, politics keep new products and capabilities at bay. You can't find capital or you know you'll step on toes if you propose new products. So, you need to have some lab time. With EM7, you spend more time working with devices and capabilities in your lab than you do installing, administering, and integrating the management platform. EM7 is significantly faster to USE than others.<br />
<br />
EM7 has a lot of the management applications you need to manage an IT / Network Infrastructure, in an APPLIANCE model. All of the sizing, OS tuning, Database architecture, etc. is done up front in defineable methods. We do this part and you get to spend your time adding value to your environment instead of the management platform.<br />
<br />
In a politically charged environment, EM7 can be installed, up and running, and adding SIGNIFICANT value before the fifedom owners even know. Em7 becomes the stndard quickly in buy vs. build scenario as there are alot of capabilities up front. But there feature rich ways to extend management and reporting beyond the stock product. Dynamic applications enable the same developers to build upon the EM7 foundation rather than attempting to redevelop foundation, then the advanced capabilities.<br />
<br />
If you're adapting and integrating Cloud capabilities, EM7 makes sense early on. Lots of both agent and agentless capabilities, service polling, mapping and dashboards, run book automation, ticketing, dynamic applications... the list goes on and on. Cloud and virtualization software and APIs can change very quickly. Standards are not always first. So, you have to be able to adapt and overcome challenges quickly.<br />
<br />
If you're in the managed services world, you need to be vigilant in your quest for capabilites that turn into revenue. If you can generate revenue with a solution, it is entirely possible to make more than it costs. Do you know how much revenue is generated by your management platform? Now compare that to what it costs in maintenance and support.<br />
<br />
Now, go back and look at things you've done to generate and instance new product. How long did it take from concept to offering? How much did it cost resource wise?<br />
<br />
In these times, when margins are lean and business is very competitive, if you aren't moving forward to compete, you're probably facing losing market share.<br />
<br />
With EM7, you get to COMPETE. You get a solid foundation with which to discriminate and differentiate your services from your competition. Quickly. Efficiently.Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-48816191859903385792010-08-15T21:53:00.000-05:002010-08-15T21:53:17.731-05:00Save the Village ID ten TEver been in one of these situations?<br />
<br />
So, you have this Manager you work with. Seems that every time you have to interact with them, it becomes a painful ordeal. See, they are the Political type. You know, the type that destroys the creativity of organizations.<br />
<br />
When you deal with them, all of a sudden your world is in Black and white... Not color. It's a 6th Sense moment. "I see Stupid People. And they don't even know they're stupid!".<br />
<br />
You ask a technical question and get the trout look. You know - Fresh off the plate! This person no more understands what the F you're talking about than the man in the moon. Nor do they want to understand. They have tuned you out. In an effort to shut you up, they proceed up the food chain to suck up to every manager in an attempt to put more red tape and BS in front of you.<br />
<br />
Well, you figure that because you are obviously not getting your technical points across, you do memos and white papers explaining the technical challenges. Then, you're too condescending and academic.<br />
<br />
You do requirements and everything becomes a new negotiation. You can get nothing done. To this person, it doesn't matter if you have valid requirements or not. What matters most is their own personal interest and growing their own empire. <br />
<br />
Its Managers like this that hold back Corporations. They kill productivity and teamwork and they can single-handedly poison your Development and integration teams. They have no idea the destruction they cause. The negative effects they have on folks. The creativity they squash. <br />
<br />
They foster Analysis Paralysis, lack of evolution and the tragic killing of products and solutions. These types of folks tend to latch on to products or functions and never release control. Such that it kills functionality and the evolution beyond the basic. Everything is about the bare minimums as you don't have to work at it much. You can float, do what you can, and go on. So what if you never ever deliver new capabilities UNLESS it is POLITICALLY ADVANCING to do so.<br />
<br />
How do you save someone like this from themselves? They refuse to listen. They are already Lunar Lander personalities - "Tranquility Base here! The EGO has landed!". They don't even have an inkling of what they don't know or understand. Yet they continue to drive out innovation.<br />
<br />
As a Manager of one of these personality types, it becomes dangerous for you. You either subscribe to the lineage of politics or they start politically going around you. So the deceit, lies, and ignorance perpetuates. And the company continues to suffer. It takes strong leadership to either control the politics or make a decision to oust the impediments to progress. Both take intestinal fortitude, leadership, and a lot of paperwork.<br />
<br />
In summary, I wish I could reach these folks. Have them understand the creativity and peoples careers they kill. All in the name of self preservation. Have them understand that a little common sense would open their whole world.<br />
<br />
In my career, I have been in this situation 4-5 times. A Situation where the only way to be successful is to compromise your principals and ethics and start sucking up. My heart goes out to my friends that endure the torture of mindless decisions , the lack of experience and unwillingness to understand, and the unending political positioning and rambling rather than doing the right thing.<br />
<br />
I'm soo elated to be away from this. Words cannot do justice to how elated and ecstatic I am to be able from the mindlessness.Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-60266649787416761032010-08-15T21:17:00.000-05:002010-08-15T21:17:12.837-05:00Event Management and Java programmersQuestion: Why is it a good idea to hire a Java programmer over someone with direct Netcool experience for a Netcool specific job?<br />
<br />
I recently heard about an open position for a Netcool specialist where they opened up the position to be more of a Java programmer slot. In part, I think its a lack of understanding regarding supporting Netcool and understanding what it takes, domain knowledge wise, to be successful at it.<br />
<br />
Most Java programmers either want to do some SOA sort of thing, Jakarta and struts, Spring Framework, Tomcat / Jboss or some portal rehash sort of thing. Most of these technologies have little to do with Netcool and rules. In fact, the lack of domain knowledge around the behavior of Network and systems devices in specific failure induced conditions, may limit a true Java programmers ability to be successful without significant training and exposure over time.<br />
<br />
Here is the ugly truth:<br />
<br />
1. Not every problem can be adequately or most effectively solved with Object Oriented Programming. Some things are done much more efficiently in Turing machines or Finite State automata.<br />
<br />
2. You give away resources in OO to empower some level of portability and reuse. If your problem is linear and pretty static, why give away resources?<br />
<br />
3. With Oracle suing Google of copywrite enfringement over the use of Java, Java may be career limiting. I mean, if Oracle is willing to initiate a lawsuit with Google which has the resources to handle such a lawsuit, how much more will it be willing to sue companies of significantly less legal resources?<br />
<br />
So, unless you plan on rewriting Netcool in Java, I'd say this repositioning is a pretty limiting move. And if you were foolish enough to think this is even close to being cost effective, I'd pretty much say that as a Manager, you are dangerous to your company and share holders.Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com1tag:blogger.com,1999:blog-5590049888603730046.post-83120470503929418302010-07-11T22:58:00.001-05:002010-07-14T14:49:17.430-05:00ENMS User Interfaces...Ever watch folks and how they use various applications? When you do some research around the science of <a href="http://en.wikipedia.org/wiki/Situation_awareness">Situation Awareness</a>, you realize that human behavior in user interfaces is vital to understanding how to put information in front of users in ways that empowers the users inline with what they need.<br />
<br />
In ENMS related systems, it is imperative that you present information in ways that empower users to understand situations and conditions beyond just a single node. While all of the wares vendors have been focused on delivering some sort of <a href="http://en.wikipedia.org/wiki/Root_cause_analysis">Root Cause Analysis</a>, this may not be what is REALLY needed by the users. And dependent upon whether you are a Service Provider or an Enterprise, the rules may be different.<br />
<br />
What I look for in applications and User Interfaces are ways to streamline the interaction versus being disruptive. If you are swapping a lot of screens, inherently look at your user. If they have to readjust their vision or posture, the UI is disrupting their flow. <br />
<br />
For example, if the user is looking at an events display and they execute a function as part of the menu. This function produces a screen that overcomes the existing events display. If you watch your user, you will see them have to readjust to the screen change.<br />
<br />
I feel like this is one of the primary reasons ticketing systems do not capture more real time data. It becomes too disruptive to keep changing screens so the user waits until later to update the ticket. Inherently, data is filtered and lost.<br />
<br />
This has an effect on other processes. One is that if you are attempting to do BSM scorecards, ticket loading and resource management in near real time, you don’t have all of the data to complete your picture. In effect, situation awareness for management levels is skewed until the data is input.<br />
<br />
The second effect to this is that if you’re doing continuous process improvement, especially with the incident and problem management aspects of ITIL, you miss critical data and time elements necessary to measure and improve upon. <br />
<br />
Some folks have attempted to work around this by managing from ticket queues. So, you end up with one display of events and incoming situation elements and a second interface as the ticket interface. In order to try to make this even close to being effective, the tendency is to automatically generate tickets for every incoming event. Without doing a lot of intelligent correlation up front, automatic ticket generation can be very dangerous. Due diligence must be applied to each and every event that gets propagated or you may end up with false ticket generation or missed ticket opportunities.<br />
<br />
Consider this as well. An Event Management system is capable of handling a couple thousand events pretty handily. A Ticketing system that handles 2000 ongoing tickets at one time changes the parameters of many ticketing systems. <br />
<br />
Also, consider that in Remedy 7.5, the potential exists that each ticket may utilize 1GB or more of Database space. 2000 active tickets means you’re actively working across 2TB of drive / database space.<br />
<br />
I like simple update utilities or popups that solicit information needed and move that information element back into the working Situation Awareness screen. For example, generating a ticket should be a simple screen to solicit data that is needed for the ticket that cannot be looked up directly or indirectly. Elements like ticket synopsis or symptom. Assignment to a queue or department. Changing status of a ticket. <br />
<br />
<b>Maps<br />
</b><br />
Maps can be handy. But if you cannot overlay tools and status effectively or the map isn’t dynamic, it becomes more of a marketing display rather than a tool that you can use. This is even more prevalent when maps are not organized into hierarchies. <br />
<br />
One of the main obstacles is the canvas. You can only place a certain amount of objects on a given screen. Some applications use scroll bars to enable you to get around. Others use a zoom in - zoom out capability where they scale the size of the icons and text according to the zoom. Others enable dragging the canvas. Another approach is to use a Hyperbolic display where analysis of detail is accomplished by establishing a moveable region under a higher level map akin to a magnifying glass over a desktop document.<br />
<br />
3D displays get around the limitations of a small canvas a bit by using depth to position things in front or behind. However, 3D displays have to use techniques like LOD or Level of Details, or Fog to enable only more local objects are attended to, otherwise it has to render every object local and remote. This can be computationally intensive.<br />
<br />
A couple of techniques I like in the 3D world are CAVE / Immersion displays and the concept of HUDs and Avatars. CAVE displays display your environment from several perspectives including top, bottom, front, left, right, and even behind. Movement is accomplished interacting with one screen and the other screens are synchronized to the main, frontal screen. This gives the user the effect of an immersive visual environment.<br />
<br />
A HUD or heads up display enables you to present real time information directly in front of a user regardless of position or view.<br />
<br />
The concept of an avatar is important in that if you have an avatar or user symbol, you can use that symbol to enable collaboration. In fact, your proximity to a given object may be used to help others collaborate and team up to solve problems.<br />
<br />
Next week, I’ll discuss network layouts, transitioning, state and condition management, and morphing displays. Hopefully, in the coming weeks, I’ll take a shot at designing a hybrid, immersive 2D display that is true multiuser, and can be used as a solid tools and analysis visualization system.Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com0tag:blogger.com,1999:blog-5590049888603730046.post-52932955098721004462010-07-11T12:03:00.002-05:002010-07-14T14:47:05.293-05:00ENMS Architecture notes...From the Architect…<br />
<br />
OK. You’ve got a huge task before you. You walk into an organization where you have an Event Management tool, a Network Management application, a Help Desk application, performance management applications, databases ad nauseum… And each becomes its own silo of a Beast. Each with its own competing management infrastructure, own budget, and own support staff.<br />
<br />
I get emails every week from friends and colleagues facing this, as well as recruiters looking for an Architect that can come in for their customer, round up the wagons, and get everything in line going forward.<br />
<br />
Sounds rather daunting, doesn’t it. Let’s look at what its going to take to get on track towards success.<br />
<br />
1. You need to identify and map out the Functional Empires. Who’s running what product and what is the current roadmap for each Functional Empire.<br />
2. You need to be aware of any upcoming product “deals”.<br />
3. You need to understand the organizational capabilities and the budget.<br />
4. In some instances, you’ll need to be strong enough technically to defend your architecture. Not just to internal customers but to product vendors. If you’re not strong enough technically, you need to find someone that is to cover you.<br />
5. You need to understand who the Executive is, what the goals are, and the timelines needed by the Corporation.<br />
ITIL is about processes. I tend to label ITIL as Functional Process Areas. These are the process areas needed in an effective IT Service. FCAPS is about Functional Management Areas. It is about the Functional Areas in which you need to organize and apply technology and workflow. eTOM adds Service Delivery and provisioning in a service environment into the mix as well.<br />
<br />
The standards are the easy part.<br />
<br />
The really hard part is merging the siloes you already have and doing so without selling the organization down the river. And the ultimate goal – Getting the users using the systems.<br />
<br />
The big 4 Wares vendors are counting on you not being able to consolidate the silos on your own. I’ve heard the term “Best of Breed” is dead and “A single Throat to Choke” as being important to customers. These are planted seeds that they want you to believe. The only way to even come close to merging in their eyes is to use only one vendor’s wares.<br />
<br />
When you deviate from addressing requirements and functionality in your implementation, you end up with whatever the vendor you picked says you’re gonna get. <br />
You need to put together a strategy that spans 2 major release cycles, and delineate the functionality needed across your design. Go back to the basics, incorporate the standards, and put EVERYTHING on the table. Your strategy needs to evolve into a vision of where the Enterprise Management system should be in the 2 major release time cycle. The moment you let your guard down on focus, the chances that something thwart movement forward, will present itself.<br />
<br />
Be advised. Regardless of how hard you work and what products and capabilities you implement, sometimes an organization becomes so narcissistic that it cannot change. No matter what you do, nothing gets put into production because the people in the silos block your every move. There are some that are totally resistant to change, evolution, and continuous improvement.<br />
<br />
And you’re up against a lot of propaganda. Every vendor will tell you they are the leader or market best. And they will show you charts and statistics from analysis firms that show you that they are leaders or visionaries in the market space. It is all superfluous propaganda. Keep to requirements, capabilities, and proving/reproving these functions and their usability. <br />
<br />
And listen to your end users most carefully. If the function adds to their arsenal and adds value, it will be accepted. If the function gets in the way or creates confusion or distraction, it will not be used.<br />
<br />
--------------<br />
Cross posted at : <a href="http://blog.sciencelogic.com/enterprise-network-management-systems-notes-from-the-architect/07/2010">http://blog.sciencelogic.com/enterprise-network-management-systems-notes-from-the-architect/07/2010</a>Dougie!!!http://www.blogger.com/profile/07830195240787895276noreply@blogger.com3