Dante's Inferno in IT...

We’ve all been through Dante’s Inferno. I know I have.

I was at one place where they developed an in house SNMP agent to be deployed on all managed systems. Because of the variety of operating systems supported and the amount of bugs in the software, many customers hated the agent and would request that the agent be taken off of their systems. But because the agent was “free”, it kept living in a miserable existence. Turns out there were many different versions supported and deployed. Additionally, the design of the agents’ sub-agents capabilities deviated significantly from industry standards such that it hamstrung the open source openness of the base agent.

Dante’s Inferno came in when you had to deal with the agent capabilities as an Architect. Saying anything negative related to the agent was Heresy. The Manager that owned the Agent would resort to Anger, Fraud, and Treachery in order to divert any negative attention to their baby. Part of the reason for hanging on to this Agent was that the ownership of more developed products promoted the Manager’s gluttony and greed. It was his silo of management technology.

While I was busy circumnavigating the Machiavellian Urinary Olympics, that Manager was working hard to put me in Limbo. Any requirements that I put forth were immediately in negotiation such that I could not finish requirements. Finally, in total frustration, I sent out a Final version of the requirements. Doing this sent the Manager into a frenzy of new Machiavellian Urinary Olympics such that my actions were elevated all the way up to a Sr. VP. Alas, I could not overcome the Marijuana Principle of Management (Harder you suck ,the higher you get!)

I left shortly afterward. So did several of my coworkers. Some are still there. All with the common experience that we’ve all been through Dante’s Inferno.

Lessons for the Architect :

• Be careful in calling someone’s baby ugly. Given the “embeddedness” of a given politician, there may be some things you cannot change. 

• Some Silos can only break down through years of pain and years of continued failure.

• In moving toward Cloud computing models, some folks may have an inclination to bring with them all of the bad habits they have currently.

• If a person has only ever seen one place, they may not understand that success looks totally different in other places.

• There is a direct cost and an indirect cost to supporting internally developed products. If your internally developed product is holding back progress and new business, it is a danger sign…

As an Architect, be wary of consensus. “Where there is no vision, the people perish. (Proverbs 29:18)”

Politically based Engineering Organizations

When an organization is resistant to change and evolution, it can in many cases be attributed to weak technical leadership. Now this weakness can be because of the politicalization of Engineering and Development organizations.

Some of the warning signs include:

- Political assasinations to protect the current system
- Senior personnel and very experienced people intimidate the management structure and are discounted
- Inability to deliver
- An unwillingness to question existing process or function.


People that function at a very technical level find places where politics prevalent, a difficult place to work.

- Management doesn't want to know whats wrong with their product.
- They don't want to change.
- They shun and avoid senior technical folks and experience is shunned.

Political tips and tricks

- Put up processes and negotiations to stop progress.
- Random changing processes.
- Sandbagging of information.
- When something is brought to the attention of the Manager, the technical
person's motivations are called into question. What VALUE does the person bring to the Company?
- The person's looks or mannerisms are called into question.
- The persons heritage or background is called into question.
- The person's ability to be part of a team is called into question.
- Diversions are put in place to stop progress at any cost.

General Rules of Thumb

+ Politically oriented Supervisors kill technical organizations.
+ Image becomes more important than capability.
+ Politicians cannot fail or admit failure. Therefore, risks are avoided.
+ Plausible deniability is prevalent in politics.
+ Blame-metrics is prevalent. "You said..."

Given strong ENGINEERING leadership, technical folks will grow very quickly. Consequently, their product will become better and better as you learn new techniques and share them, everyone gets smarter. True Engineers have a willingness to help others, solve problems, and do great things. A little autonomy and simple recognitions and you're off to the races.

Politicians are more suited to Sales jobs. Sales is about making the numbers at whatever cost. In fact, Sales people will do just about anything to close a deal. They are more apt to help themselves than to help others unless it also helps themselves. Engineers need to know that their managers have their backs. Sales folks have allegiance to the sale and themselves... A bad combination for Engineers.

One of the best and most visionary Vice Presidents I've ever worked under, John Schanz once told us in a Group meeting that he wanted us to fail. Failure is the ability to discover what doesn't work. So, Fail only once. And be willing to share your lessons, good and bad. And dialog is good. Keep asking new questions.


In Operations, Sales people can be very dangerous. They have the "instant gratification" mentality in many cases. Sign the PO, stand up the product, and the problem is solved in their minds. They lack the understanding that true integration comes at a personal level with each and every user. This level of integration is hard to achieve. And when things don't work or are not accepted, folks are quick to blame someone or some vendor.

The best Engineers, Developers, and Architects are the ones that have come up through the ranks. They have worked in a NOC or Operations Center. They have fielded customer calls and problems. They understand how to span from the known to and unknown even when the customer is right there. And they learn to think on their feet.

Dangerous Development

Ever been in an environment where the developed solutions tend to work around problems rather than confronting issues directly? Do you see bandaids to issues as the normal, mode of operation? Do you have software that is developed without requirements or User Acceptance testing?

What you are seeing are solutions developed in a vacuum without regard to the domain knowledge necessary to understand that a problem needs to be corrected and not "worked around". Essentially, when developers lack the domain knowledge around identifying and correcting problems in areas outside of software, you end up with software developed that works around or bandaid across issues. Essentially, they don't know how to diagnose or correct the problem, diagnose the effects of the problem, or in many cases, even understand that it is a problem.

In some cases, you need strong managerial leadership to stand up and make things right. The problem may be exacerbated by weak management or politically charged environments where managers manage to the Green. And some problems do need escalation.

This gets very dangerous to an IT environment for a multitude of reasons including:

• It masks a problem. Once a problem is masked, any fix to the real problem breaks the developed bandaid solution.
• It sets an even more dangerous precedent in that now its OK to develop bandaid solutions.
• Once developed and in place, it is difficult to replace the solution. (It is easier to do nothing.)
• It creates a mandate that further development will always be required because of the work arounds in the environment. In essence, no standards based product can no longer fulfill the requirements because of the work arounds.

A lot of factors contribute to this condition commonly known as "Painted in a Corner" development. In essence, development efforts paint themselves into a corner where they cannot be truly finished or the return on investment can never be fully realized. A developer or IT organization cannot divorce itself or disengage from a product. In effect, you cannot finish it!

A common factor is a lack of life cycle methodology in the development organization. Without standards and methodologies, it is so easy for developers to skip over certain steps because of the pain and suffering. These elements include:

  

• Requirements and Requirements traceability
• Unit Testing
• System Testing
• Test Harnesses and structured testing
• Quality Assurance
• Coding standards
• Documentation
• Code / Project refactoring
• Acceptance Testing.

 This is no different from doing other tasks such as Network Engineering, Systems Engineering, and Applications Engineering. The danger is that once the precedence is set that its OK to go around the Policies, Procedures, and Discipline associated with effective Software Development, it is very hard to reign it back in. In effect, the organization has been compromised. And they lack the awareness that they are compromised.

  

What do you do the right the ship?

  

Obviously, there is a lack of standards and governance up front. These need to be remedied. Coding standards, software lifecycle management techniques need to be chosen and implemented up front. Need to get away from Cowboy code and software development that is not customer driven. Additionally, it should be obvious that design and architecture choices need to be made external to this software development team in the foreseeable future.

  

Every piece of code written needs to be reviewed and corrected. You need to get rid of the bandaids and "solutions" that perpetuate problems. And you need to start addressing the real problems instead of working around them.

  

Software that perpetuates problems by masking them via workarounds and bandaids is a dangerous pattern to manifest in your organization. Its like finding roaches in your house. Not only will you see the bandaid redeveloped and reused over and over again, you have an empowered development staff that bandaids versus fixes. Until you can do a thorough cleaning and bombing of the house, it is next to impossible to get rid of the roaches.

Sometimes, development managers tend to be promoted a bit early in that while they have experience with code and techniques, exposure to a lot of different problem sets segregates the good development leaders from the politicians and wannabes.  Those that are pipelined do not always understand how to reason through problems, discern the good from bad techniques and approaches, and lead down the right paths.  Some turn very political because it is easier for them to respond politically than technically.

What are the Warnings Signs?

Typically, you see in house developed solutions that are developed around a problem in functionality that would not normally be seen in many different places.  This can be manifested in many ways.  Some examples or warning signs include:

• Non-applicability of commercial products because of one off in house solutions or  applied workarounds.
• Typical terms like "thats the way we've always done this" or "We developed this to work around..." arise in discussions.
• You see alot of in house developed products in many different versions.
• In house developed products tend to lack sufficient documentation.
• You see major deviations away from standards.
• No QA and code review is only internal to the group.
• No Unit or System test functionality available to the support organization.
• In house developed software that never transitions out of the development group.
• Software developed in house that is never finished.
• You get political answers to technical questions.

A lesson here is that it does matter that the person you setup to lead, have exposure to a lot of different problem sets, SDLC methodologies in practice and not just theory, and they have some definite problem reasoning skills. A Politician is not a good coach or development leader in these environments.

In Summary

One must be very careful in the design and implementation of software done in house.  If wrong, you can quickly paint development and the developed capabilities into a corner where you must forklift to get functionality back.   And if you're not careful, you will stop evolution in the environment because the technical solutions will continue to work around instead of directly addressing problems.