IT systems – common problems

Troublesome IT systems

Basically every Chief Information Officer has somewhere in the back of mind disturbing thought about one (or several) troublesome IT system. This thought stays dormant, but unfortunately it returns from time to time, evoked by some problem, failure or another intrusive question from business department about new functionality. But even, if none of these problems happen, the awareness that it may appear, often does not allow you to sleep peacefully.

Sources and types of problematic systems

Troublesome IT systems have different origins and different nature. However, we can feature several most common groups. The classification is not that strict and some systems can be categorized into several groups, but nevertheless the grouping makes easier to understand with what exactly we have to deal in a given case.

Systems which have never worked well

This kinds of systems simply have never worked properly and they constantly cause problems. Maybe users even got used to some faults or even developed the work methodology to bypass them. But it doesn’t change the fact that such systems can chill the blood in IT workers’ veins, who have some idea of what can be happening “under the hood”.

The old systems

Similar is the case with old systems, which once worked without fault, but have been deficient recently. Some operations take more and more time, sometimes the strange error messages come up, and it is becoming increasingly necessary to reset the server. Under normal circumstances, the provider would dealt with such problems. But in case of old systems, the providers doesn’t always want to fix the problems, and even sometimes they don’t exist anymore. If we are lucky, the providers agree to resell or even give away the source codes. But what if even with source codes, the system fails to boot up. Or what if the system built from source codes turns out to be different from version used in production? And the developers on the labor market aren’t burning to work with archaic technologies.

The “exotic” systems

The next category are systems written in “exotic” technologies. Also here the main problem is lack of specialists who know the technology, no matter whether we talk about language, platform or framework. The problems usually come up when the last expert leaves the company. Often, this specialist is a main creator of such “exotic” system, who in its time was an avid enthusiast of this solution. Unfortunately, when leaving, the developer took with self all the knowledge about the system operation.

The abandoned systems

Abandoned systems are quire specific category — once started but not finished for some reason. They can be either systems delegated to an external provider or created by own team. In the first case, with well-structured agreement, only delays can be a problem, although the system has often been paid partially or even fully, and it may be very difficult or even impossible to get reimbursement. In the case of a system created by one’s own strength, the prestige of technical department and its boss is often at stake.

The system can be abandoned under various circumstances, but most often the project can be classified into one of the following groups:

• The projects turned out to be technically difficult (in extreme cases, the goals can be impossible to achieve) while the difficulties lies in solving a complicated problem or in the appropriate selection and implementation of computational algorithms. Usually the main problem is system failure (slow operation). Improving performance requires specialized skills to locate system bottlenecks and to assess complexity and possibly optimize used algorithms.
• The project is extensive, which means that it has very large codebase and it can be very difficult to handle only by one person. The wrong design brief can cause problems in modifications and upgrades. For example, it quite often turns out that making one little change in one of the system modules results in errors or unexpected operation of other modules. Searching errors in this case can be very difficult and often requires deeper organization of the project code.
• The project falls outisde the parameters of dev team’s technological briefs. This usually happens in the case of data exchange or integration with other IT systems. If the external system only impose the solution technology and even when it is unknown technology for the team, the problem isn’t yet big. The real difficulties start when integration is supported in theory only by external system or when the documentation isn’t complete (or is gone) and there are limited possibilities to enforce it from the provider.
• A separate and quite unique type of abandoned system are delays leading to the inability to deploy the system with the established plan. It may be due to incorrect estimation of implementation time, but also due to “surprises” that can happen at every stage of programming project. These include not only an underestimation of time consumption, but also not completely agreed specification with the client or the difficulty of the topic. Unlike the previous situations, the team can always finish the project, but unfortunately it is know that without extra help deadlines will be exceeded.

Risks

As we mentioned in the beginning, troublesome system is like the sword of Damocles hanging over the Chief Information Officer’s head. At any time, it can stop working properly or even at all. Besides this, when the users see some little problems or anomalies, they begin to doubt in the correctness of the program, especially when the results returned by program are far different from expectations. Then, the pressure from business departments starts to pop up, but the problem doesn’t vanish. The IT team starts to feel insecure.

In such every case of problematic system, the basic source of problems is lack of competent person (now or at the stage of design and creation). It is highly inconvenient situation, when there is no one in the team who knows the system and how it works. There is no one who could establish in reasonable time, where the error came from or how to successfully (and quickly) get rid of it.

In this situation, the system stops responding to changing business and organization conditions, if we talk about internal system. Unfortunately it get worse, when it comes to systems for external clients. The delays in implementation often lead to contract penalties and damage to prestige. Not to mention about the costs resulting from the extended implementation.

Change, repair or maybe abandon?

When we have a troublesome system, we know what it means and what it can cause. Before solving problems, analyze every possible directions. The problematic system can be:

• repaired,
• replaced,
• rewritten,
• abandoned,
• ignored.

Repair

Repair can mean many things, depending on the problem’ nature. It can be improved implementation, configuration change, the optimization of the inefficient algorithms or completion of the abandoned project. Usually it means some investment expenses, but when it comes to troublesome project often it is impossible to establish the amount of them, at least until considerable time is invested in diagnosis. Often, it takes 90% of time to look for error, and 10% of time to remove it. It is the most practiced direction.

Rewriting the system

Probably many times you have heard that it is better to rewrite the system from the beginning than save. Indeed, it is a commonly practiced to create new system, which will replace the problematic one. It usually means using different technology in implementation.

Depending on situation, it can more or less beneficial solution. If the system is created in currently widely used technology, usually rewriting doesn’t make sense (except for exceptionally poorly designed systems). In the case of old or niche technologies it is worth to run profit and loss statement, preceded by examining the problems and estimating the costs of the repair.

It is necessary to warn that in many times, inexperienced developers try to improve the efficiency by changing the technology without previous research about the sources of inefficiency. Newer and quicker programming language or “more modern and extremely efficient library” are nothing if the real problem with efficiency wasn’t properly diagnosed. It often turns out that good specialist is able to quickly repair the system without high costs and many changes.

On the other hand, the repair only postpone the real problem for later time, because in longer perspective problems with lack of experts will pop up. That’s why in some cases, restoring the system to good state is a first step to rewrite for newer technology. In the end old (and properly working) system is the best possible form of specification.

Abandonment

No system is easily abandoned. The abandonment means wrong decisions in the past or helplessness in the face of actual problems. Nevertheless, it is worth to mention this script in analysis of possibilities.

In our work, we often dealt with systems implemented without reliable analysis of needs. It was not often dictated by pragmatics. Sometimes “the head” made the decision about “updating”, which was completely useful for some business processes. At the end, the employees unwillingly (or not at all) used the system which didn’t fulfill its function.

What is more, in such cases the system can work properly and it will still be seen as malfunctioning or incorrectly made, because it doesn’t bring the expected results. Only careful analysis of business processes can show that the system doesn't help at all, but even bother workers.

Usually it takes some time to flower to the decision about abandonment such solution. Knowing that maintaining an unsuccessful idea involves fixed costs helps to make a decision.

Ignoring the problem

The last possible direction is to ignore the problem, which doesn’t involve anything because “it happens by itself”. We don’t have to explain, that it is the worse possible decision to make and it significantly increase risk of problems. If for some reason we are not able to make decision about further steps, it is best to reach out for specialist’ opinion.

Solution – what are the options?

If the decision about repairing or rewriting has been made, we have to decide about course of action. Basically, we have two possibilities, every with a set of variants:

• own team
• external provider.

Your own team

In the case of chronic troubles with IT system, the first step of managers is thought “let’s find a person/people to do it”. It often means hiring a new person.

Employment

The process of employment usually starts from establishing who we want. In the case of problematic systems thinking generally follows the pattern “we have a problem with a system written in X technology, so we need to find a senior developer with experience in this technology”. What is more, there are usually more or less critical parameters related to experience with the programming tools used in the company, project management methodology, etc. In addition, the experience in class K systems is welcome, where K means system with which we have a problem.

Well, it would be a good solution, but usually it doesn’t want to work out. The decision to employee a specialist faces some problems. Real experts are always hard to find on labor market, they are often fickle and have huge expectations towards employer. The process of finding the potential good employee can last long time. Besides this, the chance of getting someone who knows our technology and systems may be vestigial.

The HR department or external recruitment company not only has limited elbowroom. The recruiter is often not able to understand the actual need of IT department. The initial selection is often passed through by people who actually doesn’t meet the rules, and later they need to be counterchecked by heads of an IT department.

Unfortunately, the real skills and knowledge of new hired employee are verified after some time of work. Probably every Chief Information Officer had an experience with promising candidate, who despite its great CV had difficulties with basics matters or was great technically, but cooperation did not go so well.

Employing new person is sometimes a good idea, but unfortunately the process of looking for perfect candidate and verification of its skills can take much valuable time. Hiring incompetent can bring many problems. And even if we were lucky and found our “perfect one”, we have to keep this employee in company, which isn’t always a simple thing. Sometimes ones would like to hire specialist for short period of time (maybe because of costs), but not every candidate will be pleased to work in such terms.

Brokers (HR and body leasing)

The market niche generated by mentioned above problems is domesticated by body leasing and HR companies, which spring up in the last years. They allow to quickly suplement your dev team with new developers.

Their services are basically similar – the have a wide range of experts with different level of skills and experience. The difference is that an employee recruited by the body leasing agency is formally employed there and is accounted for the time worked for the client. Intermediaries do not always solve the problem of finding the right person. The responsibility for verification and choice is often on the client side.

Nevertheless, the body leasing companies offer several benefits like easier resignation from a given person or (at least theoretically) the possibility of quick replacement or ease of contracting for a short period of time.

The practice shows that, when the number of workers is more important than experience, the best option is intermediation of people with junior and middle experience. Unfortunately establishment officers in HR and body leasing companies has the same difficulties to find high performance experts like others. Therefore, in the case of troublesome systems the use of this source is very limited.

External providers

Sometimes due to organizational reasons, especially in big business, it easier for Chief Information Officer to explain paying the invoice than hiring another person. But it is only a one reason why is it worth reaching out to third party companies. The business character of such cooperation is also important because we can expect more flexibility, better terms and higher professionalism.

There are plenty of IT companies on the market. Their offers and models of work are often highly different. How to pick the right one that will help to solve problems? There are some issues, that are need to be highlighted.

Programming outsourcing

The common way to meet the needs for programming work is outsourcing. It basically means an external programming company who creates even the most demanding project for your company. The supplier usually doesn’t provide only workforce in the shape of dev team, but also handle the projects management. Projects are usually implemented withing predetermined range at a predetermined rate.

This kind of service works great for the projects, where it is quite easy to establish what work needs to be done, especially in solutions created from scratch. Unfortunately, in situation when we already have a system which causes the problems, the programming outsourcing doesn’t work. It results from the fact that the scope of work is usually not obvious, and the diagnostics takes a lot of work.

CV Industry

Companies looking for an outsourcing provider quite often come across many options that boast of hundreds, if not thousands of developers, at hand. Unfortunately, most of these options usually turn out to be little more than a “CV factory” providing little more than a body leasing service. They have a large database of people about whom they know nothing. And because they check their competencies only casually, they don’t take any responsibility for their client’s choice of developers.

Competence outsourcing

Among the others, due to the need to separate body leasing from outsourcing, the term of competence outsourcing has started to come up recently. The word “competence” has key meaning in this case. Although the payoff model is the same as in body leasing, the offer and operating model are different. To put it simply, it can be said that body leasing usually place importance on the number of developers in offer, while the competence outsourcing focuses on quality.

The providers of competence outsourcing usually have fewer people, but with higher qualifications. They are often high-class specialists who have done many projects and can boast of extensive experience. They are supposed to work where the typical body leasing will not work.

The competence outsourcing can be quite (sometimes even the best) solution, when we deal with very difficult projects, especially when we have to take into account the possibility of hourly billing. Thanks to this service we get the expert who can be placed in the middle of project with established purpose. As long as we find a right man for our task, we can assume that the goal will be achieved.

Team leasing

Team leasing is the extension of the competence outsourcing and body leasing concept. Team leasing means company which provides a whole dev team. Different providers understand this service differently, but team leasing is much more than body leasing of group of programmers. There must also be someone with managerial competence among them.

The manager is key person in team leasing. This man will be responsible for all work, will be a team leader and also will set goals and means of achieving them with the client. The manager often will choose coworkers to given project or change them during. After all, not all available competencies in team are needed.

The smooth ability to model the team due to needs is the biggest advantage of this service. It is especially important in case of troublesome systems which can surprise anytime.

The specificity of difficult programming problems

We have already discussed programming services available on the market and their usefulness in the case of troublesome systems. It is worth to point out some important aspects related to solving difficult programming problems.

Extending the team

Most of all, extending the dev team doesn’t always bring expected effects. Unfortunately, in practice the larger team means the slower work progress. There are more and more problems with communication, the division of duties becomes unclear, the number of misunderstandings and mutual animosities increases in the team.

Defining a problem

What if it’s difficult to determine what exactly problem is to solve? Against all appearances, this isn’t a rare situation. Complex systems that use many technologies are hard to grasp, and sometimes very resistant when it comes to debugging. How to look for solution, when there is no one who could define a problem? Defining the problem properly can be the most difficult part of solving it.

Identifying needs

What is more, it’s not always clear choice, which specialist can exactly help us when it comes to troublesome systems. Reflexive “we need a senior developer in X technology” doesn’t always work.

We used to have clients who defined their needs quite incorrectly. They were looking for something completely different than they really needed. It’s hard to blame them. It’s typical of market to not clearly define solving difficult IT problems services or repairing troublesome systems services. Solution seekers often don’t know what to type in the search engine to get a little closer to finding the right offer.

The technology doesn’t solve the problem

One of the elements of incorrect identifying needs is determining the technology of the solution. It comes from quite common but naive thinking, that technology solve IT problems. Nothing could be further from the truth. The person always solve problems. Even the most modern programming language or framework won’t help, unless some incompetent programmer starts to work on it.

What to look for in provider?

And now we get to the point. It isn’t hard to guess that as a provider of team leasing and competence outsourcing services, we will direct your attention in this direction. If you are actually facing a troublesome system and have come so far, there is a good chance that you will agree that this is the right direction. But only if you find a competent provider.

How to find the best tailored provider?

As it was mentioned earlier, finding a provider who specialize in troublesome systems can be very difficult. Typing in the search engine terms such as “competence outsourcing” or “software development company” in general leads to the situation when we are drowning in the results (and adds) of companies providing not the services we are looking for. For sure, we will also find providers who will say that they have solution for our problem. And probably half of them will have. But if we choose wrong, we will risk for costs and delays. So choose wisely. What should you pay most attention to?

Actually the first and the best way of initial verification of potential provider is its website. You can find a lot of useful information on it. Providers boast about their references, used technologies, models of project management etc. With right caution, it can also be assumed that the best providers appear highest in the search engine results (it’s worth differentiating between search result and advertising).

In our opinion, from practical point of view the most valuable are two categories of information: technologies and experience.

Versatility or specialization?

Most of IT companies quite clearly specify in which technologies (or technology stacks) they work. Often it is very particular set, supported by certificates or tittles of official partnership. Such a company can be expected to perfectly master a given technology and usually we will not be disappointed with such belief.

However, it is worth considering whether the system or problem we are currently dealing, really matches to the profile of such a specialized company? Will this specialization not be too restrictive? Will provider constantly operating in a given technology not be automatically looking for solution in it, narrowing the space of possible solutions by itself?

Technological versatility comes (at least in theory) with bigger flexibility in thinking about problems and possibilities to solve them. A provider working in many fields can be expected to adjust solution to problem, not a problem to solution.

Does the lack of specialization mean shallow knowledge of technology? Of course it does, but not always.

Despite the common opinion, the knowledge of concrete technology is secondary when it comes to troublesome systems. For the vast majority of problematic projects the biggest difficulty isn’t lack of knowledge of language syntax or specific library.

Look at this from this point of view: Is it easier to enter into chaotically written and undocumented project or into good written and perfectly documented technology? It is the best to look for someone who knows a lot of technologies and can easily and without problem enter into new one if it is necessary. There can be plenty X technology specialists on the market (or maybe in the company) but problems are still not solved.

Okay, we know that it is worth choosing among “multi-technology” providers. How to approach this? You may assume, that not every provider who meets this criteria will be good at solving unconventional problems? So what else to consider?

Experience and references

Actually only reasonable criteria is experience. It is worth to look at the list of clients of potential provider. Most of them boast about them on website.

Of course list isn’t enough, because it doesn’t inform about the scope of problems and company’ work. Here can be useful references and success stories.

It is not necessarily about finding a provider who dealt with identical problem like our. Sometimes it is impossible to find such supplier. Finding provider who dealt with various problems is better approach. Such company can be expected to have wide thinking and great competences to solve unusual problems. If among technologies or projects there are ones that can be called atypical or even exotic, it is better.

It is worth to remember that we won’t know everything from other clients. Customers often require to maintain confidentiality. And even if there are no such provisions, it is sometimes a matter of ethics and good business practices not to publicly announce problems that the client has faced. Even if success story presents an anonymous company, often an attentive reader can discover who is about.

The conversation

Definitely, the surest way to verify the potential provider is phone call, or even better a personal meeting. What is more, it should focus on your troublesome system. Good provider with wide experience will certainly show us other similar problems faced in the past. If the part of them have turned the corner, then we can be sure that we have chosen right.