The art of selection the right software solution

Recently we started a project where the customer asked us to set up and manage a software selection process. The new software would replace the existing, outdated and lacking solution. One of the challenges involved in the assignment was that we needed to be open to both commercial off-the-shelf products and custom-made solutions.
Since the solution was part of their business-critical systems, simply browsing the internet and trying a few demos would not suffice.

In this blog post I will explain how we managed the selection process.

The big picture

We considered four categories: functional, services, technical and cost.
The functional category is very simple. We listed all the requirements based on several workshops with all the stakeholders. For each requirement, we indicated whether it was a minimum requirement or not.

The services category focused on project implementation aspects. We wanted to know how the vendor would perform the implementation in general and how he would continue to support the customer during the operational phase. In this category, the vendor needed to be very specific and create a tailor-made proposal.
When it came down to the technical elements we wanted to know all the capabilities of the solution. These included the different deployment models down to specific database settings and integration capabilities.

The last category, but certainly not the least, is the cost. We wanted to get the full picture. This includes the licensing model, support & maintenance for 3 years, training cost, upgrades and customizations.

Each category was given a weight. A good balance between the weights ensures that the best vendor ends up at the top of the list.

The left table above is an example of a well-balanced set of categories.
The right table however is heavily biased in favor of a vendor who scores high on the functional aspects but not on the technical aspects of the implementation.

Question everything

Once the categories and weights were defined we focused on the questions for each of the categories.
Because the customer wanted to explore the market and was open to both commercial off-the-shelf and custom-made solutions we had to tailor the questions accordingly.

To achieve this, we split most of our questions into two blocks. The first block was to learn as much as possible about the capabilities of the solution. In the second block we asked the vendor what setup or configuration he proposed for our customer.

For each question, we defined exactly which information we wanted to get. Next, we tried to avoid yes or no questions. These are often difficult to assess but can sometimes be useful.

The sample questions below demonstrate how we got both all the capabilities and a specific proposal.

“List the supported databases for your solution. Indicate what database you prefer and why.”

“Propose a database for our project and motivate your proposal.”

Answers to both questions gave us a good insight in the mind of the vendor. If he prefers one database but proposes a different one, his motivation can very well justify this discrepancy. Failing to answer these questions correctly also provides information about the vendor and that is what it’s all about.

Some questions are more important than others. So, just as with the categories we weighed each question. You can use the following table as a reference:

Essentially, the weight for a “decisive” question or an “important” is not that significant. In this table above the values are evenly spread.

In the table below the weight for a “decisive” question is twice that of an “important” question.
Vendors getting a good score for “decisive” questions will ultimately end up higher in the ranking than others.


Using the two questions above, and using table 3, the weighing could be as follows:

The overview of the supported databases is important information but the actual proposal from the vendor was crucial within our selection process.

Once all the questions have a weight we get the following table:

The unweighted total represents the sum of all the individual questions for a specific category.

If we had only used the unweighted total, the relative weight for a category would be directly impacted by the number of questions in each category.

If, for the functional category, the vendor scores 2825 he receives 20 out of a maximum 40 for that category. The same goes for all the other categories.

Scoring and motivating

Before we could start the real work, we needed to set up one more thing: how to score the answers.
Although possible, scoring each question per its individual weight is not very practical. Let’s assume you are evaluating a question with a weight of 100. How do you decide to give it 65 or 72? Having questions with different weights only complicates the problem.
A better option is to give a score from 0 to 5. A score of 0 means that either the question was not answered or the answer was very inadequate. A score of 5 indicates a perfect answer.

Let me demonstrate this with an example question.

“Describe how your solution supports integration with other systems.”

Answer from Vendor A:

“We integrate with any system you want. It really depends on what the customer wants. In the past, we have done many things such as direct database access, web services, file transfer.”

Answer from Vendor B:

“Our solution has an integration framework that allows us to expose interfaces to other systems via WCF and file transfer. The framework provides features such as monitoring, validation, error management and much more.”

I give the first answer a score of 1 and the second one a score of 5.

Based on the table above, Vendor A scores 20 out of 100 and Vendor B gets a perfect score: 100/100.

For some people, it might be obvious why Vendor A gets a lower score than Vendor B but others would argue that both gave an equally good answer and therefore deserve the same score.
This brings me to my next point: motivating your score.

Vendor B is more specific in his answer and demonstrates that he has put effort into the development of his overall solution.

Vendor A does not have a solution for integration with other systems. Each integration solution is built from the ground up which introduces additional risks and costs during the implementation phase.

Bringing these elements into the equation is important for the overall evaluation. Vendor A might score poorly on integration but he might excel at all the other elements. At least you know now where your project implementation risks are.

Wrapping up things

After carefully evaluating and scoring each answer we filled in the evaluation matrix.

Each vendor now had a score per category and a total score. This total score determined the vendor’s place in the ranking.

Looking at the results in table 8, Vendor A does a good job on all the categories except the technical one. One reason could be that he is using outdated technology.

Vendor B is the champion. He is the winner in all categories and thus he gets the highest ranking.
The last vendor, Vendor C, is a close second. He gets good scores but just not good enough to challenge Vendor B.

From the overall results, we could conclude that vendors with a commercial off-the-shelf product often scored good on both the cost and services category but sometimes lacked in functional requirements. Vendors proposing a custom-made solution mostly got good scores on the functional category but performed less in the services and cost area.

Before we made the final decision, we invited the top raking vendor for a demonstration. This allowed us to confirm our selection and it gave us the opportunity to make a second evaluation of the vendor. If something should come up during this phase we could still go to the second in the ranking and repeat the process.

Johan Celis