Forums for the Business Analyst

 
  Modern Analyst Forums  Business and Sy...  General  Evaluating Custom Software Development Companies
Previous Previous
 
Next Next
New Post 1/18/2012 8:44 AM
User is offline jsh
1 posts
No Ranking


Evaluating Custom Software Development Companies 

 I am consulting to a company that needs to purchase Medical billing software that will also support payroll given unique business rules. I worked through the requirements with the company and authored a detailed RFP which has gone out to Vendors of Off the Shelf Systems. The client wants to go with custom software if we can't find an adequate OTS solution and for a variety of reasons, I fear there's only a 60/40 chance that an OTS solution will emerge. My question: Does anyone know of good resources/methodologies for evaluating custom software development companies. I could send out a similar RFP to the one above but I fear the companies in question would just say, "yeah, we can do that" and I wouldn't have a good way (outside of asking for references, financial stability, project managment experience...) to choose the few companies to focus on. Alternatively, I've thought about sending out an RFI that would just focus on organizational strengths and experiences then provide each company with my requirements (in the form of written "The system shall...", user stories and diagrams) and ask each company to provide a white paper indicating how they would solve the problem (including communication processes, software development processes... I would then rate them based on who I believed and get them to demo similar software and follow up on references. In this latter approach, I would invite (and pay expenses?) for them to come to us so we could see how they do requirements validation. Any thoughts on alternative approaches or modifications/pitfalls to the approaches I laid out above would be greatly appreciated. 

 
New Post 1/19/2012 7:06 AM
User is offline Jarett Hailes
155 posts
6th Level Poster




Re: Evaluating Custom Software Development Companies 

 On the custom software development side most clients I work with elect to build teams by selecting individuals from various companies, rather than picking a team from one company. While they lose the potential synergy that a team with a long track record has together, and it often takes longer to build the team, they often find that they can find a higher level of overall talent this way.

I have been involved in choosing vendor teams for non-software projects before, for example a market research team. Our requirements, scope and preferred timeline were included in an RFP. We developed an evaluation criteria matrix to compare the variety of responses that we received, with no emphasis on checking off our requirements (as you indicated they can all say 'yes we can do that'). In the end we had 20 evaluation categories. At a high level we wanted to have the vendor demonstrate:

  • Experience in our industry, not just in doing the work in other industries
  • The key resources on the team had relevant experience and were going to be appropriately dedicated to our project (as opposed to working on multiple projects and not giving our project sufficient attention)
  • The vendor could articulate their understanding of our objectives and needs through their response. If people just regurgitated what we said we marked them low, if they asked insightful questions, described methodologies that would work well in our situation, etc. they scored much higher.
  • They understood the scale and scope of work by developing a work breakdown structure.
  • The deliverables proposed were appropriate given the type of work we were looking for them to do.
  • They identified risks based on our RFP and described how they can mitigate those risks (we didn't explicitly ask for this in the RFP but gave high marks to those who identified risks on their own).
  • Previous references who were either in our industry or where the work performed was very similar, and who we could contact to see how well they had done.
  • That they do quality work by evaluating the quality of the response. Spelling, grammar, visual style, etc. of the response were evaluated. 

Based on your situation I think many of the above criteria could fit well into your evaluation. I think going a full RFP is more appropriate; in my experience you don't get enough detail in an RFI and will end up having to issue an RFP afterwards anyway. After we evaluated our responses we short-listed candidates (3) and had them fly in (at their own expense) to give us a presentation that filled in some of the blanks that we felt were missing from their responses, and answer some questions that came up based on the responses.

 

 
New Post 1/20/2012 2:59 PM
User is offline Sandy
74 posts
8th Level Poster




Re: Evaluating Custom Software Development Companies 

Basically, there are 3 ways to acquire a software system:  1) buy a commercial-off-the-shelf (COTS) system - which could be a "COTS-hybrid" (blend of commerical product + custom-code); 2) contract with a vendor to build the solution to your requirements; or 3) contract with a vendor (or multiple vendors) to provide a software development team (which includes the PM, BA's, testers, etc as well as developers) to build the software to your requirements.  Option 3 is the one that Jarett suggested.  The main difference between option 2 and option 3 is the amount of involvement from your client organization.  If you bring in a contract team, your client would need to know what resources are needed to build the solution you want - and would need to be more involved in helping manage that team.  So I'll reply based on option 2, as the simpler one to start with.

Software development companies build to your requirements - they would not have a solution to demo (since they have to build yours), and a white paper or RFI would only give you a bunch of marketing material about the company.  Evaluation of the organization's capability and the quality of their work is just one part of the RFP.  The RFP needs to tell the vendors exactly what you want from them - the application requirements (which the vendor uses to scope out the application that they will build), all of the deliverables / documents that you want them to provide (requirements documents, design documents, test documents, implementation documents, as well as project management documents), what services you want from them (eg. project planning and management, requirements, design, development, testing, implementation, possibly data conversion from a legacy system, and very importantly - maintenance or support services), as well as other content for service level commitments, post-implementation warranty services, user & technical documentation, etc., etc..

The RFP should also have sections for the vendor to describe how they will oversee the project, how they will report status, how they will ensure quality, assumptions made that effect their services & price, among others.  It should also address intellectual property rights (giving your client all rights to the code & products provided, and covering any 3rd-party or proprietary components used in building the solution), service level commitments, etc.  You can ask the vendor to include resumes of the resources that will be on the project, and to provide references both for the company and for each of the proposed resources.  And last but definitely not least, the RFP has to be written in a way that allows you to compare all products, services, and prices across all the proposals received.

Since the RFP usually forms the basis for the subsequent contract with the selected vendor, it should go through a legal review as well.

So yes - writing this type of RFP and the correspondig evaluation plan is complicated and can be quite challenging if you've never done it before.  It may be worth hiring a consultant who speciallizes in preparing and evaluating RFP's of this type - it can save a lot of time & trouble in the end.  You may be abe to find some example RFP's on the internet.  I'd be happy to chat offline as well - just reply back if you'd like to follow-up with that.

Sandy

 
New Post 1/30/2012 4:30 AM
User is offline Craig Brown
560 posts
www.betterprojects.net
4th Level Poster




Re: Evaluating Custom Software Development Companies 

 Jarett Hailes wrote

 On the custom software development side most clients I work with elect to build teams by selecting individuals from various companies, rather than picking a team from one company. While they lose the potential synergy that a team with a long track record has together, and it often takes longer to build the team, they often find that they can find a higher level of overall talent this way.

Interesting statement. I agree many people operate this way, but I am sceptical of the benefit being real.  Formed teams with stable systems will outperform new teams with yet to be established systems almost every time.

 
New Post 1/30/2012 12:43 PM
User is offline Jarett Hailes
155 posts
6th Level Poster




Re: Evaluating Custom Software Development Companies 

I've seen  both work well and work poorly. I've seen thrown-together teams work well when the project is smaller (less than 10 people, 12 month timeframe). Sometimes the vendors have experienced a lot of churn within their teams so in reality their groups are not stable either.

I agree that having a well-proven system improves performance, regardless of how the team is formed. If the client has a good project methodology this can really even out the learning curve for a thrown-together team.

 

 
Previous Previous
 
Next Next
  Modern Analyst Forums  Business and Sy...  General  Evaluating Custom Software Development Companies

Community Blog - Latest Posts

Is Agile a reason to avoid documentation? I bet this question shows up again and again while working with product requirements. On one side, we have got long specifications, complicated diagrams, mystical technical design, too many prototypes and pretty obvious for engineers user guides (do we really need so much?). On the other side, can we actual...
The cloud-native application development has helped enterprises all around the globe reduce time-to-market, enhance performance, and develop agility and flexibility. Several enterprises are achieving these results by migrating their systems or traditional monolithic applications to the cloud. But to gain from the real benefits of cloud technology, ...
So you’ve found the perfect time and place to study and you’re ready to finally get some work done. You’ve pulled out your laptop, your textbook, and your notes, and four different highlighters. After five minutes of reading your textbook, you start zoning out and thinking about puppies. Then, you go on Tumblr and look at cut...

 



Upcoming Live Webinars




 

Copyright 2006-2021 by Modern Analyst Media LLC