Tell me more ×
Answers OnStartups is a question and answer site for entrepreneurs looking to start or run a new business. It's 100% free, no registration required.
up vote 2 down vote favorite

I am considering buying some software, which would do very advanced calculations/simulations for me. If this software provides wrong results, I could go bankrupt.

I specifically did not want to mention what kind of software I am talking about, as I am interested in a general answer.

My question How can I judge the reliability of commercial software?

My best ideas so far are:

  • Version number (horrible)
  • Brand name (horrible)
  • Large or small company developing the software (does not say much either)
  • References (it does not say how much the large company depends upon the software creating correct results)

What other factors could I use?

EDIT: A couple of you have mentioned testing, which is fine, but in my case it could take months of work to do more than basic testing. I have considered hiring someone do to this, but it seems like I am really doing the job of the software company. In addition, would only be able to do black box testing.

share|improve this question
@Tim The existence of my startup (and others in a similar situation) will depend on the decision, so it should be on topic. – David May 16 '11 at 15:00
1  
@Tim this is a valid question since many small businesses lack the resources to do a full analysis of software options, and need to know the best way to evaluate a purchase for quality and ability to meet their needs. – Elie May 16 '11 at 17:42
Well, small businesses also need to evaluate cars, copy machines, etc. Large businesses also need to evaluate software - as do individuals. It is not specific to startups at all. It is specific to this person's needs, and he does not give enough details to let anyone help. If the software is that important and could bankrupt the company then it is very specialized. This is not a good quality question IMO. – TimJ♦ May 16 '11 at 17:48
1  
Thank you @Tim for share with us (especially @David) your reasons for flagging. this type of immediate feedback is really helpful. I have opened a question on Meta to discuss the issue that you raise. That is the appropriate location for the continued conversation. meta.answers.onstartups.com/q/483/8791 – Joseph Barisonzi May 16 '11 at 20:10
4  
I'll vote to reopen if it does get closed. The question is fine. I still think it is not startup related - other than a startup has this problem, but it is actually far better than a lot of other questions. I have changed my mind – TimJ♦ May 16 '11 at 21:20
show 1 more comment

4 Answers

up vote 3 down vote accepted

David: I've been in the software industry for over a decade, and to be be blunt, I don't see how any of your ideas could give you an insight into the reliability of the software you are considering.

With one exception: References

When the software is used for highly complex or mission-critical tasks, your best bet is to look at who the existing customers are. You seem wary of this because a company could use the software only rarely/have low operational dependence on the software. I would counter this argument by saying that in a high-risk environment companies will not ever use - in any part of their process - software which is not reliable. They can't afford to.

So, look to the case studies, search out discussion boards and blogs (they will be out there), look at the list of users the company has on their website.

And once you make the decision, do a test drive - this is easier/more prevalent with web-based/thin clients than it is with thick clients, but speak to a rep and see if you can do a test drive, and check the results for yourself.

Best of luck.

share|improve this answer
References are provided by the company for the clients with which the software worked -- has anyone ever been successful in "Could I have some references from people who used the software with disasterous results?" – Joseph Barisonzi May 16 '11 at 15:03
As someone who has been involved in numerous startups and small businesses, I can assure you I would never pass on info from an unhappy customer. Period. – gef05 May 16 '11 at 15:16
Although you won't hear about the bad experiences, you can still tell quite a bit about a product from a happy customer. Yes, you won't get a product bashing out of it, but even happy customers may have had some smaller negative experiences that they will share (e.g. it's great, though we had some issues getting the reports printed properly, but the company helped us resolve that). – Elie May 16 '11 at 17:44
I agree. References are an important part of the qualification process. Is this product real? Qualifying the references of course is also helpful. But it can't be the end all. If it is mission critical software -- a recommendation will not validate if the software will meet your business requirements. – Joseph Barisonzi May 17 '11 at 4:12
up vote 2 down vote

After you have qualified the software as doing on paper what you need:

I would propose you have to: Test it.

Develop a test of standardize calculations that you will be running. Run them on the software. See with what reliability the correct answer is delivered.

If they wont give you a version to run test on -- then it is not the software for you.

share|improve this answer
up vote 0 down vote

A couple of you have mentioned testing, which is fine, but in my case it could take months of work to do more than basic testing. I have considered hiring someone do to this, but it seems like I am really doing the job of the software company, I am would only do black box testing.

No. Absolutely no!! The testing will enable you to establish if it's producing the figures/results you expect to see in the format in which you expect to see them. This speaks to business rules, selection of mathematical models, etc.

It has nothing to do with bugs.

share|improve this answer
But it would really be nice to save those months of tedious work and focus on the real value adding. But so far, it seems like there is no quick way around this. – David May 16 '11 at 20:27
up vote -2 down vote

The company does not provide a trial version of their software? How about a money back guarantee?

  1. If they do not provide either, I would get in touch with them and talk directly to the sales people, letting them know your position. They would either provide you with enough info to boost your confidence in the product or be lenient and have some kind of trial period for users to test the software.

AND

  1. Buy through Paypal - buyers are always more protected than sellers, especially with intangible goods. If the software is not what they touted it to be, let Paypal know and get a reversal.

OR

  1. Go blackhat. Get the software bootlegged, probably an earlier version, test it out. If it is what you need and is a success, I highly advise buying the software, don't just skip out on the developers that made a great product (if it is great ;)

I would scour Google and try to find people that have been ripped off, but make sure you get quality feedback - not some guy that couldn't find the button that was in his face.

share|improve this answer
you can downgrade all you want, but the reality is that when you have a need for software to get you from point A to point B, it could make you go bankrupt, and the product doesnt have a trial period - blackhat is an option. – Flignats May 16 '11 at 18:39
1  
Just because something is possible doesn't make it a viable option. And by viable I mean life-affirming. You are free to suggest it but others are free to note its cost. – Kenneth Vogt May 17 '11 at 0:26
It's not a suggestion, it is an option. Don't be too narrow. – Flignats May 17 '11 at 2:13
I appreciate that @Flignats has posted the option (suggestion -- I am not sure I can tell the difference) of going to Blackhat. It provides the community the opportunity to express it's perspective on that option. That is what the community board is about. As someone who works hard to help companies monetized their IP and develop software-based products and services I feel good that the community has responded so quickly. – Joseph Barisonzi May 17 '11 at 4:17
I really appreciate the portion of @Flignats answer which is to talk to sales -- I think that this is really important. And I strongly feel that a company that won't let you kick the tire a little for a mission critical software is probably not the right one for you to bet your company on. – Joseph Barisonzi May 17 '11 at 4:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.