Xavier Russo
Posted: 02 June, 2010 |
|
|
Software Shortlist aims to produce high quality, independent reviews that you can trust and rely upon in your software selection process.
No doubt your prior experience has taught you that “not all reviews are equal”. Common concerns that people have about software reviews on the web include: Is it an unbiased judgement or a paid opinion? How qualified is the reviewer? Did they use a systematic process, or just rely on gut feel?
Here at Software Shortlist, we’re confident that our software reviews are among the best and most reliable available. Consequently, we have decided to be completely open and transparent about our independent review process and associated policies by publishing this in-depth article.
For convenience, this article is based around common questions people ask. If you have any questions or concerns that aren’t addressed here, please contact us and we’ll answer them for you … and then revise this article accordingly. Why are our software reviews independent? Information is plentiful on the web, but trusted, reliable and independent information is relatively scarce. If you’re going to make an important, costly and hard-to-reverse decision (such as buying software for your business), we strongly believe it is important that information you rely upon in making that information is credible, accurate and reliable.
Software Shortlist was founded to help small & mid-sized businesses make better software decisions. The quality and independence of our software reviews helps us achieve that mission, and serves as a real point of difference for us compared to the many other websites out there. Who are our software reviewers and why should you trust them?Software Shortlist engages independent experts to conduct our reviews. We select them based on (1) their expertise in the relevant software type, (2) their communication and analysis skills, and (3) their ability to be unbiased in assessing the merits of each software product.
Of particular importance is that third criterion. We work to ensure our reviewers are not resellers, channel partners, or otherwise closely aligned to any software product in the category or class that they are reviewing. This rules out many otherwise well-qualified experts from being reviewers for Software Shortlist, but it is a fundamental reason why our reviews are so well-regarded.
Reviewers are paid by Software Shortlist to complete an agreed number of reviews. They are editorially independent – free to form their own judgement and communicate it in their reviews. Software Shortlist staff conduct some limited editing of the reviews, mostly related to word count, sentence structure & formatting, but we do not adjust ratings or materially change the core messages submitted by the independent expert. (Full disclosure - in the past, we occasionally used multiple reviewers in a category and had to do some adjustments for consistency, but this is not our process going forward) How do we select software for inclusion in a review?Software included in a review pack is selected by the independent expert engaged by Software Shortlist to conduct the reviews, based on their judgement of which are most suitablem, and in consultation with us to ensure a good mix of products. We also have one-page online reviews of products that vendors may request and pay for, but these still adhere to our editorially independent process.
Software Shortlist maintains a long list of products in our software directories, based on our own research and vendors getting in contact with us. The independent expert engaged to do the reviews is responsible for screening this long list down to 10-20 products for review.
The screening process is based on each product’s reputation in the market, its suitability for small and mid-sized businesses and the reviewer’s initial judgement about product’s quality.
Sometimes there may be good products that miss out on inclusion, but it is a balancing act for us between the cost of the creating the reviews and the number of products included. What is the review process and how robust is it? Software Shortlist has developed its own proprietary review process and associated systems, designed to ensure that reviews are conducted in a robust, reliable and consistent way. The independent expert is responsible for conducting the reviews according to this process.
The review process includes the following major elements:
- Category Definition: We give the software category a name and define it clearly (harder than it sounds!), in order to limit the scope of the review and guide the screening process.
- Screening: The reviewer screens the long list of candidate products down to 10-20 for inclusion in the reviews, based on reputation, suitability for SMBs, and perceived quality.
- Testing Plan: Software Shortlist and the reviewer agree on a testing plan, based on common user actions and scenarios. This helps ensure consistency in how products are reviewed.
- Demo / Trial: The reviewer opens up a demo account or downloads a trial version for each product in the review, and runs through the testing plan themselves. Where this is not possible, they contact the vendor and get a guided demo that covers the same testing plan.
- Rating Guidelines: Based on their experience in the demo/trial, the reviewer rates each product on the dimensions defined by Software Shortlist. A set of rating guidelines are provided to ensure consistency in how the star ratings are assigned.
- Features: The reviewer develops a standard list of 20-30 features that are important in the category, and maps which products have which of these features. If products have multiple versions, the features are assessed based on the most relevant option for a typical SMB. (Note: features change rapidly, especially for software-as-a-service products, so please be aware that the feature map may be slightly out of date. We have a regular cycle for updating the reviews, but may update more frequently if required)
- Written reviews: The reviewer prepares various elements of written commentary, as per our standard review templates. This typically includes an overall assessment of the product, plus what they liked and what they didn’t like.
- Editing: Reviews are submitted to Software Shortlist by the independent reviewers. We then edit for consistency, grammar, format, etc to get the reviews into the defined formats for publication as Review Packs and for inclusion in our online software comparison service.
What is the role of vendors during the review process? Given the independent nature of the reviews, vendors’ role is limited. Importantly, vendors cannot pay for favourable reviews. Vendors can request and pay for "off-cycle" reviews (e.g. for online one-page reviews) but our usual standards of editorial independence in conducting these reviews still apply.
Vendors may be contacted by the reviewer during the review process to assist with the demo/trial, to confirm their understanding of features present, and to explore any other concerns or issues encountered. However, vendors are not permitted to contribute to the ratings or written reviews.
If vendors feel after publication that their review is unfair, they may raise an objection with Software Shortlist . We then communicates this to the reviewer for investigation and possible clarification. However, the reviewer’s decision is final and changes to the reviews are only made in the event of an error or major misunderstanding. How do we earn money if our reviews are independent? Software Shortlist earns revenue primarily by providing lead generation services and premium directory listings to software vendors, resellers and service providers. We generate qualified sales leads when people who download our reviews or use our online comparisons give permission that we may connect them with relevant providers. We also offer online one-page reviews that are sometimes a non-paid by-product of the review packs, and that sometimes vendors who missed out on inclusion in the review packs or from areas not covered by our review packs may request and pay for a review.
The review process has been carefully designed to be independent of any relationship that Software Shortlist have with vendors or service providers. We engage independent, third party experts to conduct the software reviews. These reviewers operate under a principle of editorial independence and their names and contact details are published with the reviews. Any questions? We hope the above information reassures you of the quality and independence of the software reviews available on Software Shortlist. If you have any questions or concerns, please contact us and we’d be happy to discuss them with you. This article will be revised from time-to-time to ensure it answers the most common questions. |
| |
 |
|
|
|
 |
 |
 |
Xavier Russo is the Marketing Director and co-founder of Software Shortlist. He has been a freelance strategy & marketing consultant; worked for management consulting firms such as Booz & Co and Gemba Group; and led the marketing team that launched online art community RedBubble.com.
http://www.softwareshortlist.com
|
|
| |
|
 |
|
You've Chosen Business Software, Now What? The Importance of End-User Training
So you’ve made the decision, selected your software, implementation is underway and you are shortly due to go live. So much to organise and yet often the crucial process of incorporating an end-user s |
The Human Side of ERP
When people hear that a new “system” is coming, their first reaction can be “Oh my. They’re going to automate my job and get rid of me,”. So if you're implementing a new system, it's important that |
Enterprise Decision-Making: The Limits of BI Tools
Business intelligence software can be a powerful aid to decision-making in the enterprise environment. However, it is important to keep in mind some inherent limitation about how we process informatio |
15desks Review
Basic academic based learning management system built with blended learning and self-paced eLearning in mind. |
AppShore CRM review
A user friendly, well designed CRM system with just the right features to manage sales, activities and marketing for SMBs. |
AimCRM review
A good tool for businesses where capturing, processing and distributing/aggregating leads is the primary objective... |
Thinking Cap LMS Review
A true LMS for managers and administrators, offering highly functional career paths, dynamic enrolments, and course reuse in different programs for various audiences. |
Learn.com Review
A blended LMS solution geared towards multiple learning audiences. |
Top reasons for buying a CRM
The right CRM software can help you understand, anticipate and respond to your customer's needs in a way that delivers real value to them and to your business. |
Soffront CRM review
Soffront provides an all-inclusive CRM/ERP/Back Office program with some unique twists. It is an impressive offering overall. |
| Add your own Article |
|
|
|
|
|
 |
|
SoftwareShortlist is an innovative online platform
that helps small to medium businesses
compare, select and buy the right
software for their specific needs.
read more... |
|
|
