Contents

• Software Testing
  1. How do I know how much testing is required for my product?
  2. What kinds of testing should I use?
  3. Should I build my own test department?
  4. What does it cost to have an effective test department?
  5. Where does the test department belong in an organization?
  6. What is the difference between SQA and software test?
  7. When should I consider test automation?
• Independent Test Lab
  1. When does it make sense to use an independent test lab?
  2. How can independent test labs be more efficient than my in-house department?
  3. How can an independent test lab learn my software well enough to test it
  4. What advantages does an independent test lab provide over using contract test engineers?
• Revision Labs
  1. What types of projects does Revision Labs undertake?
  2. What information do I need to provide to get a proposal from Revision Labs, Inc.?
  3. How long does it take to propose a project?
  4. How fast can a project be started by Revision Labs?
  5. What hardware and software is available to use on my project?
  6. What product guarantees does Revision Labs offer?
  7. How often are defects reported?
  8. What reports do I get?
  9. What does a testing project cost?
  10. Who owns the work products?

Software Testing

1. How do I know how much testing is required for my product?

It is really a function of a) what quality standards your market requires and b) how good a job you do in building the product in the first place. Having well-defined release criteria are a great help in determining the types and amount of testing required - it gives a "finish line", a means of knowing when testing is done. But the amount depends on the quality of the software as it is built. If a piece of software has many critical defects, it may well take several full regression tests to be convinced that the product reaches its required quality. If it is well designed and the code developed is solid, it is possible to just complete one test cycle and be confidant that the product is ready to ship. 

2. What kinds of testing should I use?

There are many kinds of testing ranging from functionality to ease-of-use to unit to hardware compatibility. Which types you use depend on your product and your market requirements. In general, the more testing, the more reliable a product and the more types of testing conducted, the less likely that major problems will show up later.

3. Should I build my own test department?

We believe that all companies building software should be at least as good at testing it as building it. Testing should really be seen as "proving" that the product meets its design goals. Whether you do this using an internal test group, an external group or some other method depends on your size, your products and the relative return-on-investment for alternative activities.

4. What does it cost to have an effective test department?

An effective software test department takes years to build and requires at least a) a seasoned software test manager; b) a well-developed testing methodology; c) tools to support the testing methodology and d) competent software engineers to complete the testing work. A company could easily spend $500,000 to get a department to that point.

5. Where does the test department belong in an organization?

In our view, the test department is a tool of the executive management of the company. If used properly, it serves almost as an "audit" function; providing senior managers with the information they need to manage the company. Therefore, a test department should not be part of the product engineering group, but should be distinct and report to some other function which "owns" the issue of quality assurance for senior management.

6. What is the difference between SQA and software test?

Testing software is a very specific set of activities designed to "measure" the product against a pre-defined set of criteria or specifications. It provides managers with knowledge of the state of a product and answers the question: "When is the product ready to ship?" (This assumes a clear set of criteria for release). Software Quality Assurance (or any quality assurance function) is responsible for assuring management that the overall process used to specify, design and test a product meets the corporate goals relative to software products. Testing is a tool of the SQA function but is not itself adequate SQA.

7. When should I consider test automation?

Test automation is a major investment, both in infrastructure and process. A good rule of thumb is that it may take 5 times the effort to manually test a product to build the test automation for that product. Therefore, there needs to be a clear return-on-investment in terms of product quality to justify test automation. And note, this 5x rule assumes you already have the infrastructure and processes in place for test automation.

1. When does it make sense to use an independent test lab?

When all factors in testing are considered, an independent test lab makes sense when there are occasional peaks in test projects or when there are "extra" unplanned projects. These factors include lab set-up; project management; test staff management; recruiting and training QA staff; and direct labor costs. 

2. How can independent test labs be more efficient than my in-house department?

It depends on how efficient and effective your current test department is. For a group with a stable, well-developed test department, it is difficult for an outside lab to be more efficient on those activities that the internal group is competent at. In these cases, an independent lab can be as efficient and effective as an internal lab once it has completed the learning curve on the products.

On the other hand, if an organization does not have a professionally managed, stable test group, an external test lab can be much more efficient and effective because 1) it has a stable group of experienced software test engineers and managers and 2) it has a well-developed testing infrastructure and methodology.

3. How can an independent test lab learn my software well enough to test it?

Revision Labs staff have tested a wide range of software and are adept at quickly learning a new application. While there is a learning curve for the first test project, subsequent projects are even more efficient because of the experience gained in the first project.

4. What advantages does an independent test lab provide over using contract test engineers?

The two key advantages are continuity and accountability. It is rare that a specific contractor will be available from project to project if there are gaps between these projects. Thus, using contractors requires constant replacement and training. Secondly, contractors require day-to-day supervision and are not responsible for deliverables. Although you can fire a contractor, you are still held accountable for what he or she didn't get accomplished.

An independent test lab, however, addresses both of these problems with contractors. Staff are stable and are much more likely to be available when a second or third project comes to it for your products. Plus, with some forward planning, this probability can be increased greatly. And an independent lab takes responsibility for deliverables - schedule, budget and tasks. This can be a great load off of overworked managers.

1. What types of projects does Revision Labs undertake?

Revision Labs concentrates on full functionality testing. This includes test planning, test case design, test automation and test execution. We can apply these skills for stress and load testing; software interoperability testing; GUI testing; white box testing and even unit testing under the right circumstances. We also do localization testing; client/server testing and internet or web-based applications testing. 

2. What information do I need to provide to get a proposal from Revision Labs, Inc.?

Revision Labs staff interviews your technical staff to assess project scope, timetable, equipment requirements, etc. Specifications, early software, test documentation, etc., are requested to evaluate the size and resources required. A formal proposal is submitted with deliverables and cost estimates.

3. How long does it take to propose a project?

Unless a product is very complex, we can usually provide a completed proposal in less than a week. For very complex projects, we may start with a short "investigation" project to develop the strategy and estimate the size of the project. Clients typically pay for this investigation.

4. How fast can a project be started by Revision Labs?

We start many projects on the day that a contract is awarded. Typically, it takes 1-3 weeks to start larger projects.

5. What hardware and software is available to use on my project?

Revision Labs has configured test beds with up to 90 late-model PC's for use on Microsoft-based projects. We also have Novell licenses, full frame-relay access to the internet and dozens of current applications. In addition, we have the latest versions of local Microsoft operating systems for all supported languages. We can rent Macintosh and UNIX systems on short notice. We do not, however, maintain a hardware compatibility test lab. There are a number of good compatibility labs around the country.

6. What product guarantees does Revision Labs offer?

We do not guarantee the quality of your product. We guarantee that we effectively execute all test cases in a plan approved by our customer and report all defects found.

7. How often are defects reported?

Defects are typically reported daily but could be reported more or less often, depending on the specific product and circumstances. We use email or fax to report defects and maintain our own defect database for each project. If you have a defect tracking system, we can report directly into it or in a format which allows easy input.

8. What reports do I get?

Beside defect reports, you get weekly written and verbal status reports and a final test project report.

9. What does a testing project cost?

Prices are set for each project and range from several thousands to tens of thousands of dollars, depending on the size of the project. Costs are based on estimated person-days of effort and equipment and other expense costs.

10. Who owns the work products?

Unless arranged differently, you own the test plans, test cases and any other work products or deliverables which were specifically developed for your project.

For more information, send e-mail to sales@revlabs.com , complete a Project Request Form, or call 503-439-1275 in the Portland area, or 425-861-1302 in the Seattle area infoseattle@revlabs.com. Please send any comments to webmaster@revlabs.com