You’re in the homestretch; IT is working hard to meet the product launch date, but something’s got to give.  What features can you delay until a later release yet not disappoint customers who’ve been waiting months for this new product?  Once you’ve made the tough choices, how do you ensure the product is tested and ready to go?  And have you prepared sales, marketing, and customer service to launch? The stakes are getting higher by the minute and your CEO, customers, and the industry are all watching.  Are you ready?

This is the final of six serialized sections of “Getting Your Product out the Door: Product Development Basics. Article Six – Feature Prioritization, Testing, and Launch” provides the tools you need to bring your product successfully across the finish line.

Introduction

Endless pages of requirements and hours of meetings and the product still doesn’t look like you’d hoped.  A looming deadline and an impossible decision about what features to cut from the launch. When you’re new to product development, such setbacks can seem daunting, but they can be overcome – even avoided completely.

Getting Your Product out the Door: Feature Prioritization, Testing and Launch is the last article in a six-part series compiled by product owners with decades of experience in creating and launching successful subscription products.  The series is full of specific tips, checklists, examples and tools as well as best practices and foundational knowledge that will help you make informed decisions to prioritize product features, test thoroughly and be ready to launch your terrific product when it’s ready to go.

This series is written with the beginner in mind.  The content is explanatory and foundational, designed to give someone new to product leadership the practical tools necessary to make good choices, participate in product testing and plan well for launch. 

---

Getting Your Product out the Door: Product Development Basics

Part 6: Feature Prioritization, Testing, and Launch

Last but Not Last: Launch

That’s no typo.  While prioritization, test, and launch are the last article in our series, each of these activities begins almost immediately after you start developing your product.  In order to build customer personas, complete competitive analysis or write user stories, you need to have a pretty clear idea what you want the end product to look like.  Further, while you may not designate certain features as such, you likely have a sense for what must be included in a product in order to have a meaningful launch, versus what could wait, if necessary, for a post-launch release.

---

 “Begin with the end in mind.”

-Stephen Covey

---

What’s often overlooked is the amount of coordination, discipline, and planning that goes into a successful product launch, not to mention keeping the focus and measuring success post-launch. Launch planning should be a formal part of the development process from the very beginning, and the team members involved in each aspect of launch will need to be kept abreast of progress, and brought into the process at the right time.  You will be surprised how many departments will be involved in getting your product out the door!

With that in mind, this article is broken down to address key actions through the launch process.  They are:

1. Project Planning: what’s due when, and who’s getting it done?
2. Prioritization: which features are “must haves?”
3. Minimum Viable Product: what is the most basic version of the product that can be taken to market?
4. Acceptance Testing: is the product ready to launch?
5. Launch Plan: how will we let the market know about this new product?

The following detail each of these steps.

1. Project Planning

What Is It? 

The project plan is what keeps your project on schedule, and makes sure everyone knows who is responsible for what.  It can take the form of a Gantt chart and task list built with the help of software such as Zoho, or as simple as an Excel spreadsheet or even a to-do list.  Whatever form it takes, it should include:

Project Planning Checklist

• Specific tasks. “Create wireframes” or “perform acceptance testing” are good examples of tasks.

• RACI assignments. We reviewed the RACI chart in the second article in this series, “Getting Your Product out the Door: Who Does What in Product Development?” and the name of the people reviewing, accountable, consulted and informed about each task should be included in your project plan.

• Milestone/deliverable dates.  These are the major due dates when the technical development should be complete, when content for the first issue or box must be finished, when the direct marketing campaign will begin, etc.

• Dependencies.  Product code has to be written before it can be tested, and tested before it can be approved.  These milestones should be included in your plan, and the plan should be carefully reviewed to ensure that dependent tasks are scheduled serially.

• Updates on scope, schedule and budget. You should update your plan on a periodic basis so that anyone on the team can check on progress at a glance. While there are many ways to do this, a best practice is to colorize with red (behind), yellow (at risk) and green (on schedule) indicators for each of the three areas of scope, schedule and budget.Scope = are you on schedule to deliver all of the features in the requirements?Schedule = are you going to deliver on time?
  Budget = are you using extra resources or funds to hit the deadline?

---

TIP:  
Be sure to include the words “behind,” “at risk” or “on schedule” in addition to your color-coding, in case you print reports in black and white!

---

Who Helps Create It?

Every organization is a little different in how it handles project management. Larger organizations may have a project management group, and you may be assigned a project manager to help keep your product development moving along. In other businesses, this responsibility falls to someone in IT or the product owner. 

Whatever the case, one mistake you should avoid is assuming that your engineers are managing the project because they are using time management or bug tracking software.  That planning is only a subset of the overall product development plan which should include internal communications pre-launch as well as the customer-facing launch plan.

What Does It Look Like?

Below are a variety of project plans and reports. There are no right or wrong ways to document your project plan, as long as whatever format you use clearly defines due dates and responsibilities, and is regularly updated and used as a communication tool.

2) Minimum Viable Product

If you’re developing a completely new product, you need to understand its most basic unit of value to your customer.  What are the product features you must give them in order for them to be interested enough to test the product? This set of features is called the “Minimum Viable Product (MVP).” In product development, the MVP has just enough features to validate assumptions and continue learning.

The first MVP row begins with several product iterations that aren’t usable; you can’t launch them and you can’t learn anything. This is definitely a good example of how not to create an MVP! The second row, which is often held up as a good way to create an MVP, begins with such a broad lack of assumption that your company mission would have to be unrealistically wide-open in order to follow a customer through the learning process. The process may work in theory, but not in the real business world.

But the third row illustrates that a solid set of assumptions have been made against a flexible but efficient vision for the business. An MVP is built that assumes a customer persona that is looking for safety, capacity and speed, which the company is prepared to meet. The MVP can be tested and improved upon, without materially changing the direction of the business. This is the most realistic MVP progression.  Following are some ideas for MVP progressions in the subscription world:

It’s important to be aware of your company culture and whether your board, management or customers have the patience for testing a product everyone acknowledges is in very rough form. Online video games can often deliver a basic set of features and glean significant input to drive future product releases; B2B software subscriptions should be more robust and secure before the market test.  It’s even possible that your buyers would accept an iterative process, but your company doesn’t want to risk its position in the market by delivering a less-than-robust product.

---

That’s what I believe is the biggest misunderstanding about the minimum viable product concept. That is the V in the MVP.” 
-Daniel Ek, co-founder and CEO, Spotify

---

 Even with requirements built for an MVP, there still may come a time in the development process when hard choices have to be made. Knowing how to prioritize effectively will be important.

3) Prioritization:

What Is It?

In product development, prioritization puts some features ahead of others in the product development cycle. Even with a clear idea of your minimum viable product, you will often be asked by your engineers to make some tradeoffs in order to get Version 1.0 out the door.  It’s also likely that you’ve created requirements and/or user stories with features that might not be absolutely necessary to the customer right out of the gate – after all, if you’ve done your market research, you have a pretty good idea what you want a fully-formed solution to look like.  While it’s tempting to insist that all features are “top priority,” you will only create confusion and frustration by adopting such a position.  A more realistic and productive approach is to establish a prioritization framework.  A framework will not help you clarify your own priorities, it will allow others involved in the project to understand why one feature is more important than another – even to assist in the prioritization process.

Who Helps Create It?

While the final prioritization of features usually rests with the product owner, it’s critical to gather input and recommendations from the engineering, sales and market research team members. They will be the ones who can best answer questions about the time or difficulty to code a feature, what will sell and what your competition has on the horizon.

What Does It Look Like?

Feature prioritization can be performed in myriad ways, from customer need to implementation difficulty to estimated risk of failure.  Likewise, the amount of time, consensus building and approvals of a prioritization can be nil to infinite. Here we return to our common theme of “do what works best for your business.” The only recommendation is that you document how you make priority decisions, and then use that method consistently in the actual practice of feature prioritization.

A practical way to begin prioritizing is to use three metrics: Value (to the customer), Cost (in resources or actual dollars) and Complexity (how easy is it to get done), scored on a 1-to-5 scale.  For example:

While this is a quick prioritization tool, it isn’t infallible, as you can see above.  Even if the relative usefulness of a feature is low, as in this case with the decorative shutters, it can still get a high score if it’s very easy and inexpensive to build. A more sophisticated variation is to attach a weight to each score, as shown below:

As a beginner, it’s much less important to implement a complex or time-consuming ranking system than it is to do the following:

• Use some system that is understood by your team and the organization as a whole.

• Use all scoring levels.  Your goal should be reasonably even distribution across the rough “high, medium, low” bands of importance.

• Don’t be a slave to your system.  As you saw in the first chart, numbers don’t always tell the whole story.  Rather, use the matrix to inform a decision, and to force you to justify a decision that is outside of the scoring results.

Prioritization Methodsrequirements.com, offers a few that go beyond the basics previously discussed:

• Value – This approach focuses on the business benefit of any given requirement; the requirements that will return the greatest business or economic value are given the highest priority. This focus on value helps to ensure “quick wins” for the organization.
• Cost – With an eye toward funding, this approach may be implemented a number of ways-implementing the least expensive requirements first or first implementing requirements with the greatest ROI (return on investment).
• Risk – This approach prioritizes the riskiest requirements first, with the logic that should they fail, the project can be abandoned with a minimum of investment. This approach often makes sense when a controversial or untested initiative is planned.
• Difficulty of Implementation – The inverse of the risk approach, a focus on the difficulty of implementation places the highest priority on the requirements that are the easiest to implement-the safest bets. This approach allows a project to get some project benefits deployed quickly, enabling customers to get familiar with the project and give critical feedback before you deploy more difficult aspects of the project.
• Relationship to Other Requirements – Requirements often intermingle in complex relationships of interdependence. With this approach, requirements that support other high-priority requirements are also given high priority.
• Stakeholder Agreement – With this approach, stakeholders must come to a consensus on which requirements are most important, and then those are given highest priority. Within most organizations, stakeholder agreement is likely to be at least a partial factor no matter what other prioritization methods are employed.

Again, the method you choose should be logical for your own organization and easy for all to use. Once prioritization has streamlined the way for development to progress, you and the engineers can begin thinking about product testing.

4) Product Testing/Acceptance Testing

What Is It? 

In many technical product development methods, testing happens throughout the process, and is performed by many different people to ensure all facets of the product are working as they should.  This segment, however, will focus on the final product testing and acceptance, and the roles and goals thereof.

Product testing is the performance of product use scenarios that evaluate a system’s compliance with the business and technical requirements of the product and verify that it’s ready to deliver to your actual subscribers.

Who Helps Create It?

Creation of the tests is largely the province of the engineering team, but you should be involved in testing the product from the perspective of the user.  Ideally, the more hands on a product during a test the better.

What Does It Look Like?

While the engineering team will perform tests to ensure the systems your product is built on can stand up to an appropriate number of simultaneous users, and make sure that the product is integrated with other systems (such as billing and accounts), you will likely be most involved in the Acceptance Testing portion of the review, and identifying and prioritizing any bugs in the product to be fixed prior to acceptance.

Acceptance testing, a testing technique performed to determine whether or not the software system has met the requirement specifications. The main purpose of this test is to evaluate the system’s compliance with the business requirements and verify if it is has met the required criteria for delivery to end users.

The test plan lays out user actions as well as system procedures that, while possibly triggered by a user, are not actually performed by a user (e.g., sending an error message, calculating tax on an order), and the result anticipated.  Then the quality assurance team performs each of the activities and validates that it performs as expected.  Following is an example of some test scenarios:

Test Scenario	Step	Expected Result	Actual Result
Verify that the registration form validates for a business email address.	Enter a gmail email address in the registration form.	An error message, “Please enter your business email.”	The error message was received.
Verify that a new registration for the Annual “Gold” Subscription Plan sends the correct invoice amount to the invoicing system.	Place an order for the “Gold” Subscription Plan.	Accounts Receivable System receives the command to send an invoice for $599.	AR correctly received the notice to generate the $599 invoice.
The drop-down menu to select state of residence on the registration form should allow selection of only one state.	Attempt to add more than one state by hitting CTL-State to try to add a second state.	Sound error “ping.”	Error ping heard.

Categorizing Bug Fixes

Much like feature prioritization, categorizing bug fixes gives the developers a roadmap to follow in fixing problems with the product.  A good set of categories for this effort is shown below:

Critical: Cannot launch without fixing. Generally, these bugs include critical portions of the workflow, such as sign-in or search, not working. They could also include data anomalies (e.g., searching for articles by James Doe, who writes a regular column in your weekly news magazine, and the search results coming back with no matches) or missing functionality that impedes the use of the product.  Other items that could be considered Critical are incorrect information, typos or grammatical errors (although often these are considered less important outside of the publishing industry), or pricing errors.

Important: It is preferable to launch with these fixes, but not absolutely necessary. Sometimes “Important” bugs can be the absence of significant workflows, pricing, and sign-up options or other information.  If the product can be used without a fix but loses some of its value, you may categorize it as Important, but not Critical.  What actually constitutes the difference between the two will depend on your product and your customers.

Ideal: The “nice to have” fixes. These may be tweaks to the user experience that will make it easier to use your product. “Ideal” items may include the re-ordering of a drop-down menu, adding some clarifying text to a subscription option or swapping out an image.

Out of Scope: These items  are entirely new features discovered during testing to be added to the next phase of product enhancements. While you want to capture all good ideas generated in the testing process, you should avoid the never-ending bug fix list, and keep new ideas for future product releases.

Using JIRA, BugTower or other bug-fix and product development software is especially ideal when the team is distributed in more than one office, or even working from home or in an outsourced facility.

---

TIP:
As discussed earlier, understanding who within the organization “accepts” the finished product should be understood by the whole team at the outset.  A best practice is for the engineering lead to stamp approval on the systems structure and performance, and the product owner to approve the feature set and utility of the product.

---

The Importance of Being Involved in Product Testing

While it may be tempting to leave product testing in the hands of the engineers and quality assurance department, it’s a mistake on several levels.  First of all, as the product owner, you have the most at stake if the launch fails, and therefore, it’s in your best interests to be involved.  But more importantly, you are part of the team and testing benefits from the knowledge of each team member.  You bring some of the best knowledge of the customer to the process, so be sure to perform some hands-on testing yourself and, where possible, get Sales, Marketing, and Customer Service in on it too. And of course wherever possible – customers!

---

68% product development executives indicate they have too many projects for their resources.  
–Product Development and Management Association

---

Pitfalls to avoid:

Testing only part of the workflow.  This is a common pitfall when the product owner leaves all testing up to IT.  While the engineering team may ensure that the systems operate appropriately and functionality isn’t impaired when a high volume of users are online, they are not focused on the customer experience or performance beyond whether or not a task can be completed.  For example, engineering may sign off on the fact that an email “Thank You” is sent following the submission of an order, but you will confirm that it says what it’s supposed to, and includes the right contact information, and can be answered properly by the person who receives it.

Providing engineering with vague fix requests. If you find a bug, it’s important that you provide as much detail as possible to assist the developers in finding, replicating and repairing the bug.  Following are some helpful tips on bug reporting:

Best Practices in Bug Reporting:

Unhelpful Bug Reporting	Helpful Bug Reporting
Describe what you see.	Take a screen capture and make notes right on the image.
Documenting the error only.	Documenting what you did to get the error.
Getting the error once and reporting.	Replicating the error a couple times to make sure you have all the data.
Sticking to your bug-fix program for all interactions.	Stopping by or meeting with engineers 1:1 and as a team in addition to leveraging the bug tools.
Reporting an error on its own.	Offering the reason why something is an error, its impact on the customer, and the desired fix.
Calling everything a “bug.”	Understanding the difference between a bug and an addition to the requirements.
Making everything a critical priority.	Prioritizing based on real need.
Single pass through the product, then done.	Rapid and repeated iterations as necessary.
Testing only how the software aspect of the product works.	Validating the results of the software and performance from the customer’s perspective (E.g., what happens when you call the Help Desk number?  What does an invoice look like?).

---

TIP:
Beware of under-prioritizing.  While product owners seldom err on this side of the equation – it’s more tempting to identify all fixes as Critical – both sides of the spectrum should be guarded against.  Be as clear-eyed as possible about what you need to launch, and don’t launch with any less than will allow you to meet your goals with subscribers.

---

 5) Launch Plan

What Is It?

0)]This is the step all your hard work has been leading up to, putting the product in your customer’s hands.  The launch plan is your roadmap to building market awareness of your new product, while also making sure your organization is ready to promote, sell and support the product. The launch plan will detail the steps of your public relations, marketing, training efforts, and should begin almost as soon as your product development initiative gets approved. Some workflow terms you should know include:

Go/No Go Decisions.  Initially, your “go or no go” may be deciding whether or not you should even launch the product.  As you move forward, the “no go” usually refers to the launch date you’re working toward.  If the code isn’t stable (i.e., not working well enough to test), legal is concerned about plagiarism or other product issues, the marketing isn’t ready to go or even a natural disaster could be a reason to push back launch. 

Dependencies.  Also discussed earlier in the context of technical product development. In order to train the sales force, you have to create the training materials.  In order to create the training materials, you need to have the contract ready, the selling strategy prepared and the compensation details worked out. These are examples of dependencies, which need to be appropriately timed within the project plan to ensure each deliverable is enabled by the elements required in it.

Optimal Launch Date: While a great product will do well whenever you launch it, timing your launch with a major trade show, the season of the year or local event may give you an extra boost of awareness. Holding a launch for 3 months to time it with a trade show may not make sense, but doing so for a couple of weeks might be very smart. 

Who Helps Create It?

1)]Depending on your organization, launch plan creation may be the responsibility of a product marketing manager, or you may be the owner of this process.  If the former, this will be the time for you to step up as a supporter of another team member, and help ensure that the launch is executed well by communicating the progress of product development, as well as any changes to features you will be launching or the launch date itself, continuously and thoroughly.  While you may not be the “Decider” or “Approver” of the launch plan, your performance in the “Contributor” role will be essential to your product’s success.

When it comes to delivering the elements in the launch itself, you will be surprised how many people your product depends on! You will call on nearly every department in the company for some input.  Legal should complete a trademark search and perhaps, with the engineers, file a patent; you may also need to have them create contracts or approve terms and conditions. Finance will have to approve pricing and their IT support gets the new pricing into the accounts receivable system with a new product code. Editorial will have to create content that can be edited prior to launch.  Customer service needs to be prepared for calls and sales must be ready to sell. Marketing needs plenty of time to create social media buzz, website copy, press releases and marketing outreach such as emails and ads. If you’re building an app, you will need to submit it to the various app stores for acceptance.

So, while creating the plan itself is likely in the hands of product marketing or you, be aware that the actual elements reach into every corner of the company.

What Does It Look Like?

The launch plan may be part of your overall project plan, or may be managed separately (Subscription Insider recommends the former). While the project plan usually focuses on the engineering/technical aspects of getting your product out the door, the launch plan will focus on marketing and sales; the outward-facing activities of any product development effort.  

---

TIP:
Don’t forget to allow for completion of parallel procedures such as app store approval, trademark search, legal review. And – the content!

---

 

The planning chart below is a good example of what should be happening when.  Of course, every organization has its own processes and responsibilities, and ever product development effort is different.  Complex products can take a year or more to develop, and testing may go on for months.  This, however, is a representative sample of both how actions should follow one to another, as well as an illustration of the roles played by the various functional areas within a company.

2)]

A Note on Post-Launch

Note the inclusion of a “Post-Launch” column in this planning chart.  Often a product owner is so focused on getting the product out the door they forget the follow-through that’s crucial to make it a market success.  Be prepared with product content, new offers and follow-up marketing and measurements of success (Key Performance Indicators, or KPI’s) for at least several weeks beyond your stated launch date.  By preparing post-launch activities, you officially kick off the ongoing product development lifecycle that will keep your new product growing and successful!

Summary:

Reid Hoffman, co-founder of LinkedIn, once said, “[Product development is like] you jump off a cliff and you assemble an airplane on the way down,” and there’s a great deal of truth to that feeling.  Without new products, your business can’t grow – but a product that takes too long to build, or that launched poorly can decimate the company. As the product owner, you’re responsible – for getting everyone else to do the work. And once it’s launched?  The real work begins.

As noted in article two, “Getting Your Product out the Door: Who Does What in Product Development?” product development is not for the faint of heart! But you can do this.  By creating a vivid picture of what customers want, delivering customer-centric user stories and making sure everyone involved is clear on what they’re supposed to do, you win.

Use the tools of product development to help you. The RACI chart from article two, the product requirements sample from article four and project and launch plans in this article are templates that will serve you well.  Also, take the time to understand the needs of your engineering team – of all the teams you work with – and learn from them.

Thank you, and good luck getting your product out the door!