Where We Left Off:
In part one of this blog post series, I explained a four-step process to shape the direction of your solution:
- Define the problem you solving
- Define the purpose your application will play in solving the problem
- Identify the audience will you be a hero to
- Identify the functionality your application will have
In part two, I explained how to gain a significant amount of advantage for your applications by working on a loose methodology involving:
- Innovation – brainstorming on every idea we could think of to solve for each piece of functionality.
- Emulation – analyzing competitors and innovative web sites to determine how they solved for various pieces of functionality.
- Integration – Combine your innovation and emulation findings to create a set of workflows that will be needed to support your functionality.
- Authentication – Verify the validity of your workflow by documenting your proposed workflows
In this part, I’m going to help you define your success. No seriously. In doing this, you will help refine your problem statement, and make smart investment decisions for your startup as it matures. Failure to take the time to work on step 8 WILL result in a failed startup – either because your customers aren’t successful and you won’t know how to correct it, or because your investors who don’t understand the problem as well as you will define a metric that doesn’t make sense for your business model, and then will judge you by it! Stick with me here. I know I’m long-winded.
Step 8: Define Success
“What gets measured gets managed” — Peter Drucker
It’s true that you can do the work I’ve presented to you in the previous posts, set yourself up beautifully for a project, and then fail miserably. Time and time again, we’ve seen new companies or products make a big splash with marketing and an influx of curiosity from customers, only to quickly fade away. Any of you still using Friendster?
What is it that causes a company with, what seems like a great idea to fall apart so quickly? In my view, it’s a failure to define for yourself what success means, and then do everything you can to become successful by your own standards.
Now, many of you may be ready to throw the towel in on this post saying “Oh no, here comes the mission statement and the meaningless metrics.” We ALL have reason to hate metrics. Time after time again, we’ve been given a not-so-clear goal of improving a random metric because someone decided that it would determine whether a project was successful or not. To your dismay, the metric didn’t improve, so the stakeholder of your project decided to dump money into a campaign that would help improve that metric. Your metric improved, but your project failed or was dumped. No wonder developers hate measuring things and loath the topic any time it comes up. Can’t we just follow the same mysterious voice Kevin Costner did in “Field of Dreams” – ‘if you build it, they will come’, right? Well, sure. Honestly, if you’ve done everything right to this point, you’re going to attract attention and people WILL come. However, even if you do solve a problem for someone, your ability to track how well you are doing at solving that problem and then learn from each success or failure your customers have in using your product to solve that problem is what will keep your project on track.
Let me ask it to you this way:
Would you give up time with your friends and loved ones and invest your life savings into a project that was just going to attract people who looked at your project then walked away? There are very rare circumstances where this is a good idea and a startup usually isn’t in that position to meander that long. I think many of you would agree that would just be foolish. Yet, what is it that many of you want to track the minute your web application goes live? Say it with me: page views!
“Page Views" and Other Roads to Ruin
I’m sure that somewhere in the silicon valley is a homeless shelter filled with failed entrepreneurs who will shiver any time you mention the phrase “page view”. The page view metric is today’s equivalent of window shoppers. Window shoppers don’t make your business successful. Hell, people walking in your store don’t make your brick-and-mortar a success by almost any standard. you won’t stay in business long if you just try to improve the traffic in your store. Yet marketing organizations in big companies are judged successful if they can just drive up page views on your site.
If you’ve set up your project using the method I’ve described in the previous post, and you decided that you’re going to stop at nothing to drive up page views, you might as well just start handing me your money. No seriously, there’s a donate link right … here. By the time you are done reading this post, you’ll understand that page views, and other related metrics, are really just symptoms of a well or poor performing application or web project.
Refining your “problem” by defining the metric
So if not page views, what should you measure? To answer that, let me step away from software and give you another example and then we will come back to your application – weight loss.
According to BusinessWeek, Americans spent $40 billion a year on weight-loss programs in 2008. Today, AOL Small Businesses claims that number is now $60 billion spent on products, and services of various types. It’s all a gimmick. Jenny Craig plasters images of dieters like Carrie Fisher who lost 30 pounds in 18 weeks. Weight Watchers, a company with the absolutely wrong metric in their company’s name, boasts that it is also rated the #1 best plan for weight loss. These companies are set up to help customers solve a problem – their weight. Seems noble. It makes a good business industry, obviously. Yet it also leaves many people stuck in yo-yo diets because they set their expectation and focus on the wrong measure of success – weight. So what’s wrong with measuring weight? That’s the goal for most of us, right? Not really. I challenged myself to 21 days of exercise and proper diet just to kick start my own personal fitness plan (blog post forthcoming). The first couple of days, my weight just shed off. After that, I had some days where I weighed more than I did the days prior. It would have been frustrating if my metric for success was weight. Those on those diets from Jenny Craig and Weight Watchers would have said that the program just wasn’t working. However, I also had the sense to track several other metrics such as:
- Steps-per-day using a Fitbit to track my movements.
- Body Fat, muscle mass and water weight using a Omron Body Composition Monitor
- Calorie intake using Fitbit’s online services and a Newline Food Scale
- Calories burned using Fitbit’s online services and RunKeeper
Sure, I do track weight. But weight is a symptom of solving the problem. The real problem is that I was doing all of the things above wrong. I wasn’t moving enough in the day. I wasn’t eating right and wasn’t getting exercise. An amazing thing started happening the day that I started tracking those metrics instead of just measuring my weight – I felt successful with my every act, not just when I stepped on a scale to see if my weight dropped. By valuing the right metrics, I get instant gratification when I make the right food choice, go for a short walk just to keep my steps up for the day or burn calories on a bike ride. The thing that Weight Watchers and Jenny Craig are missing here is that they are solving a problem for customers – they are just solving the wrong one. Their metric only let’s their customers see success once a week when they step on a scale – and often too slowly to keep the customer interested. My metric allows me to see success early and often. You can see that by defining what success means, I actually have a way to refine and focus my problem statement, improve my business model, and make customers more successful on a regular basis.
So What’s Being Measured ?
So now let’s back to the world of web applications. Page Views are not a metric to concentrate on improving. Increased PVs are just a symptom of a web application that is solving a problem correctly for customers.
Many businesses incorrectly assume that they can game the system by spending their money to gain more views. This is a trick to try and get investors fooled into putting their money in your company. They can see your upward tick of page views and determine that you’re app must be doing well. You’d never catch Dave McClure or Garry Tan making that mistake. A good investor can tell if a business is wisely spending their money by asking what metrics they are tracking. If page views are it, then they know the business is just gaming the system. The business might even try to throw in a “conversion rate” or other traffic-centric metric. Those too are easy to spot as useless because, as stated above, what get’s measured, gets managed. You can tell a lot about how a company is managed based on what they are measuring. If the metrics are centered around customer success, that company is destined for good things. If the metrics are centered around a one-sided win for the company, you can be sure that company is hoping for an early windfall and an exit strategy. If nothing is being measured, that company is in deep need of help!
If you want to be successful, you MUST DEFINE SUCCESS — for the customer, for yourself, and for the potential investor. You have to dig deeper into your problem and determine what metrics will give you more insight into determining if your customers are successful at solving their problem with your solution.
Let’s pull our example from the previous two blog posts into play here. My fake application is a content curation system. The problem , if you recall, is as follows: “The content that is most relevant to a developer’s evolving needs is buried in a pile of useless material and is never discoverable when they need it.” My purpose is: “to curate the best developer content targeted at an individual’s needs and to deliver it to the customer how they want and when they want it.” My audience consists of professional and experienced developers who build standards-compliant web applications and are passionate about software development. The functionality that I determined was important was: low-barrier community contribution, rewarding community interaction, compulsive browsability, personalizable content, and diverse delivery.
To me, the metrics for this application seem pretty clear, but this is an acquired skill that didn’t come easily for me as a developer. What would you choose to measure to determine success? Go ahead and jot some ideas down. I’ll wait.
Seriously go ahead.
OK, so what’s being measured? If what you jotted down had anything to do with how successfully the customers used the functionality that you determined was important to successfully solve the problem, then you can give yourself a pat on the back. All of your metrics should indicate if customers are successful with your functionality. The best place to start would be to go back to your Evernote (or other planning system of your choosing) and look at your functionality notes. Remember that we speced out “Low-barrier community contribution” and we determined one workflow is:
- User posts a link to twitter using a cell phone and uses hash tag #CR8ME (curate me).
- User receives a response tweet instructing them to connect their twitter to the app
- User clicks on link, authorizes app, and new post is added to curation queue
Your metrics should then include things like
- “Number of customers using #cr8me” tag to initiate sign-up process
- % who are sent an authorize link but do not click
- % who click on authorize link
- % who click on authorize link by do not authorize
- “Number of customers using #cr8me” tag to add new items to queue
Why would you do this for every feature? Because, if you truly believe that “low-barrier community contribution” is vital to the success of your project, and a month after launch you determine that people aren’t using your solution to that problem, you can manage that however you feel best. You could either supplement or substitute your functionality with additional functionality. You could determine that people aren’t aware of the feature and market it better. The point is, you how can manage the things that are relevant to the solution you have defined.
What Doesn’t Get Measured
What doesn’t get measured is a question very frequently not asked. More data is always good, right? Wrong. There are times in life when it’s better to not allow data to be collected. Collecting data that means nothing allows someone to later look at data that has no relevance to your project and start rat-holing on it. For instance, many web analytics packages allow you to track things like “cost per end action”. This is great if you are actually running a marketing campaign, have defined end-actions, and want to determine the effectiveness of your campaign. If you haven’t done those things, but are just collecting and reporting that metric because you can, an investor, stakeholder, or customer can look at it and say “well, they can’t be that good because their CPA is really bad”. Additionally, going back to the adage, “what gets measured, gets managed”, you’ll find yourself trying to improve a metric that has no meaning to you. It will waste your time and resources. If you don’t have a need to track it, and you do anyway, you or your investors will say “well, maybe I should be running a marketing campaign so we can track this better.” Then you’ll spend countless hours working on a plan and process and marketing campaign so you can improve your numbers. All the while, those resources could have been better spent tracking and improving metrics that are relevant to the problem you are trying to solve and the functionality you determined solves that problem.
If it ain’t planned, don’t track it!
Document your Metrics
Document all of the metrics and reasoning behind metrics in your project document system (again, I’m using Evernote, but you might just have hand-written paper and a file folder or two). For me, in my Evernote, I added the metrics under each workflow, such as in the following screenshot:
Measuring success of a web application is not a cookie-cutter process. If you haven’t defined success for your application before you started coding, you’ll likely find yourself trying to get meaningful data when it’s too late – like when your business is failing or an investor comes calling and wants more information. With any initiative in life, it’s always good to know what the end game is to keep you focused on your vision and to help refine your problem statement.
- track workflow metrics – Your workflows are tried to functionality which in turn is tied to your problem statement. So it makes sense to start by looking at the workflows you created for each piece of functionality and determining what data is needed to track your success or failure.
- set your limits – Determine in advance what values or trends you’ll accept as “success” and what values mean you need to regroup.
- avoid mindless collection – Don’t track what can’t be tied back to a problem-solving functional workflow, or to a process or marketing program that supports your current business plan. Collecting “just in case” will waste resources, time and may make you lose out in the future.
- document your metrics – Keep track of what metrics you intend to collect and document them with the rest of your project. Once again, showing an investor or other stakeholder what you are doing to solve the problem, and how you are tracking success with your approach gives you credibility and your stakeholder gains confidence.
I’m “counting” on your feedback (ba-dum dum). I intend to revise this post and keep it updated based on community feedback so please give some! Thanks for reading.
I would appreciate if you would tweet, “like” or otherwise recommend this post if you feel it has been helpful. I’m not tracking these as metrics for success, but it’s always nice to know when someone finds your work helpful. Thanks again for reading!