Why does a customer stay or go?
Subscription businesses are built on customers returning over and over, which means we need to understand why users stay or go. We do this by:
- Defining what a successful customer retention looks like
- Identifying what leads to success or failure
- Adjusting our offering and re-evaluating the situation
Framed supports you through each of these steps. We'll walk through the process below.
Define what success looks like
Defining success is the foundation of analysis, because it lets us measure which customer behaviors help our success, hurt our success, or have no effect on our success.
Example: a viral video brings thousands of visitors to your site, but those who watch the video return no more frequently than those who did not watch the video. We can conclude the video has no effect upon your long term customer retention.
Framed comes with several built-in definitions of success:
|Used your service within...||Good for...|
|1 day||Games, ultra sticky sites like deals and social networking|
|3 days||Most routine sites and tools|
|7 days||Specialty sites and business tools|
Pick the engagement period that makes the most sense for your business. You can see your list of reports in the upper-left hand side of the user interface.
If your desired success metrics is not included in our default set, contact email@example.com and we can adjust your reporting.
Identify factors that predict success or failure
Who likes our product? What keeps people coming back? What is driving people away? Everything is speculation until you look at the data. Framed measures the events and properties correlated with engaging your service and summarizes them in the variable importance summary.
To use the variable importance summary, you first instrument your product to track any positive or negative usage events. The variable importance chart will then tell you which good events are best and which bad events worst. In other words, the chart lists the most important things that indicate success or failure for your customers:
Some of the success signs will be obvious, but others will require deeper investigation to explain.
Example: negative customer feedback events have an obvious impact on retention. But suppose time zone was very strongly correlated. In order to understand why some time zones are having a negative experience, you will need to identify people in the affected time zone and investigate why their experience is different from others.
If you need to investigate particular groups, and why their experience is positive or negative, Framed gives you user lists that bucket users based upon whether they are high or low in a given signal. You can find user lists at the bottom of the retention report next to their tipping point analyses, described below:
Adjust your offering to improve strengths and reduce weaknesses
Once you have identified strong positive or strong negative signals, it's up to you to experiment on your service to increase positive signals and decrease negative signals. Framed provides tipping point analyses to describe approximately how far you need to move an indicator in order to get people to retain.
Example: your company sells widgets. A tipping point analysis shows that buyers on average look at 10 products, while non-buyers look at only two. User research reveals that many buyers cannot find what they are looking for until they have scanned at least 10 items. The development team adjusts the product so that more users and shown more items early in their session.
In the tipping point analysis below, you can see that 50th percentile of retained users (the second row) makes 13 visits per time period, while non-retained users (the first row) make less than 5.
Re-evaluate impact and repeat
After you have deployed a change to your service, check back Framed within the next few days to determine whether your changes had an effect. There are a small number of possibilities:
- If your tipping point got lower, then you made your product easier to use, more sticky or more catchy. Customers will naturally want to use it more and retain longer, starting from earlier in their usage of you.
- If your user behavior got closer to the tipping point, you have successfully encouraged people to use the product more, and become more likely to retain.
- If neither tipping point nor user behavior improved then your experiment had no effect upon your retention. However, you should also check your variable importance summary to determine whether your change disambiguated any new strong positive or negative signals. Even if your retention has not improved, it is possible to improve your understanding of your situation by developing clearer instrumentation and tests.
By repeating these steps you can in turn improve retention or improve knowledge required to improve retention. This is how you iterate towards success in an intentional manner, not merely trying all possible A/B tests, or interviewing any random users.
Did the instructions above make sense? Our team would love to get feedback or talk specifically about your company's needs.
- Thomson is a professional data scientist and scholar at NYU Courant.
- Elliot is a software engineer trained in classical qualitative and quantitative sociology.