I. Under­standing Your Market

Being agile doesn't mean being aimless.

‍

Startups fail because they misunderstand the market they’re trying to serve. It’s like building a bridge over the wrong river. It doesn't matter how impressive your bridge is if it's not solving a real problem for real people. Without an intimate understanding of the different factors shaping your industry, you risk developing a product that’s out of touch with actual market demands or really just missing the window of opportunity altogether.

‍

I’ve seen this firsthand with startups that rush into development, only to realize they need to pivot when it's too late. That’s why validating product-market fit isn’t just about gut feeling or surface-level surveys—it’s about really understanding the people you’re building for—what frustrates them, what keeps them up at night, and where existing solutions fall short.

‍

By digging deep into market research, running experiments, and continuously refining your offering based on real-world feedback, you minimize the risk of a product or feature that fails post-launch. 

‍

TLDR:

• Market misunderstanding, not a lack of ideas, is why most startups fail. To succeed, you must deeply understand both market dynamics and customer pain points.
• Market research is your competitive edge. It’s not just about trends—it’s about understanding your customer’s frustrations, motivations, and unmet needs.
• Product-market fit must be data-driven. Move beyond instinct and validate your assumptions through rigorous research, experimentation, and real-world feedback.
• Go beyond superficial metrics. Focus on retention drivers, behavior patterns, and predictive analytics to inform strategic decisions.
• Competitor analysis should reveal market gaps. Don’t chase feature parity—find where competitors are failing and innovate there.
• Validation is continuous. Test and validate at every stage, from problem discovery to post-launch performance, ensuring alignment with customer needs and business goals.
• Controlled experimentation minimizes risk. Build precise hypotheses tied to measurable outcomes, using lightweight experiments to test assumptions before scaling.
• Post-launch optimization is critical. Use real-time data and ongoing A/B testing to drive engagement and long-term product success.

‍

Tools and Techniques for Effective Market Research

Focus Groups and Interviews

Focus groups and interviews aren’t just for superficial observations. Use these research tools to identify the subconscious motivations behind customer behavior, understanding the nuance in feedback, and avoiding the pitfalls of confirmation bias.

‍

Unmoderated Testing

Unmoderated testing is a way to scale behavioral insights without constant researcher oversight. The key here isn’t just task completion—it's analyzing task performance, friction points, and behavioral patterns in a way that drives iterative design decisions.

‍

Sites like UserTesting.com and UsabilityHub provide scalable remote environments for unmoderated tests. The focus is on leveraging these tools for explicit scenarios such as testing a complex customer flow under time constraints or gathering insights from diverse cohorts with specific behavioral expectations.

‍

Imagine running tests across completely different personas, such as early adopters in emerging markets versus late adopters in more saturated regions. The goal isn’t just to see if they can complete tasks, but to understand how their contextual behaviors—cultural expectations, habitual workflows, and even cognitive biases—affect their interaction with the product. These detailed insights can inform a global product rollout strategy or even help identify where a localized feature set could unlock new market potential.

‍

While running tests in Nigeria during my time at World’s Greatest Videos, I quickly realized the complexities of working in environments with unreliable internet and electricity. Participants experienced frequent disconnections, and we had to adjust our testing expectations to reflect these limitations. It was eye-opening to consider how critical infrastructure differences affected people’s experiences. We needed to make sure our app (which had a lot of time-sensitive elements) would still function smoothly under these real-world constraints. This allowed us to account for things like the app’s ability to save actions locally when offline and relay them to the server when back online. It wasn’t just about understanding how people interacted with the app but also factoring in the larger context of how and when they could access it.

‍

Bonus Tip: Move beyond strictly validating hypotheses to using unmoderated testing as a tool for market differentiation. For example, test radically different UX approaches or experimental features that could set your product apart from competitors. By analyzing not just task completion but the cognitive load, emotional responses, and even the multitasking behavior of people, you can refine your product to offer a truly unique value proposition. This strategic use of testing transforms it from a validation tool to a driver of competitive advantage.

‍

Deep-Dive Interviews

1-on-1 interviews and focus groups should target behavioral and emotional drivers, especially for complex B2B solutions or niche markets. Product teams should focus on exploring how the product integrates into broader workflows, what emotional triggers are driving use, and identifying any hidden barriers to adoption.

‍

For example, you could use interviews to explore how senior decision-makers at Fortune 500 companies prioritize automation in their processes vs. mid-market firms. Through in-depth interviews, you uncover that larger enterprises often struggle with internal friction and slower decision-making, while smaller firms are more agile but struggle with resource allocation. These insights reveal not just a preference for automation features but the psychological and operational contexts that inform those preferences, allowing you to design solutions that speak directly to these needs.

‍

Observe the emotional reactions people have as they navigate your product—whether it's frustration during a poorly designed workflow or excitement when discovering a feature that saves time. Body language, pauses, and moments of hesitation. These responses are critical data points. Integrate them into your design processes by mapping these highs and lows against specific product touch points, and then iterating on the parts of the experience that elicit the most visceral reactions. 

‍

While I’m generally not a fan of focus groups, when needed, make sure individual voices are heard by structuring activities that promote independent thinking before group discussions. Techniques like silent brainstorming or asking participants to write down their thoughts before sharing with the group can help mitigate groupthink and bring out diverse ideas.

‍

If conducting remotely, you can pair apps like Zoom or Google Meet with Otter.ai or Sonix for automated transcription and sentiment analysis. Use these tools not just for transcription but for mapping emotional cues and insights across multiple interviews. Integrate feedback into product requirement documents and personas in real time.

‍

Bonus Tip: Instead of directly asking people about a feature, probe into decision-making patterns: “What factors influenced your last major purchasing decision?” or “Can you walk me through your workflow when you're under pressure?” By focusing on the process rather than outcomes, you can identify friction points or moments that aren’t immediately apparent in their day-to-day.

‍

Data-Driven Insights

Real data-driven insights are not just about counting button clicks and page views. They’re about building predictive models that identify emerging opportunities and creating features that don’t just respond to customer needs—but anticipate them.

‍

In many organizations, data dashboards are filled with vanity metrics that look impressive but fail to actually drive meaningful action. Focus on the metrics that tell the real story of your customer’s behavior and how they interact with your product.

‍

I learned this lesson the hard way. Several years back, while running iteration sessions on an onboarding flow with my team at the time, we saw a really high click-through rate on one specific button. Excited by this, we focused on that button, iterating to make it even more prominent, even reducing the hierarchy of other buttons. Eventually, we removed those other buttons entirely. At first, it seemed like a success—more people were making it through onboarding. However, soon we noticed that conversions for paid features were drastically decreasing. We shifted focus to optimizing the pricing page and building new ways to educate people about paid features, trying to recover lost conversions.

‍

Looking back, we were too focused on that one high-click button. What we should have done was zoom out and take a broader approach. Through goal flow analysis, we could have examined the typical paths leading customers to become paying customers. We might have discovered earlier that customers engaging with that high-click button were not actually converting into high-value customers. In fact, they were costing more in support and churn. The real opportunity was with another group—those who entered through a different, less obvious path—who were far more likely to convert into paying customers. Instead of over-focusing on a single metric, optimize for the process that truly leads to paying customers, looking at the entire journey across various cohorts rather than just isolated click data.

‍

Build a full picture before making decisions:

1. Define the entire customer journey: Map out the key stages that lead customers from awareness through conversion. Don't just focus on isolated points like onboarding or pricing—understand how each stage connects and influences the next.
2. Use cohort analysis: Break down your customer base into meaningful cohorts (ex. behavior, demographic, or acquisition channel) and analyze how each group moves through the journey. This helps you identify patterns in how different segments convert over time.
3. Focus on high-impact metrics: Focus on metrics that tie directly to your core business goals—like customer lifetime value, retention, and engagement with key features.
4. Leverage goal flow and funnel analysis: Use tools like Mixpanel, Amplitude, or Google Analytics to trace customer paths. Identify the most efficient paths that lead to conversions and spot friction points where people drop off.
5. Constantly validate assumptions: Every time you iterate, make sure that your decisions are based on comprehensive data, not just a single metric. Regularly revisit the full customer journey to avoid over-optimizing one step at the cost of another.

You don't need to be a data scientist to start extracting valuable insights from your data. By focusing on the entire picture and understanding how all the pieces fit together, you can pull smarter insights from your data and make decisions that not only drive short-term wins but also align with long-term business goals.

‍

Bonus Tip: Prioritize data quality over quantity. It’s tempting to drown in data, but a clean data pipeline ensures your metrics are reliable, consistent, and actionable. Prioritize events that directly correlate with key business goals (ex. retention, net promoter score, customer lifetime value, etc…) and make sure those data points are captured with high fidelity across all platforms.

‍

Competitor Analysis

Anticipate industry shifts before they happen, allowing you to shape your strategy in ways that outpace competitors. Create a future-proof roadmap by understanding the trajectory of the market, emerging threats, and opportunities that others may not yet see.

‍

Beyond Feature Parity

Chasing feature parity with competitors is a losing game. The real value lies in positioning your product strategically to solve problems that your competitors haven’t yet thought to address. By going beyond basic feature comparisons, you focus on identifying white space opportunities—the gaps in the market where no one else is playing effectively. This could be an underserved niche, an emerging technology, or a novel integration that shifts the competitive landscape in your favor. Similar to chess, you're not just reacting to your opponents actions, you're predicting them, you're constantly thinking several steps ahead trying to anticipate their next move.

‍

Apps like Crayon and Klue aggregate competitor updates, including product changes, messaging shifts, and go-to-market strategies, while SimilarWeb provides data on traffic sources, demographics, and referral strategies. Look for patterns in these updates that suggest future strategies. For example, if you notice a competitor suddenly investing heavily in AI-driven features, consider whether they’re targeting an evolving segment of the market or preparing for a pivot. Use this intelligence to shape your roadmap and align your development efforts with where the market is going, not where it’s been.

‍

Customer Sentiment

Deeply analyze reviews, feedback, and community conversations to identify areas where competitors are underdelivering or overpromising. These gaps in customer satisfaction present opportunities for you to differentiate your product by addressing unmet needs or improving on weak points.

‍

Use tools like Brandwatch or Sprinklr to analyze sentiment trends across social media, forums, and review platforms. Look for pain points that consistently appear in competitor products. For example, if people frequently complain about the complexity of a competitor’s onboarding process, take that as an opportunity to redesign your own onboarding experience to be frictionless. It’s not just about being better—it’s about solving problems that your competitors don’t even realize are hurting them.

‍

Bonus Tip: Aggregate competitor intelligence with market trend analysis. Firms like Gartner and Forrester Research offer deep insights into long-term market forecasts and emerging technologies. Use these reports to not just validate your current strategy but to proactively shape it. By consistently aligning your product roadmap with both current competitor intelligence and market trends, you maintain a dynamic competitive edge that evolves as the landscape changes.

‍

Proactive Product Validation

Validation isn’t something you only do at the start or end of a cycle—it’s continuous and iterative. The key is to integrate validation into every stage of product evolution, making sure that you’re constantly aligning business goals with market needs and customer expectations. This involves layered testing frameworks that span from early problem discovery through post-launch performance tracking, making sure you’re always validating the right assumptions at the right time.

‍

Problem Discovery

The earliest stage of feature development is about validating potential problems against your target goals. This phase focuses on understanding how significant the problem is, how much it’s holding your business back from your goals, and how widespread it is among your target audience. 

Before I joined Trim by OneMain, product development on the team was largely developer-driven and fragmented. Solutions were being built around ideas, often without thoroughly validating the problems they were meant to solve. Features were rolled out based on assumptions, and the result was a cycle of rework, technical debt, and customer dissatisfaction. This might sound familiar.

‍

As Principal Designer, I introduced a more holistic, customer-centric, and collaborative approach that extended the discovery phase but ultimately streamlined development. This approach wasn’t just about slowing down to be cautious—it was about making sure that every minute spent on development was aligned with validated customer needs and business goals.

‍

Stop diving into potential solutions right away. Instead, focus on understanding and validating the problem. Emphasize problem discovery and validation, which should become a cornerstone of your process.

Your teams should start asking critical questions:

‍

• How significant is this problem?
• How many people are affected by it, and how deeply does it impact their workflow or outcomes?
• Does solving this problem move the needle on our goals?

Analyze customer feedback, behavior patterns, and pain points with tools like FullStory and Heap, allowing you to quantify the impact of unresolved problems affecting things like retention, engagement, and overall satisfaction. Integrate these insights into your decision-making, allowing you to confidently prioritize the right problems to solve.

‍

By addressing real pain points through the proper channels, you can develop more effective solutions, reduce technical issues, and improve team morale. Product development becomes more streamlined and strategic when every problem is thoroughly vetted and aligned with a goal-centric approach.

‍

Solution Exploration

Once the problem has been validated during discovery, the next phase isn’t about jumping straight into design or prototyping—it’s about validating the right solution to test. This phase requires a disciplined approach, balancing creativity with strategic decision-making to make sure that the solutions being considered are aligned with your business goals and have the highest potential for impact.

‍

Structured Brainstorming

The first step is bringing together department leads and key colleagues for a targeted brainstorming session. The goal is to generate diverse solution ideas that draw from various functional perspectives—product, engineering, marketing, customer success—each bringing unique insights to the table.

‍

Techniques like Crazy 8’s are especially effective during these sessions. This method requires participants to sketch eight ideas in eight minutes, forcing them to quickly get the obvious solutions out of the way and tap into deeper creativity in the latter minutes. By working independently and then sharing with the group, you make sure that no one is swayed by early dominant voices, maximizing the diversity of ideas.

‍

For the host, I want to stress how important it is to create a supportive and safe space where everyone feels comfortable sharing their thoughts. This starts long before the session itself. People need to feel valued and understood, regardless of their role, and believe that their ideas can contribute something meaningful to the conversation. For example, I’ve often worked with less vocal engineers or team members who felt that their role—whether in development, legal, or another non-design function—meant their voice didn’t carry the same weight as others. One specific experience that stands out is when I was brainstorming at the table with a lawyer who always began his contributions with, “I’m not a designer, but…”—and that’s exactly the point. His legal perspective was valuable because it was different from mine. Helping participants recognize the power in their unique point of view is key to unlocking real creative solutions.

‍

Similarly, I’ve had engineers in the room who were hesitant to speak up, feeling like that their ideas were too technical or not ‘creative’ enough for the session. My role as the host was to highlight their strengths, showing them how their technical perspective IS important. They weren’t expected to think like marketers or designers—what we needed was their unique lens to make sure that the ideas generated were grounded in what was technically feasible or could push the product forward in technically interesting ways.

‍

In the end, a productive brainstorming session comes down to creating an environment where people feel comfortable sharing their unique insights without feeling like they need to be someone else or speak from any perspective but their own. When participants feel encouraged to lean into their strengths, the best ideas surface organically, and you get the diversity of thought necessary to generate meaningful and effective solutions.

‍

Categorization and Clustering

Once all ideas are on the table, the next step is to categorize them. This is where patterns begin to form. Group similar ideas together into an affinity diagram, identifying common themes or approaches. For example, some solutions might focus on the usability of specific flows, while others address performance enhancements or monetization strategies. The goal here is to start distilling the wide range of ideas into manageable categories, each representing a distinct direction that could be taken to solve the validated problem.

At this point, you’re not discarding ideas. The purpose of this categorization is to help you focus your attention on key areas. Rather than sorting through dozens of individual ideas, you now have grouped themes that represent different strategic directions. This makes the next step—strategic filtering—more focused and actionable.

‍

Narrow down categories and choose a direction:

1. Align categories with key objectives: Start by assessing how each category connects to your business goals. Focus on the ones that directly address your immediate needs, whether it's improving retention, reducing churn, or increasing engagement. Deprioritize categories that aren't critical to your current objectives—such as focusing on monetization strategies when retention is your main priority.
2. Evaluate impact and feasibility: For each category, evaluate both the potential impact and the resources needed. For example, usability improvements might address critical pain points and improve retention with minimal effort, while performance enhancements could require significant development for only marginal gains. Prioritize categories that offer the most substantial impact with the least resources.
3. Use data to validate categories: Use data to confirm which categories will deliver the most value. If questions come up during the meeting—such as whether usability improvements or performance enhancements impact retention more—stop to check the data. For example, analytics might show higher churn due to onboarding issues, leading you to prioritize onboarding usability. Don’t hesitate to stop and revalidate assumptions with data before proceeding.
4. Prioritize based on feasibility and data insights: Once you've validated your categories with data and assessed feasibility, narrow your focus to one or two key areas. These will be the categories that both align closely with your goals and show the most promise in terms of impact and ease of implementation.

Bonus Tip: If you’re using FigJam as your whiteboarding tool, take advantage of its AI-driven clustering features. Keeping everyone focused and engaged during this organizational step can be challenging—often requiring you to pause the meeting and organize the board separately. FigJam’s AI tools can automatically group similar ideas together, allowing you to maintain the momentum of the meeting and focus on prioritizing the most promising solutions. While it’s still important to organize your solutions in a way that makes sense for your business, using FigJam’s AI can streamline this process in some cases, helping keep the team aligned and the meeting productive.

‍

Frameworks for Filtering

Three frameworks I find particularly valuable at this stage are:

‍

• Matrix Charts: By plotting ideas against two key variables—such as impact vs. effort or potential value vs. risk—you can visually see which ideas are worth exploring further. For instance, low-effort, high-impact solutions might be prioritized for immediate testing, while high-risk, high-reward ideas may require additional validation steps before moving forward.
• RICE Scores: This framework allows you to quantify ideas based on Reach, Impact, Confidence, and Effort. Scoring each idea across these dimensions provides a clear, objective ranking that helps prioritize which solutions to pursue first.

• ~Reach: How many people will be affected by this solution?
• ~Impact: How deeply will this solution move the needle on your key metrics?

• ~Confidence: How confident are you that this solution will have the desired effect, based on your data and past experiments?

• ~Effort: What’s the estimated cost—time, resources, technical debt—of implementing this solution?

• Weighted Scoring: This method assigns different levels of importance (weights) to various criteria based on your business goals. For example, if increasing customer engagement is more critical than cost-efficiency, you might assign a higher weight to engagement in your scoring model. Each idea is then scored across multiple factors—such as potential ROI, risk, customer impact, and development time—using the pre-defined weights. The weighted scores are calculated to produce an overall score, allowing you to prioritize solutions that align most closely with your strategic objectives.

At the end of the day, it's about having the discipline to say no even when it's tough. These frameworks not only help you identify the most promising opportunities but also allow for decisiveness in rejecting ideas that don’t align strongly with your goals, even if they seem attractive or exciting.

‍

Reviewing the Data

Before making any final decisions, it’s important to revisit the original data that prompted the need for a solution in the first place. This quick but critical step should provide a concise, data-driven recap—highlighting the validated pain points, the key performance indicators at stake, and any relevant research or behavioral analytics. By taking a few minutes to review this information, you make sure that the team remains aligned with solving the core problem rather than drifting toward unrelated challenges.

‍

For example, if the goal is to reduce churn during the onboarding process, reviewing the data that highlights where people drop off will refocus attention on the right solutions. It's a reminder of why you're here, helping the team make decisions rooted in the validated problems. Pausing to align keeps the conversation on track and prevents last-minute pivots.

‍

Once the team has clarity on the problem and the metrics, you can move forward confidently. By this stage, you will have sorted through a broad set of potential solutions—grouped them into strategic categories, filtered them based on impact and feasibility, and validated them with data. This process narrows your options to a select few high-impact candidates that align both with your business goals and the validated customer pain points.

‍

Arriving at a Clear Decision

Now, with the shortlisted ideas in hand, you can make a clear and confident decision. This isn’t about picking the most exciting or loudest idea—it’s about choosing solutions that will deliver measurable results. This structured approach ensures that your decisions are grounded in both creativity and practicality. You’re not just responding to surface-level problems; you’re addressing the deeper, validated needs that will drive sustainable growth.

‍

By following this rigorous yet streamlined process, you not only bring clarity and consensus to your team but also lay the groundwork for strategic, data-informed product decisions that align with long-term goals. 

‍

You can now confidently move forward to validate the chosen solutions through experimentation—setting the stage for the next phase of product development.

‍

Designing an Experiment

Once you’ve narrowed down your potential solutions to a shortlist of high-impact candidates, the next step is to validate these ideas through controlled experimentation. The goal of this phase isn’t to build a full-featured solution but to design a precise experiment that tests your hypothesis quickly and effectively. This approach makes sure that you can either validate or invalidate a solution with minimal resource investment, allowing you to iterate rapidly based on real data.

‍

Develop a clear hypothesis

At the core of every good experiment is a testable hypothesis. Your hypothesis should be a clear, measurable statement that connects your proposed solution with the expected outcome. This ensures that you build your experiment around specific, measurable outcomes tied directly to your strategic objectives, creating a focused approach with success criteria defined in advance.

‍

A simple template for a hypothesis might look like:
"If [Action], then [Target Market] will [Expected Behavior] which will result in [Primary Metric]."

‍

For example:
"If we implement an automated appointment reminder feature within our booking platform, then first-time customers of local wellness clinics (ex. massage therapy centers, physical therapists, and beauty salons) will be more likely to attend their scheduled appointments, which will result in a 10-15% increase in overall appointment attendance rates within 3 months."

‍

It’s important to note that while a detailed hypothesis provides direction for the experiment, it doesn't actually define any specific experiment design. In my example, the exact strategy for reminding customers (ex. email vs. SMS vs. push notifications) has not been defined yet. A well designed hypothesis should be able to be validated or invalidated independently by multiple parties through various experimentation. This ensures that the focus stays on testing the underlying assumption, not just a single method.

‍

Define your metrics

Once you have a hypothesis, the next step is to define the metrics you’ll use to measure success. Your metrics should be directly linked to the goals you’re targeting and should offer concrete data that can validate or invalidate your hypothesis.

‍

Consider asking the following:

‍

• What is the primary key performance indicator we’re trying to move?
• What secondary metrics will provide additional context?
• How will we measure engagement or behavior changes in response to the solution?

For example, if your solution involves an automated appointment reminder feature, you might track:

‍

• Primary Metric: Increase in overall appointment attendance rates by 10-15% within 3 months.
• Secondary Metrics: Reduction in no-show rates for first-time customers and an increase in rebooking rates after the initial appointment.

These metrics will serve as your benchmark throughout the experiment, guiding your analysis and determining whether your hypothesis is valid.

‍

Build an experiment

Instead of committing a significant amount of time and resources to a full solution, the goal is to create an experiment. This should be the smallest, simplest version of your solution that will still allow you to test your hypothesis effectively. Running experiments can be a quick and low risk way of gathering actionable data without over investing in development.

‍

Experiments for initial validation:

• Button Test: If you want to gauge interest before developing a feature, use a button test. Add a button in your UI that displays a new feature but isn’t fully functional yet. Measure how many people click the button to assess interest and validate if building the feature is worthwhile. When people click the button in this scenario, you’d typically show them a modal with a ‘coming soon’ message or a brief survey asking what they’d like from this feature. 
• Landing Page: Create a simple, single-page marketing site that describes the feature or product and allows people to sign up for updates. This helps gauge initial interest without building the actual feature.
• Demand Testing: Offer a service or product for pre-order before it’s built. If people are willing to pay up front or sign up early, it indicates there is demand worth pursuing. Just make sure you understand the laws for your industry around taking money prior to delivering a product or service and have a clear plan for managing and delivering on pre-orders.

Experiments for service delivery and feasibility:

• Wizard of Oz: Create the illusion of an automated system, but handle certain processes manually behind the scenes. This lets you validate whether people are interested in a feature without building a full backend. For example, if testing a new reminder assistant, people may think they're interacting with a fully automated system, but you or your team manage reminders manually.
• Concierge Testing: Provide a new service manually to a small group of people to validate interest before automation. For example, manually create personalized workout plans to test the value of a recommendation feature before building the algorithm. Unlike Wizard of Oz, people are fully aware that the service is manual.

Usability and experience feedback:

• Prototype Testing: Create a lightweight prototype or mockup instead of building the entire feature. Use tools like Figma to build interactive prototypes that people can actually use, allowing you to observe their interactions and gather early feedback.

Experiments for comparative and real-word testing:

• Feature Flag: For features that need real-world testing with actual people, use tools like LaunchDarkly to deploy feature flags. This allows you to enable specific features for a subset of people in real-time, which is great for collecting data before a full rollout.
• A/B Testing: Present different versions of a feature to separate groups of people to determine which version performs better against specific metrics, such as conversion rates or engagement.

In the end, the experiment you choose should answer one key question: Does this solution validate or invalidate our hypothesis, even in its most basic form?

‍

Select your testing environment and audience

The environment where you test your hypothesis is just as important as the experiment itself. You’ll want to select a testing environment that closely mimics real-world conditions to make sure that the data you collect is reliable and meaningful.

‍

I knew a group of college students who were developing an app to help people with visual impairments to navigate their surroundings using their phones. Rather than limiting testing to controlled environments, they ran experiments in real-world conditions, such as helping students with visual impairments navigate a busy college campus or helping their older relatives differentiate between medication bottles. To make sure the app was practical, they also compared its performance against traditional support methods, such as guide dogs. This allowed them to verify if their product not only fit into people’s existing routines but if it could potentially outperform conventional solutions as well.

‍

In the end, they identified who would benefit most from the app (and why) and focused on developing features that addressed those needs, rather than trying to replace methods they couldn’t outperform—such as a guide dog's ability to physically stop its owner in unsafe situations.

‍

Depending on your product, you’ll have to decide whether it makes sense to run your experiment in a live environment with a targeted customer segment or within a staged testing environment that allows for more controlled conditions. Start with a test audience that aligns with your hypothesis. If your solution is geared toward improving retention for enterprise customers, select a small, diverse sample of enterprise customers to gather meaningful insights that are representative of the broader customer base.  

Run Your Experiment

Now that you’ve designed your experiment with a clear hypothesis, defined metrics, and a well-constructed test, the next step is to run the experiment. The goal here is to make sure that you gather accurate, actionable data while minimizing risks and resource investment. Running your experiment effectively requires careful management of both the process and the insights it generates.

‍

Set the stage

Before launching your experiment, make sure that all stakeholders are aligned on the objectives, timeline, and desired outcomes:

‍

• Review Experiment Plan: Start by revisiting the experiment's goals, hypothesis, and metrics. Make sure that everyone understands the rationale behind the hypothesis, the expected outcomes, and how success will be measured. This alignment is important for keeping the team focused and avoiding misinterpretation of results.
• The Right Environment: Confirm that the testing environment mirrors real-world conditions as closely as possible. Depending on your needs, this could mean running the experiment in a live environment, staging it within a controlled test group, or using production feature flags to limit exposure to a specific customer segment.
• Audience Segmentation: Make sure the correct customer group has been selected for testing. Whether you’re focusing on a specific demographic, customer tier, or behavioral group, make sure that the segment you’ve chosen aligns with your hypothesis.
• Align Stakeholder Expectations: Clarify the roles and responsibilities of each team member involved in the experiment. Make sure that timelines, checkpoints, and reporting structures are established, and that each stakeholder knows what to expect and when. This ensures the whole team is coordinated and responsive throughout the experiment.

Launch the experiment

Start by communicating the exact launch timing to all stakeholders. Clearly define when the experiment will go live, what they should be observing, and provide instructions on who to contact if any issues arise during the process. This transparency ensures everyone is aligned and prepared for the next steps.

‍

Verify that all monitoring and tracking tools are properly set up and configured. This means making sure that tools like Mixpanel, Google Analytics, or any others intended for tracking the metrics defined in the design phase are fully operational.

‍

Once everything is set, it’s time to officially launch your experiment!

‍

Monitor in real-time

While your experiment runs, it’s important to monitor progress in real-time. Real-time data provides early insights into how people are interacting with your experiment, allowing you to catch critical issues before they escalate.

‍

Tools like Mixpanel or Google Analytics help you track the metrics you defined in the design phase, such as engagement, conversion rates, or task completion. Additionally, qualitative tools like Hotjar, Fullstory, or Lookback.io give you a deeper understanding of customer behavior, giving you context to the quantitative data.

‍

Key questions to consider during this phase:

‍

• Are people interacting with the solution as expected?
• Are there any unforeseen friction points?
• Is the data aligning with the hypothesis, or are unexpected patterns emerging?

Flexibility in execution

Running an experiment doesn’t mean everything has to go perfectly the first time. Be flexible and agile, ready to adjust the experiment as needed. This could mean adjusting the rollout process, redefining customer segments, or iterating on the experiment itself if data shows early signs of failure.

‍

One of the key advantages of tools LaunchDarkly for feature flags is their ability to adapt mid-experiment. For example, if a particular customer segment shows signs of struggling with the new feature, you can quickly adjust the experiment by limiting exposure to that group while gathering insights from other segments. This agility allows you to fail fast, learn fast, and iterate rapidly.

‍

Post-Launch

Collect and Consolidate Data

Once the experiment has run for its set amount of time, consolidate the data from all relevant sources—quantitative metrics, qualitative insights, and feedback from stakeholders. Analyzing this data comprehensively will help you determine whether the hypothesis is validated, invalidated, or if further iteration is necessary.

‍

A few best practices for data consolidation:

‍

• Cross-reference quantitative data with qualitative findings. For example, if engagement metrics are strong but qualitative feedback highlights customer frustration, you’ll need to balance those insights to form a complete picture.
• Look for correlations across metrics. Did the increase in feature engagement also correlate with improvements in retention? Did a decrease in task completion rate show up alongside spikes in customer complaints?

Tools like Amplitude, Mixpanel, or even custom SQL queries can help you sort through the data efficiently, and surface key insights that should guide your next steps.

‍

Debrief and reflect

Once the experiment is complete, hold a post-experiment debrief with your team. The goal is to discuss the results, share insights, and determine what worked, what didn’t, and why. This reflection phase is important for documenting learnings that will inform future experiments and broader product strategy.

‍

Were the results aligned with the hypothesis? What unexpected learnings emerged that weren’t initially considered? What was the broader impact of the experiment on key business objectives?

‍

By systematically reflecting on the outcomes, your team can apply these learnings to future product development cycles and create a culture of continuous improvement.

‍

Determine the next steps

Based on the results of your experiment, the path forward should be clear:

‍

• Validated Hypothesis: If your hypothesis is validated, you now have the data to move forward with scaling the solution. The experiment has de-risked your decision-making, allowing you to confidently invest resources into full development.
• Invalidated Hypothesis: If your hypothesis was invalidated, the data will help guide the next iteration. Use the insights gathered to either pivot the solution, adjust the hypothesis, or explore alternative approaches.

The outcome of running the experiment, regardless of success or failure, informs your next steps in a data-driven, objective manner, ensuring that your product development efforts are always rooted in validated insights.

‍

Bonus Tip: Even if your hypothesis has been validated, resist the temptation to get overly excited or rush into large-scale changes. Use the data and insights gathered from the post-launch phase to plan carefully for iterative releases that keep your product evolving in response to customer needs and market demands. 

‍

Integrating Market Research into Your Strategy

Refining the Product Vision:

Customer feedback is a powerful tool that should not only validate your assumptions but also shape the evolution of your product. Successful startups don’t just listen to feedback—they integrate it into their strategy, allowing the product to continuously align with customer needs and market demands.

‍

When incorporating feedback into your product roadmap, the key is to maintain a balance between addressing immediate concerns and staying true to your long-term vision. The feedback you gather can help refine your priorities, clarify feature sets, and even inform strategic pivots that better serve your target audience. For example, you might learn that people are consistently requesting a feature that you had initially deprioritized, signaling an opportunity to pivot your focus.

‍

Real-World Example

Instagram is a great example of refining product vision based on customer behavior and feedback. Originally conceived as a location-based check-in app called Burbn, its founders noticed that people were primarily engaging with the photo-sharing feature. They pivoted based on this insight, focusing exclusively on photo sharing and social interaction, which ultimately led to Instagram’s explosive growth as a dominant social media platform.
‍

To refine their product vision, they:
‍

• Identified Patterns: Looking for consistent feedback patterns that indicated where the product was falling short or where there was potential for improvement.
• Prioritized Customer-Centric Features: Adjusting their roadmap to reflect the most critical customer needs, balancing short-term wins with long-term growth.
• Measured Impact: Tracking how changes based on feedback impacted key metrics such as customer satisfaction, retention, and engagement.

By continuously refining your product based on real-world insights, you ensure that your product remains relevant, adaptable, and aligned with the needs of your target market.

‍

‍

Conclusion

Understanding your market isn't a one-time task—it's an ongoing journey that demands careful attention, adaptability, and a commitment to continuous learning. Remember, startups fail not because they lack ideas, but because they misunderstand their market and the real problems faced by their real customers. Embrace thorough market and customer research and position your product beyond survival and towards true market dominance.

‍

Continuous validation through every stage of your product's development ensures that you're always aligned with customer needs and market demands. This proactive approach transforms market research from a checkbox into a competitive advantage, allowing you to anticipate industry shifts, outpace competitors, and deliver solutions that resonate deeply with your audience.

‍

Remember, a well-informed strategic vision allows you to prioritize effectively, allocate resources wisely, and stay focused on your ultimate goals without getting sidetracked by false opportunities.

‍

As you move forward, consider how these insights can be harnessed to translate your business objectives into a cohesive product vision. Next, we'll explore strategies to align your team's efforts with your overarching goals in order to make sure that every step you take thoughtfully balances addressing customer needs with advancing your most important business objectives.
‍