The Optimal Path

A framework for decision-driven research with Roberta Dombrowski | Maze

Episode Summary

Roberta Dombrowski, Research Partner at Maze and former VP of User Research at User Interviews, talks about her framework for decision-driven research and how to enable the organization to make confident, informed decisions.

Episode Notes

The Optimal Path is a podcast about product decision-making from the team at Maze. Each episode brings in a product expert and looks at the stories, ideas, and frameworks they use to achieve better product decision-making—and how you can do the same.

You can follow Roberta on Twitter (@robertalearns) or check out her website.

Resources mentioned:

Follow Maze on social media:

To get notified when new episodes come out, subscribe at maze.co/podcast. See you next time!

Episode Transcription

Ash Oliver:
Welcome to The Optimal Path, a podcast about product decision-making brought to you by Maze. I'm your host, Ash Oliver, UX Designer & Design Advocate. Great products are the result of great decisions, decisions that deliver value for customers and the organization. In this podcast, you'll hear from designers, product managers, and researchers about the ideas informing decision-making across all aspects of product development.

Ash Oliver:
Today, I'm joined by Roberta Dombrowski. Roberta is the VP of UX Research at User Interviews. In her free time, Roberta is an adjunct professor through Boise State University's Organizational Performance and Workplace Learning program and she's a mindfulness teacher. Roberta, it's so great to have you here.

Roberta Dombrowski:
Yeah. Thanks so much for having me.

Ash Oliver:
All right. So we're going to get into with the relationship between research and decision-making in a bit, but I thought maybe you could start with some context on your background and career.

Roberta Dombrowski:
Yeah, happy to. I think context as a UXer is always important. Yeah, background is I actually am a dabbler. So I was a product manager before I decided to specialize in UX research. And so in my role as a product manager, it's all about decision-making. It is just like, what features should we build? What are the opportunities? Why is this important? How are we going to measure impact, especially after you release?

Roberta Dombrowski:
And what I notice is that my favorite part around being a product manager was actually doing the research, connecting with customers, whether it was discovery research or evaluative research after the fact. And I think it's because that process of going through research helped clarify some of the decisions that I was having to make on a daily basis. And when I use research, I use it very broadly. It could be data science, so quant research. It could also be qualitative user research. When you're actually sitting down talking to customers, there's obviously lots of different methods.

Roberta Dombrowski:
But I really see that as the benefit of research: it's really helping to clarify decision-making and hopefully in the long run, making your customer's lives easier. You're releasing an amazing product, their life is changing, and you're having an impact on society really as a whole. So I do see research and decision-making really being so linked together.

Ash Oliver:
Yeah. I love that. It's also like informing those decisions with some of that confidence. Well, I figured what we could do is jump right into the mechanics because I want to have you focus on the details of the framework. So maybe you could describe what is decision-driven research.

Roberta Dombrowski:
Yeah. So I joined the User Interviews team—it's really like six months ago at this point—and shortly after I joined, I wanted to create an operating model. I was the first researcher and so I was like, how are we going to operate with the product team? How can I really start to showcase more of the impact, the value around research, its purpose, where it can come into the product development process, and the type of decisions the teams were making? So I really went through this process of looking at different models. I was heavily influenced by Kolb's learning models and models I had learned from learning, because my background's in organizational development, and charted this framework for decision-making.

Roberta Dombrowski:
And so essentially there's a number of different steps. It really starts off as like, hey, there's a decision we need to make. So some type of like trigger that takes place. And then usually the next thing you want to do is really start to scope evidence of some sort. I found that typically the default for a lot of teams is actually jumping to conduct research because it's easier rather than looking at historic research. So we wanted to make sure our teams were taking advantage of all of the rich data that we already had. If we find out that like, "hey, we already have the research, we don't actually need to go out", then you actually don't need to do any research, which is great.

Roberta Dombrowski:
But there's going to be cases when you still need to actually go out and conduct a study, which is the third step. You really define your approach, pick out the research methods, really hammer down like, what's my question? And then once you have your approach, it's really pretty similar to other ways of conducting research. You go out, you explore, you talk to customer or you do a usability test. And then afterwards you really want to reflect on what you've learned and then incorporate that into your decision that you're going to make. I find that over the years, a lot of teams don't do that reflection part. They typically will be like, oh, I learned this observation, let me go apply. And you're like, wait, what does that actually mean? And so I put it together in this quick framework and our team's been using it for the past really six months since I started. So it's been really cool to see it.

Ash Oliver:
That's really exciting to construct something like that and then implement it. And now you're six months into seeing how it's been working. I mean, one of my questions for you later on, we could definitely ask it now is, what kind of benefits have you observed at least in this first six-month period from working within the framework?

Roberta Dombrowski:
Yeah. I've done a lot of training with the framework too. So after I created it, I created a training program for our product teams too. And we really dove into methods and everything. It's been really interesting to see people more sit down, really chart out "what's the decision that we have to make, what are the research questions?" And thinking about methods rather than just jumping to a specific research method like, "hey, we're going to do a survey. We're going to do this." But actually like "is that needed?" There's been cases where I have actually talked to one of our product managers a few weeks ago. We were going back and forth around answering this question, how are we going to do it? And we were like, we don't have the time. We don't have the time for this. We're not going to do research in this case. And that's okay. We don't need to kill ourselves for this.

Roberta Dombrowski:
I'm also seeing a lot of mixed methods integration, which is really cool. I've been working a lot with our data science team to really try and tell a holistic picture of what our customers lives and experiences are like. So we're getting really crafty with the research techniques too, because we're exploring different approaches in the framework.

Ash Oliver:
That's interesting. It's kind of giving the teams the guidelines to be able to work within, but then have kind of that contained exploration. It sounds like even autonomy too.

Roberta Dombrowski:
Oh yeah. Our products culture is pretty interesting at User Interviews in that. Before I joined, there was research typically already being done by the product teams. Usually it was the product managers, designers. We utilized Teresa Torres' continuous discovery framework. And so when I came in, I wasn't starting from scratch. I'm really coming in and adding more rigor, different types of methods, really giving more a solid foundation for the team. I don't want to prevent them from talking to customers at all. Like, let's keep doing that. That's absolutely amazing. I've been at places where that does not even happen. So like let's just give them a jumping off point to just keep doing that even better.

Ash Oliver:
Yeah. What a difference to come into an organization that already had that really solid foundation running as an engine and then you get to come in and amplify that and remove any other obstacles. I mean, based on what you said, it sounds like that's kind of a rarity. Is any of that connected to, I mean, User Interviews kind of being in the nature of the space? Do you think that you're at the forefront of this kind of work so it informs the team's rituals or ways of working?

Roberta Dombrowski:
Absolutely. I had a cognitive dissonance when I started actually because I've been in so many different organizations, I did consulting too, and I've gone into so many places where I have to immediately day one put on a cape and like, "here's why research is important, here's why it's valuable" and keep selling it to stakeholders. And in my first week I was like, wait, I don't understand. I don't have to do that. I can just like do the work. And I talked to our CEO about it and he's like, "Yeah, Roberta, the whole purpose value of our company is to help enable people connecting, gathering customer insights." And I was like, "Oh yeah, that makes sense."

Roberta Dombrowski:
And we talk to researchers every single day, people who do research is what I call them. So that could be product managers, designers, user researchers. There's also people who do clinical research too. And so I hear patterns about research being done in so many different contexts and it changes our approach to how we do research too. I think it's made me very aware of some of the craft skills I want to keep working on myself. It's really interesting to see workarounds that teams do around research too. 

Ash Oliver:
That's really fascinating to see that unfold. Going back to the framework itself, when you first came in, I mean the teams were working through continuous discovery already, but they hadn't applied this framework. You've implemented this framework. Can you describe what some of the mechanics of them are? I know that there's the phases that you described. Maybe you can go in a little bit more detail into each one of the phases within the framework.

Roberta Dombrowski:
Yeah, for sure. So usually in the first phase of the framework is like decision trigger. So there's some type of decision. And so at that point I really ask the teams to get crystal clear on what's the type of decision, the level of decision that you would like to make. I gave four different levels to the team. The first is usually vision decision. So like, what's the thing we should be focusing on in terms of what's our competitive advantage in the market? How do we want to package, price? What's the philosophies of our customers or mental models? That's where a lot of those questions lie.

Roberta Dombrowski:
Then there's strategic decisions, which are typically like, "we know the problem we need to solve. What's the strategy and the way that we want to get there?" After that, there's typically definitions decisions. This is if like you follow Teresa Torres, the Opportunity Solution Tree. So you have a few opportunities you want to go narrow down into, what's the best way to do that? That's when things like usability testing might happen, preference testing. And then finally, the last level of decision-making is evaluation. So evaluative. You release it, maybe you're doing an A/B test. You're measuring "what does success look like?" Comparing it. And it's really like a cycle.

Roberta Dombrowski:
It's also important to note that I listed four types of decisions. You're probably making like 20 decisions a day. You're jumping from level to level. It's all crazy. And so really getting crystal clear on the type of decision is the first step. And then I ask teams to flip it. So, you know you have this decision, now, what questions do you have? What type of information do you need to inform that decision? And that helps to go into how you scope the evidence. Like, do you actually have that existing evidence to inform your decision or do you need to go out and gather more evidence to make your decision?

Ash Oliver:
Yeah. I love that you have the flip it aspect in there because when you think about a decision, at its root is a question. So I love that. It's really trying to understand, okay, what type of decision is this? But then, what are the root of the questions that we're ultimately trying to get more clarity on?

Roberta Dombrowski:
Yeah. And one thing that's interesting is, typically with user researchers, we typically start with the questions. And what I've noticed is that, at least through talking to researchers in my own experience too, is that we want to measure impacts. We want to measure the impacts that our research is having over time. If you take one step back and don't immediately go to the questions, but what are the decisions? I also ask people to track out what's the business impact that we think. Is there an OKR that's related to that decision? Is there some kind of KPI? And that also helps you measure the impact of your research over time as well. So that's one of the benefits of starting first with the decision, rather than just going into the question first too. It's been really interesting to do that because I do that mental flip, but it's different doing that and then marking it down and being like, this isn't actually something I do. I didn't really know that I did that before, but okay.

Ash Oliver:
Yeah. Yeah. Definitely. I was thinking about the scoping of evidence part because you have talked about in that phase of the framework, like the difference between deciphering facts, opinions, and guesses. So I was wondering if you could speak a little bit about that, but then I also want to know how this might differ if you have someone that believes something is a fact or an opinion.

Roberta Dombrowski:
Yes. It's funny you ask that because since we made the blog posts and started sharing out the framework with other people, I've actually trimmed it down to just facts and opinions because my team was like, "What is the difference between an opinion and a guess?" And I'm like, "Yeah, valid point. I don't know." Essentially facts are something that is based on data or you have evidence. It absolutely can be anecdotal. It can be qualitative. It can be quantitative. But ideally it's not something that is that gut instinct that you feel inside your body and you're like, huh, like a hunch. So there's some type of evidence, whether it's qual or quant.

Roberta Dombrowski:
An opinion, a guess, whatever we want to call it, is that hunch. It's the, "huh, I think this", or "I think we have this," but you're not really sure. There's no way to validate it. There's no customer quote. There's no highlight reel. There's no data point or analytics around it. And so typically I'll flag that as an opinion or guess, and then I typically also have that form into an assumption section of my research plan. So we're not going to completely exclude it. We'll note it, keep it in mind. And then that also informs my storytelling after the fact as well.

Roberta Dombrowski:
And this process of pulling things apart as well, it's actually brought really fruitful conversations with my stakeholders, the leadership team that I work with, because they're like, "wow, we actually don't know this for sure at all." And so we end up, when we come out of sessions, we're like, "Hey, maybe we should submit a request to the analytics team to try and start pulling this and investigating this a little bit more." So it's been really helpful for that.

Ash Oliver:
That's really interesting. I mean the stakeholder alignment piece I'm sure is a huge asset to incorporating this framework. So within the four levels of decision-making that you've just described, what are some of the associated UX Research methods that would be best utilized based on the types of decisions that you're going after?

Roberta Dombrowski:
Yeah. In UX, kind of typical UX fashion, it depends. You knew I was going to say it. You started smiling immediately. It really depends, but there are some similarities. So like vision decisions, typically that's going to be a lot of generative qualitative methods. It might be things like diary studies, one-on-one interviews. It might be competitive analysis. Really going out and like broad, being like, what is the philosophies of the market, of the customers? Where do we sit in this industry and what we're hoping to do?

Roberta Dombrowski:
With strategic decisions, it's a little similar. Typically, it's going to lean more towards generative, going to be things like one-on-one interviews. You might start to get into reviewing data analytics a little bit more to looking at existing state flows, where's drop-off rate, where are people just not being retained within the experience as well. When you start to get into definition, this is where Maze hangs out. This is like usability testing. This is the preference testing. A lot of cool prototyping is happening at that. You're trying to get feedback as you're going through opportunities and mapping out potential solutions to the problem.

Roberta Dombrowski:
And then evaluation is pretty similar, really looking at things like analytics from A/B tests. It might also be doing interviews again. It ends up being a loop. You're trying to see what does success look like from this opportunity that we either did an A/B test on or released into the wild in some way. So typically those are some of the methods. Really, it varies from study to study, of course, as always.

Ash Oliver:
Absolutely. When you're describing these different types of decisions and the methods here, this might be a very specific question given that I know that your team is working through continuous discovery, but do you find that it's a little bit segmented based on what type of role might conduct the type of research? So I'm wondering like with vision research, is it kind of more of the skillset based on your specialized researchers versus something that might be definition or evaluative that might be more on the designers or even engineers, if you are incorporating those into continuous? 

Roberta Dombrowski:
That's a great question. So typically, and part of when I was trying to figure out this framework, I mapped out like, okay, this is where researchers live, this is where products live because I was just like, how can we have these two types of research coexist in the same place and trying to make sense of it all. I believe very heavily in democratization. And so we hang out as a research team really in vision and strategy a lot of the time. The product teams, product pods will hang out through strategy, through evaluation, but any member of the research team can jump into anything.

Roberta Dombrowski:
So I typically will do coaching with product managers, with designers around definition decisions or even A/B test results, and what opportunities should we start working on? So there's a lot of coaching going on. I find also that the engineering team is involved in the continuous discovery that we use as well. So they sit in on customer interviews, they're working with the UX designers, the product designers through the definition decisions. So we definitely hire product-led engineering leaders as well. So they're involved in that too. They don't sit as much in the vision sessions. Sometimes they might observe and jump in if I'm leading an interview or doing research. But yeah, it's very communal in that sense. We're very democratized.

Ash Oliver:
I love that. At Maze, we believe in that and I see teams and what they're able to accomplish through incorporating things like this, but it's really exciting to speak to someone about it. Okay. This is a different question I want to ask. When we're talking about the different methods, are there ever instances when you select methods based on a known preference or a bias with the stakeholder?

Roberta Dombrowski:
Absolutely. Absolutely. And it's like, if a researcher or person who does research tells you that they don't have a bias towards a method, they're lying to you. Right? So I call myself a mixed methods researcher, but I lean heavily qual. I am like qual all day, all night. So I find myself sometimes leaning more towards diary studies, one-on-one interviews. And I have to actively prevent that bias. I'm doing a lot of just talking with our analytics team more, trying to incorporate that into telling the whole story as well. And our stakeholders, the people who do research on the team have similar biases as well. Everyone has biases.

Roberta Dombrowski:
I know that my fellow members of the leadership team, lean heavily quant, which makes sense. So I incorporate that into my storytelling too. We do a lot of quantifying the qualitative data from our studies and segmenting them. Sometimes we'll increase sample sizes too, to make sure that the customers we do qualitative research with, we have larger segments so we can try to quantify it a little bit more, or maybe follow up with a survey or looking at the analytics data too. Yeah, it comes back into the storytelling and getting buy-in as well.

Ash Oliver:
Yeah. Well, I'm personally really intrigued by storytelling with data. How does, if at all... So I'm assuming that the UX research study plan that you would put together would differ based on the method that you would select in this case, but is there anything else that differs in those research study plans based on the decisions or types of decisions?

Roberta Dombrowski:
So when I started, I created a research playbook with a bunch of templates to just make it easier for people to do research and us to scale research. Our study plans are pretty similar to traditional research study plans that you see, but there's a few differences. The first thing at the very top is like, what decision are you making? I'll also put, there's a section on business impact. A lot of the times researchers, people who do research, sometimes aren't exposed to that. I feel very lucky to be on the leadership team level. I can pull up our operating model and be like, here are our goals for revenue and other things. And I put that right in the plan. And so I'll also map out the assumptions based off of like the opinions as well, and that all forms into the next section, which is typically what our actual questions are and then the methods. And then that's the traditional research study plan that you might see and gets all nerdy. 

Roberta Dombrowski:
I just love craft. I just made like an amazing plan a few weeks ago. And I was with my team, I was like, "This is the most beautiful research plan I've ever seen in my life." And my team was like, "Yeah, it is, Roberta. Cool." But it's like, I just really love it. And there's so much that goes into that point that you even get it on paper, right? It took a few weeks to pull it out of the stakeholders to have the conversation, to pick the methods. And seeing it in a plan format, whether it's in Miro or in a Google Doc, that is a milestone that should be celebrated by anybody who does research. 

Roberta Dombrowski:
When you think about it, I talk a lot with teams about this, that the act of product, the act of actually creating a service or product is like magic a lot of the time because you're overcoming all of these constraints and obstacles to actually birth something into life. And so it's really important to celebrate those little things of just, oh my gosh, we're all aligned on this right now, like on this plan of talking to customers of this method or whatever it might be. And that should be celebrated. And often it's not. So I'm a huge nerd when stuff like that happens.

Ash Oliver:
I'm curious if there are any favorite ways that you've either shared or that you've seen others share like learning and insights.

Roberta Dombrowski:
I'm very like, when you talked about the transparency with the plan, I carry that through everything throughout the process. As I'm actually going through that pulling of the research plan, I typically do it in Miro a lot of the times through workshops. So I'll use the same Miro board and I create interview snapshots similar to the continuous discovery framework. After I get out of a session, so somebody might be note-taking or observing, we'll do a 15-minute debrief like, "Hey, what did you notice? What insights did you pull out?" Immediately put it into the interview snapshot. And then I write up a quick summary and put it in Slack to share it with stakeholders so they can see what's going on. And they see the insights along the way.

Roberta Dombrowski:
After we do sessions, I'll typically do more of the rigorous, just analysis. And so that might be actually going through reading transcripts, depending on the study, creating highlight reels and stuff like that. And I'll usually start to weave in the storytelling a bit more through a rapport or a PowerPoint presentation or a video share out, whatever it might be. But there's like levels of analysis in sharing out, going through the whole time. And so when we get to that big share out at the end, whatever it might be, and it has recommendations of like, hey, this team should probably do this, this team should probably do that, it's not surprising because everyone was seeing the Slack update, seeing videos, seeing highlights. Sometimes they sat in on the sessions themselves. So they're like, "Yeah, this makes sense." And in that sense, it's kind of not as monumental as it might be because they're just like, "Yep, we've been here." But it's still really great. There are really great insights to the recommendations as well. So yeah, I'm pretty collaborative throughout the whole research process.

Ash Oliver:
That's interesting. And yeah, maybe not as big of a reveal, but certainly a huge impact to have everybody aligned from start to finish. So Roberta, I'm going to ask you our hat-trick questions. These are three more personal questions that we ask every guest. I'm sure you could guess why I've entitled it hat-trick questions.

Roberta Dombrowski:
Oh yeah. It's the hockey. It's that Upstate New York. Yeah.

Ash Oliver:
That's right. Okay. So what is one thing that you've done in your career that has helped you succeed that you feel very few other people do?

Roberta Dombrowski:
I think I've mentioned earlier, I flutter around things. So my career has not been linear at all. It was instructional designer, UX designer, product manager, now researcher. And so that hopping, not normal. I don't know if I'd recommend it to other people too, but it gave me a lens in different perspectives so that now when I talk to people, I have so much empathy that I understand the types of decisions that need to be made. Like I can empathize. I'm like, "Yeah, that really sucks. You got to make that decision with not a lot of information. What can I do to help out?" And I think it builds a lot of empathy with my teammates too, because they understand, I understand. We can talk the same language.

Ash Oliver:
Yeah. Well, as someone who made a huge career transition, I admire that you speak openly about that and are a model for how much of a skillset and unique skillset that that can bring into someone's career. Okay. What is the industry-related book that you've given or recommended the most?

Roberta Dombrowski:
Teresa Torres, Continuous Discovery Habits. I love it. I hope to speak with her and meet her at some point in the future. I haven't then. I've just been like Twitter fangirling her for a few years. So I recommend it to so many people, whether you're a product manager, designer, researcher. Just hired a new researcher for our team and I do it as an onboarding gift so they can understand the culture that we're building on the team too.

Ash Oliver:
Oh, I love the onboarding aspect too. You're building that into the culture. That's huge.

Roberta Dombrowski:
I feel like, especially it's hard to see company cultures that do both. We can still do the rigorous research. You still can have the continuous discovery framework too. And so it helps. We even use it for the product managers that onboard too. We send it to them. You can see the communal aspect and understand it and then you're still going to have to process it when you get in the team too. 

Ash Oliver:
Awesome. My last question for you is, what is an unusual habit or an absurd thing that you love?

Roberta Dombrowski:
I don't know if it's unusual, but I love forensics. So when I was growing up, my undergrad was in biomedical photography and I was like, I'm going to be a forensic photographer obviously, but didn't end up doing that. Last night, me and my partner just watched Forensic Files for hours. And it's like, I don't know if it's COVID or we just love that stuff, but we were just like, "Oh, this is so calming after a day of work," which is horrible and super creepy and weird, but it's just, I love it.

Ash Oliver:
Interesting though that your career started there because it's kind of that investigative aspect from the very beginning it seems like, has that been a through-line for your career?

Roberta Dombrowski:
I think so. My mom said when I was younger, I was like an owl. I would just watch everything around me. I actually ended up switching out of that specialization in undergrad of biomedical photography because I wanted to get out of the lab. I wanted to get out of behind the microscope. And so it's interesting, now I'm in applied research, that is talking to people every day, but still doing that rich data analysis too.

Ash Oliver:
Amazing. This has been so fun. I really appreciate you being here and sharing all of this. It's been so incredible to see, not just in the framework, but also in the real experiences inside your team. So thank you so much for being here.

Roberta Dombrowski:
Yeah. Thank you so much for having me.

Ash Oliver:
The Optimal Path is hosted by Ash Oliver and brought to you by Maze, a product research platform designed for product teams. If you enjoyed this episode, you can find resources linked in the show notes. If you want to hear more, you can subscribe to The Optimal Path by visiting maze.co/podcast. Thanks for listening. And until next time.