We are thought leaders and visionaries bound by a common mission: if it’s worth doing, it’s worth doing right. Positioned at the intersection of innovation and strategy, we deliver perspectives and insight on topics impacting data, design, and business scale.

You Can’t Test Taste (with Steven Gianakouros)

Steven Gianakouros
Design Manager, Product Creative - Netflix

A conversation with Steven Gianakouros, a Design Manager on the Product Creative Team at Netflix, Los Angeles, on the term “data” and a specific metric – engagement. Engagement is often used to describe a whole group of behaviors, but what does it actually look like to measure engagement and try and incorporate those insights into both the design process, and the visual designs themselves? 

[00:07] Welcome to It’s Worth Doing Right. I’m your host, Olivia Hayes: resident creative pragmatist, and the Head of Product Strategy at Accomplice. Today we’re going to dig into the term data a little more deeply and talk about a specific metric: engagement. Engagement is often used to describe a whole group of behaviors – but what does it actually look like to measure engagement, and try and incorporate those insights into both the design process and the visual designs themselves? To get an experienced perspective, we went straight to a company that tops the engagement charts: Netflix. And not only does Netflix have an incredible amount of user data they can track – they also have a reputation for using that data to design a more compelling customer experience.


[00:40] My name is Steven Gianakouros; I’m one of the design managers on the Product Creative Team at Netflix in Los Angeles. Within the Product Creative Team, we’re responsible for the creation of artwork for our shows and the Netflix service. We have teams that work directly with many licensed partners around the globe, like Disney, NBC, and ABC in the US. We also have producers that develop artwork for all of our originals on the service, and we have a large team to make sure that those assets that we create get on the Netflix streaming service. My team specifically focuses on testing new creative strategies for artwork. We work with our UX, product, and engineering teams in the Bay Area to develop these tests and bring new experiences to our members.


[01:25] Today, we wanted to dig into the idea of engagement as a metric. Engagement is a little bit of a buzzword and digital design. So I wanted to ask you: what’s your idea of the real meaning of the term “engagement”?


[01:37] Well, first I’m glad the buzzword is “engagement” and not “storytelling” anymore. Engagement simply is the way somebody will interact with your product, your service, and your brand. Traditionally in TV and print advertising, engagement has been more around connecting with a specific message or feeling. But that medium is really hard to gauge impact. The beauty of digital is that engagement is measurable. That’s what it means to me; I think you can see what people are doing, how they’re interacting with what you’re putting out there. Then you have data or proof that your message is actually resonating. So if you’re successful, those metrics will validate what you’re doing, and if not, you could try a new direction pretty easily with digital, and compare results.


[02:19] What do you feel like makes a design engaging, versus attention-grabbing or flashy?


[02:25] It goes back to measurement. To me it’s not about the style, but about what actually works. When I first started in digital, going back like almost 20 years, it was all about the number of impressions and clicks you got on your ad. The true story is what happens after that. Does somebody learn more about that product, or they actually buy or do something? Take some action? Sometimes flashy and attention-grabby can just fill the funnel, but usually you’re not getting qualified, engaged folks. It also depends: if you look at kids content, and marketing toys, that stuff is kind of over the top and flashy for a reason. That tends to work a little different at Netflix because we let users decide if “flashy”, or maybe a little bit more conservative, works.


[03:13] With so many customers using their service on a daily basis. Netflix usage statistics aren’t the only data they use to measure engagement. So how do they wade through all of their data to get a deeper understanding of what it means for their customers to be engaged?


[03:26] We measure a lot of Netflix. I think what we can share is what my team does when designing artwork. We produce a lot of images: we produce all the artwork for the titles on the service, and we can design and test a lot of different assets when it comes to each title. So there are some that could be seen as maybe a little bit more click-baity, and some that are a little bit more conservative. But we don’t just measure clicks, right? We measure viewing. So that’s something that’s really unique to us. If a member chooses to engage with an image on their Netflix homepage, and they actually watch that show, we believe we’ve done a good job with that image and connecting that image to that member. We test a variety of designs like I mentioned; we can do frame stills or poster artwork to weird, attention-grabbing pieces, but we quickly can know and figure out which image is right for that member to watch that show.


[04:20] Yeah, I remember you talking about that in your SXSW talk. About how design can really impact a viewer’s behavior, and how that’s something you track.


[04:29] Weird thing is that’s why I don’t think it can be just attention-grabbing or even pretty straightforward. It really depends on that member, and that piece of content. We don’t know what’s going to work. So we try a wide, diverse set of imagery to test and tell us what works. If you’re clicking on a title and you’re watching more of it, you love it. So then we determined that that image is the right one for that show. Obviously I can’t share a lot of those metrics, but it’s pretty straightforward.


[04:57] It’s not like you have a specific demographic. Basically everyone watches Netflix.


[05:01] Four or five years ago, we were just in a handful of countries. And now we’re in over 190 countries. So we don’t use demographic behavior to target this artwork. It’s really about the member and what they’re watching. That’s what we really care about. So we don’t care where you live, what age you are, what sex you are. If you’re watching similar shows, that’s how we cluster what title is surfaced to you, and also what image you get.


[05:26] So is it fair to say that every individual user is getting a unique Netflix experience?


[05:32] Definitely. Everybody’s homepage is different, right? So what we do, and what the Netflix algorithm does, is it will cluster people with similar viewing behavior, and share content in that way. So it could be, you could be a 17-year-old male teenager in Germany, or a 90-year-old abuelita in Mexico, and you could have the same viewing behavior. We don’t judge, we don’t care. But if you have that same viewing behavior, we’re going to surface similar titles to you. We can honestly say there’s probably no two Netflix homepages that are alike.


[06:06] Although Netflix has access to all this viewer data, they aren’t just relying on the numbers for direction. Their process involves utilizing all of the skills they have as a team, and the information available to them, in order to make design changes they think will be the most impactful.


[06:20] We talk about data a lot, but we also use our gut, and our intuition, and some market research to make decisions on what we do in the artwork. I wouldn’t say we’re data-driven, but we’re heavily informed by the data.


[06:34] Do you think that designers naturally understand how to incorporate these data points into their process, or is it a skill you have to teach them?


[06:41] I don’t think it’s natural. I think the passion is something that designers develop over time. Most designers I know came out of art school, and art school is not really stressing data and user-centric design. They’re doing obviously a lot more with UX classes, and web development classes, and things like that. But it’s still honing the craft of design and the artists. What I’ve seen is – especially with younger designers, or more traditional designers moving into digital, or even just starting into digital – it’s leveraging both. So you want to have the craftsmanship and the aesthetic on the one side, but you also need to understand the data. So I think the ones that are successful – at least, at Netflix, too – are the ones that are probably not going to come in knowing a lot about data and how to use and how to test. But they have a passion for it. And if you have a passion for it, that will drive you to learn how to use it to your advantage.


[07:34] Do you all internally at Netflix have a formalized process for taking those engagement metrics and applying them to your design, or is it more of an informal process?


[07:42] It’s pretty informal. We don’t have a playbook where we say, “this is what you need to do for a standup special versus a film.” We do look at trends, and we have analysts on my team that will look at the artwork. And we’ll work with our Science and Algorithms Team up in Los Gatos. (That’s a fancy title… fancy name for a team.) But they go in and do some analysis around the creative, and see what’s actually resonating with people – and with what titles. But we don’t want to just defer to historical performance, because then that will just limit us, and we won’t be learning and we won’t be innovating. We use some other tools – we use our gut. But those metrics and data will keep changing with the new things that we do. So it’s still a pretty informal process, and it makes us more flexible.


[08:35] How are you able to know if your design changes have made a design more engaging? Is it something that you can measure? Can you measure progress in that way?


[08:43] Yeah, that’s what we do with image testing. We know that when we test multiple images for a title, we increase engagement by increasing the number of people watching that show. So we’re looking to do is to grow that over time. It could be testing more assets, it could be testing different types of assets. We’re always trying new things. But again, it all boils down to our people – our members – being happy watching the show, and that’s where we are today. We’re in service to our members.


[09:15] What is the most difficult part about trying to get those designers to consider this engagement data as feedback just as critical as any other kind of feedback?


[09:24] In Los Angeles, we have designers that are probably more creative-focused or -minded, where in the Bay Area, our designers come from a UX or Interaction background. I would say the people that we bring up on the UX side all have that experience. I think in LA, we work with a lot of creative agencies with super amazing creative capabilities. But we go back to having somebody there to have the passion or the understanding to learn from the data – instead of just being worried about the subjective beauty of a piece of artwork.


[09:57] So you set the expectation for your designers out of the box that they need to have that service-minded design mindset – and also, the flexibility you mentioned. They need to be flexible in their design.


[10:10] Exactly. And we talk about that internally too. We’re not designing for ourselves, we’re not designing for our portfolios – but we’re designing for our members. And that’s what our charter is. Where it’s service to our members, not to print something and hang it up on the wall in our offices. We have to design with our members in mind.


[10:29] Are there any downsides of focusing on engagement as a design metric?


[10:34] Sometimes when you rely too much on the data, or engagement, or any metric, the aesthetics can be seen as secondary. For us, it is important to have something that looks good while representing the brand. We have the craftsmanship there, but it shouldn’t be at the expense of engaging with the member, in our case. Sometimes you’ll have folks that can lean very heavily on the data side, where some of the creativity gets lost. So we have a good challenge where we try to balance both.


[11:09] That makes a lot of sense. There’s a lot of conversation around how some constraints can be really good for creativity, and can really make the design a lot better, but it’s probably a very fine line.


[11:20] Exactly, I agree, If you don’t have any constraints, then you have artwork. We’re not working with a blank canvas here. We have goals and objectives of what we’re trying to do, whether it’s from a UX perspective or an artwork perspective. Having those boundaries is great. That little sandbox so you don’t go too crazy. We have a defined goal with with what we’re doing.


[11:43] Netflix is such an established brand; obviously, you have brand guidelines and parameters to stay within. But it sounds like you also have the space for flexibility and experimentation to take it to the next level, as you mentioned.


[11:55] When I started at Netflix, I was working on the brand team as a design lead. I helped redo the logo, create the visual system – and it’s funny, but on the brand side as well, we still have that flexibility. Typically at other places, you’ll see documents with brand guidelines and strict rules. We don’t even have strict rules around the brand. We have things that are foundational, but we want to encourage people in the company, or agencies, or creative folks to really expand on that, and to push it forward. Because if we did have really strict brand guidelines and didn’t change that for three to five years, then that limits you. We’re looking for growth and trying to move fast. So if we find something new out of an exploration, that’s great. We’ll not throw out the brand guidelines, but we’ll evolve them. Nothing is concrete here at Netflix, nothing is set in stone. We have our lined principles in what we’re trying to do, but having that flexibility is great because you can try new things.


[12:56] Are there are other metrics that are important to designers?


[13:00] I don’t think there are solid metrics that you can say like, “these are really important.” Again, it goes back to the aesthetics. What we call “brand love” internally is a great metric. It’s really hard to measure. You can’t test taste, and you can’t test what people are feeling, really. But we can measure what they’re doing. So we have our own design aesthetic metrics that are not really tied to say business metrics, but you know, we try to balance both.


[13:28] Over the course of your career, what has changed most significantly about the idea of engaging design?


[13:35] I know it’s cliche, but it is the data: when I first started about 20 years ago, my first job out of art school was designing VHS and DVD covers. The only “data” we had was the CEO is saying he doesn’t like wood, so we couldn’t use any wood textures on any of the artwork. And also to make sure the title was on top, so when you would walk down the video store at a Walmart, you would see the title. Now, we have so much data, it’s really empowering for a designer. So you can validate your designs so much better. I know some people really can’t lean into that. It’s really difficult. But again, the way I see design now is that I don’t consider it artwork. I consider it service design, where we’re trying to help a member. We’re trying to get somebody to be comfortable enough to do something. So we have to service them. Having the balance of art and science is awesome.


[14:41] So that’s what you’re anticipating for the future of Netflix, and the future of design at Netflix.


[14:47] My first introduction into testing was testing emails back in the day. This is probably 15+ years ago, where we were designing an email for a retail client. It was the time where iPods were out. And they had one agency that designed this beautiful email – and this was not Apple, this was another retailer – but they just had iPods Photoshopped in the snow with holiday decorations around it. When you’re looking at an email, I think you just need to be quick and easy, and just get that information across with a very simple grid-based design. We did a head-to-head test with grid-based design, and had 3x more click-throughs. That was my introduction to it. Learning about data and using it has been so amazing. And at Netflix, every designer understands that, values that, and uses that to their benefit – not as a deterrent.


[15:43] And you guys have so many data points that you can track. It’s kind of amazing.


[15:48] Yeah, it is. And again, to the designers listening? It’s not all about data. There are things that we do even if the data says it may not be good. If we have a belief that it’s something that we should do from a design perspective, we’ll do it. We’ll build upon it from there. So data is not the end-all, be-all. There is still human intuition and aesthetics and tastes that go into it.


[16:15] So you hire designers who you really trust to make those calls.


[16:20] We don’t have a hierarchical company here. Every designer is empowered to do what they feel is right for each project, and we share, debate, and talk a lot about what we do. But yeah, it’s really about having somebody own their project and seeing it through.


[16:37] Steve, thanks so much for talking to me today. I really appreciate it.


[16:40] My pleasure. I loved chatting with you. We are hiring in LA and in Los Gatos, California. We’re looking for some talented folks that are passionate about art and design, so… jobs.netflix.com.


[16:53] With companies that value innovation in the way that Netflix does, engagement is just a fluid and evolving measurement that shifts along with human behavior. And those shifts won’t look the same from one service or product to the next. So being open to experimentation, minding your metrics and trusting your gut are all ways that Steve’s team at Netflix are able to leverage their strongest skills – or peak viewer engagement.


[17:16] Thanks for joining us for today’s conversation. If you’d like to learn more about our family of agencies, see more content from the Accomplice team, or leave us feedback, visit us at itsworthdoingright.com, or drop us an email at podcast@itsworthdoingright.com. And remember: if it’s worth doing, it’s worth doing right.

Meet Your Host

Kenzie Haynes

As a daughter of a teacher and a technologist, Kenzie was encouraged to ask questions and build answers. Now, as an Experience Designer focused on digital products, she uses that same inquisitive skill to drive outcomes across industries. When she’s not podcasting, Kenzie helps businesses ask the most critical questions and build market-leading answers using design and technology.

Understanding new perspectives is her passion and probing topical conversations is her forté, she brings both to the mic at It’s Worth Doing Right.

Interested in being a part of the conversation?