This talk was originally given at UXPA 2023 with the following talk track…

Redesigning the design interview

How candidates and businesses can find the right “fit” by rethinking hiring practices from first principles

Hi everyone. It's great to meet y'all. My name is Dani and I'm currently the head of design at Stemma - working on building a B2B data catalog.

Lessons Learned From..

5 -> 55 people

Discuss.io

~2.5k, after acquisition: ~30k

Pivotal Labs

50 -> ~500 people

Chipper

11, early stage

Stemma.ai

numbers above = growth during tenure
I've spent most of my career at startups. Some stayed small, while others hyper-scaled. One problem they all had in common was “hiring”.

When you're small and your total headcount for the year is in the single digits, every hire counts for a lot… and mistakes can cost you more than just a few months - it can lead to failed launches, frustrated teams, and worse.

When teams are scaling up, which a lot of folks found themselves doing between the end of 2019 and mid-2021, hiring managers end up having many more opportunities to find a good fit, however they also have less time to spend really crafting the hiring process - this can lead to declined offers, biased hiring decisions, or worse.

Agenda

An Initial Redesign

Updating our practices while growing

Q&A / Conversation

Let's chat about these ideas

Looking ahead

AI, Economic Downturn, etc.
I originally wrote this talk while in this second state. I was scaling up a design team at a hyper-growth company. Today, I'll be sharing the lessons I learned there, and do my best to connect them with larger trends I've seen as both a candidate and colleague to other design leaders.

When I started at Chipper as employee 54 or so, I was the first design hire. By the time I left, I had hired a design team consisting of 15 other members - including ux, visual, content and brand design - as well as researchers and middle managers.

We did a lot right… and we also stumbled a bit. Throughout this first part of the talk, I'll be sharing how feedback from candidates and a focus on first principles helped shape a more equitable, respectful and focused interview loop.

After sharing these ideas with you, I'd love to hear a bit from you all. I'd like to know if you've tried similar practices? And what worked for you?

If you've been a candidate on the other side of these practices, how did they make you feel? We'll have plenty of time for Q&A in the middle, so please feel free to save up your questions and comments for that time.

In the back half of the talk, we'll look at the topic of hiring from a more hypothetical perspective. I'll try my best to answer questions like, “how do we evaluate skills as AI becomes more prevalent in portfolios and products?” And “how do we deal with the huge influx of applicants we're seeing due to layoffs?”

All settled? Great. Let's dive in.

The way we do design hiring today is broken

The way we do design hiring today is broken. Candidates are required to do too much up-front and still have to complete a long chain of interviews to get an offer.

Even before the mass layoffs from this last year, hiring managers find themselves overwhelmed by applicants - making it difficult to decide who to move forward with and how to efficiently evaluate skills during the interview process.

The standard interview loop

This process can take anywhere from 2 weeks to 12!

InterviewLoop.png

This is what a standard interview loop looks like. Whether you're at a big company like Google or you're a startup who is trying to hire your first designer. “Everyone does it this way” for better or for worse.

ScreeningLoop.png

Hiring starts with an application screening - where a recruiter or the hiring manager takes a look at a large number of candidates’ resumes and portfolios.

Based on a rubric (or sometimes just a “gut feel”), the person reading through the applications will select a handful of candidates to move onto at least one “screening call” with the recruiter and/or hiring manager.

PortfolioReview.png

If this initial call goes well, the candidate is usually required to do a portfolio review with all or some of the members of the “interview loop”. This panel is usually made up of members of the company that will be responsible for asking questions of the candidate throughout the rest of the loop and helping the hiring manager make their final decision. But sometimes, this “panel” is just one person.

A quick side note here: if you're doing portfolio reviews, please just require everyone in the interview loop to be there. Unless you're absolutely sure an interview will have nothing to do with past work, omitting a member of the interview team from the portfolio panel will lead to candidates repeating themselves multiple times throughout their loop…

When we were still doing portfolio presentations, this was by far the most frequent feedback we heard from candidates - citing feelings that they “were not being listened to” when they had to repeat the same stories. In some cases, they event had to pull up their portfolio again in other interviews, giving the same presentation twice or even three times 😬

CritLoop.png

Then, the most time-consuming part of the loop begins. Sometimes combined with the portfolio review, a whiteboard interview is a chance for candidates to solve a problem that they will likely never face in the role they're being hired for. Over the course of an hour, sometimes longer, the candidate is asked to “redesign an ATM” or “design an app to coordinate ping-pong games at the office”.

In the worst case scenario, they're asked to design something the company is actively working on. Why is this the worst case?

Well, we initially had something like this at Chipper and quickly abandoned it after hearing reports from candidates that their work from other company's design challenges was actually showing up in those company's applications. These candidates were not paid for their contributions, nor were they hired onto those teams they interviewed for…

Now, I don't know the full story behind those situations. It's very possible the features that shipped at those companies ended up being similar to the work these candidates produced for them - without any work actually being stolen.

But one candidate's negative experience and view that they were taken advantage of can poison a whole well of possible candidates. So we took this concern as a cautionary tale, and I'd implore others to do the same if they're considering whiteboarding challenges or take-home assignments that have any possibility of being shipped.

This “take home assignment” is usually next. It's often just a higher fidelity version of the whiteboard challenge, but other times it can be a completely new assignment. Either way, the candidate is expected to come back with some polished work to present to the next round of interviewers so that they can defend their design decisions, discuss how they might test them, and answer any number of hypotheticals that the company has written in their interview guide.

In a minute, I'll share why the take home assignment is problematic - especially for those transitioning from one career to another or moving companies.

TraditionalLoop.png

But let's finish out the traditional loop.

It's at this time that the candidate is allowed to participate in the types of interviews that are more common across all industries.

These “culture” and “technical” interviews usually have questions like, “Tell me about a time that you had to settle a dispute” or “Compare the differences between mobile-first and responsive design.”

These interviews are usually the first chance that a candidate has to really meet the people that they might actually work with on a daily basis. Even if they were in the portfolio panel, there likely wasn't much time to ask about the company itself and how their potential future colleagues feel about their jobs.

Good interviews usually have time for both - evaluative questions and ample time for the candidate to ask questions in return.

Offer.png

At most large companies (like Google), this whole process can take 8-12 weeks from the first interview until the offer is negotiated and signed. At smaller companies, it's usually quite a bit shorter… but it's rare to see this loop take less than 2 weeks - given folks availability and the volume of other candidates being interviewed.

What's wrong with this picture?

For those just entering the field, this process can feel like a lot of new things, all at once. For designers who are looking for a new job, the amount of time these loops take can make it difficult (if not impossible) to transition to a new position or company.

But the process we just walked through is not just overwhelming. It's also…

What's wrong with this picture?

It's…

Inequitable

1. Inequitable - Not everyone has the time to create/update their portfolio, schedule time off at their existing job or work on “take home” assignments outside of work. By insisting that this amount of work is required to be hired, we're implicitly screening out candidates who are working multiple jobs or in hostile work environments, have family to take care of, or have hobbies outside of work.

What's wrong with this picture?

It's…

Inequitable

Disrespectful

It's…
2. Disrespectful - When a candidate is rejected after 5-10 hours of time dedicated to the process over multiple weeks, it can feel like a complete waste of their valuable time here on Earth. And, when design challenges are actually representative of a “real world” situation for the company, it can also feel like candidates are being asked to work for free.

What's wrong with this picture?

It's…

Inequitable

Disrespectful

Focused on the wrong things

Finally it's usually…
3. Focused on the wrong things - Not all design roles are the same. Some roles are more UX-oriented while others focus on visuals. Due to the collaborative nature of design, past work shown in a portfolio review rarely represents the skills required for a given role and is usually judged on a surface level - prioritizing work that “looks good” over the actual design process.

You've likely heard a lot of these critiques before. If you've been involved in hiring as either a candidate or hiring manager in the past decade or so, you've likely called out similar issues.

I'm not here to re-litigate whether or not it's worth addressing these problems. Instead, I want to share what we what happened at Chipper when we asked ourselves…

Why do we do this to each other?

After reflecting on all the ways that this hiring process disadvantages diverse applicants, I had to take a step back and ask how we got here. What were we trying to achieve with the current hiring process?

At Chipper, we identified that we were really interested in three things:

Our hiring principles

Avoiding misalignment

• We wanted to answer some important questions as a hiring team:
• “Does the candidate share our values?”
• “Can they resolve conflicts cross-culturally?”
• “Can we afford them?”

Evaluating design process foundations

• We needed to evaluate candidates on the some basic criteria:
• Can they break down problems into more accomplishable steps?
• Can they prioritize problems based on user needs?
• Do they respond well to feedback?
• Can they actually “ship it” (as in, can they work with devs, adjust to business constraints, etc.)?

Filling gaps in our team

• We had an existing team and wanted to ensure that the folks we were hiring could fill the gaps
• What skills do they have already (are they UX experts? Visual designers? Or maybe they just have some Platform-specific expertise)
• What skills do they want to learn/grow in to? (These could be IC skills, Management skills, whatever helped us track their progress post-hiring - and avoid conflicts with other team member's growth trajectory)

After a few rounds of experimentation, we came up with a process that requires no “take home” work, and combines the “culture”, “technical”, and “whiteboard” interviews into a series of interviews with the minimal number of interviewers…

What we tried…

How might we create a more equitable hiring process that respects candidates’ time while ensuring that we hire designers with the skills, values and interests needed for the roles we're trying to fill?

InterviewLoop-New.png

It looks something like this. At first glance, there's a lot that's the same here. But, like most things in our field, there's some specific design details that ended up significantly improving the lives of our candidates, team members and recruiting team.

It wouldn't be a design talk without a “how might we” statement… but I think it's helpful to frame this effort in a way that specifically targets the problems we're trying to solve. The one we came up with was:

How might we create a more equitable hiring process that respects candidates’ time while ensuring that we hire designers with the skills, values and interests needed for the roles we're trying to fill?

Before we kicked off the hiring process, the hiring manager sat down with the Talent team to align on the priorities for the role:

• This conversation ended with a job post and a rubric for evaluating skills throughout the process

Portfolio presentations for visual design roles only

For more visual roles, portfolios were part of this evaluation. Otherwise, they were not.

Portfolio presentations for visual design roles only

No screening on years of experience

We also agreed that years of experience could not be used to screen candidates in/out of the role.

This was specifically because we were specifically hiring for more developing markets and those markets didn't have the same education systems, training or access to opportunities that our recruiters were used to when hiring from the Global North.

Convincing the Talent* team

*Recruiting/HR/etc.
It was important to us to get the Talent team on board with this new way of evaluating candidates or this whole thing would fall apart.

Many of the points I've made so far about the limitations of our current hiring process was made to our recruiting partners and Head of People to get them to understand why this process needed to change.

By letting them shadow the new interview loops and allowing time for feedback from them (and candidates themselves) at the end of each loop, we gained their trust and quickly became the model they shared with other teams for how to improve their own hiring processes.

What we tried…

How might we create a more equitable hiring process that respects candidates’ time while ensuring that we hire designers with the skills, values and interests needed for the roles we're trying to fill?

InterviewLoop-New.png

After the initial resume screen, usually by the recruiter assigned to the role, but sometimes by the hiring manager themself, candidates got to talk to the recruiter. This was where the role was discussed in detail and the timing of the interview loop agreed upon.

This call helped to make sure the candidate understood the scope of the position before they committed to the rest of the interview loop.

This framing conversation was usually very casual and allowed for candidates to familiarize themselves with our unique interview process, share their timelines, or just bow out early if the role immediately doesn't feel like a fit.

Candidates were then set up with a hiring manager interview…

The “hiring manager” interview

Visual Design Prompt

Show me one project that stands out to you as your best work. Feel free to skip the process, if that's not relevant. There will be a more formal portfolio review in the full-day interview.

• For roles with a visual design focus:
  This call is framed with the prompt, “Show me one project that stands out to you as your best work. Feel free to skip the process, if that's not relevant. There will be a more formal portfolio review in the full-day interview.”

The “hiring manager” interview

Other Design Roles

Please share one or two projects that you think demonstrate your understanding of research, design and collaboration. You can show these if you want, or speak to them.

• For roles without a visual design focus:
  The call starts with, “Please share one or two projects that you think demonstrate your understanding of research, design and collaboration. You can show these if you want, or speak to them.”

The “hiring manager” interview

What's the point?

These calls were very open ended and facilitated by the hiring manager to ensure that candidates could show the best side of themselves right up front.

If a candidate didn't perform well in this interview, it was easy to make a call right then or get back to them within a couple of days with a personalized rejection email or phone call.

For candidates that did perform well, this up-front conversation with a hiring manager gave them ample opportunity to ask about the role and evaluate whether or not the rest of the interview process was worth their time.

Other than the “design studio” interview, which replaced the traditional “white boarding” and “take home” assignments, it was this hiring manager interview that got the most positive feedback from candidates.

What we tried…

Portfolio reviews for Visual Design roles only

Interview-visual.png

Visual design candidates then presented at panel-style portfolio review, where they shared 2-4 pieces of work that demonstrate their understanding of research, design and collaboration with all members of the interview loop.

The panel that was assigned for this interview had an understanding of Chipper's Design Principles and how to look for them within a portfolio. They were also the folks who would be interviewing the candidate 1on1 later on.

In our process, the portfolio review was for visual design roles only. For non-visual roles, we were finding very little signal from that given the rest of the loop.

What we tried…

Replacing “homework” with “together work”

Interview-together.png

That “take home” assignment and the whiteboard interview that preceded it was replaced by two “workshop” style interviews

First, a member of the Product or Growth team - someone who would likely be working with them on a daily basis - presented a business problem framed by data. The candidate can ask questions, discuss options and frame a problem that can be solved by design.

That's it. That was the first interview. The output wasn't graded. They could take notes or not. How much the candidate understood of the presentation would be evaluated in the next step. The rubric in this “product interview” was instead focused on how well they asked questions and collaborated with the product owner to break down the problem into smaller pieces.

Then, a follow up call with a member of the Design team starts by asking the candidate to introduce the problem in the context of a Design Studio. The candidate and the interviewer must then sketch multiple possible solutions to the design problem and share their sketches with one another.

At the end, the candidate is asked imagine how the solution(s) might be built and next steps they'd take if it were their project.

This is probably the most casual interview of them all, since candidates get to work alongside the designer running the interview to generate ideas and give each other feedback.

It's, also, by far the interview we got the most positive feedback on and where we stood out when being compared to other companies that the candidate was considering.

What we tried…

For senior roles, there were only two additional interviews

interview-senior.png

For more senior roles, there were only two additional interviews

One was the “Values” interview. Usually run by a member of our Operations team, this interview focuses on common and unique situations that occur in a remote startup. Candidates are asked how they might deal with them or have dealt with them in the past.

The other interview was with a member of our Engineering team. In this call, the candidate and interviewer discuss tradeoffs between the worlds of design and engineering. Past experiences are compared to ideal situations as the interview focuses on how the two roles are involved in product development.

What we learned

With fewer steps and specific goals for each interview, we were easily able to incorporate our company's goals for interviewing designers into this more streamlined process.

In addition to allowing candidates to opt-out early if they felt like the role isn't a good fit for them, the recruiter screen at the front also allowed for candidates to candidly ask for accommodations if the timing/approach doesn't work for them.

We actually ended up incorporating modified version of the “design studio” interview to work for disabled candidates - including a text-only script writing studio that assesses the same skills without the visual element.

We got lots of positive feedback from candidates as these new interviews came together. But I think it's time I asked…

What do you think?

Up next: Looking ahead
(AI, Economic Downturn, etc.)
What do you think? Do you prefer this approach? Did we miss something totally obvious?

For folks in the room who haven't been on the hiring side before, how would you feel going through a loop like this? Is there a more equitable process that we could adopt instead?

Looking ahead

I haven't been in the hiring manager seat for the past year or so, since I moved to a much smaller company. But I've been speaking a lot to my colleagues who are still hiring and I've noticed a few trends that I think are worth bringing up in the context of how we hire.

AI killed the portfolio star

The first of which is AI. It's everywhere. From cover letters generated to ChatGPT to entire portfolio presentations built with tools like Notion, Tome and even PowerPoint.

Just the other day, I stumbled upon this video that showed how Framer could be used to generate a UX portfolio website with just a single prompt.

@zander_supafast
Example Site: https://uxpa2023-example.framer.ai/

video.mp4

The person in the video runs a UX bootcamp - one of the many that's sprouted up over the past few years - to support the huge number of people who want to get into our field.

In the interest of transparency, I will actually show you a case study I generated using the same tooling. https://uxpa2023-example.framer.ai/

AI killed the portfolio star

It's not great. But it's really just the beginning. Swap out this content with a better trained model, or just some clever ChatGPT prompts, and you've got all the text we'd expect.

Add in some images from Midjourney or some new model trained exclusively on dribbble shots, and you'll end up with a portfolio site that's virtually indistinguishable from the hundreds of sites hiring managers have to review every day.

The line has officially blurred between the things that people produce with the machines and the things that machines can produce with limited input from people.

In mentorship spaces, social media and university, we talk a lot about how portfolios should be laid out - what information hiring managers are looking for in them - and how important they are to convey the communication skills that employers value at a premium.

But regurgitating content into prescribed formats is one of the things that generative AI is best at. So now that a portfolio that fits into the model we've been teaching designers to produce is just a prompt away, are they even worth reading anymore?

The Magnet of Mediocrity

Source: Prakhar Mehrotra

mediocrity2.png

I know I'm not the only one who spends less than a minute on a portfolio site or pdf to qualify or disqualify a candidate. Every role has different evaluation criteria but it's pretty easy to separate the folks at the bottom of this graph, the “Novice"s from the people further up.

The Magnet of Mediocrity

Source: Prakhar Mehrotra

mediocrity1.png

In the article that this image is sourced from, this area in green is described as the place where people are good enough at a skill that they can get paid for it. It's the people in this area that we're looking for when we're evaluating resumes, cover letters, portfolios and responses to our interview questions.

The Magnet of Mediocrity

Source: Prakhar Mehrotra

mediocrity5.png

But what happens if we introduce an AI into this graph? Well, it ends up sitting right here - at the middle.

After all, ChatGPT and other generative models like it are essentially producing the most average of all content available on the internet.

But the average was in our green area.

The Magnet of Mediocrity

Source: Prakhar Mehrotra

mediocrity4.png

And, as it turns out, having everyone below the average looking a lot like the average really crowds up the space we're trying to evaluate. Especially when we're trying to hiring junior and associate designers.

The Magnet of Mediocrity

Source: Prakhar Mehrotra

mediocrity6.png

In his post, Prakhar also posits that above-average folks will also be drawn to use ChatGPT - bringing them down into the middle, as well. I'd tend to agree, especially if you consider my point in the first half of the talk about designers who don't have time to make a portfolio because they're working or spending time with their family.

So we are left with a landscape where only two tiers of skill exist

Source: Prakhar Mehrotra

mediocrity3.png

So we are left with a landscape where only two tiers of skill exist. This top-level genius tier, and the flood of designers all drawn toward the magnet of mediocrity.

In my experience hiring and talking to other hiring managers, portfolios are not usually used to identify this top-tier, genius level. These folks usually come in from recommendations or “head hunting” recruiters.

Instead, we tend to use portfolios as a bar - sometimes an intentionally low bar - to easily screen out candidates who don't have enough experience to contribute meaningfully to a design team.

Now that it's much easier to clear this bar with AI tools, I'd like to know…

Has AI killed the portfolio?

And, if so… how do we deal with the reality that we can't interview everyone

We can't interview everyone

This was the most painful part of being a hiring manager. And I'm sure many of you in this room feel the same way. I know, in my heart of hearts, that there's amazing people that slipped through the cracks during my initial screening.

Maybe it was their resume, unconventional education, or a poorly formatted portfolio. But I'm sure I missed a few perfectly shaped needles in the haystack.

There's a reason why more than 30% of jobs are filled by referrals (source)… it's really, really hard to evaluate candidates without talking to them. But we have to pre-screen somehow.

This has gotten even worse in the past year or so. As so many companies have been doing layoffs, the market is filled with talented people who would love to work where we're hiring.

I can list on both hands people in my network who are currently looking for work - from new grads to folks who have led dozens of teams. And all of them have been met with a faceless wall of rejection emails… or just ghosted altogether.

At Chipper, we spent multiple quarters redesigning our interview process from the first interview onwards. But I think it's time we expanded the scope of our problem to include initial screening, as well.

To do that, I'd like to pose a slightly modified version of my problem statement to y'all.

How might we create a more equitable hiring process that evaluates the skills, values and interests of candidates (not their AI assistants) while still respecting their time, effort and lives?

Full transcript and slides can be found at: dreamindani.com/talks/uxpa2023

If you're a hiring manager, I'd like to invite you to bring this back to your HR team and try to solve it together. If you're a candidate, I invite you to try to solve this yourself and add it to your portfolio. I know that if I saw a case study trying to solve something like this, it'd certainly stand out above the magnet of mediocrity.

And, if there's time, I'd like to try to solve this together with all of you. Can we solve this? If not, what barriers are there between us now and a solution in the future?