My last job search lasted 100 days. I went through 11 onsites at companies like Google, Airbnb, Lyft, Uber, Amazon, Square, Stripe — you name it and I did an onsite there. Out of the 11 I landed 10 offers, mostly as Staff Engineer, ultimately choosing Airbnb.

The process changes drastically as you move into the Senior / Staff levels. I wanted to write this essay to draw an outline of the kind of preparation and change in focus that's needed and to do well in these interviews. I've seen little written about this, and I hope it's useful to you. This essay is formatted like a "100 day curriculum" of sorts. I hope you enjoy it!

Day 0: Foundations

To kick off the search, there are a few mindset shifts that if done well, can make everything else simple. In fact, I’ll venture to say that some of these mindset shifts when generalized can change your perspective… on life itself!

Okay okay that’s a serious statement, but give this a read, try it yourself and let me know what you think after 😊.

Foundations I: Attitude

Most people dread the job search. They think it’s a kind of performance, and “they just want it to be over with”. This kind of attitude can shoot you in the foot. If you dread something, you’re less likely to invest the time needed to get great at it. That can produce negative experiences, which reinforces your opinion and causes a spiral of bad outcomes. Plus, if you “just want it be over with” you’re more likely to choose the first few offers and miss out on the opportunity that’s really right for you.

Let’s shift this. Learn to love the job search. You can look it as an infrequent, leveraged opportunity for you to grow. This is your chance to select exactly the right next move for your career. It’s one of the best times to negotiate your role and your compensation. Plus, it’s a great opportunity for you to expand and deepen your relationships: you’ll lean on your community throughout the search and meet many talented people at some amazing companies.

Well, that’s a lot of real good reasons to love the job search. Internalize this attitude, and every step that follows turns into a meaningful part of your life story.

Foundations II: Community

Most people treat the job search as a solo activity. They apply for jobs by themselves, they negotiate themselves, and they choose themselves. I think this is because people associate asking for help with dependance, and they’re afraid to make themselves vulnerable.

Now, this is fine, it’s an independent mindset. But we can go further. Let’s shift to an interdependent mindset. Think about it, teams achieve great things, not individuals.

Form a team around your search and involve them in every phase. Who are the mentors who can show you what you didn’t know you didn’t know? Who are the peers that can train you and give you advice? Who are friends who can refer you to the right companies? Engage them from the start.

For example, before starting the search I met up with friends and mentors to get their advice on what to do. I can honestly tell you that these conversations opened paths I could never have conceived before. People who champion you can give you ambitious advice that you may be afraid to see, and that changes the game. A special shoutout to my previous co-founder and life-long friend Sebastian Marshall, whose strategic advice changed was invaluable to me at this stage.

When I did start the search, I created a document that tracked the companies I was aiming for, alongside todos. This made it easy to share where I was at with my community, and easy for my community to discover ways they can help. Through this I received referrals at all 11 companies I aimed for.

When I began, two friends took the time to write up detailed advice from their last job search. Many, many friends took the time to train me. One of them was 12 timezones away and made time to skype for hours. They showed me where the bar was at, and gave me the feedback I needed to iterate.

Had 1 day in New York, 3 life-long friends took the time to give me personal training

Abraham Sorock, Alex Reichert, Andrew Wong, Daniel Woelfel, Giff Huang, Jacky Wang, Kam Leung, Mark Shlick, Robert Honsby, Sean Grove special shoutouts for the support and advice you gave me 😊. There’s many more — if you’re reading this and I haven’t included you, it’s mainly because I thought you preferred privacy — let me know and will update!

You’re starting to get a sense how interdependent this was. If I listed every person, I think it would surpass a hundred people. I truly believe if you embrace this mindset and look at problems as a team, you’ll see a huge change in all you go for, not just the search.

Foundations III: Narrative

We get to the final part of the mindset, how you look at your career. Most people, if you ask them “Why are you looking for a new job”, will give you either a generic answer — “I’m looking for a change”, “I want to be impactful”, or some negative story — “I was bored”, “_I didn’t like X”. This hurts you in at least three ways. First, it’s what everybody says, and it demonstrates that you haven’t thought deeply about your career. Second, it makes it hard for people to help. If you tell your mentors “you’re looking to be impactful”, that doesn’t narrow much down, and They can’t just open up their rolodex to email every company they know. Third, it makes it impossible to be a _strong fit for any company. What can a recruiter note down if you say “you’re looking for a change”, which will let them think you’re a perfect fit for their company? If you’re looking for any company, then by definition you can’t be a strong fit for a specific company.

Instead, let’s shift your viewpoint and take charge of your career. Look at your career as an adventure. Sit down and reflect: What got you here? What did you love about your last job? What would the most amazing opportunity look like? As you go deeper, you’ll discover reasons that will get your thrilled, and show you how unique and awesome your path has already been. It will make the search a lot more fun, it will make it easier for people to help, and get recruiters very excited about you. Most importantly, it will change how you think about yourself and your career. Let me be clear, this isn’t some trick to change your viewpoint artificially. If you dig deep, you’ll discover real, true reasons, and that will reverberate throughout your life.

Day 1 → 5: Kick Off

Okay, now we’ve learned to love the job search, we see it as a team endeavor, and we have a strong conviction about our career. It’s time to kick off the search. To do that, let’s do two things:

Kick Off I: Communication

In every interview you do, people aim to extract signal and scope. Signal is what demonstrates that you can do your job — from coding to designing systems. Scope is what demonstrates the kinds of problems you can take on — do you lead a team, a team of teams, or the whole org?

The combination of signal and scope is what determines your level. Your level, and where you fall on the spectrum is the most leveraged decision for determining your compensation and the kinds expectations you have.

Many people get annoyed about this. It feels very bureaucratic. Why should some number define you? You may also know people who have “high levels” but produce less. Thinking this way can hinder you. Another way you can look at it is this: it’s how things are for now, and it’s the best way we know so far to align your skills with the company. Let’s lean in and work with it.

Your goal for the first 5 days is to form a conviction about your level. Go over expectations, and prove to yourself that you are the level you are. If you want to do go deeper on this, visit our course and see the “Communication” module — it gives you a leveling guide to calibrate. Once you’ve done this, update your resume and LinkedIn to communicate that level. When you speak with recruiters and interviewers, you should be able to freely communicate it as well. Finally, go to levels.fyi, and form a conviction about the kind of compensation ranges you’re looking for too.

Kick Off II: Schedule Screens

From consulting with your community and forming your narrative, you should have about 10 or so companies that you could see yourself being thrilled to work at. Now, engage your community and find a referral for each company. You can do this by searching through your LinkedIn, or sharing a document of your top choices with your community and asking them if they know people who can refer. Do the best you can to go through referrals, as having someone to champion you helps in many ways throughout the process. If you can’t find someone, it’s okay to send a direct application, but do that as a very last resort.

With all recruiters, ask to schedule a screen about 20–30 days in advance. They may be surprised, but you can tell them you want to prepare, and that will send positive signal too. With that, you’ll have a deadline, and a real good reason to start studying 😊.

Day 5 → 30: Preparation

Now comes the hard work. It’s time to ramp up for the technical interviews.

Preparation I: Key Mindset

The mindset at this stage I want to stress, is to give it all you’ve got. Sometimes, as a way of coping with rejection, people give themselves an out — “Oh I didn’t really study anyways”. My suggestion is to frame your success on how hard you prepare, not on the results. Aim to remove any reason to say, you didn’t try your absolute best.

Fun story on that — months before the search, I had trip planned with one of my closest friends to go to Israel and Jordan, and it fell right during preparation time…

Elements of Programming Interviews, 6am at a Bedouin camp

Well then, we need to travel and study at the same time 🙂. A special shoutout for Jacky Wang for being such an awesome travel partner, and being an encourager for the studies every day. Jacky’s the kind of guy who not only pushes the envelope to explore (just check out his photos), but he’ll indulge and converse with excitement about programming questions for hours on long car rides!

Preparation II: General Advice

There are three pieces of advice that apply generally to all interview types. First, the process to study: I recommend self-study first, then schedule practice interviews, then try the real thing, then calibrate and repeat the process if necessary. Second, approach every interview as if you are solving the problem for real. Most people seem to “pretend” when they solve the problem. They may not go deep enough, write code that won’t run, explain things for the sake of explaining, or create a system that won’t scale. Imagine a team at work set up a meeting and asked you to solve the problem, and act accordingly. Third, most people are blissfully unaware of what an excellent performance looks like. They assume the interviewer will read their minds and see how amazing they are. As you can guess, this won’t work. Study each interview type deeply, and get feedback on how you’re doing from your peers. Calibrate, make sure you are aware the bar, and you are well above it.

…Now, for some more specific advice

Preparation III: Algorithm Interviews

Unless you’re a specialist, this is the interview type to master first. No matter how skilled and senior you are, algorithm interviews play a significant role. I suggest you pick one book — Joe and I prefer Elements of Programming Interviews — and work through it linearly every day (personal nit: skip the first chapter in this book on bitwise operators, this is the most annoying part of the book, and isn’t needed for most interviews). If you get stuck on a question, just look at the answer, but make sure to write it with your own code, line by line. By the end of it, you’ll be surprised with all that you learn, and I guarantee you, at least if you go through this book, you’ll rarely be asked a question that you can’t map to something you’ve learned.

One common pitfall here is, people go on leetcode and do a bunch of random questions. I suggest avoiding this — since the questions are random, you won’t see improvement quickly, and you may not learn what’s needed. Leetcode is great though for finding and doing specific questions.

A week or so after you’re into the book, schedule practice interviews. My favorites are https://interviewing.io and https://pramp.com. A special shoutout to the founder of interviewing.io, Aline Lerner. There was an obscure 19 year-old on hackernews who was having trouble coming to the U.S. She took the time to help, and even gave him free access to her lawyers. That 19 year-old was me, and I am still grateful 🙂.

The third or fourth week, schedule practice interviews with your friend group. This is where you’ll really get a sense of where you’re at. If the feedback is positive, you’re ready for the screens. If not, _move them. Y_ou’ll be surprised how many times you can do that. We go quite deep on this, and the kinds of questions you should be able to answer, in the “Algorithm Interview” module of jobsearch.dev.

Preparation IV: UI Interviews

For UI Engineers, I suggest three steps. First, get comfortable with online editors, you’ll be using them all the time, so you should be able to set up an environment quickly. Second, ramp back up on some UI concepts that you may not have worked on recently — things like drag and drop, selection apis, etc. If you’re looking for more specific inspiration, we’ve included some homework in the “UI Interview” module of the course. Third, explore the latest and greatest in JS — look up what interests you — from react hooks to all that’s new in GraphQL. If you’re looking for some courses, Wes Bos has some great content to ramp you up. The key to these interviews is to demonstrate your ability to build UIs and your love of UI engineering. If you follow these steps it’ll come through naturally.

Preparation V: System Design Interviews

For System Design, I also suggest three steps. First, master the structure — scope down, go broad, outline a complete solution, go deep on as many components as you can, then go the extra mile, to share how you would actually launch the system. The “System Design” module in the course does a great job of going deep on this. Next, as you go through the practice interviews, you’ll start to notice places where you can improve. Start noting those down and build a plan to learn. For example, if you don’t know how to scale, you can search “{Company Name} InfoQ” and find some awesome talks. If you’re unsure about concurrency, the book 7 concurrency models in 7 weeks can ramp you up. Don’t know much about databases? 7 databases in 7 weeks. Pick what interests you and start going deep. This is my favorite interview type, because as you go deeper, the skills and knowledge you learn transfers into making you a better architect.

Preparation VI: Experience Interviews

The purpose of the Experience Interview is to understand your scope — the kinds of problems you can solve — and whether you are a culture fit. This interview type largely takes care of itself if you’ve centered on your narrative (you know what you want and where you’re going), and your communication (you know what level you are). The first coneys your culture fit, and the second your scope. You can go deeper on this in the “Experience Interview” module in the course.

Preparation VII: How to Focus for these 20 days

Now you’ve got an outline of each kind of interview, and how to prepare. For the first 20 days, I suggest focusing heavily on Algorithms. If you’re a UI engineer, still do algos, but spend maybe 30–40% of the time building UI components too. You want to focus on this because the screens are mainly this type of interview. As you pass the screens, you’ll have to go deeper on systems before you head into the onsites.

Day 30 → 60: Execution

Okay, we’re now ready to execute. Over the next month, you should be finished with all your screens and on-sites.

Execution I: Key Pieces

Three key pieces of advice here:

Come in to every interview with a positive mindset, and treat everyone with class. These people could become your friends, and it’s fun to learn about how a company works. Come with that mindset, and the confidence will seep through.

Batch all the interviews. Make sure all the screens and onsites are scheduled around the same time. The preparation you can do part time, but this piece Joe and I highly suggest, you figure out how to do full-time. When offers land around the same time, you’ll have a lot more leverage.

Communicate your level and your narrative. Keep signal and scope top of mind, and communicate clearly during every phase — recruiter screens, technical screens, and and onsites. We go deeper on this in the “Interview Phases” module of the course.

Execution II: Dealing with the Unexpected

As you go through the onsites, two things can happen:

You’ll fail at some interview

As you may have noted, I did 11 onsites, and got 10 offers…that means I failed one of em 😄. Even with some of the most intense prep, sh*t happens. When it does, do not over-react, embarrass yourself by being mean, or change your plans and cancel all upcoming interviews. Instead, treat everyone with a smile and with class. Then, be kind to yourself, take out what lessons you can, and continue on with the plan. If you see failure on more than one or two interviews, consider re-scheduling and re-calibrating.

You’ll get something unexpected

Life can throw curve ball at anytime, and you’re unlikely to avoid it during the job search. Case in point:

Don’t try huge jumps on your skis, when you don’t know how, especially during your job search

On Christmas break, Joe and I went skiing. I did an unintended backflip, and landed in the ER. When something like this happens, again, avoid over-reacting. Treat surprises as an opportunity to be malleable, and make it part of your story. For example, the day I visited the ER, I had a conversation with one of the best recruiters I’ve ever met, Twitter’s Matt Robbins. I sent him that picture, and I wager that will be the most unique recruiter conversation either of us will have in our career.

Day 60 → 100: Choice

Around Day 60, offers should start to roll in, and it’s time to choose. The first surprise that people may have here, is the number of days. Most people take about a week or two to decide…Day 60–100 is 40 days! Does it really take that long to choose a company, even after you’ve received your offers?

Yes!

Let me be clear, this isn’t about negotiation. You’ve already done 90% of the legwork for that through your preparation and execution — if you communicated your level well, you’ve aced the interviews, and you landed your offers around the same time, you can safely assume every company will be competitive and get you the best offer. There are specific things you can do at this point — the main advice being to use clear with what you would sign right away with. We go deeper on this in the “Negotiation” section of the course. Patio11’s Salary Negotiation and Rands’ The Business are great reading too.

The reason this takes so long is because the purpose for this phase is to choose the company that’s the best fit for you. It’s your turn to interview back. Meet your team, meet your manager, really align on the kinds of work you would do, and let your future team show you how they could be a part of your narrative.

To give you context, Stripe, Airbnb, Square, and Google were my final choices. I ended up meeting 6 managers at google, and visited Stripe, Airbnb and Square offices at least 7 times each. By the end of these meetings, you’ll have formed friends, and have gotten a great sense of what it’s like to work at each company.

The final step before making the decision, is to get feedback from your community. I suggest you write out an essay that goes over your decision, and discuss with your peers and mentors — do they agree with your thinking, what do they see that you don’t, do they have any suggestions?

To get a sense of how this looks, here’s a compressed screenshot of the essay I sent my friends and mentors: “On Choosing”

Don’t squint, it’s not meant to be read

It’s a bit too personal to share outright, but I wanted to give you a sense of the kind of deliberation that went behind this: thousands of words, and insights from more than a dozen of my closest friends and mentors. These were amazing companies and the decision was close.

If you do this well, you’ll have clear, strong conclusions for exactly why you would choose a certain company. Compensation will be far away from the main reasons, and you’ll be thrilled to sign.

Day 100: Breathe

This will be a day where you breathe and realize how exciting, yet how tiring things were. Take a break and plan a celebration with your community.

With that, we’ve covered the advice 😊.

Final Notes

Want to go deeper?

If you made it this far, and you’re pumped, get ready to go deep into the rabbit hole. Joe, Luba, and I, with the help of Aamir Patel and Jordan Yagiello, expounded on all the lessons learned here, and created a video course that will walk you through it all. From leveling guides, to example narratives, to deep explanations on system design, UI interviews, and more, you can get it there. Best of all, it’s 100% free. We were thinking of making it paid course initially, but after some long talks, we wanted to share this with our community 😊. Go to https://jobsearch.dev and see for yourself.

Onwards from Airbnb

My best friend and co-founder Joe and I left Airbnb, to start a company of our own. We are currently exploring fitness, figuring out what it takes to reach advanced levels of performance. If that kind of stuff interests you, sign up for our waitlist here.

This blog is built using https://essay.dev

I didn't write any code, nor I configured/deployed anything to get it up.

I just opened GitHub app on my iPhone and created an issue in essay.dev's repo. That's it. 🚀

No computer

Looks pretty neat! Didn't know blogging would be so easy. What's stopping you from blogging now? Just create issues and write out your mind. Put your ideas in public and see them evolve.

Why I write (and why you should too)

The end goal is to think clearly. Writing helps me get the thoughts on to paper and give them a direction.

And when you have a thought or idea clearly formed, publish it so others get inspired.

More about me - https://aravindballa.com/

Do you already have a blog? Share it below 👇

When it’s released, React 18 will include out-of-the-box improvements including:

• Automatic batching
• New APIs like startTransition
• A new streaming server renderer with built-in support for React.lazy

The React team has also taken a new step by creating the React 18 Working Group to provide feedback, ask questions, and collaborate on the release. The Working Group is hosted on GitHub Discussions and is available for the public to read.

React 18 Working Group

Members of the working group can leave feedback, ask questions, and share ideas. The core team will also use the discussions repo to share their research findings. As the stable release gets closer, any important information will also be posted on the React blog.

Because an initial surge of interest in the Working Group is expected, only invited members will be allowed to create or comment on threads. However, the threads are fully visible to the public, so everyone has access to the same information. The team believes this is a good compromise between creating a productive environment for working group members, while maintaining transparency with the wider community.

No specific release date is scheduled, but the team expects it will take several months of feedback and iteration before React 18 is ready for most production applications.

• Library Alpha: Available today
• Public Beta: At least several months
• Release Candidate (RC): At least several weeks after Beta
• General Availability: At least several weeks after RC

More details about the projected release timeline are available in the Working Group.

Create React App with Vite's React Template

yarn create @vitejs/app ajcwebdev-react18 --template react

Install dependencies

cd ajcwebdev-react18
yarn

Start development server

yarn dev

Install react@alpha and react-dom@alpha

package.json

If we look in our package.json we'll see the following dependencies included from the Vite template.

"dependencies": {
  "react": "^17.0.0",
  "react-dom": "^17.0.0"
},

Install alpha versions.

yarn add react@alpha react-dom@alpha

Check your dependencies for the new versions.

"dependencies": {
  "react": "^18.0.0-alpha-e6be2d531",
  "react-dom": "^18.0.0-alpha-e6be2d531"
},

Use esbuild.jsxInject to automatically inject JSX helper imports for every file transformed by ESBuild:

// vite.config.js

export default defineConfig({
  plugins: [reactRefresh()],
  esbuild: {
    jsxInject: `import React from 'react'`
  }
})

main.jsx

import ReactDOM from 'react-dom'
import './index.css'
import App from './App'

const root = ReactDOM.createRoot(
  document.getElementById('root')
);

root.render(<App />);

App.jsx

import logo from './logo.svg'
import './App.css'

function App() {
  return (
    <div className="App">
      <header className="App-header">
        <img
          src={logo}
          className="App-logo"
          alt="logo"
        />

        <p>
          React 18 Deployed on Netlify with Vite
        </p>
      </header>
    </div>
  )
}

export default App

Deploy to Netlify

touch netlify.toml

[build]
  publish = "dist"
  command = "yarn build"

Create a blank GitHub repository at github.new.

git init
git add README.md
git commit -m "first commit"
git remote add origin https://github.com/ajcwebdev/ajcwebdev-react18.git
git push -u origin main

Connect your GitHub repository to Netlify.

The build commands are included from your netlify.toml.

$ yarn build
yarn run v1.22.4
warning package.json: No license field

$ vite build
vite v2.3.7 building for production...
transforming...
✓ 26 modules transformed.
rendering chunks...

dist/assets/favicon.17e50649.svg   1.49kb
dist/assets/logo.ecc203fb.svg      2.61kb
dist/index.html                    0.51kb
dist/assets/index.7cb030a3.js      0.39kb / brotli: 0.20kb
dist/assets/index.0673ce28.css     0.76kb / brotli: 0.40kb
dist/assets/vendor.9aeda92c.js     134.00kb / brotli: 37.26kb

Done in 4.86s.
​
(build.command completed in 5.1s)

Set a custom domain.

Go to your new domain.

You can find all the code for this article on my GitHub.

abc do-re-mi

Want to set up GraphQL subscriptions on GitHub with your custom GitHub app?

Here's a video showing how to set that up:

This is also a test of embedding mov files directly in GitHub issues to be displayed in OneBlog.

This is a test

limited supply crisis continues to disrupt thousands of mchenry utilities customers in the castro distrcit

just tryingt for step in range(10):print(step)

Be sure to subscribe to my blog for more content.

Before we get into the details of what localizer is and how it came to be, it's crucial that we look at what developer environments were and the motivations behind the ones I create.

What is the Goal of a Developer Environment?

Ever since I wrote my first ever developer environment for the now-defunct StayMarta, I've always focused on one thing: ease of use. For a docker-compose based development environment, this was a relatively simple task. Create Docker containers, write a compose file, declare ports, and they'd be available on my local machine. It was as simple as docker-compose up. While this approach didn't necessarily match what production looked like, it was really the best option available for containers at the time. Options that exist now, such as Kubernetes, weren't available, which rendered equality between production and development a work-in-progress dream.

Scaling Up

Fast forward to 2017, at my first larger-scale startup, Azuqua, I had a chance to reimagine what a developer environment looked like under a whole new set of constraints. While Docker Compose works for small teams, it falls apart when you map it to production systems. At the time, our production system was based on Chef. Docker Compose doesn't map to Ruby-based Chef configuration files. When I joined Azuqua, the pain around having separate tooling for infrastructure had become incredibly clear. It wasn't sustainable to have an entire team; write the configuration for our services, communicate with developers why infrastructure can't infinitely scale without good software design, and do it all without blame or single points of failure. Fundamentally this is why I take issue with DevOps teams and prefer promoting the Google SRE Model instead. While at Azuqua, we started a transition to an SRE model and used Kubernetes as a way to help facilitate that.

Introducing Kubernetes

While at Azuqua, I identified the need to run Kubernetes to ease cloud resources' scalability and improve the developer experience. At the same time, this drastically decreased the developer experience. While this may seem contradictory at first, it's essential to consider the multiple ways that developer experience presents itself. While at first, it may seem like it's just tooling to test/write code, it's a combination of testing code, writing code, and deploying that code. With the Docker Compose environment at StayMarta, we made the test/build cycle incredibly simple but shifted the deploy aspects onto a bespoke team, the DevOps model. That approach works for small teams, but as you grow, this quickly doesn't scale. If you can't deploy code efficiently, the developer experience is frustrating and promptly turns to an unhealthy relationship with the team responsible for that cycle.

So, how exactly does Kubernetes make this better then?

Going back to how Kubernetes improved the developer experience, which I assure you it does. The benefit to Kubernetes was that it brought a control plane into the mix and focused on being just a container orchestrator. The DSL it does have is highly specific to orchestrating containers and making them work. The control-plane allows self-healing and building tooling to enable all different types of use-cases consistently. While it lacks abstractions, it brings something to developers they've never had before, the ability to deploy code reproducibly. With the introduction of minikube, KinD, and others, you could now run the same stack you run in production, but locally.

The ability to reproducibly deploy helps both deployment confidence and the amount of time needed to get from nothing to running in production. However, it's not without its faults. Whenever you start moving deployment tooling onto developers, you've decreased the developer experience. It's unavoidable because you're introducing net new materials, DSLs, and more for the developers to learn. While KinD and minikube are great projects, they all suffer from needing developers to understand how to work with Kubernetes. You need to be able to build a Docker image, push the docker image into the cluster, delete the application pod, verify if it's even using the correct tag, wait for Kubernetes to recreate the container, and make sure you've configured an ingress route to access it outside of your cluster or use kubectl port-forward to access it. The second that breaks, you're now required to dig into why service connectivity isn't working, why your Docker image isn't in the containerd cache, or other not so easily solved areas. While to someone who's worked with Kubernetes for years now, this isn't very difficult, this is hardly achieving the "ease of use" goal I have.

How do we make developing on a Kubernetes developer environment easier?

Solving these problems is not easy. Debugging Kubernetes is difficult and requires knowledge that can only be solved with education, more tooling, or a combination of both. Unfortunately, these are not cheap. Education is hard to do correctly and tends to result in lots of time writing quickly out of date material that is hard to maintain. While at Azuqua, we encountered this same problem. Very few engineers wanted to learn Kubernetes or invest time in the technology around it. We decided to go back to the basics and focus on a tooling based approach. How do we get the same developer experience level, at a minimum, of Docker Compose with Kubernetes? Our answer ended up being a tool called Telepresence. Telepresence describes itself as:

It seemed perfect, a tool that enabled your local machine to act as if it were running in Kubernetes and allow your services to be targeted from inside of Kubernetes. We rolled it out at Azuqua after initial tests showed it worked well, and we went with it. Unfortunately, it didn't last.

The Problem With Current Local Service Development Tooling

While Telepresence solved the problem of communicating with our local Kubernetes cluster, it spectacularly failed at letting services inside the cluster talk to our local services. When it worked, it worked amazingly, but nine times out of ten, it'd fail in a magnitude of ways. Whether that's failing to clean up after random crashing or slowly taking down the computer's internet connection, it generally wouldn't work. Luckily for us at Azuqua, we had a pretty well-defined system for service discovery and few services that needed to talk to one another directly. The few that needed that could just run outside of the cluster. That allowed us to accept those pains and be successful with Telepresence. To be clear, this is not a hit on Telepresence. It worked very well for us at Azuqua, but it's not a complete solution.

When I started developing a new development environment for Outreach, I again tried to use Telepresence as the solution to bridge the gap between our local cluster and our local machine. Unfortunately, it didn't work. We did not have a well-defined service discovery mechanism to work around this, we had a much larger engineering team, and almost all services needed to talk to each other. We found more and more edge cases with Telepresence, and our developer experience was suffering. Our NPS score for our developer environment was at a low of -26.

It was pretty clear that Telepresence was not going to solve our use-cases. It was time to look into alternatives.

What are other alternatives out there?

Telepresence was interesting since it used a VPN to give you access to your resources. However, it also required a hack DNS injector component, which tended to be the primary source of network problems. There weren't many projects out there that seemed to solve this, but one interesting one was kubefwd. If you're not aware of Kubernetes port-forwarding, there is a command that allows you to bring down a set of ports for a given service and have them be available on your local machine via kubectl port-forward. Think of the -p argument to docker run, but for Kubernetes. Unfortunately, this didn't support FQDNs (.svc.cluster.local), statefulsets, or all namespaces automatically. It also didn't support reverse tunneling. I wanted to keep the spirit of Telepresence by continuing to be a one-stop tool. Aside from kubefwd, there were seemingly no tools that could do this. Reverse tunneling into Kubernetes alone was seemingly an unsolved project. To fill that gap, I decided to write Localizer.

Introducing Localizer

Localizer is a rewrite of Telepresence in Golang, but without the complex surface area. Its goal is to be a simple, no-frills development tool for application developers using Kubernetes. Its goals are to be easy to use, self-explanatory, and easy to debug. Localizer is also a daemon. When trying to expose multiple services, developers generally had to have many terminal windows to run Telepresence or other reverse tunnel solutions. Localizer moves away from that by having all commands send a gRPC message to the daemon. More on that later. At the core of Localizer is three commands.

Default: Creating Tunnels into Kubernetes

When you run localizer without any arguments, it automatically creates port-forwards for all Kubernetes Services (this is important), a loopback-interface powered IP address, and adds them to /etc/hosts. Out of the box, this allows code to communicate with Kubernetes services despite being outside of the cluster.

Expose: Creating a Reverse Tunnel

The expose command takes namespace/service as an argument that allows you to point a Kubernetes service down to your machine's local service instance. When run, it automatically scales down any endpoints that already exist for the service. Localizer then creates an OpenSSH pod with the Kubernetes service's selector, which makes Kubernetes route traffic to the created pod. Localizer then creates an SSH reverse proxy over a Kubernetes port-forward (which enables firewall bypassing for remote clusters) to the OpenSSH pod created, which exposes and routes traffic to your local service. In essence, it creates an SSH reverse tunnel for you.

List: Listing the status of the tunnel(s)

The arguably most useful command that localizer provides is list. It enables developers to view the various tunnels running and provide insight into the health and status.

How Localizer Solved Our Problems (sort of)

Rolling out Localizer was a massive success for our developer environment's stability at Outreach, but it's ultimately not a complete solution. It helps enable the developer experience's build/test aspect, but it doesn't solve the deployment complexity aspect of Kubernetes. Ultimately, it's out of scope for Localizer to fix these problems and is just another tool that helps bridge the gap between Kubernetes and a developer's local machine.

The lack of abstractions is a fundamental problem for Kubernetes right now, and I look forward to writing tooling that can help developers focus on the business problems and not waste time on the details.

Looking to the Future

Localizer has a lot planned for the future! Namely proper daemon support, but also trying to improve the visibility into different failure modes. Stability and recoverability are going to be a constant focus for this project.

Are you interested in Localizer? Check it out on Github!

Special Thanks: Mark Lee for editing!