<ul>
<li>When working on complex systems, there are often more than one test parameter that changes. For example when working on a component for your design system, you need to test across all these different parameters:
<ul>
<li>Responsive Breakpoints</li>
<li>Dark/Light Mode</li>
<li>Accesibility options (screen reader, mouse-only, keyboard-only)</li>
<li>Different browser versions</li>
</ul>
</li>
<li>A good abstraction to help cope with this increasing complexity is by thinking of combinational matrixes.
<ul>
<li>In code, this can be multi dimensional arrays or nested loops. <a href="https://jestjs.io/docs/api#testeachtablename-fn-timeout">Jest even has a feature for it</a>.</li>
<li>In PR descriptions, it's helpful to draw a table to make sure you test all possible combinations throghoutly.</li>
</ul>
</li>
<li>Increasing the dimensions can make a full test run very costly though, so you can’t just blindly add more.</li>
<li>A very common pattern where this is userful is for manual pull request descriptions where you want to provide proof that you have exhaustively tested a new feature. A two-dimensional table might be used to show screenshots of e.g. different browsers and different responsive breakpoints.</li>
</ul>

Test Matrixes

Clear and comprehensive communication skills are important for <a href="https://spiess.dev/note/engineering/great-engineers">good engineers</a>. Since a lot of day-to-day communication tends to be in written documents (be it in design docs, issue trackers, or Slack/Discord), I am always striving to be better at it. This is a collection of notes on strategies I've found useful to improve my writing:
<h2>1. Know Your Audience</h2>
This is a technique that a more senior engineer at Meta shared with my pretty much after the first document I wrote there. The idea is simple: define the kind of person that your writing is for in advance. And then write only for them.
I use this to get an understanding of what kind of previous knowledge I can assume from the audience and will make me avoid falling into the trap of using too much jargon or technical terms when writing for product teams and customers or being too broad and unspecific when writing for engineers.
The second thing I really like about this technique is that it will give you a cue when your writing is addressing an audience that is too broad. I realized this when I wrote project status reports that were read by my peers but also product managers, designers, and leadership. These reports should not be focused on technical details so instead I split up the report into two parts: one for my team and one for the rest of the organization.
<h2>2. Align on a Shared Goal</h2>
This is especially helpful whenever you reach out to someone for the first time. Before going deep into the details, give some context and align on a shared goal.
I use this all the time when I reach out to different engineering teams (within the organization or broader community), especially platform teams (so teams working on frameworks and infrastructure). People on these teams have a lot of requests from other teams so aligning to their goals is going to give you the best chance of getting your request prioritized.
What I do is usually a paragraph with the following content:
<ol>
<li>Some context as to what I am trying to do (what is the feature you are building and why is it important)</li>
<li>Connect to their goals (e.g. we want this feature to be reliable, fast, or efficient.)</li>
</ol>
<h2>3. Follow a Cohesive Train of Thought</h2>
Whenever I work on a longer document, I try to come up with one cohesive thread ahead of time that I try to follow with a relatively fixed pacing.
This is a bit hard for me to put in writing since I rely on intuition for this. One example is when I write a proposal for a new feature, I start with setting up the context and goals first, then explain the problem in more detail, then discuss the different solutions considered and narrow down on the proposed one before I conclude with the next steps necessary to move along. I try to give all sections the same amount of detail so that the reader can better orient themselves in the text.
I don't think the actual structure matters too much as long as it's consistent within the piece.
<h2>4. Avoid Hierarchical Components</h2>
Do you know the feeling when you read a book and it lingers in your mind for months afterwards? This happened to me this year with <a href="https://spiess.dev/note/other/reading-list">The Courage to Be Disliked</a> by Fumitake Koga and Ichiro Kishimi. This book introduced me to Adlerian psychology and the concept of vertical relationships—relationships with a hierarchical component like boss/employee or teacher/student.
It's hard in a work environment to not have vertical relationships, however I do think that relying on this hierarchy will help you write more effectively. Instead, I try to meet eye-to-eye with the reader and try to write from the same level. Creating a shared goal helps a lot with this.
<h2>5. Do One Level Of Homework</h2>
Whenever I write something, I take a few minutes afterwards to think about the immediate next question that a reader might have and then incorporate the answer into the writing. This works great for any documents but is especially helpful when I need to ask for help as it will sometimes even answer the question that I had in the beginning.
I started to call this "doing homework" and my intention is to save time for the reader and thus improve clarity. I need to work hard to only do this for one level, though, since it can be easy to get distracted this way.
<h2>6. Keep It Short and Simple</h2>
I used to be really impressed by Software Engineers that write long prose with lots of fancy vocabulary even if I never fully understood what they were trying to communicate. If I don't understand what they are trying to say and it "sound smart", I concluded that I must be stupid.
This, of course, is nonsense and making your readers feel small is not something to aspire to. Instead, I try to keep my writings as inclusive as possible by focusing on simple and concise wording, even if this means that I have to put in more work.
<h2>Conclusion</h2>
Effective written communication is hard and the only real way I found to improve them is by practice. The points above might seem trivial to some but they have not always been trivial for me. Who knows, maybe you can try one of these ideas the next time you write something? If so, I'd love to hear about your experience!

Effective Writing

A friend asked me a great question this week: “Who are great engineers you know and why are they great?”. And while I immediately had a few people in mind, I had a really hard time answering why I think of them as being so great.
<h2>The Archetypes</h2>
All of the people that came to mind were very senior engineers that have very specific areas that they excel in. This is typical for Staff level engineers and it's why it's so hard to compare them to each other and to give Senior engineers a clear path to Staff.
There are numerous attempts to define the different archetypes and if you haven't read Will Larson's <a href="https://staffeng.com/guides/staff-archetypes/">Staff archetypes</a> before, I recommend you look it up right away. I don't think the specific areas are too important though (e.g. there are different definitions across different companies) and I also think in reality, most people are a mix of all of them.
Here are just some of the archetypes that I have worked with:
<ul>
<li>The excitement machine: This is a person that loves the area they work in so much that just talking to them makes you more enthusiastic. They often would work on a prototype for a new feature that would impress the whole organization. They have a unique skill to extrapolate the current trends into longer term visions and also know how to make the next logical step toward that vision. Their excitement helps to align different functions to work towards the same goals.</li>
<li>The business expert: This person knows the industry they work in by heart. This allows them to make very good tradeoffs between when to focus on technical excellence and when to focus on the business impact. They are usually working really closely with the customers and the sales team and are able to translate the business needs into technical requirements.</li>
<li>The tech lead: Having built the companies most complex features before, this person has an extremely deep expertise of the codebase. They have also learned, though, that it's not enough to just code solo and now use their experience to work on more strategical planning and mentoring. They bring alignment to the broader organization and help make sure projects are delivered on time and with the right quality.</li>
<li>The hyperspecialized: This person is so deeply focused in one area that they start to use lingo that only a few people will understand. They are able to connect new problems to past research and will tell you about this one 1992 paper that they feel is massively undervalued.</li>
</ul>
<h2>What They All Have In Common</h2>
After thinking of some concrete examples, I was trying to come up with common traits of these folks, too. I wanted to think of something more actionable than "go find your niche". In my experience, great engineers have the following in common:
<ul>
<li>Broad technical knowledge: Being an engineer still means you are thinking about code and so all these folks are obviously very good at that. Even the hyperspecialized person has excellent mental models of areas around them. They all have a strong base for the <a href="https://en.wikipedia.org/wiki/T-shaped_skills">T-shaped skills model</a>.</li>
<li>Strong communication skills: When I started out, I never thought how important communication skills are. Just show me some code, everything else is irrelevant, right... right? In reality, this can't be further from the truth. The best engineers I know are able to communicate their ideas in a way that is easy to understand for the target audience. They usually demonstrate their skills by writing technical documents or prepare talks that help shape the direction of the company or industry.</li>
<li>Large networks: To scale their work beyond themselves, great engineers have built and maintain connections to lots of different people in the organization or the broader industry. They are like Saul Goodman in breaking bad and always "know someone" for whatever problem they face. These networks usually span across the engineering organizations and might contain sales, leadership, or even customers. It takes a village to raise a child, right?</li>
</ul>
<h2>The Rest?</h2>
I had a daunting realization after thinking through the above that I know so, so many great people that did not came to my mind at all when I thought about great engineers. This is odd because these are people that I would immediately hire and that definitely cover the common traits listed above. So why have I not thought about them?
The answer I arrived at is scale: These engineers, while being absolutely amazing individually, do not scale their impact beyond only a few people. Which lead me to my final conclusion: Great engineers have figured out how to scale their work. They do this by influencing the product/technical direction, upleveling others, being role models or simply giving excellent advice. They are the people you look up to for feedback and that will make you better at what you're doing simply by being around them.

Great Engineers

I remember a situation from early on in my career where I asked my more senior coworker for some help. They came to my desk and showed me how things are done. Full of awe, I reached out to them afterward and told them: "Thank you so much for showing me this, I learned a ton from you!". To my surprise, they responded with: "No worries! I learned a lot from you too!"
This was really confusing to me. The more senior folks are supposed to be the teachers, and I am just the student. What is there to learn from me?
However, being fortunate to have been on the other end, I now get it.
Good teachers are not the ones who only preach their own way of thinking. Instead, they will be your peers when exploring the problem and will solve it with you, not for you. In doing so, they will pick up your way of articulating the problem, your way of thinking about the world, your way of approaching the problem, and so much more.
Learning, at its best, is a bidirectional relationship away from any hierarchical structure. Everyone gets better as a result of it.

Learning is Bidirectional

Large companies are inherently number-driven. This is not a bad thing per se: Aligning around measurable objectives has huge benefits and allows an organization to keep itself accountable.
However, if you work in larger tech companies, you will quickly come into contact with goals like this:
<ul>
<li>"We are working on improving our sign-up rate by 20% in the next quarter."</li>
<li>"The goal of the platform rewrite is to reduce defect rate by 50%."</li>
</ul>
These numbers seem fine at first but once you start digging into it, you learn how extremely arbitrary these are, because:
<ul>
<li>It is entirely unclear to the team how to increase sign-up rate at all.</li>
<li>The platform rewrite is going to increase reliability, but there is no certainty into how it will affect the metrics.</li>
</ul>
When I worked on web performance at Meta, I came into contact with this very early and my rational personality was quickly overwhelmed: How can I come up with a number that is truly achievable? Do I need to scope out all exact work streams ahead of time and hope that they are all successful? How are goals like this made?
Luckily, I had an amazing colleague back then who told me the one secret about goals at Meta, which she called the 50-50 goal: Choose a target that feels achievable 50% of the time (so you will have an equal chance of not hitting it). At the time this was confusing but I've seen this used extensively at many platform teams at Meta and only now, after many years, I finally understand why this system works:
<ul>
<li>50-50 goals are about embracing failure: If there's an equal chance of you missing the goal, you set yourself up for failure. And, in this case, that's a good thing! No one actually cared whether the sign-up rate increased by 20%. It's important that it increases at all. The trick is to remove the psychological burden of missing a goal and instead focus on actually driving these goals.</li>
<li>These goals help with alignment: Setting an ambitious target will make it clear to everyone in the room that they need to get creative. It embraced risk-taking and encourages people to think outside the box.</li>
<li>It creates excitement across the organization: By choosing ambitious goals, you will more effectively communicate the goal of the project. This helps to get buy-in from other teams and stakeholders and sets the stage for better collaboration.</li>
</ul>
So, if you ever think about setting a target for your team, think about the 50-50 goal. Oh and don't forget to communicate that it's a 50-50 goal. You don't want to set wrong expectations.

The 50-50 Goal

When faced with hard choices, I'm often tempted to search for the better option as if it's hiding just beneath the surface. But most of the time, there isn't one. Hard choices are hard not because they're unsolvable puzzles, but because each option carries its own set of values and complexities that defy simple comparison.
<h2>Most Choices Are Not Permanent</h2>
When standing before a hard choice, it's easy to fall into the trap of seeing it as a permanent, life-altering event. This type of thinking has had a huge impact on me previously and made me spiral down the "thinking-train" even more. If I think just a bit more, I might avoid making a big mistake.
In reality, however, most choices are not permanent and it's important to keep that in mind when facing a hard choice. If it's only a temporary choice, it doesn't have to be the best choice, does it?
<h2>The False Dichotomy</h2>
We often box ourselves into thinking there is only option A or B. But what if neither option is the right choice at the moment? Take the example of moving to a new house—if neither option feels quite right, perhaps the best choice is to wait.
Decision-making is not just about choosing between given options, though. Instead, we have the power to actively shape those options and the reasons behind our choice.
Not choosing now doesn't prevent you from making a similar choice in the future.
<h2>The "Gut Feeling"</h2>
I have a bit of a gripe with this term because while my gut does many great things, it certainly doesn't have feelings. It took me long to understand what is meant by it but I now think a better term would be emotional feeling.
If you have no rational explanation as to why an option is better or worse than the other, but you have a strong emotional feeling, don't undervalue this signal. Emotions have, after all, made humans as successful as they are today.
<h2>Embrace the Complexity</h2>
If you haven't seen it, I highly recommend you check out <a href="https://www.youtube.com/watch?v=8GQZuzIdeQQ">Ruth Chang's talk on how to make hard choices</a>. It has highly influenced me in this area. The core takeaway I have from this talk is that I now start to embrace the complexity of hard choices in life and see them as opportunity to steer my life's narrative towards who I truly want to be.
It's extremely freeing to not think of hard choices as confining moments but as rare opportunities that you can use to grow.

Hard Choices

You can find me on <a href="https://www.goodreads.com/user/show/50192165-philipp">Goodreads</a> for my up-to-date reading list. In the past months, I really enjoyed science fiction. Let me know if you have any recommendations!
Here are a few books I rated 5 stars recently:
<ul>
<li><a href="https://www.goodreads.com/book/show/20518872-the-three-body-problem">The Three-Body Problem</a> by Cixin Liu</li>
<li><a href="https://www.goodreads.com/book/show/16158581-the-collini-case">The Collini Case</a> by Ferdinand von Schirach</li>
<li><a href="https://www.goodreads.com/book/show/54493401-project-hail-mary">Project Hail Mary</a> by Andy Weir</li>
<li><a href="https://www.goodreads.com/book/show/41160292-exhalation">Exhalation</a> by Ted Chiang</li>
<li><a href="https://www.goodreads.com/book/show/77566.Hyperion">Hyperion</a> by Dan Simmons</li>
<li><a href="https://www.goodreads.com/book/show/112537.Rendezvous_with_Rama">Rendezvous with Rama</a> by Arthur C. Clarke</li>
<li><a href="https://www.goodreads.com/book/show/43306206-the-courage-to-be-disliked">The Courage to Be Disliked: How to Free Yourself, Change your Life and Achieve Real Happiness</a> by Fumitake Koga and Ichiro Kishimi</li>
<li><a href="https://www.goodreads.com/book/show/61714633-the-wager">The Wager: A Tale of Shipwreck, Mutiny and Murder</a> by David Grann</li>
<li><a href="https://www.goodreads.com/book/show/18373.Flowers_for_Algernon">Flowers for Algernon</a> by Daniel Keyes</li>
<li><a href="https://www.goodreads.com/book/show/48484.Blindsight">Blindsight</a> by Peter Watts</li>
</ul>
Last update: Aug 11, 2024

Reading List

You can deploy Cloudflare Workers via the HTTP API and thus dynamically create Workers. The canonical solution for this is to use <a href="https://developers.cloudflare.com/cloudflare-for-platforms/workers-for-platforms/">Workers for Platforms</a> however you can also do it with a free Cloudflare account. Documentation for this feature is sparse but thanks to <a href="https://community.cloudflare.com/t/deploy-a-worker-via-http-api/108439/4">this Cloudflare Community post</a> I came up with this:
Update: Well, turns out there is actually quite good documentation on that API in <a href="https://developers.cloudflare.com/api/operations/worker-script-upload-worker-module">the Cloudflare API Docs</a> 🙃.
<h3>1. Bundle your Worker</h3>
If you already have <code>wrangler</code> set up, you can bundle your worker by running <code>wrangler build</code>. This will build the worker into the <code>dist/</code> folder.
<h3>2. Prepare your Metadata</h3>
Worker metadata is a configuration format that contains the runtime info from your <code>wrangler.toml</code> file. You can learn more about it <a href="https://developers.cloudflare.com/cloudflare-for-platforms/workers-for-platforms/reference/metadata/">in the Worker metadata docs</a>.
Without any bindings, the minimum metadata might look like this:
<div class="remark-highlight"><pre class="language-json"><code class="language-json">{"main_module":"worker.js","bindings":[]}
</code></pre></div>
<h2>3. Upload your Worker</h2>
Now upload your worker:
<div class="remark-highlight"><pre class="language-bash"><code class="language-bash">curl -X PUT "https://api.cloudflare.com/client/v4/accounts/&#x3C;ACCOUNT_ID>/workers/scripts/&#x3C;SCRIPT_NAME>" \
 -H "Authorization: Bearer &#x3C;CF_API_TOKEN>" \
 -H "Content-Type: multipart/form-data" \
 --form 'worker.js=@"dist/worker.js";type=application/javascript+module' \
 --form 'metadata="{\"main_module\":\"worker.js\",\"bindings\":[]}"'

{
 "result": {
 "created_on": "2024-04-02T21:27:40.527498Z",
 "modified_on": "2024-04-02T21:27:40.527498Z",
 "id": "&#x3C;SCRIPT_NAME>",
 "tag": "aabbcc",
 "tags": [],
 "deployment_id": "aabbcc",
 "tail_consumers": null,
 "logpush": false,
 "etag": "aabbcc",
 "handlers": [
 "fetch"
 ],
 "last_deployed_from": "api",
 "usage_model": "standard"
 },
 "success": true,
 "errors": [],
 "messages": []
}
</code></pre></div>
Notes on the params:
<ul>
<li><code>ACCOUNT_ID</code> can be found in the Cloudflare dashboard URL</li>
<li><code>SCRIPT_NAME</code> is the name of your new worker and will be part of the default subdomain</li>
<li><code>CF_API_TOKEN</code> must be an <a href="https://dash.cloudflare.com/profile/api-tokens">Edit Cloudflare Workers API token</a></li>
</ul>
<h3>4. Publish the Worker via Subdomain</h3>
The last step is to publish the Worker to be accessible via subdomain. The domain for this will depend on the <code>workers.dev</code> subdomain assigned for your Cloudflare account, e.g.: <code>&#x3C;SCRIPT_NAME>.myapp.workers.dev</code>.
<div class="remark-highlight"><pre class="language-bash"><code class="language-bash">curl -X POST "https://api.cloudflare.com/client/v4/accounts/&#x3C;ACCOUNT_ID>/workers/services/&#x3C;SCRIPT_NAME>/environments/production/subdomain" \
 -H "Authorization: Bearer $CF_API_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{"enabled": true}'
{
 "result": null,
 "success": true,
 "errors": [],
 "messages": []
}
</code></pre></div>

Deploy Workers Programatically

The VS Code extension runtime uses Node.js (via Electron) and you can thus use Node.js related debugging tool.
Here's a quick guide on how to record performances traces of VS Code extensions:
<div class="remark-highlight"><pre class="language-bash"><code class="language-bash">/Applications/Visual\ Studio\ Code.app/Contents/MacOS/Electron --inspect-extensions=9333
</code></pre></div>
After VS Code is started, you want to head over to Chrome and go to <code>chrome://inspect</code> which should open up a view like this:
<img src="https://gist.github.com/assets/458591/0a17881b-5449-48d5-a53e-5556f4f2dedd" alt="tracing overview">
From here you want to configure the inspect server that we just started on port 9333. To do this, click on "Open dedicated DevTools for Node", go to the "Connection" tab if that is not open yet, and make sure to add <code>localhost:9333</code> to the list like this:
<img src="https://gist.github.com/assets/458591/972ce113-88f0-482a-99b7-5e51957981ef" alt="add localhost to the list">
Now, head back to the <code>chrome://inspect</code> tab and you should see a new remote target that you can inspect. For me this looks like this:
<img src="https://gist.github.com/assets/458591/06b2e293-aea7-42e8-a9cc-592863b6fb07" alt="tracing overview with a new electron option">
Clicking this will open a (somewhat reduced) DevTools view. Great! We've almost got it. From here you can go to the "Performance" tab to record a trace. Then, swap tabs to the VS Code window and interact with the extension. Come back later to stop the recording and export it. Voila!
<img src="https://gist.github.com/assets/458591/d590636b-31e5-4436-8039-ee62b7a8e59f" alt="screenshot of the performance tab in the nodejs profiler">

Profiling Extension in Production

Feature flags are a powerful technique that every engineer should have in their toolbox. They allow you to ship changes incrementally and reduce risk when releasing new features. Here are my thoughts on why you should use feature flags:
<ul>
<li>
Avoid big bang releases: With feature flags, you can push out changes regularly without affecting all users at once. This makes it easier to review pull requests since the changes are smaller. No more waiting days or weeks to merge in order to batch changes.
</li>
<li>
Reduce risk: New features can be released to a small percentage of users first. If issues come up, simply turn off the flag before rolling out widely. This protects your whole user base from potentially breaking changes.
</li>
<li>
Enable experimentation: By targeting feature flags to different user segments, you can run A/B tests and collect data on how new features are performing. This allows you to validate ideas before fully releasing (and allows you to collect awesome graphs for your next review period 😉).
</li>
<li>
Simplify rollbacks: If a new feature causes problems after being released, you can instantly roll back by disabling the flag (often referred to as a kill switch). This is especially handy if you can remotely control those flags.
</li>
<li>
Improve code hygiene: Make sure to clean up unused flags over time. Centralize flags in one place like an enum to avoid combinatorial complexity.
</li>
</ul>
Feature flags don't have to be complex, either. Something as simple as a config boolean can get you started. And you don't necessarily need third-party tools for simple use cases. Just try it out on your next feature branch!

Feature Flags

Good engineers ship. That's no big secret. Shipping is, after all, one of the very core pillars of the engineering culture and celebrated a lot.
But have you ever noticed that great engineers ship all the time? They ship, even if the product is not updating. They ship by sharing quick demos and building excitement and urgency by sharing their prototypes with other engineers. I started to refer to this as "demo culture" and it's something you see to great extend in high performing teams.
Demo culture is the practice of regularly sharing quick demos to build hype and validate ideas in the cheapest possible way. It's a way of showing off your work and getting feedback from your team and stakeholders without all the red tape. It's a way to make every team more engineering driven.
So how can you start building a demo culture in your team? Here are a few tips:
<ol>
<li>Share early and often. Someone needs to start setting an example. If you can hack something together 30. The more brittle, the better, so others understand that there's no need for quality.</li>
<li>Make it fun. Demos should be engaging and exciting. I like to create little stories of potential use cases to lighten the mood.</li>
<li>Keep it short and sweet. Demos should be quick and to the point. If it's over 2 minutes, it's probably too long. Make sure your audience has your attention and make it easy for them to consume the information.</li>
<li>Encourage and provide feedback. After your demo, ask for feedback and suggestions. This will help you improve your work and make sure you're on the right track. Similarly, if someone else gives a demo, make sure to share your thought. And do not forget to show them how excited you are about the demo. 🔥</li>
</ol>
So, the next time your meeting ends 5 minutes early, just share your screen and say "hey I have a quick demo to show you" and start a transformation in your team.

Demo Culture

If you've never come across it, I highly recommend you read up on <a href="https://en.wikipedia.org/wiki/Theorycraft">Theorycraft</a>. Because of course there exist a huge subculture in the internet that is trying to mathematically optimize video games and of course I’m going to start my article with it.
Whenever arguments in video game Discord chats arise about how to optimally play in a certain scenario, people would cite the research made by Theorycraft. But sometimes people insist that playing in a different way is better. Well, you know the internet folks, these people will be called words. One of the words that stuck with me was "Feelscraft". Huh, what an insult!
But what is Feelscraft software? Well, for me it is acting on intuition and experience rather than data and common knowledge when making product decisions. It is shipping a feature even though you do not have the metrics to back it up. And, as usual, I have ambivalent feelings about it.
Much like the Theorycraft folks in the video game scene, I strive to make the best product choices. Working on the most requested feature that is also sure to increase revenue and that has measurable impact on our user base. And, in a perfect world, we also have all the data necessary to make these decisions and there is only one valid way to solve a problem to begin with. However, in reality, this is hardly the case and this is when Feelcraft comes in handy.
I do think that intuition is often overused, though. When it's used, it's very hard to understand why certain choices were done for people outside of the decision metrics process. One example of Feelscraft Software is from my time at Meta when I read a product announcement that stated that a new implementation of an existing feature is being shipped because it shows no regression in usage metrics over the old one. And that after weeks and weeks of improvements on the new implementation... I think that's a low standard for pushing a new product, don't you think?
Another more recent example that I have seen are numerous changes of an LLM prompt in the hope to improve the product. LLMs are inherently prone to variation in their outcome and making decisions on whether a prompt is better than another normally requires thousands and more different samples. That's why we at <a href="https://sourcegraph.com/cody">Cody</a> are investing so heavily into online and offline evaluation methods. Yet, every now and then I see the industry making changes on prompts based on intuition.
I have been guilty, too, of course! In my most recent work on <a href="https://sourcegraph.com/blog/the-lifecycle-of-a-code-ai-completion">Cody autocomplete</a> I have often made subtle decisions in the because it felt better. Things like setting initial debounce times too low or cut-off ratios too aggressive. However, we had to get the product out in some way or another and only after getting usage we were able to validate these initial assumptions and make informed decisions (which, of course, resulted in my tweaking these values later).
So, the next time you try to understand how a software decision has been made, remember to think about the Feelcraft behind it. 😉

Feelscraft Software

Understanding and processing natural language is a complex task. Embeddings are a powerful tool in todays ML/AI discussions to address this problem.
<h2>Mental model</h2>
In my mental model, embeddings are used to encode words into meaning. Embeddings are encodings in n-dimensional vector space where every parameter is a signed a specific meaning in the object space.
I can't really imagine n-dimensional vectors but for an embedding system with two vectors, I like to visualize it like this:
<div class="remark-highlight"><pre class="language-unknown"><code class="language-unknown">terrain (y-axis)
|
|
| o (mountain bike)
|
| o (road bike)
|
|
| o (horse)
|
| o (car)
|
|
+------------------------------- speed (x-axis)</code></pre></div>
In this example, each point represents a mode of transportation. Items with similar speed and terrain capabilities are placed closer together. For example, a mountain bike and a road bike are closer to each other because they have similar speeds and are both used on different terrains, while a car is further away since it operates at higher speeds on more paved terrains.
There are two super-powers from being able to map natural language onto an n dimensional vector space:
<ul>
<li>We can use relatively simple vector algorithms to quickly find relevant words. Algorithms like <a href="https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm">kNN</a> are easy to be optimized with spacial maps so lookup is fast. In our example, we can see that a mountain bike is closer to a road bike than it is to a horse.</li>
<li>Embeddings compress insanely well. It's entirely possible to compress pages of written language into a vector size of a few hundred vectors.</li>
</ul>
There are also some downsides, though:
<ul>
<li>Training an embedding system is like training any other ML system: It's a complex and expensive task. And even with a fully trained model, you will need a lot of compute power to convert a piece of text into embeddings. That's why it's currently common to outsource embeddings generation to third-party APIs like OpenAI.</li>
<li>When a model is trained, we don't really know what meaning each value of the vector is assigned to. This makes it hard to understand why a specific pair of text is considered close to each other or not.</li>
</ul>
<h2>How are embeddings used?</h2>
Because of the fast way to find related vector pairs, embeddings are often used to answer search problems. In the current AI discourse, embeddings are used to quickly and efficiently find relevant context in order to put it into the prompt of a large-language model (LLM).
Imagine an AI system to ask questions about your code documentation. We can, ahead of time, create embeddings for every paragraph in the documentation and store it in a vector database (a database that can hold the embedding vectors for us and has some index to efficiently query it. Like the <a href="https://github.com/pgvector/pgvector">pgvector</a> extension for Postgres).
Now, when the user enters a question in natural text into our AI system, we can find the most relevant paragraphs into our documentation. We usually want to pick as many as we can fit into the prompt of the LLM. The LLM now has an understanding of your documentation when answering the question. Fascinating!
One important thing from the paragraphs above is that we have to split the documentation into meaningful chunks. Since we can only retrieve similarity in windows of this chunk site, you can easily see how important it is to pick a proper system.
<h2>Some links I want to remember</h2>
<ul>
<li><a href="https://huggingface.co/blog/getting-started-with-embeddings">Getting Started With Embeddings</a></li>
<li><a href="https://huggingface.co/blog/mteb">MTEB: Massive Text Embedding Benchmark</a></li>
<li><a href="https://huggingface.co/spaces/mteb/leaderboard">MTEB Leaderboard</a></li>
</ul>

Embeddings

When building an AI code assistant like <a href="https://about.sourcegraph.com/cody">Cody</a>, you often find yourself in need for a quick way to rank the similarity of two blocks of text relative to each other.
This is where <a href="https://en.wikipedia.org/wiki/Jaccard_index">Jaccard similarity</a> is helpful. Jaccard similarity is a metric to find similarity between two sets of data, typically used for comparing text data.
The equation for Jaccard distance is:
<div class="remark-highlight"><pre class="language-unknown"><code class="language-unknown">J(A, B) = |A ∩ B| / |A ∪ B|</code></pre></div>
Here's a simple example in TypeScript to calculate the Jaccard distance between two sentences:
<div class="remark-highlight"><pre class="language-typescript"><code class="language-typescript">function jaccardDistance(sentence1: string, sentence2: string): number {
 const set1 = new Set(sentence1.split(' '));
 const set2 = new Set(sentence2.split(' '));
 
 const intersection = new Set([...set1].filter(x => set2.has(x))).size;
 const union = new Set([...set1, ...set2]).size;
 
 return intersection / union;
}
</code></pre></div>
The Jaccard similarity comes in handy for finding similar text, like detecting plagiarism or comparing documents. It might not be the most advanced method for text comparison, but it's simple and fast, making it great for many applications.
You can even do some pre-processing on the text to increase its accuracy. The above example is naively comparing words but you could use <a href="https://en.wikipedia.org/wiki/Stemming">stemming</a> to make it easier to compare words with the same meaning.
Why not give it a try with some sample texts and see how it works for you?

Jaccard Similarity

A lot of companies love ClickHouse for it's great performance as LDAP system, incredible compression, and its ease of deployment.
This is a list of features that are perhaps less known about but also quite powerful:
<ul>
<li><a href="https://clickhouse.com/docs/en/sql-reference/statements/alter/column#materialize-column">Materialized Columns</a>: This feature allows you to add new columns (which, given the fact that ClickHouse is column based, is very cheap) that derive the data from existing fields. You can imagine a generic log table with a json text field and you want to extract a specific field from the JSON into a custom column. With this feature, you can start your log table in an unstructured form and gradually migrate to a schema over time.
Check out <a href="https://www.youtube.com/watch?v=pZkKsfr8n3M">this video from Altinity</a> for more informations.</li>
<li><a href="https://clickhouse.com/docs/en/guides/developer/ttl">TTL</a>: ClickHouse allows you to define how long data inside tables and even specific columns is being retained. I think this is an amazing feature for sensitive data, ensuring the data is only retained as long as it is necessary for analytical purposes.</li>
<li><a href="https://clickhouse.com/docs/en/interfaces/schema-inference#json-formats">Automatic schema inference from JSON</a>: Granted, this is a very niche feature but I assume it could be amazing in combination with materialized columns to gradually and automatically convert an untyped JSON payload into a typed column structure.</li>
<li><a href="https://clickhouse.com/docs/en/sql-reference/statements/create/quota">Quotas</a>: A feature to allow you to limit resources in a multi-tenant ClickHouse deployment. You don't care about it until you absolutely need it, and here you just get it for free.</li>
</ul>

Niche ClickHouse Features

This is a list of non-trivial problems I have come across in the front-end community over the past few years. It mostly exists as a personal reminder. <a href="https://twitter.com/PhilippSpiess">Let me know</a> if you have something that should be added here!
<ul>
<li>
Accessibility: Making web apps accessible because it’s so easy with the Web fundamentals to break things.
</li>
<li>
Bundle coverage: A lot of the code that is shipped in today’s bundles (CSS, JS, etc.) is not needed for the first render.
</li>
<li>
Data waterfalls: Waterfalls is a term used to describe serial loading dependencies. E.g. to render a list of users we first download the list component, wait for it to load, then the list component makes a request to an API, waits for it to load, etc.
This is the number one issue that I have seen causing websites to be slow. It often manifests itself as a myriad of uncontrolled loading states that sometimes end in another loading state popping up. 😵‍💫
An interesting observation: Most applications could be able to resolve all the data needed for the initial page load on the server side based on the URL, params, and authentication tokens.
</li>
<li>
Overfetching/underfetching: A client should only ever get as many data as it needs to render. Anything more is not needed and everything less is not enough.
</li>
<li>
Interaction delays: If I interact with an item on the website, it should show an instant response.
</li>
<li>
Layout shifting: Once the first elements on a website are shown, you don't want to have them jump around. This only causes confusion to the user and results in a poor experience.
</li>
<li>
Developer experience: I think this term is often overused but the current way we write apps is definitely not the end game. We need constant improvements to the tooling because the cost of building is still way too high.
</li>
<li>
Hydration: When you have a server-rendered HTML page and it needs to become interactive ASAP.
</li>
<li>
Reliability: A meta-term that is all about knowing if your app functions. There are many vectors this can be done, however easy mechanisms also have a weak correlation to actual user problems.
<ul>
<li>Error handling/tracking. Unfortunately knowing that an error was thrown does not often relate to a broken website—Is the error recoverable by a user or is it crashing the whole page?</li>
<li>Browser crashes/freezes/loops</li>
<li>Uptime</li>
<li>Tracking core interaction uptime</li>
</ul>
</li>
<li>
Performance: This is a very convoluted term that justifies it’s own note. Until then, here are some thoughts:
<ul>
<li>Measuring page loads is usually simpler than measuring interactions (navigations, popups).</li>
<li>You can measure performance in a lab but this will only ever be a proxy to the real-user performance metrics (aka RUM—Real User Monitoring).</li>
</ul>
</li>
<li>
Efficiency: Another meta-term for a few lower-level optimzations that rarely cause problems (but when they do, they are noticeable):
<ul>
<li>Memory usage</li>
<li>Heavy CPU usage</li>
<li>Uncontrolled scheduled tasks (e.g. <code>setInterval</code>)</li>
<li>Browser rendering issues</li>
<li>Network/IO overhead</li>
</ul>
</li>
<li>
Testing: While software testing is hard in every environment, front-end has a couple of special complexity with things like visual regression testing, accessibility testing, the reliance on a backend API, or the flakiness of browser instrumentation.
</li>
<li>
Observability: Knowing how your app functions is a complex problem in itself. Traditionally, the web platform was very limited in how much data we have available with a big focus on analytics. Lately there have been a few amazing developments though:
<ul>
<li><a href="https://github.com/tc39/proposal-async-context">Async Context</a> is a recent proposal to bring zones.js to the platform which will allow for easer tracing APIs in the future.</li>
<li><a href="https://github.com/WICG/js-self-profiling">JavaScript Self-Profiling API</a> brings process sampling features to the web platform.</li>
</ul>
</li>
</ul>

Modern Front-End Problems

<ul>
<li><input type='checkbox' disabled /> Input methods
<ul>
<li><input type='checkbox' disabled /> Set up the <a href="https://github.com/patrick-zippenfenig/us-with-german-umlauts#:~:text=Mac%20U.S.%20keyboard%20layout%20with%20German%20Umlauts,-This%20layout%20is&#x26;text=The%20layout%20is%20a%20modified,option%20%2B%20u%20%3D%3E%20%C3%BC">US-German Keyboard layout</a></li>
<li><input type='checkbox' disabled /> Disable natural scrolling</li>
<li><input type='checkbox' disabled /> Maximize key repeat rate and delay until repeat</li>
<li><input type='checkbox' disabled /> Map <code>tab</code> key to <code>escape</code> in Keyboards settings</li>
</ul>
</li>
<li><input type='checkbox' disabled /> <a href="https://brew.sh/">Homebrew</a></li>
<li><input type='checkbox' disabled /> <code>brew install --cask 1password</code> (set up 1Password immediately)</li>
<li><input type='checkbox' disabled /> Connect iCloud and enable Find My Mac</li>
<li><input type='checkbox' disabled /> Customize Dock</li>
<li><input type='checkbox' disabled /> Connect Google account for the Calendar integration</li>
<li><input type='checkbox' disabled /> Oh My Zsh
<ul>
<li><code>git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions</code></li>
<li><code>git clone https://github.com/zsh-users/zsh-syntax-highlighting.git ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-syntax-highlighting</code></li>
</ul>
</li>
<li><input type='checkbox' disabled /> <a href="https://mise.jdx.dev/getting-started.html">Mise</a></li>
<li><input type='checkbox' disabled /> <code>brew install git</code></li>
<li><input type='checkbox' disabled /> <code>brew install --cask warp</code></li>
<li><input type='checkbox' disabled /> <code>brew install --cask raycast</code></li>
<li><input type='checkbox' disabled /> <code>brew install --cask discord</code></li>
<li><input type='checkbox' disabled /> <code>brew install --cask google-chrome</code></li>
<li><input type='checkbox' disabled /> <code>brew install --cask firefox</code></li>
<li><input type='checkbox' disabled /> <code>brew install --cask hiddenbar</code></li>
<li><input type='checkbox' disabled /> <code>brew install --cask spotify</code></li>
<li><input type='checkbox' disabled /> <code>brew install --cask visual-studio-code</code></li>
<li><input type='checkbox' disabled /> <code>brew install --cask obsidian</code></li>
<li><input type='checkbox' disabled /> <code>brew install --cask rectangle</code></li>
<li><input type='checkbox' disabled /> <code>brew install --cask monitorcontrol</code></li>
<li><input type='checkbox' disabled /> <code>brew install --cask telegram</code></li>
<li><input type='checkbox' disabled /> <code>brew install --cask whatsapp</code></li>
<li><input type='checkbox' disabled /> <code>brew install --cask flux</code></li>
<li><input type='checkbox' disabled /> <code>brew install --cask the-unarchiver</code></li>
<li><input type='checkbox' disabled /> <code>brew install --cask stats</code></li>
<li><input type='checkbox' disabled /> Fonts
<ul>
<li><input type='checkbox' disabled /> <code>brew tap homebrew/cask-fonts</code></li>
<li><input type='checkbox' disabled /> <code>brew install --cask font-fira-code</code></li>
</ul>
</li>
<li><input type='checkbox' disabled /> Sync <a href="https://github.com/philipp-spiess/dotfiles">dotfiles</a></li>
<li><input type='checkbox' disabled /> Set up SSH keys for GitHub and remote servers</li>
<li><input type='checkbox' disabled /> From the App Store
<ul>
<li><input type='checkbox' disabled /> Microsoft To Do</li>
<li><input type='checkbox' disabled /> Things</li>
<li><input type='checkbox' disabled /> Prologue</li>
</ul>
</li>
<li><input type='checkbox' disabled /> For Work Macs
<ul>
<li><input type='checkbox' disabled /> Slack</li>
<li><input type='checkbox' disabled /> Loom</li>
<li><input type='checkbox' disabled /> Carbon Black</li>
<li><input type='checkbox' disabled /> Dev Environment</li>
<li><input type='checkbox' disabled /> Zoom</li>
<li><input type='checkbox' disabled /> Set up a second Chrome profile and change the UI theme of one of the profiles (pref. work to blue)</li>
</ul>
</li>
<li><input type='checkbox' disabled /> For Personal Macs
<ul>
<li><input type='checkbox' disabled /> VPN Client and helper</li>
<li><input type='checkbox' disabled /> Sketch</li>
</ul>
</li>
<li><input type='checkbox' disabled /> Configure backups</li>
</ul>

New Mac Checklist

<ul>
<li>Being not constraint to a specific domain is one of React's killer features.</li>
<li>In React DOM, you can always fall back to creating custom <code>&#x3C;div></code> elements with your own styles if something doesn't work.
<ul>
<li>There's a maintainance overhead by doing so but it also unblocks so many use cases.</li>
<li>Even at the new Facebook.com, custom containers with with the full fallback to the browser were very common.</li>
</ul>
</li>
<li>Every good design systems that I have seen embraces this and allows their users to forward e.g. custom class names to their inner working components.</li>
</ul>

React’s Escape Hatch

A collection of my random thoughts. This is a place for me to write down raw ideas, so don’t expect a high bar for quality.
Inspired by Josh Branchaud’s <a href="https://github.com/jbranchaud/til">til</a>, Lee Byron’s <a href="https://leebyron.com/til/">til</a>, and Shawn Wang’s <a href="https://www.swyx.io/learn-in-public/">Learn In Public</a>.

Philipp Spiess

Test Matrixes

Other Notes

About the author